Method and apparatus for recording multimedia data

Abstract

The embodiment of the application discloses a recording method and device of multimedia data, wherein the method comprises the following steps: performing real-time analysis on multimedia data to obtain voice content and at least one demonstration action of a target object; wherein the multimedia data comprises simultaneously collected first audio data and image frame data; determining semantic correlation between each demonstration action and the voice content; according to the semantic correlation, in the case that content indicated by any demonstration action is inconsistent with the voice content, performing video understanding on an image frame where the corresponding demonstration action is located to convert the demonstration action into second audio data; and dynamically inserting the second audio data into the first audio data to update the multimedia data and record.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of image processing technology, and in particular to a method and apparatus for recording multimedia data. Background Technology

[0002] However, during podcast recordings or live streams, the speaker may make some impromptu demonstrations, such as gesturing a shape, demonstrating a scene, or showing objects or charts. This visual information cannot be conveyed to the audience through audio.

[0003] There's no perfect solution right now. It might require the host to be constantly alert, interrupting the speaker to explain things to the audience, but this demands skill from the host, and if both sides are fully engaged, they might overlook this information. Another option is to supplement this information in the audio through post-production, but this is time-consuming and laborious, and it doesn't solve similar problems in live streaming scenarios. Summary of the Invention

[0004] In view of this, the embodiments of this application provide at least one method and apparatus for recording multimedia data.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] On one hand, embodiments of this application provide a method for recording multimedia data, the method comprising:

[0007] Multimedia data is parsed in real time to obtain the voice content of the target object and at least one demonstration action; wherein the multimedia data includes first audio data and image frame data acquired simultaneously; the semantic relevance between each demonstration action and the voice content is determined; based on the semantic relevance, if the content indicated by any demonstration action is inconsistent with the voice content, video understanding is performed on the image frame where the corresponding demonstration action is located to convert the demonstration action into second audio data; the second audio data is dynamically inserted into the first audio data to update the multimedia data and record it.

[0008] In some implementations, determining the semantic relevance of each of the demonstration actions to the speech content includes: determining that the first demonstration action is related to the speech content if the duration of the first demonstration action and / or the host's attention time is greater than a first threshold; or, determining that the second demonstration action is related to the speech content if the second demonstration action is a recognized action that represents a strong emotion; or, determining that the third demonstration action is related to the speech content if the similarity between the features of the extracted third demonstration action and the features of the extracted speech content meets a second threshold.

[0009] In some implementations, dynamically inserting the second audio data into the first audio data to update the multimedia data and record it includes: dynamically inserting the second audio data into the first audio data to obtain target audio data; replacing the first audio data in the multimedia data with the target audio data; synchronizing the target audio data and the image frame data to obtain updated multimedia data and record it.

[0010] In some implementations, dynamically inserting the second audio data into the first audio data to obtain target audio data includes: determining the action occurrence interval corresponding to the demonstration action related to the voice content in the first audio data; filtering at least one idle time period from the first audio data that is closest to the end time of the action occurrence interval; determining a target time period among the at least one idle time period that meets the duration requirement; and inserting the second audio data into the target time period to obtain the target audio data.

[0011] In some implementations, the step of performing video understanding on the image frame containing the corresponding demonstration action to convert the related demonstration action into second audio data when the content indicated by any of the demonstration actions is inconsistent with the audio content, based on the semantic relevance, includes: determining the content indicated by the corresponding demonstration action when the semantic relevance indicates that any of the demonstration actions is related to the audio content; performing text recognition on the image frame containing the corresponding demonstration action to obtain a descriptive text for the same demonstration action when the audio content does not include the content indicated by the demonstration action; and generating the second audio data based on the descriptive text.

[0012] In some implementations, the real-time parsing of multimedia data to obtain the voice content of the target object and at least one demonstration action includes: performing speech recognition on the first audio data in the multimedia data to obtain the voice content of the target object; and performing behavior recognition on the image frame data in the multimedia data to obtain at least one demonstration action of the target object.

[0013] In some embodiments, the method further includes: saving the start and end positions of the inserted second audio data and marking them with labels for subsequent editing.

[0014] In some implementations, the multimedia data is generated in a live or recorded broadcast scenario, and the target audience is the broadcaster or presenter.

[0015] In some embodiments, the method further includes: outputting a prompt message when the content indicated by any of the demonstration actions is inconsistent with the audio content; wherein the prompt message is used to remind the scene host to control the recording progress of the multimedia data.

[0016] On the other hand, embodiments of this application provide a multimedia data recording device, the device comprising:

[0017] The data parsing module is used to parse multimedia data in real time to obtain the voice content of the target object and at least one demonstration action; wherein, the multimedia data includes first audio data and image frame data acquired simultaneously;

[0018] A data matching module is used to determine the semantic relevance between each of the demonstration actions and the speech content;

[0019] The image conversion module is used to perform video understanding on the image frame where the corresponding demonstration action is located, based on the semantic relevance, in the case where the content indicated by any of the demonstration actions is inconsistent with the speech content, so as to convert the demonstration action into second audio data;

[0020] The data update module is used to dynamically insert the second audio data into the first audio data to update the multimedia data and record it.

[0021] In another aspect, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0022] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0023] In this embodiment, multimedia data is first parsed in real time to obtain the voice content of the target object and at least one demonstration action. Then, the semantic relevance between each demonstration action and the voice content is determined. Based on this semantic relevance, if the content indicated by any demonstration action is inconsistent with the voice content, video understanding is performed on the image frame containing the corresponding demonstration action to convert the demonstration action into second audio data. Finally, the second audio data is dynamically inserted into the first audio data to update the multimedia data and record it. Thus, by parsing multimedia data in real time to filter out image frames containing missing demonstration actions in the voice content, and then performing video understanding on the image frames to automatically convert the demonstration actions in the image frames into corresponding audio data, which is then dynamically inserted into the previously acquired audio data, complete multimedia data is formed and recorded. This reduces the problem of missing important information that may exist in related technologies, and automatically supplements content without post-production, making it suitable for real-time recording scenarios.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0026] Figure 1 A schematic flowchart of an optional multimedia data recording method provided in an embodiment of this application;

[0027] Figure 2 A schematic flowchart of an optional multimedia data recording method provided in an embodiment of this application;

[0028] Figure 3 A schematic flowchart of an optional multimedia data recording method provided in an embodiment of this application;

[0029] Figure 4 A schematic diagram illustrating a scenario for a multimedia data recording method provided in an embodiment of this application;

[0030] Figure 5 A logic flowchart of a multimedia data recording method provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the composition structure of a multimedia data recording device provided in an embodiment of this application;

[0032] Figure 7This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0037] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0038] Artificial Intelligence (AI) is an interdisciplinary and emerging field based on Computer Science, integrating computer science, psychology, philosophy, and other disciplines. It studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0039] AI-generated content (AIGC) refers to content generated using artificial intelligence technology. It has been widely applied in various scenarios, such as text-based video generation, which can transform text into video and enrich the user's perceptual experience.

[0040] Video understanding is a crucial task in computer vision. In recent years, with the application of deep learning, especially supervised learning, video understanding has experienced rapid development, achieving remarkable results in tasks such as video action classification and video segment summarization. Video understanding encompasses video classification, action recognition, temporal action detection, and video summarization.

[0041] Automatic Speech Recognition (ASR) is a technology that converts human speech into text.

[0042] Video captioning (VC) is a technology that uses machine learning to generate corresponding natural language sentences from videos. It is widely applicable to human-computer interaction scenarios, such as helping visually impaired people describe video content, generate video titles or summaries, etc.

[0043] The data acquisition or recording process is performed through digital conversion. During acquisition, the source material is first converted into a digital signal and then stored on the computer's hard drive. Recording involves outputting the data, which is stored digitally on the computer's hard drive, to a connected recording device for recording.

[0044] This application provides a method for recording multimedia data, which can be executed by a processor of a computer device. The computer device refers to a device capable of recording multimedia data, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1 The following is a schematic diagram of an optional flowchart of the multimedia data recording method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps S110 to S140:

[0045] Step S110: Perform real-time analysis of the multimedia data to obtain the voice content of the target object and at least one demonstration action.

[0046] Here, the multimedia data includes simultaneously acquired first audio data and image frame data. The first audio data includes audio of all objects in the current scene, as well as background music, and can be acquired using an audio acquisition device. The image frame data consists of several images of the target object acquired using an image acquisition device. The audio acquisition device and the image acquisition device may be on the same device or not, but the timestamps of the first audio data and the image frame data must remain consistent.

[0047] In some implementations, the target object can be photographed using an image acquisition device to obtain image frame data; in other implementations, image frame data containing the target object can also be obtained by extracting from the acquired video stream, and the specific implementation is not limited.

[0048] In practice, general speech recognition technology can be used to perform speech recognition on the first audio data in the multimedia data to obtain the speech content of the target object, and video understanding technology can be used to perform behavior recognition on the image frame data in the multimedia data to obtain at least one demonstration action of the target object.

[0049] The at least one demonstration action may include, but is not limited to, facial expressions, mouth movements, body movements, and gestures combined with objects, such as gesturing a shape, demonstrating some scenes, or taking out some objects or charts to gesture, etc. This application embodiment does not limit this.

[0050] Step S120: Determine the semantic relevance of each of the demonstration actions to the speech content.

[0051] Here, each demonstration action is identified and converted into corresponding text content, and the speech content is also converted into corresponding text content. Then, the similarity between the text content of each demonstration action and the text content converted from the speech content is compared in turn. If the similarity is greater than a certain threshold, the semantic relationship between the corresponding demonstration action and the extracted speech content is determined.

[0052] Step S130: Based on the semantic relevance, if the content indicated by any of the demonstration actions is inconsistent with the speech content, perform video understanding on the image frame where the corresponding demonstration action is located, so as to convert the demonstration action into second audio data.

[0053] Here, the content indicated by any of the demonstration actions is obtained by recognizing the corresponding demonstration action. If the audio content does not include the content indicated by the demonstration action, then the content indicated by the demonstration action cannot be transmitted to the audience in real time via audio.

[0054] In practice, this application embodiment first filters out the image frames containing inconsistent demonstration actions as key frames, then extracts information from the key frames using a trained text recognition model to obtain the text description corresponding to the demonstration action in each key frame; finally, each text description is converted into second audio data.

[0055] Step S140: Dynamically insert the second audio data into the first audio data to update the multimedia data and record it.

[0056] Here, the second audio data is dynamically inserted into the first audio data at an appropriate time according to the context, and then the processed first audio data and the image are synthesized to obtain the updated multimedia data.

[0057] In this embodiment, multimedia data is first parsed in real time to obtain the voice content of the target object and at least one demonstration action. Then, the semantic relevance between each demonstration action and the voice content is determined. Based on this semantic relevance, if the content indicated by any demonstration action is inconsistent with the voice content, video understanding is performed on the image frame containing the corresponding demonstration action to convert the demonstration action into second audio data. Finally, the second audio data is dynamically inserted into the first audio data to update the multimedia data and record it. Thus, by parsing multimedia data in real time to filter out image frames containing missing demonstration actions in the voice content, and then performing video understanding on the image frames to automatically convert the demonstration actions in the image frames into corresponding audio data, which is then dynamically inserted into the previously acquired audio data, complete multimedia data is formed and recorded. This reduces the problem of missing important information that may exist in related technologies, and automatically supplements content without post-production, making it suitable for real-time recording scenarios.

[0058] In some embodiments, step S110 may further include the following steps: performing speech recognition on the first audio data in the multimedia data to obtain the speech content of the target object; and performing behavior recognition on the image frame data in the multimedia data to obtain at least one demonstration action of the target object. Thus, by using speech recognition technology to recognize the speech in the first audio data and extract the speech content of the target object, and by using video understanding technology to perform behavior recognition on the image frame data in the multimedia data to obtain at least one demonstration action of the target object, rapid parsing of the multimedia data is achieved.

[0059] In some embodiments, step S130 may further include the following steps: if the semantic relevance indicates that any of the demonstration actions is related to the speech content, determine the content indicated by the corresponding demonstration action; if the speech content does not include the content indicated by the demonstration action, perform text recognition on the image frame where the corresponding demonstration action is located to obtain a descriptive text of the same demonstration action; and generate the second audio data based on the descriptive text.

[0060] Here, after determining that there are demonstration actions related to the speech content in the image frame data, the demonstration actions with no corresponding audio are further identified as descriptive text. Then, the descriptive text is converted into second audio data through audio editing technology. This enables real-time understanding and audio conversion of the image frames where important demonstration actions are located. This allows for real-time explanation to the audience without interrupting the interview, provided that both parties are sufficiently engaged. It also facilitates post-production.

[0061] In some embodiments, the multimedia data is generated in a live or recorded scenario, and the target audience is the broadcaster or speaker. Thus, the multimedia data recording method provided in this application embodiment can achieve real-time explanation of the content indicated by the speaker's impromptu demonstration actions to the audience without interrupting the speaker's speech in a live or recorded scenario, which is beneficial for attracting the audience's attention and engagement.

[0062] In some embodiments, after step S130, the method further includes the following step: if the content indicated by any of the demonstration actions is inconsistent with the audio content, outputting a prompt message; wherein the prompt message is used to remind the scene host to control the recording progress of the multimedia data. Here, the prompt message can be an audio prompt or a visual prompt, such as reminding the scene host to pay attention to the demonstration actions of the target object by flashing a red dot, reducing interruptions and ensuring the recording effect of the multimedia data.

[0063] based on Figure 1 , Figure 2 A schematic flowchart illustrating an optional multimedia data recording method provided in an embodiment of this application is shown below. Figure 2 As shown, step S120, "determining the semantic relevance of each demonstration action to the speech content," may include the following steps S210 to S230:

[0064] Step S210: If the duration of the first demonstration action and / or the attention time of the scene host are greater than a first threshold, determine that the first demonstration action is related to the voice content.

[0065] Here, the first threshold is an empirical value that can be set according to the actual scenario. If the duration of the first demonstration action is relatively long or the scene host pays attention to the first demonstration action for a long time, it indicates that the target object is gesturing some important content through the first demonstration action, and it can be determined that the first demonstration action is related to the voice content extracted in the same scene.

[0066] Step S220: If the second demonstrated action is a recognized action that represents a strong emotion, determine that the second demonstrated action is related to the speech content.

[0067] Here, the second demonstration action may be the same as or different from the first demonstration action. The commonly accepted actions representing strong emotions, such as waving hands or clapping to express excitement or agitation, and pounding the chest or stomping the table to express anger, are generally considered to accompany emotional excitement with physical gestures, that is, expressing emotions through body language. In such cases, audio corresponding to the gesture instructions is often not captured, requiring special attention.

[0068] Step S230: If the similarity between the features of the extracted third demonstration action and the features of the extracted speech content meets the second threshold, it is determined that the third demonstration action is related to the speech content.

[0069] Here, the third demonstration action may be the same as or different from the first and second demonstration actions. The second threshold is a similarity threshold, such as 0.9 or 0.85. In implementation, the content indicated by the third demonstration action and the speech content are first converted into their respective text information, and then features are extracted from the text of both to determine the similarity. Based on the similarity, it is then determined whether the third demonstration action is related to the speech content.

[0070] In the above embodiments, the semantic relevance between the demonstration action and the speech content is determined by at least one aspect, such as the duration of the demonstration action, the attention time of the scene host, whether the demonstration action belongs to the recognized actions that represent strong emotions, and the similarity between the features of the demonstration action and the features of the speech content extracted in the same scene. This allows for subsequent audio conversion of important demonstration actions, reduces the parsing and conversion of actions arbitrarily triggered by the target object, and improves the real-time recording effect of multimedia data.

[0071] based on Figure 1 , Figure 3 A schematic flowchart illustrating an optional multimedia data recording method provided in an embodiment of this application is shown below. Figure 3 As shown, step S140, "dynamically inserting the second audio data into the first audio data to update the multimedia data and record," may include the following steps S310 to S330:

[0072] Step S310: Dynamically insert the second audio data into the first audio data to obtain the target audio data.

[0073] In some implementations, a second audio data point is automatically inserted during a selected idle period in the first audio data to merge the first and second audio data into a single target audio data segment. The idle period indicates a pause by the speaker, such as the target audience or the presenter, during which no audio information is captured. This insertion of the second audio data, representing the transition between demonstration actions, into the original first audio data facilitates the later recording of complete multimedia data.

[0074] In some implementations, the demonstration action related to the voice content is determined to correspond to an action occurrence interval in the first audio data; at least one idle time period closest to the end time of the action occurrence interval is filtered from the first audio data; a target time period with a duration requirement is determined from the at least one idle time period; and the second audio data is inserted into the target time period to obtain the target audio data.

[0075] Here, the timestamp information of the image frames containing the demonstration action related to the audio content is first determined, including the timestamp of the image frame where the corresponding demonstration action first appears and the timestamp of the image frame representing the end of the demonstration action. Since the image frame data and the first audio data are acquired simultaneously, the action occurrence interval corresponding to the demonstration action is determined in the first audio data based on the timestamp information. Then, based on the context position of the action occurrence interval, a target time period with idle time and a duration longer than the second audio data is selected from the first audio data, and the second audio data is dynamically inserted into the target time period, thus realizing the fusion of the first and second audio data.

[0076] Step S320: Replace the first audio data in the multimedia data with the target audio data.

[0077] Here, the first audio data in the multimedia data generated in the current scene is replaced with the processed target audio data, which supplements the important visual information brought by the demonstration actions of the target object, and can improve the listening experience of the audience.

[0078] Step S330: Synchronize the target audio data and the image frame data to obtain updated multimedia data and record it.

[0079] Here, the target audio data and image frame data are aligned in time based on the timestamp information and then synthesized to obtain updated multimedia data, which is then recorded.

[0080] In practice, the target audio data can be separated into multiple audio frames to ensure that the audio frames and the acquired image frames correspond one-to-one. The corresponding video frames and audio frames have the same timestamp, thereby synchronizing the target audio data and image frame data.

[0081] In the above embodiments, a target time period that meets the requirements is selected according to the context, and the second audio data converted from the relevant demonstration actions is dynamically inserted into the first audio data. This achieves the fusion of the first audio data and the second audio data, so that the fused target audio data completes the real-time explanation of the demonstration actions of the target object to the audience. In this way, complete multimedia data can be recorded in real time without the need to supplement important visual information in the first audio data through post-production, thus simplifying the recording process.

[0082] In some embodiments, the start and end positions of the inserted second audio data are saved and marked with labels for subsequent editing. Here, the annotation function built into the audio editing software can be used to mark the start and end positions of the inserted second audio data. For example, labels can be used to indicate that this data is automatically generated by the system, which can facilitate later editing.

[0083] The method for recording multimedia data described above will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0084] Podcasts are becoming increasingly popular; they are a type of audio media, such as... Figure 4 As shown, the host 41 interviews the guest 42 on a specific topic. The guest 42 is equivalent to the target object described in the embodiments of this application, and in some cases, may also be referred to as the host or speaker. However, during podcast recording or live streaming, the speaker may make some impromptu demonstrations, such as gesturing a graphic, demonstrating a scene, or showing objects or charts. This visual information cannot be conveyed to the audience through audio. Currently, there is no good solution. The host may need to be constantly alert and interrupt the speaker to explain to the audience, but this requires skill from the host, and if both parties are very engaged, they may overlook this information. Another option is to supplement this information in the audio through post-production, but this is time-consuming and laborious, and cannot solve similar problems in live streaming scenarios.

[0085] This application provides a podcast recording all-in-one device with an added camera, which monitors the speaker while recording audio. Utilizing AI technology, when the AI ​​detects important demonstration actions by the speaker besides their voice—such as gesturing a shape, demonstrating a scene, or showing objects or charts—it automatically converts these actions into explanatory audio (equivalent to secondary audio data) after assessing their importance. The importance is characterized by the semantic relevance between the demonstration action and the extracted audio content, and can be determined by factors such as the duration of the demonstration action, the host's attention span, whether there are generally recognized actions conveying strong emotions, the use of a particular image for explanation, and the relevance of the action itself to the explanation.

[0086] Then, based on the context, at an appropriate time, the explanatory audio is automatically inserted into the existing podcast audio (equivalent to the first audio data). Figure 5 A logic flowchart of a multimedia data recording method provided in this application embodiment is shown below. Figure 5 As shown, the podcast all-in-one machine 51 with an added camera performs video capture to obtain image frame data 52 containing the target object; simultaneously, it captures audio of the current scene to obtain first audio data 53. Through video understanding, the relevant demonstration actions contained in the image frame data 52 are converted into second audio data 54. By selecting an appropriate time through real-time understanding, the second audio data 54 is dynamically inserted into the first audio data 53 to obtain the target audio data 55. In some embodiments, the position where the second audio data 54 is inserted can also be saved and marked for easy post-editing.

[0087] When the presenter makes an important demonstration through video analysis, a flashing red dot can alert the host. This allows for real-time explanations to the audience without interrupting the interview, while ensuring both parties are fully engaged, and also facilitates post-production.

[0088] Based on the foregoing embodiments, this application provides a multimedia data recording device. The device includes various modules, sub-modules, and units included in each module, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0089] Figure 6 This is a schematic diagram of the composition structure of a multimedia data recording device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device 600 includes: a data parsing module 610, a data matching module 620, an image conversion module 630, and a data updating module 640, wherein:

[0090] The data parsing module 610 is used to parse multimedia data in real time to obtain the voice content of the target object and at least one demonstration action; wherein, the multimedia data includes first audio data and image frame data acquired simultaneously.

[0091] The data matching module 620 is used to determine the semantic relevance between each of the demonstration actions and the speech content;

[0092] The image conversion module 630 is used to perform video understanding on the image frame where the corresponding demonstration action is located, based on the semantic relevance, in the case where the content indicated by any of the demonstration actions is inconsistent with the speech content, so as to convert the demonstration action into second audio data.

[0093] The data update module 640 is used to dynamically insert the second audio data into the first audio data in order to update the multimedia data and record it.

[0094] In some possible embodiments, the data matching module 620 includes: a first determining submodule, configured to determine that the first demonstration action is related to the voice content if the duration of the first demonstration action and / or the attention time of the scene host is greater than a first threshold; or, a second determining submodule, configured to determine that the second demonstration action is related to the voice content if the second demonstration action is a recognized action representing a strong emotion; or, a third determining submodule, configured to determine that the third demonstration action is related to the voice content if the similarity between the features of the extracted third demonstration action and the features of the extracted voice content satisfies a second threshold.

[0095] In some possible embodiments, the data update module 640 includes: an insertion submodule for dynamically inserting the second audio data into the first audio data to obtain target audio data; a replacement submodule for replacing the first audio data in the multimedia data with the target audio data; and a synchronization submodule for synchronizing the target audio data and the image frame data to obtain updated multimedia data and recording it.

[0096] In some possible embodiments, the insertion submodule includes: a first determining unit, configured to determine the action occurrence interval corresponding to the demonstration action related to the voice content in the first audio data; a filtering unit, configured to filter at least one idle time period from the first audio data that is closest to the end time of the action occurrence interval; a second determining unit, configured to determine a target time period among the at least one idle time period that meets the duration requirement; and an insertion unit, configured to insert the second audio data into the target time period to obtain the target audio data.

[0097] In some possible embodiments, the image conversion module 630 includes: a fourth determining submodule, configured to determine the content indicated by the corresponding demonstration action when the semantic relevance indicates that any of the demonstration actions is related to the speech content; a recognition submodule, configured to perform text recognition on the image frame where the corresponding demonstration action is located to obtain a descriptive text of the same demonstration action when the speech content does not include the content indicated by the demonstration action; and a generation submodule, configured to generate the second audio data based on the descriptive text.

[0098] In some possible embodiments, the data parsing module 610 includes: a speech recognition submodule, used to perform speech recognition on the first audio data in the multimedia data to obtain the speech content of the target object; and a behavior recognition submodule, used to perform behavior recognition on the image frame data in the multimedia data to obtain at least one demonstration action of the target object.

[0099] In some possible embodiments, the device further includes a storage module for storing the start and end positions of the inserted second audio data and marking them with labels for subsequent editing.

[0100] In some possible embodiments, the multimedia data is generated in a live or recorded broadcast scenario, and the target object is the broadcaster or presenter.

[0101] In some possible embodiments, the device further includes an information output module for outputting a prompt message when the content indicated by any of the demonstration actions is inconsistent with the audio content; wherein the prompt message is used to remind the scene host to control the recording progress of the multimedia data.

[0102] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0103] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0104] It should be noted that, in the embodiments of this application, if the above-described multimedia data recording method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0105] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0106] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0107] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0108] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0109] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0110] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0111] Processor 701 typically controls the overall operation of computer device 700.

[0112] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0113] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0114] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0119] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0120] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0121] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for recording multimedia data, the method comprising: Multimedia data is parsed in real time to obtain the voice content of the target object and at least one demonstration action; wherein, the multimedia data includes first audio data and image frame data acquired simultaneously; Determine the semantic relevance between each demonstration action and the speech content; wherein, when the similarity between the text information corresponding to the demonstration action and the text information corresponding to the speech content is greater than a threshold, the demonstration action is determined to be semantically related to the speech content. Based on the semantic relevance, if the content indicated by any of the demonstration actions is inconsistent with the speech content, video understanding is performed on the image frame where the corresponding demonstration action is located to convert the demonstration action into second audio data. The second audio data is dynamically inserted into the first audio data to update the multimedia data and record it.

2. The method according to claim 1, wherein determining the semantic relevance between each of the demonstration actions and the speech content comprises: If the duration of the first demonstration action and / or the attention time of the scene host are greater than a first threshold, it is determined that the first demonstration action is related to the voice content; or, If the second demonstrated action is a recognized expression of strong emotion, then the second demonstrated action is determined to be related to the spoken content; or... If the similarity between the features of the extracted third demonstration action and the features of the extracted speech content meets a second threshold, the third demonstration action is determined to be related to the speech content.

3. The method according to claim 1, wherein dynamically inserting the second audio data into the first audio data to update the multimedia data and recording comprises: The second audio data is dynamically inserted into the first audio data to obtain the target audio data; Replace the first audio data in the multimedia data with the target audio data; Synchronize the target audio data and the image frame data to obtain updated multimedia data and record it.

4. The method according to claim 3, wherein dynamically inserting the second audio data into the first audio data to obtain the target audio data comprises: Determine the interval in the first audio data where the demonstration action related to the voice content occurs; Filter the first audio data to at least one idle time period closest to the end time of the interval in which the action occurred; Determine the target time period whose duration meets the requirements within the at least one idle time period; The second audio data is inserted into the target time period to obtain the target audio data.

5. The method according to any one of claims 1 to 4, wherein, based on the semantic relevance, in the case where the content indicated by any of the demonstration actions is inconsistent with the speech content, video understanding is performed on the image frame containing the corresponding demonstration action to convert the relevant demonstration action into second audio data, comprising: If the semantic relevance indicates that any of the demonstration actions is related to the speech content, the content indicated by the corresponding demonstration action shall be determined. If the audio content does not include the content indicating the demonstration action, text recognition is performed on the image frame where the corresponding demonstration action is located to obtain the descriptive text of the same demonstration action; The second audio data is generated based on the description text.

6. The method according to any one of claims 1 to 4, wherein the real-time parsing of multimedia data to obtain the voice content of the target object and at least one demonstration action comprises: Speech recognition is performed on the first audio data in the multimedia data to obtain the speech content of the target object; Behavior recognition is performed on the image frame data in the multimedia data to obtain at least one demonstration action of the target object.

7. The method according to claim 4, further comprising: Save the start and end positions of the inserted second audio data and mark them with labels for subsequent editing.

8. The method according to any one of claims 1 to 4, wherein the multimedia data is generated in a live broadcast or recorded broadcast scenario, and the target object is the anchor or speaker.

9. The method according to claim 8, further comprising: If the content indicated by any of the demonstration actions is inconsistent with the audio content, a prompt message is output; wherein, the prompt message is used to remind the scene host to control the recording progress of the multimedia data.

10. A multimedia data recording apparatus, the apparatus comprising: The data parsing module is used to parse multimedia data in real time to obtain at least one demonstration action and audio content of the target object; wherein, the multimedia data includes first audio data and image frame data acquired simultaneously; A data matching module is used to determine the semantic relevance between each demonstration action and the speech content; wherein, when the similarity between the text information corresponding to the demonstration action and the text information corresponding to the speech content is greater than a threshold, the demonstration action is determined to be semantically related to the speech content. The image conversion module is used to perform video understanding on the image frame where the corresponding demonstration action is located, based on the semantic relevance, in the case where the content indicated by any of the demonstration actions is inconsistent with the speech content, so as to convert the demonstration action into second audio data; The data update module is used to dynamically insert the second audio data into the first audio data to update the multimedia data and record it.

Images