Multimedia data processing method, apparatus, device, storage medium, program product

Abstract

The present document provides a multimedia data processing method, device, equipment, storage medium and program product, which can improve the accuracy of the generated description information of the multimedia data. The method comprises: acquiring multimedia data, generating first description information of the multimedia data; the multimedia data is divided into a plurality of multimedia segments, and for each multimedia segment, the related information of the multimedia segment is acquired; the related information is used to represent the content of the associated segment; for the multimedia segment, the second description information of the multimedia segment is generated according to the first description information and the related information; and the third description information of the multimedia data is generated based on the second description information of the multimedia segment.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a multimedia data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Currently, multimedia data creators have a need to generate descriptive information for their multimedia data. For example, video creators need to generate storyboards for their videos to gain a deeper understanding of the shooting process. While various technologies offer methods for generating descriptive information for multimedia data, the accuracy of the descriptive information generated by these technologies is generally poor. Summary of the Invention

[0003] In one scenario, a multimedia data processing method, apparatus, device, storage medium, or program product is provided, which can improve the accuracy of the descriptive information in the generated multimedia data.

[0004] Firstly, a multimedia data processing method is provided, including: Acquire multimedia data and generate first description information of the multimedia data; The multimedia data is divided into multiple multimedia segments, and for each multimedia segment, relevant information is obtained; the relevant information is used to represent the content of the associated segment. For the multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information; Based on the second description information of the multimedia segment, the third description information of the multimedia data is generated.

[0005] Secondly, a multimedia data processing apparatus is provided, comprising: The first description generation unit is used to acquire multimedia data and generate first description information of the multimedia data. The relevant information acquisition unit is used to divide the multimedia data into multiple multimedia segments, and acquire relevant information for each multimedia segment; the relevant information is used to represent the content of the associated segment; The second description generation unit is configured to generate second description information for the multimedia segment based on the first description information and the related information. The third description generation unit is used to generate third description information of the multimedia data based on the second description information of the multimedia segment.

[0006] Thirdly, an electronic device is provided, comprising: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method described in the first aspect.

[0007] Fourthly, a computer-readable storage medium is provided for storing computer-executable instructions that, when executed by a processor, implement the method described in the first aspect.

[0008] Fifthly, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] In one scenario, the process involves first acquiring multimedia data and generating a first description of the multimedia data. Next, the multimedia data is segmented into multiple multimedia segments. For each segment, relevant information is acquired, which represents the content of associated segments. Then, for each multimedia segment, a second description is generated based on the first description and the relevant information. Finally, a third description is generated based on the second description. Since the third description combines the first description and the relevant information of the multimedia segments, the first description provides a global overview of the multimedia data, aiding in understanding the overall picture. The relevant information represents the content of associated segments, helping to understand the relationships between them. Therefore, by addressing both the global picture and the relationships between multimedia segments, the accuracy of the second description of the generated multimedia segments is improved, ultimately enhancing the accuracy of the third description. Attached Figure Description

[0010] To more clearly illustrate the technical solution, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description only record some implementation situations. For those skilled in the art, other drawings can be obtained from these drawings without creative effort. Figure 1 This is a schematic diagram illustrating an application scenario of a multimedia data processing method provided in one particular situation. Figure 2 A flowchart illustrating a multimedia data processing method provided in one scenario; Figure 3 A flowchart illustrating a multimedia data processing method provided in another scenario; Figure 4 This is a schematic diagram of a multimedia data processing device provided in one scenario. Figure 5 This is a schematic diagram of the structure of an electronic device provided in one scenario. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in one or more of the scenarios described below, the technical solutions in one or more of the scenarios described below will be clearly and completely described below in conjunction with one or more accompanying drawings. Obviously, the scenarios described are only some scenarios, not all scenarios. Based on one or more of the scenarios described below, all other implementations obtained by those skilled in the art without inventive effort should fall within the protection scope of this document.

[0012] It is understood that before using any of the technical solutions described below, relevant parties should be informed of the type, scope of use, and usage scenarios of the information involved and their authorization obtained in an appropriate manner in accordance with relevant laws and regulations.

[0013] For example, upon receiving a user's proactive request, a prompt message can be sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media that perform the technical solution, based on the prompt message.

[0014] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0015] It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of the various scenarios described below. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the various scenarios described below.

[0016] In one scenario, a multimedia data processing method, apparatus, device, storage medium, or program product is provided, which can improve the accuracy of descriptive information in the generated multimedia data. The multimedia data processing method can be applied to and executed by a terminal device, including but not limited to laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smartphones, smart speakers, smartwatches, smart TVs, in-vehicle terminals, and various other types of terminal devices. The multimedia data processing method can also be applied to and executed by a server, which can be a single server or a server cluster consisting of multiple servers.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a multimedia data processing method provided in one particular situation, such as... Figure 1 As shown, the scenario includes terminal device 101 and server 102. Figure 1 In this configuration, terminal device 101 can be any user's terminal device. Server 102 can be a single server or a server cluster consisting of multiple servers. Figure 1 In this process, the user can send the multimedia data to be processed to the server 102 through the terminal device 101. The server 102 obtains the third description information of the multimedia data through the multimedia data processing method and returns the third description information to the terminal device 101. The terminal device 101 can display the third description information.

[0018] In one example, the multimedia data includes video data, and the third descriptive information includes the storyboard script for the video data. Figure 1 In this process, the user can send the video data to be processed to the server 102 through the terminal device 101. The server 102 obtains the storyboard of the video data through the multimedia data processing method and returns the storyboard to the terminal device 101, which can then display the storyboard.

[0019] I can understand. Figure 1 The scenario described is an illustrative example; this video processing method can also be executed independently by the terminal device.

[0020] Figure 2 This is a flowchart illustrating a multimedia data processing method provided in one scenario, such as... Figure 2 As shown, the method includes: Step S202: Acquire multimedia data and generate first description information of the multimedia data; Step S204: Divide the multimedia data into multiple multimedia segments, and obtain relevant information for each multimedia segment; the relevant information is used to represent the content of the associated segments. Step S206: For the multimedia segment, generate second description information for the multimedia segment based on the first description information and related information; Step S208: Based on the second description information of the multimedia segment, generate the third description information of the multimedia data.

[0021] In one scenario, the process involves first acquiring multimedia data and generating a first description of the multimedia data. Next, the multimedia data is segmented into multiple multimedia segments. For each segment, relevant information is acquired, which represents the content of associated segments. Then, for each multimedia segment, a second description is generated based on the first description and the relevant information. Finally, a third description is generated based on the second description. Since the third description combines the first description and the relevant information of the multimedia segments, the first description provides a global overview of the multimedia data, aiding in understanding the overall picture. The relevant information represents the content of associated segments, helping to understand the relationships between them. Therefore, by addressing both the global picture and the relationships between multimedia segments, the accuracy of the second description of the generated multimedia segments is improved, ultimately enhancing the accuracy of the third description.

[0022] In step S202 above, multimedia data is acquired, and first descriptive information of the multimedia data is generated. In one scenario, the multimedia data may include video data, which can be any type and length of video data such as music videos (MVs), short dramas, short videos, movies, and TV series. The video data may also be audio-visual data with audio information. In another scenario, the multimedia data may include multiple images played at intervals, for example, one image played every second. The multimedia data may or may not have corresponding audio information.

[0023] In one scenario, the first descriptive information includes at least the plot information, scene information, and character information of the multimedia data. The first descriptive information essentially provides a global description of the multimedia data. By setting the first descriptive information to include at least the plot information, scene information, and character information of the multimedia data, it achieves the effect of providing a global description of the multimedia data.

[0024] In one scenario, the first descriptive information for generating multimedia data includes: By analyzing multimedia data using the first multimodal processing model, at least one of the following information is obtained: the plot outline of the multimedia data, the narrative order of the multimedia data, the story background of the multimedia data, the scene environment of the multimedia data, the identity of the characters in the multimedia data, the appearance of the characters in the multimedia data, and the relationship between the various characters in the multimedia data. Based on the plot summary and narrative order, generate plot information; based on the story background and scene environment, generate scene information; and based on the character's identity, appearance, and the relationships between the characters, generate character information.

[0025] In this scenario, the first multimodal processing model can be an AI (Artificial Intelligence) model capable of processing multiple modalities such as video, text, and audio. Since multimedia data typically includes accompanying audio information and subtitles in addition to visual information, the first multimodal processing model can achieve the effect of processing information from various modalities within the multimedia data.

[0026] In this scenario, multimedia data is input into a first multimodal processing model, which analyzes the multimedia data. The first multimodal processing model can output at least one of the following information: a plot summary of the multimedia data, the narrative order of the multimedia data, the story background of the multimedia data, the scene environment of the multimedia data, the identity of the characters in the multimedia data, the appearance of the characters in the multimedia data, and the relationship between the various characters in the multimedia data.

[0027] In practical implementation, the first multimodal processing model can output the following fields: The logline field: This field is used to summarize the entire multimedia data in one sentence. For example, the content of this field is: Character A went to the city park to play alone. From being reserved and nervous when entering the park, to getting to know small animals and running and playing in nature, and finally returning home peacefully with the sunset, completing a healing and relaxing half-day park visit. The world_setting field: This field is used to record the worldview and scene tone of multimedia data. For example, the worldview is a modern urban park. For example, the scene tone is a soft overall color tone, with natural light as the main light during the day and warm orange sunset tones in the evening. The atmosphere is quiet and serene at the beginning, lively and bustling in the middle, and gentle and lazy at the end. It is a daily leisure scene full of life. The `character_profiles` field records the identity, appearance, and relationships between core characters in the multimedia data. Core characters can be preset key characters in the multimedia data, such as the protagonist. For example, the content of this field is: Character A: Light-colored casual sweatshirt + sports shorts, introverted and slow to warm up, initially afraid of unfamiliar environments, but gradually becomes lively and playful. Supporting characters: Park animals, stray orange cat, sparrow, and ducklings by the lake, are Character A's interactive playmates in the park. Relationships between characters: Character A and the animals go from being strangers and tentative to playing partners. The scene_priors field: This field is used to record the main scenes and recurring visual elements in the multimedia data. The main scenes / recurring visual elements can be scenes / elements that appear more frequently than the frequency threshold. For example, the content of this field is: Core scenes: park entrance iron gate, tree-lined lawn, artificial lakeside, flower bed and flower bushes, park exit walkway; Recurring visual elements: colorful bubbles, character A's yellow canvas shoulder bag, sycamore leaves falling in the wind, lake ripples. The narrative_arc field: This field is used to record the main narrative thread and the narrative progression path. An example of the content of this field is: Beginning: Entering the park: A stands at the park entrance, hesitant to enter. Strolling around the park: A is attracted by a cat and walks into the lawn, chasing sparrows and feeding ducks in the lake, gradually letting go of himself. Climax: Playing to his heart's content: A blows bubbles, runs and rolls in the flower bushes, and plays with the small animals. Ending: Leaving in the evening: As the sun sets, A waves goodbye to the small animals and slowly walks out of the park.

[0028] Among the fields mentioned above, the plot summary of multimedia data can include the content of the logline field; the narrative order of multimedia data can include the content of the narrative_arc field; the story background of multimedia data can include the world view part of the world_setting field; the scene environment of multimedia data can include the scene tone of the world_setting field and the content of the scene_priors field; and the character identity, character appearance, and relationships between various characters in multimedia data can all be included in the content of the character_profiles field.

[0029] Next, based on the plot synopsis and narrative order, plot information is generated; based on the story background and setting, scene information is generated; and based on character identities, appearances, and relationships between characters, character information is generated. For example, the plot synopsis and narrative order can be used as plot information, the story background and setting as scene information, and character identities, appearances, and relationships as character information. Furthermore, the plot information, scene information, and character information from the multimedia data can be combined into the first descriptive information of the multimedia data.

[0030] Through the above process, the first description information of the multimedia data is obtained using the first multimodal processing model, so that the first description information can provide a global description of the multimedia data. This ensures that the second description information of each subsequent multimedia segment matches the global description of the multimedia data, and that the second description information of each subsequent multimedia segment is relatively coherent, without losing the plot context, character identities, and visual themes of the multimedia data.

[0031] In some cases, to enrich the initial descriptive information, after generating the plot information, scene information, and character information of the multimedia data, the following can also be added: Construct entity descriptions corresponding to multimedia data; entity descriptions are used to define the naming of entities; multimedia data includes entities; entities include at least one of characters, scenes, and props; Add the entity description to the first description information so that the first description information also includes the entity description.

[0032] Multimedia data includes multiple entities, which can be one or more of the following: characters, scenes, and props. Large Language Models (LLMs) can be used to identify each entity in the multimedia data, assigning each entity a unique name. Based on this unique name, an entity description is constructed. Furthermore, the entity description is added to the first description information, ensuring that the first description information also includes the entity description.

[0033] In one example, the entity descriptions are as follows: Entity 1: Silver-haired singer; Entity 2: Masked dancer; If the above entities appear in the multimedia data, please strictly use the above naming and do not invent your own terms.

[0034] By constructing entity descriptions corresponding to multimedia data, the same name can be used for the same entity in the multimedia data when generating the second description information for each multimedia segment, thereby improving the consistency and coherence of the second description information for each multimedia segment.

[0035] In step S204 above, the multimedia data is divided into multiple multimedia segments. In one example, the multimedia data includes video data, and the multimedia segments include shot clips; the video data is divided into multiple shot clips. In another example, the multimedia data includes multiple images played at intervals, and the multimedia segments include image clips, each image clip including one or more images; the multimedia data can also be referred to as an image sequence.

[0036] In one example, classic vision algorithms can be used to perform segment detection on multimedia data, obtaining multiple multimedia segments and the millisecond-level start and end times and segment numbers for each segment. Furthermore, 1-3 keyframes can be extracted from each multimedia segment, such as the first, middle, and last frames, to serve as static visual reference information for each segment. A mapping relationship is then established between each multimedia segment and its corresponding keyframes. By segmenting the multimedia data into multiple segments, the effect of breaking it down into the smallest analytical units can be achieved, improving the accuracy of the subsequent output of the second descriptive information for each multimedia segment. Moreover, obtaining the millisecond-level start and end times for each multimedia segment avoids the possibility of incorrectly inferring these times during the generation of the second descriptive information.

[0037] In step S204 above, for each multimedia segment, relevant information is obtained. This relevant information indicates the content of associated segments of the multimedia segment. Associated segments are the segments associated with the multimedia segment among the various multimedia segments of the multimedia data. Among the various multimedia segments of the multimedia data, associated segments can be one or more of the preceding and following segments of the multimedia segment.

[0038] In some cases, the associated segment includes the preceding segment of the multimedia segment; the aforementioned acquisition of relevant information about the multimedia segment includes: Obtain one or more first video frames at the end of the previous segment, and obtain the content description information of the previous segment; Based on the first video frame and / or content description information, relevant information about the multimedia segment is obtained.

[0039] When multimedia data includes video data, each multimedia segment comprises multiple video frames. When multimedia data includes multiple images played at intervals, each image can be considered a video frame. It is understood that for the first multimedia segment in the multimedia data, there is no preceding multimedia segment. Therefore, in this case, we will use any multimedia segment other than the first multimedia segment in the multimedia data (hereinafter referred to as multimedia segment A) as an example for explanation.

[0040] For multimedia segment A, one or more first video frames from the end of the preceding segment in the multimedia data are retrieved, along with content description information from the preceding segment. One or more of the first video frames and content description information are used as relevant information for multimedia segment A. The one or more first video frames at the end can be consecutive or non-consecutive video frames at the end. The content description information describes the content of the preceding segment. For example, the content description information could be: Character A is running in the rain, followed by a car.

[0041] In one example, for multimedia segment A, the last video frame of the previous segment in the multimedia data is obtained as the first video frame, and the second description information of the previous segment in the multimedia data is obtained as the content description information of the previous segment. The first video frame and the content description information are used as the relevant information of multimedia segment A.

[0042] By acquiring one or more first video frames at the end of the previous segment, and acquiring the content description information of the previous segment, relevant information of the multimedia segment can be obtained based on the first video frames and / or the content description information. This enables the understanding of the behavior logic and transition logic of the current multimedia segment when generating the second description information of each multimedia segment, by combining the content of the previous segment and the video frames at the end. This allows the second description information to accurately describe the behavior logic and transition logic of the multimedia segment.

[0043] In some cases, the associated segment includes the segment following the multimedia segment; the above-mentioned acquisition of relevant information about the multimedia segment includes: Obtain one or more second video frames at the beginning of the next segment, and obtain relevant information about the multimedia segment based on the second video frames.

[0044] Similarly, when multimedia data includes video data, each multimedia segment comprises multiple video frames. When multimedia data includes multiple images played at intervals, each image can be considered a video frame. It is understood that for the last multimedia segment in the multimedia data, there is no subsequent multimedia segment. Therefore, in this case, we will use any multimedia segment other than the last multimedia segment in the multimedia data (hereinafter referred to as multimedia segment B) as an example for explanation.

[0045] For multimedia segment B, one or more second video frames are retrieved from the beginning of the next segment of multimedia segment B in the multimedia data, and these second video frames are used as relevant information for multimedia segment B. The one or more second video frames at the beginning can be consecutive or non-consecutive video frames. For example, the first video frame at the beginning of the next segment of multimedia segment B in the multimedia data is retrieved, and this first video frame is used as relevant information for multimedia segment B.

[0046] By acquiring one or more second video frames at the beginning of the next segment, and obtaining relevant information about the multimedia segment based on the second video frames, it becomes possible to understand the transition logic of the current multimedia segment when generating the second description information of each multimedia segment, by combining the video frames at the beginning of the next multimedia segment. This allows the second description information to accurately describe the transition logic of the multimedia segment.

[0047] In some cases, the above methods can be combined to obtain relevant information for multimedia segments. Specifically, for the first multimedia segment in the multimedia data, its relevant information includes one or more second video frames at the beginning of the second multimedia segment, such as the first video frame of the second multimedia segment. For multimedia segments other than the first and last multimedia segments in the multimedia data (which can be referred to as the current multimedia segment), its relevant information includes one or more first video frames at the end of the previous segment, such as the last video frame, one or more second video frames at the beginning of the next segment, such as the first video frame, and the content description information of the previous segment, such as the second description information. For the last multimedia segment in the multimedia data, its relevant information includes one or more first video frames at the end of the previous segment, such as the last video frame, and the content description information of the previous segment, such as the second description information. In this example, it can be guaranteed that each multimedia segment has relevant information.

[0048] In the above process, by acquiring relevant information for each multimedia segment, the generated second description information can provide a coherent and unified description of the content of each multimedia segment, thereby improving the accuracy of the second description information.

[0049] In step S206 above, for each multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information of the multimedia segment.

[0050] In one example, the second descriptive information is used to represent the content and camera movement strategy of a multimedia segment. The camera movement strategy refers to the transition strategy between video frames / images within the multimedia segment. In other examples, the second descriptive information can also be used to describe the transition logic of the multimedia segment. By setting the second descriptive information to represent the content and camera movement strategy of a multimedia segment, it is possible to provide a comprehensive and accurate description of the multimedia segment from multiple perspectives.

[0051] In some cases, based on the first descriptive information and related information, second descriptive information for the multimedia segment is generated, including: The multimedia segment is analyzed using the second multimodal processing model based on the first descriptive information and related information to obtain analysis results. The analysis results include one or more of the following: the events described in the multimedia segment, the narrative atmosphere of the multimedia segment, the character states in the multimedia segment, the camera movement information of the multimedia segment, and the transition information between the multimedia segment and related segments. The content of a multimedia segment is determined based on one or more of the event, narrative atmosphere, and character state. The camera movement strategy for the multimedia segment is determined based on camera movement information and transition information.

[0052] In this scenario, the second multimodal processing model can be an AI (Artificial Intelligence) model capable of processing multiple modalities such as video, text, and audio. Since multimedia data typically includes accompanying audio information and subtitles in addition to visual information, the second multimodal processing model can effectively process information from various modalities within the multimedia data. The second multimodal processing model can be the same model as the first multimodal processing model, or they can be different models.

[0053] In this scenario, for each multimedia segment, its relevant information and first description information are input into the second multimodal processing model. The second multimodal processing model analyzes the multimedia segment based on the first description information and relevant information, obtaining at least one of the following: the event described in the multimedia segment, the narrative atmosphere of the multimedia segment, the character state in the multimedia segment, the camera movement information of the multimedia segment, and the transition information between the multimedia segment and related segments. This transition information at least refers to the camera transition method, such as how the next segment is obtained from the previous segment through a change in camera angle. Appropriate prompts can be input into the second multimodal processing model to prompt it to output the above information. In the example of entity description above, "If the above entities appear in the multimedia data, please strictly use the above names and do not invent your own terms" can be considered as a prompt for the second multimodal processing model. Since the multimedia data has been divided into multiple multimedia segments, the second multimodal processing model only needs to focus on how to output the above information for each multimedia segment; it does not need to determine the start and end times of each multimedia segment.

[0054] In a specific example, any multimedia segment is taken as the current multimedia segment. The current multimedia segment, its keyframes, related information, and first description information are input into the second multimodal processing model. Appropriate prompts are also input into the second multimodal processing model to prompt it to enhance the narrative continuity based on the relevant information, understand the action logic and transition logic of the current multimedia segment based on the relevant information, and maintain the consistency of character, prop, and scene naming based on the first description information, so as to avoid using different descriptions for the same entity in different multimedia segments.

[0055] The second multimodal processing model can output the following information based on the input information: the main action and events of the current multimedia segment, the camera movement and shot changes of the current multimedia segment, the lighting atmosphere, color tone and visual style of the current multimedia segment, the transition method of the current multimedia segment and its relationship with the transitions of the preceding and following segments, and the performance state and narrative function of the characters in the current multimedia segment. Furthermore, the second multimodal processing model can output the above information based on the following principles: strictly using predefined templates for output, focusing the output content on "camera language" rather than generalized post-viewing impressions, prioritizing the naming of entities in the first descriptive information, and displaying relevant information from the first descriptive information and each multimedia segment for related content in different multimedia segments.

[0056] In the examples above, the events described in the multimedia clips include the main action, events, and narrative functions of the screen; the narrative atmosphere of the multimedia clips includes the lighting atmosphere, color tone, and visual style; the character states in the multimedia clips include the performance states of the characters; the camera movement information of the multimedia clips includes the camera movement methods and changes in shot size; and the transition information between the multimedia clips and related clips includes the camera transition methods and the transition relationship with the preceding and following clips.

[0057] In one example, taking multimedia data including video data and multimedia clips including shot clips as an example, the second multimodal processing model outputs the following information: Shot 5: Medium shot (slow zoom in) Structured description: Subject and event in the shot: The medium shot focuses on Xiao A, who stops, bends down to pick up a wildflower that has fallen on the grass, gently pinches the petals with her fingertips, and looks up with a smile.

[0058] Camera movement and shot composition changes: the camera slowly zooms in from a wide shot to a medium shot, and the focus gradually shifts from the park environment to Xiao A's actions.

[0059] Lighting, color scheme, and visual style: Sunlight shines obliquely on Xiao A, creating soft side lighting, and the color scheme continues the fresh and warm tone, highlighting the character's innocence.

[0060] The transitions between shots and their relationship to the transitions between shots: a hard cut from the previous shot of a panoramic view of the park; continuing the action of "entering the park" in the previous shot, showing the interactive details of the protagonist.

[0061] The character's performance and narrative function: portraying Xiao A's curiosity and ease, and advancing the narrative line of "exploring the park".

[0062] In the above process, one or more of the events, narrative atmosphere, and character states will be determined as the content of the multimedia segment, camera movement information and transition information will be determined as the camera movement strategy of the multimedia segment, and the content of the multimedia segment and the camera movement strategy of the multimedia segment will be combined into the second descriptive information.

[0063] In this scenario, the second multimodal processing model can be used to accurately obtain the second descriptive information for each multimedia segment, improving the accuracy of generating the second descriptive information. Furthermore, the second descriptive information is used to represent the content and camera movement strategy of the multimedia segment, improving the comprehensiveness of the second descriptive information.

[0064] In some cases, a second multimodal processing model is used to analyze multimedia segments based on the first descriptive information and related information, yielding analysis results, including: The first descriptive information, related information, and multimedia fragments are input into the second multimodal processing model; By using the second multimodal processing model and referring to the first descriptive information, the multimedia segment is understood, and one or more of the following are obtained: the events described in the multimedia segment, the narrative atmosphere of the multimedia segment, and the state of the characters in the multimedia segment. The second multimodal processing model is used to determine the camera movement information of multimedia segments based on the shot change information in the multimedia segments; The second multimodal processing model determines the transition information between multimedia segments and related segments based on relevant information, the entrance video frame of the multimedia segment, and the exit video frame of the multimedia segment.

[0065] In this scenario, the first descriptive information, related information, and multimedia segments can be input into the second multimodal processing model. During processing, the second multimodal model can refer to the first descriptive information to understand the content of the multimedia segments, obtaining one or more of the events described in the multimedia segments, the narrative atmosphere of the multimedia segments, and the character states within the multimedia segments. Through this process, the events described in the multimedia segments, the narrative atmosphere of the multimedia segments, and the character states within the multimedia segments can be matched globally with the multimedia data, and the descriptions of events, narrative atmosphere, and character states across all multimedia segments can be kept consistent and coherent.

[0066] Furthermore, using a second multimodal processing model, the camera movement information of the multimedia clip is determined based on the shot change information within the multimedia clip. For example, the camera movement information of the multimedia clip could be described as "slowly zooming in from a wide shot to a medium shot, with the focus gradually shifting from the park environment to the action of Little A," which represents the shot change information within the multimedia clip.

[0067] Furthermore, through the second multimodal processing model, the transition information between multimedia segments and related segments is determined based on relevant information, the entrance video frame of the multimedia segment, and the exit video frame of the multimedia segment. The entrance video frame of the multimedia segment can be one or more video frames / one or more images at the beginning of the multimedia segment, and the exit video frame of the multimedia segment can be one or more video frames / one or more images at the end of the multimedia segment. Based on relevant information, the entrance video frame of the multimedia segment, and the exit video frame of the multimedia segment, the model can accurately understand the transition information between multimedia segments and related segments, such as obtaining the transition information: "A hard cut from the previous segment's panoramic view of the park; continuing the action of 'entering the park' in the previous segment, showing the interactive details of the protagonist."

[0068] In this scenario, the first descriptive information and related information can be used to help the second multimodal processing model accurately understand the content and camera movement strategy of each multimedia segment, thereby accurately obtaining the second descriptive information of each multimedia segment.

[0069] In step S208 above, third description information of the multimedia data is generated based on the second description information of each multimedia segment. In one case, the multimedia data includes video data, and the multimedia segments include various shot segments of the video data. The second description information of each shot segment can be combined into a storyboard for the video data, and the storyboard for the video data can be used as the third description information. In another case, the multimedia data includes multiple images played at intervals, and each multimedia segment includes at least one image. The second description information of each multimedia segment can be combined into the third description information of the multimedia data.

[0070] In another scenario, based on the second description information of each multimedia segment, third description information of the multimedia data is generated, including: Acquire the first rhythm information of the multimedia data and the start and end time information of each multimedia segment, and determine the second rhythm information of each multimedia segment based on the first rhythm information and the start and end time information; The third descriptive information of the multimedia data is obtained based on the start and end time information of each multimedia segment, the second rhythm information of each multimedia segment, and the second descriptive information of each multimedia segment.

[0071] In this scenario, the first rhythm information of the multimedia data is also obtained. For example, multimedia data is read using a tool library, and the first rhythm information of the multimedia data is obtained. This first rhythm information includes the BPM (Beats Per Minute) of the multimedia data, the millisecond-level timestamps of the accented beats, the millisecond-level timestamps of the starting notes, and the millisecond-level timestamps of significant rhythmic changes. These millisecond-level timestamps of the accented beats, the starting notes, and the significant rhythmic changes can also be referred to as the main beat distribution information. The first rhythm information of the multimedia data can be marked on the beat timeline of the multimedia data.

[0072] Next, the start and end time information of each multimedia segment is obtained, including the start and end times. Based on the first rhythm information and the start and end time information, the second rhythm information of each multimedia segment is determined. Each multimedia segment is marked as having hit a key beat by aligning the beat timeline of the multimedia data with the start and end times of each multimedia segment. Key beats can be downbeats, drumbeats, strong beats, etc. The second rhythm information includes whether each multimedia segment hits a key beat, as well as the timestamp when hitting a key beat (e.g., multimedia segment start beat / multimedia segment end beat) and the rhythm density change of each multimedia segment. Taking a multimedia segment consisting of one shot as an example, the rhythm density change of the multimedia segment refers to the number of multimedia segments (e.g., shots) per unit time, used to represent the density of multimedia segments (e.g., shots). Higher density indicates shorter shots and more frequent cuts (fast cuts, explosions), while lower density indicates longer shots and longer image dwell times (slow motion, lyrical).

[0073] Next, based on the start and end time information of each multimedia segment, the second rhythm information of each multimedia segment, and the second description information of each multimedia segment, the third description information of the multimedia data is obtained.

[0074] Based on the above, when obtaining the third description information of multimedia data, the second rhythm information of each multimedia segment and the first rhythm information of the multimedia data are also considered, making the third description information of the multimedia data more comprehensive.

[0075] In some cases, based on the start and end time information of the multimedia segment, the second rhythm information of the multimedia segment, and the second description information of the multimedia segment, the third description information of the multimedia data is obtained, including: For each multimedia segment, the start and end time information, the second rhythm information, and the second description information of the multimedia segment are fused to obtain the fusion result of the multimedia segment; The parts of the fusion result that do not meet the content requirements are corrected, and the third description information of the multimedia data is obtained based on the corrected fusion result.

[0076] In this process, firstly, for each multimedia segment, the start and end time information, the second rhythm information, and the second description information of that multimedia segment are fused together to obtain the fusion result of that multimedia segment. During fusion, for each multimedia segment, the second description information and the second rhythm information can be merged using the segment's sequence number and start and end time information as the primary key to obtain the fusion result of that multimedia segment.

[0077] Next, it is determined whether there are any parts in the fusion result of each multimedia segment that do not meet the content requirements. These non-compliance includes, but is not limited to: different names used for the same entity in different multimedia segments; out-of-order sequence numbers of the multimedia segments; logical problems in the second description information of multiple related multimedia segments (e.g., conflicting descriptions of a character's behavior in consecutive multimedia segments); missing start and end time information, second rhythm information, or one or more of the second description information in the multimedia segments. If non-compliance is identified, it is corrected. For example, the same entity is renamed to the same name; the sequence numbers of the multimedia segments are arranged in order; the second description information of multiple related multimedia segments is corrected to eliminate logical problems; and missing information in the multimedia segments is added. It is understood that during correction, only non-compliance parts can be corrected, while parts that meet the content requirements do not need to be corrected.

[0078] Finally, the corrected fusion results are combined into third-party descriptive information for multimedia data, such as storyboards for video data. This third-party descriptive information can be structured JSON data, human-oriented third-party descriptive information, or data in a format that can be displayed, such as tables or reports.

[0079] In addition, in some examples, the parts of the fusion results that do not meet the content requirements can be marked by the large model, so that they can be manually reviewed.

[0080] Therefore, in this scenario, for each multimedia segment, the start and end time information, the second rhythm information, and the second description information of the multimedia segment can be fused to obtain the fusion result of the multimedia segment. Furthermore, parts of the fusion result that do not meet the content requirements can be corrected. Based on the corrected fusion result, the third description information of the multimedia data, such as the storyboard of the video data, can be obtained. Thus, accurate and comprehensive third description information can be generated based on the start and end time information, the second rhythm information, and the second description information of the multimedia segment.

[0081] Figure 3 A flowchart illustrating a multimedia data processing method provided in another scenario, such as Figure 3 As shown, this process, using multimedia data including music videos (MVs) as an example, includes the following steps: The input music video undergoes preprocessing and material extraction, a process comprising three parts: first, segmenting the video into scenes and outputting the start and end times of each shot; second, extracting keyframes from each shot, including the first, middle, and last frames; and third, extracting the beats of the music video to obtain BPM and main beat distribution information. Finally, a full-length pre-scan of the music video is performed to obtain a global prior, including plot, scene, and character information.

[0082] Next, based on the start and end times of each shot segment, keyframes, BPM, main beat distribution information, plot information, scene information, and character information of the music video, a shot-level parsing task is constructed. During the parsing task execution, context enhancement processing is first performed, including sliding window memorization (obtaining the second description information of the previous shot segment) for each shot segment, visual frame interpolation for each shot segment (obtaining the last frame of the previous shot segment and the first frame of the next shot segment), and obtaining the entity description of the music video (or entity description dictionary). During the parsing task execution, the above information is input into a multimodal parsing model for shot-by-shot parsing to obtain the second description information for each shot segment.

[0083] Then, the second description information of each shot segment is aligned and assembled with the start and end times of each shot segment, the keyframes of each shot segment, and the BPM and main beat distribution information of the music video to obtain the storyboard of the music video, which is then exported in a structured form. The structured export can be exported as structured data such as tables. During this process, the BPM and main beat distribution information of the music video are marked on the beat timeline of the music video. The second rhythm information of each shot segment can be obtained by aligning the beat timeline of the music video with the start and end times of each shot. Furthermore, by assembling the keyframes of each shot segment, the keyframes of each shot segment can be used as static visual reference information for each shot segment in the second description information.

[0084] By using the above methods, we can improve the accuracy of the second descriptive information of the generated shot segments by considering both the global perspective of video data and the contextual relationships between shot segments, thereby improving the accuracy of the generated storyboard.

[0085] Figure 4 This is a schematic diagram of the structure of a multimedia data processing device provided in one scenario, such as... Figure 4 As shown, the device includes: The first description generation unit 42 is used to acquire multimedia data and generate first description information of the multimedia data. The relevant information acquisition unit 44 is used to divide the multimedia data into multiple multimedia segments, and acquire relevant information for each multimedia segment; the relevant information is used to represent the content of the associated segment; The second description generation unit 46 is configured to generate second description information for the multimedia segment based on the first description information and the relevant information. The third description generation unit 48 is used to generate third description information of the multimedia data based on the second description information of the multimedia segment.

[0086] Optionally, the first description information includes at least the plot information of the multimedia data, the scene information of the multimedia data, and the character information of the multimedia data.

[0087] Optionally, the first description generation unit 42 is specifically configured to: analyze the multimedia data through a first multimodal processing model to obtain at least one of the following information: the plot summary of the multimedia data, the narrative order of the multimedia data, the story background of the multimedia data, the scene environment of the multimedia data, the character identities in the multimedia data, the character appearances in the multimedia data, and the relationships between the various characters in the multimedia data; generate the plot information based on the plot summary and the narrative order, generate the scene information based on the story background and the scene environment, and generate the character information based on the character identities, the character appearances, and the relationships between the various characters.

[0088] Optionally, the first description generation unit 42 is further configured to: after generating the plot information, the scene information, and the character information, construct an entity description corresponding to the multimedia data; the entity description is used to define the naming of the entity; the multimedia data includes the entity; the entity includes at least one of a character, a scene, and a prop; and add the entity description to the first description information so that the first description information also includes the entity description.

[0089] Optionally, the associated segment includes the preceding segment of the multimedia segment; the relevant information acquisition unit 44 is specifically used to: acquire one or more first video frames at the end of the preceding segment, and acquire content description information of the preceding segment; and obtain the relevant information of the multimedia segment based on the first video frames and / or the content description information.

[0090] Optionally, the associated segment includes the next segment of the multimedia segment; the relevant information acquisition unit 44 is specifically used to: acquire one or more second video frames at the beginning of the next segment, and obtain the relevant information of the multimedia segment based on the second video frames.

[0091] Optionally, the second description information is used to represent the content of the multimedia segment and the camera movement strategy of the multimedia segment.

[0092] Optionally, the second description generation unit 46 is specifically configured to: analyze the multimedia segment using a second multimodal processing model based on the first description information and the related information to obtain analysis results; the analysis results include one or more of the following: the event described by the multimedia segment, the narrative atmosphere of the multimedia segment, the character state in the multimedia segment, the camera movement information of the multimedia segment, and the transition information between the multimedia segment and the associated segment; determine the content of the multimedia segment based on one or more of the event, the narrative atmosphere, and the character state, and determine the camera movement strategy of the multimedia segment based on the camera movement information and the transition information.

[0093] Optionally, the second description generation unit 46 is further configured to: input the first description information, the relevant information, and the multimedia segment into a second multimodal processing model; understand the multimedia segment by referring to the first description information through the second multimodal processing model, and obtain one or more of the events described by the multimedia segment, the narrative atmosphere of the multimedia segment, and the character states in the multimedia segment; determine the camera movement information of the multimedia segment based on the camera change information in the multimedia segment through the second multimodal processing model; and determine the transition information between the multimedia segment and the associated segment by using the second multimodal processing model according to the relevant information, the entrance video frame of the multimedia segment, and the exit video frame of the multimedia segment.

[0094] Optionally, the third description generation unit 48 is specifically configured to: acquire first rhythm information of the multimedia data and start and end time information of the multimedia segment; determine second rhythm information of the multimedia segment based on the first rhythm information and the start and end time information; and obtain third description information of the multimedia data based on the start and end time information of the multimedia segment, the second rhythm information of the multimedia segment, and the second description information of the multimedia segment.

[0095] Optionally, the third description generation unit 48 is further configured to: fuse the start and end time information of the multimedia segment, the second rhythm information of the multimedia segment, and the second description information of the multimedia segment for the multimedia segment to obtain a fusion result of the multimedia segment; correct the parts of the fusion result that do not meet the content requirements; and obtain the third description information of the multimedia data based on the corrected fusion result.

[0096] The multimedia data processing devices described above can implement each process of the multimedia data processing method described above and achieve the same effect and function, which will not be repeated here.

[0097] In one instance, an electronic device was also provided. Figure 5 This is a schematic diagram of the structure of an electronic device provided in one scenario, such as... Figure 5 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 501 and memories 502, with the memory 502 storing one or more application programs or data. The memory 502 can be temporary or persistent storage. The application programs stored in the memory 502 may include one or more modules (not shown), each module including a series of computer-executable instructions from the electronic device. Furthermore, the processor 501 may be configured to communicate with the memory 502 and execute the series of computer-executable instructions stored in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input or output interfaces 505, one or more keyboards 506, etc.

[0098] In one specific scenario, the electronic device includes a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following process: Acquire multimedia data and generate first description information of the multimedia data; The multimedia data is divided into multiple multimedia segments, and for each multimedia segment, relevant information is obtained; the relevant information is used to represent the content of the associated segment. For the multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information; Based on the second description information of the multimedia segment, the third description information of the multimedia data is generated.

[0099] The electronic devices described above can implement each process of the multimedia data processing method described above and achieve the same effect and function, so they will not be repeated here.

[0100] In one embodiment, a computer-readable storage medium is also provided for storing computer-executable instructions that, when executed by a processor, perform the following process: Acquire multimedia data and generate first description information of the multimedia data; The multimedia data is divided into multiple multimedia segments, and for each multimedia segment, relevant information is obtained; the relevant information is used to represent the content of the associated segment. For the multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information; Based on the second description information of the multimedia segment, the third description information of the multimedia data is generated.

[0101] The computer-readable storage media described above can implement each process of the multimedia data processing method described above and achieve the same effect and function, which will not be repeated here.

[0102] In one scenario, a computer program product is also provided, the computer program product comprising a computer program that, when executed by a processor, implements the following process: Acquire multimedia data and generate first description information of the multimedia data; The multimedia data is divided into multiple multimedia segments, and for each multimedia segment, relevant information is obtained; the relevant information is used to represent the content of the associated segment. For the multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information; Based on the second description information of the multimedia segment, the third description information of the multimedia data is generated.

[0103] The computer program products described above can implement each process of the multimedia data processing method and achieve the same effect and function, which will not be repeated here.

[0104] In all the above scenarios, the computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0105] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0106] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0107] The systems, devices, modules, or units described above can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0108] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing the above scenarios, the functions of each unit can be implemented in one or more software and / or hardware components.

[0109] Those skilled in the art will understand that one or more of the above-described embodiments can be provided as methods, systems, or computer program products. Therefore, one or more of the above-described embodiments can take the form of entirely hardware, entirely software, or a combination of software and hardware aspects. Furthermore, one or more of the above-described embodiments can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] The above description refers to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the above descriptions. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] One or more of the above scenarios can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more of the above scenarios can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0115] The above scenarios are described in a progressive manner. Similar or identical parts between scenarios can be referred to interchangeably. Each scenario focuses on its differences from the others. In particular, the system is described more simply because it is fundamentally similar to the method; relevant parts can be found in the method section.

[0116] The above description is only a partial description of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A multimedia data processing method, comprising: Acquire multimedia data and generate first description information of the multimedia data; The multimedia data is divided into multiple multimedia segments, and relevant information for each multimedia segment is obtained. The relevant information is used to represent the content of the associated fragment; For the multimedia segment, a second description of the multimedia segment is generated based on the first description information and the relevant information; Based on the second description information of the multimedia segment, the third description information of the multimedia data is generated.

2. The method according to claim 1, wherein the first description information includes at least the plot information of the multimedia data, the scene information of the multimedia data, and the character information of the multimedia data.

3. The method according to claim 2, wherein generating the first descriptive information of the multimedia data includes: The multimedia data is analyzed using a first multimodal processing model to obtain at least one of the following information: the plot summary of the multimedia data, the narrative order of the multimedia data, the story background of the multimedia data, the scene environment of the multimedia data, the character identities in the multimedia data, the character appearances in the multimedia data, and the relationships between the various characters in the multimedia data. Based on the plot synopsis and the narrative order, the plot information is generated; based on the story background and the scene environment, the scene information is generated; and based on the character identity, the character appearance, and the relationships between the characters, the character information is generated.

4. The method according to claim 3, further comprising, after generating the plot information, the scene information, and the character information: Construct entity descriptions corresponding to the multimedia data; the entity descriptions are used to define the naming of the entities; the multimedia data includes the entities; the entities include at least one of the following: characters, scenes, and props. The entity description is added to the first description information so that the first description information also includes the entity description.

5. The method according to claim 1, wherein the associated segment includes the preceding segment of the multimedia segment; and the step of obtaining relevant information of the multimedia segment includes: Obtain one or more first video frames at the end of the previous segment, and obtain content description information of the previous segment; The relevant information of the multimedia segment is obtained based on the first video frame and / or the content description information.

6. The method according to claim 1, wherein the associated segment includes the next segment of the multimedia segment; and the step of obtaining relevant information of the multimedia segment includes: One or more second video frames at the beginning of the next segment are obtained, and the relevant information of the multimedia segment is obtained based on the second video frames.

7. The method according to claim 1, wherein the second description information is used to represent the content of the multimedia segment and the camera movement strategy of the multimedia segment.

8. The method according to claim 7, wherein generating second description information of the multimedia segment based on the first description information and the related information comprises: The multimedia segment is analyzed using the second multimodal processing model based on the first description information and the relevant information to obtain the analysis results. The analysis results include one or more of the following: the events described in the multimedia segment, the narrative atmosphere of the multimedia segment, the character states in the multimedia segment, the camera movement information of the multimedia segment, and the transition information between the multimedia segment and the associated segment; The content of the multimedia segment is determined based on one or more of the event, the narrative atmosphere, and the character state; the camera movement strategy of the multimedia segment is determined based on the camera movement information and the transition information.

9. The method according to claim 8, wherein analyzing the multimedia segment using a second multimodal processing model based on the first descriptive information and the relevant information to obtain analysis results includes: The first description information, the relevant information, and the multimedia segment are input into the second multimodal processing model; By using the second multimodal processing model and referring to the first description information, the multimedia segment is understood to obtain one or more of the following: the event described by the multimedia segment, the narrative atmosphere of the multimedia segment, and the character state in the multimedia segment. Using the second multimodal processing model, the camera movement information of the multimedia segment is determined based on the camera movement information in the multimedia segment; Using the second multimodal processing model, the transition information between the multimedia segment and the associated segment is determined based on the relevant information, the entrance video frame of the multimedia segment, and the exit video frame of the multimedia segment.

10. The method according to claim 1, wherein generating third description information of the multimedia data based on the second description information of the multimedia segment comprises: Acquire the first rhythm information of the multimedia data and the start and end time information of the multimedia segment, and determine the second rhythm information of the multimedia segment based on the first rhythm information and the start and end time information; The third description information of the multimedia data is obtained based on the start and end time information of the multimedia segment, the second rhythm information of the multimedia segment, and the second description information of the multimedia segment.

11. The method according to claim 10, wherein obtaining the third descriptive information of the multimedia data based on the start and end time information of the multimedia segment, the second rhythm information of the multimedia segment, and the second descriptive information of the multimedia segment comprises: For the multimedia segment, the start and end time information, the second rhythm information, and the second description information of the multimedia segment are fused to obtain the fusion result of the multimedia segment; The parts of the fusion result that do not meet the content requirements are corrected, and based on the corrected fusion result, the third description information of the multimedia data is obtained.

12. A multimedia data processing apparatus, comprising: The first description generation unit is used to acquire multimedia data and generate first description information of the multimedia data. The relevant information acquisition unit is used to divide the multimedia data into multiple multimedia segments, and acquire relevant information for each multimedia segment. The relevant information is used to represent the content of the associated fragment; The second description generation unit is configured to generate second description information for the multimedia segment based on the first description information and the related information. The third description generation unit is used to generate third description information of the multimedia data based on the second description information of the multimedia segment.

13. An electronic device, comprising: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method described in any one of claims 1-11.

14. A computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the method described in any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images