Video content text generation method and device, equipment, storage medium and program product

Abstract

The invention provides a video content text generation method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of image processing. The method comprises the following steps: determining video segment contents in a target video based on content information of a video clip in the target video; voice information in the video segment content is converted into text information; generating a video description text based on the text information and the video segment content; and carrying out fusion processing on the text information and the video description text to obtain a video content text. The video content can be analyzed and summarized in combination with the description content and the text information of the video, and the video content is perfected and supplemented, so that a user can receive more details in the video while keeping immersion experience, and the user can have deeper understanding and feeling on the video content.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the field of video content description technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for generating video content text. Background Art

[0002] The content description of the video is to track the behavior changes of key characters and provide key descriptions to users when watching the video, which can help users better understand what is happening on the current screen. However, the video content description implemented in the related art usually generates descriptions for a single scene or clip, which cannot capture the overall narrative and the relationship between the scenes, and cannot solve the problem of generating comprehensive and context-related descriptions for complex content. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for generating video content text.

[0004] In a first aspect, an embodiment of the present disclosure proposes a method for generating video content text, including: determining the video segment content in the target video based on the content information of the video clip in the target video; converting the voice information in the video segment content into text information; generating a video description text based on the text information and the video segment content; and fusing the text information and the video description text to obtain the video content text.

[0005] In a second aspect, an embodiment of the present disclosure proposes a video content text generation device, comprising: a video segment content generation module, configured to determine the video segment content in a target video based on content information of a video clip in the target video; a text information generation module, configured to convert voice information in the video segment content into text information; a video description text generation module, configured to generate a video description text based on the text information and the video segment content; and a video content text generation module, configured to fuse the text information and the video description text to obtain a video content text.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the video content text generation method described in any implementation method in the first aspect when executing.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the video content text generation method described in any implementation manner in the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the video content text generation method as described in any implementation manner in the first aspect.

[0009] The video content text generation method, device, equipment, storage medium and program product provided by the embodiments of the present disclosure perform content description on the processed video content by processing such as identification and grouping of the video content, analyze and summarize the video content in combination with the description content and the text information of the video, thereby improving and supplementing the video content, thereby allowing the user to maintain an immersive experience while receiving more details in the video and having a deeper understanding and feeling of the video content.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0013] Figure 2 A flowchart of a method for generating text from video content provided by an embodiment of the present disclosure;

[0014] Figure 3 A flowchart of an implementation method of determining the content of a video segment in a target video based on content information of a video clip in the target video provided by an embodiment of the present disclosure;

[0015] Figure 4 A flowchart of another implementation of determining the content of a video segment in a target video based on content information of a video clip in the target video provided by an embodiment of the present disclosure;

[0016] Figure 5 A flowchart of generating a video description text based on text information and video segment content provided by an embodiment of the present disclosure;

[0017] Figure 6A flowchart of fusing text information and video description text to obtain video content text provided in an embodiment of the present disclosure;

[0018] Figure 7 A flowchart of another method for generating text from video content provided by an embodiment of the present disclosure;

[0019] Figure 8 A schematic diagram of a flow chart of a method for generating text from video content in an application scenario provided by an embodiment of the present disclosure;

[0020] Fig. 9 A flowchart of a method for generating text from video content in another application scenario provided by an embodiment of the present disclosure;

[0021] Fig.10 A schematic diagram of a flow chart of a method for generating text from video content in another application scenario provided by an embodiment of the present disclosure;

[0022] Fig.11 A structural block diagram of a video content text generation device provided by an embodiment of the present disclosure;

[0023] Fig.12 A structural block diagram of another video content text generation device provided by an embodiment of the present disclosure;

[0024] Fig.13 A schematic diagram of the structure of an electronic device suitable for executing a method for generating text from video content provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the video content text generation method, apparatus, electronic device, and computer-readable storage medium disclosed in the present invention can be applied.

[0028] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0029] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications for realizing information communication between the terminal devices 101, 102, 103 and the server 105 can be installed, such as short video social applications, video live broadcast applications, video playback applications, instant messaging applications, etc.

[0030] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0031] It should be noted that, in addition to being obtained from the terminal devices 101, 102, 103 through the network 104, the information such as the original image, video, sorting information, and transition mode can also be pre-stored locally in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally, it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0032] Since the process of video processing based on images requires more computing resources and stronger computing power, the video content text generation method provided in the subsequent embodiments of the present disclosure is generally performed by the server 105 with stronger computing power and more computing resources, and accordingly, the video content text generation device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations handed over to the server 105 through the video playback application installed thereon, and then output the same result as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but when the video playback application determines that the terminal device where it is located has stronger computing power and more remaining computing resources, it can allow the terminal device to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105, and accordingly, the video content text generation device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0034] Please refer to Figure 2 , Figure 2 A flowchart of a video content text generation method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0035] Step 201: Determine the video segment content in the target video based on the content information of the video clip in the target video.

[0036] This step is intended to be performed by the execution subject of the video content text generation method (for example Figure 1 The server 105 shown in the figure obtains the target video to be processed. In this embodiment, the target video can refer to a pre-loaded video clip (for example, a video clip of a preset duration such as 10 minutes or 20 minutes), or a live clip obtained in real time (for example, a live clip of a volleyball, basketball, football or other event), etc., and the present disclosure is not limited thereto.

[0037] For the target video, the execution subject can segment the video, extract the content information in the segmented video segments, and determine the video segment content of the target video based on the content information. In this embodiment, the content information mainly includes the character information, scene information, time information, event information, etc. of the target video. Through this information, the relationship between the video segments can be further determined, thereby generating video segment content, which is equivalent to the combination of the target video based on the content information in the video segments after segmenting the target video.

[0038] Step 202: Convert the voice information in the video segment content into text information.

[0039] This step is intended to extract voice information from the video segment content by the above-mentioned execution subject and convert the voice information into text information. In this embodiment, this step is mainly for identifying the lines, lyrics, broadcast content, etc. that appear in the video segment content. In actual applications, if the target video is accompanied by subtitles, the subtitles will not be removed during the process of generating the video segment content in step 201. Therefore, the subtitle information will still be attached to the video segment content. In this way, after the voice information of the video segment content is identified and converted into text information, the text information can be further compared with the attached subtitle information, so as to proofread the identified text information.

[0040] Step 203: Generate video description text based on the text information and the video segment content.

[0041] This step is intended to generate a video description text for the video segment content by combining the generated video segment content and the text information obtained by identification and conversion by the above-mentioned execution subject. The video description text is mainly used to explain the actual video content of the video segment content. For example, during the live broadcast of a volleyball game, in a certain round, the commentator's report may be: Player A scored 1 point by attacking. But the actual process of obtaining this 1 point is: Player A received a pass from setter B, and scored 1 point by fast attack and smash. For this scene, the text information is "Player A scored 1 point by attacking". Through the video content text generation method of this embodiment, the execution subject can generate a video description text for the segment in combination with the text information as "Player A received a pass from setter B, and scored 1 point by fast attack and smash".

[0042] Step 204: The text information and the video description text are merged to obtain the video content text.

[0043] This step is intended to combine the video description text with the text information by the above-mentioned execution subject, judge the similarities and differences of the contents therein, and obtain the video content text for the video segment content. The video content text is mainly used to explain the video playback content of the video segment content, and further, it can be used to supplement and assist the text information. Taking the example of the above-mentioned volleyball match live broadcast process as an example, the text information is "Player A attacks and scores 1 point", and the video description text is "Player A receives the pass from setter B and quickly attacks and smashes to score 1 point". After judging the similarities and differences between the contents in the video description text and the text information, the video content text finally obtained by combining the text information and the video description text can be "Player A receives the pass from setter B and quickly attacks and smashes to score". Therefore, by combining the video content text with the text information, the content of this video segment can be enriched. In this way, even if the user does not directly see the video segment content, only through text description or voice broadcast, the video content text description can also have a deeper understanding and feeling of the video content at that time.

[0044] The video content text generation method provided by the embodiment of the present disclosure performs content description on the processed video content by processing such as identification and grouping of the video content, and analyzes and summarizes the video content in combination with the description content and the text information of the video, thereby improving and supplementing the video content, thereby allowing users to maintain an immersive experience while receiving more details in the video and having a deeper understanding and feeling of the video content.

[0045] In practical applications, in the process of determining the content of the video segments in the target video based on the content information of the video segments in the target video in step 201, since the target video is segmented and combined, it is possible that the video content of some segments does not conform to the overall scene. In order to avoid such problems, the video segment combination process can be further optimized. Figure 3 As shown, step 201 mainly includes:

[0046] Step 201a: Perform content recognition and content understanding on the video clip in the target video to obtain content information of the video clip.

[0047] In this embodiment, the process of content recognition of video clips can be mainly implemented based on known deep learning models, such as convolutional neural networks (CNNs). By training a large amount of video data, the deep learning model can recognize and extract text, voice and other information in complex backgrounds. Natural language processing (NLP) technology can also be used to improve the accuracy of text extraction and semantic understanding ability. Through the above-mentioned content recognition process, facial features, character action features, etc. of characters in video clips can be recognized, and scene information, etc. in video clips can also be recognized.

[0048] The process of understanding the content of video clips is mainly based on the existing theories and methods in related fields such as computer vision, image processing, pattern recognition, and deep learning. The specific process mainly includes:

[0049] Feature extraction: Use computer vision algorithms to extract key features from video frames. These features can be low-level features such as the shape, color, and texture of the target object, or high-level features such as the target object's motion trajectory and behavior pattern. The accuracy of feature extraction directly affects the effect of subsequent analysis.

[0050] Target detection and tracking: Specific targets in the video (such as pedestrians, vehicles, animals, etc.) are identified and located through target detection algorithms, and target tracking is performed between consecutive frames to obtain the target's motion trajectory and behavior pattern.

[0051] Behavior analysis and event recognition: Analyze and classify the target's behavior based on its trajectory and movements (such as human activity detection, abnormal behavior recognition, etc.), and identify specific events in the video (such as the process of a game, conversations between people, traffic accidents, fires, etc.).

[0052] Step 201b: Match the content information with the preset scene information and / or target object.

[0053] In this embodiment, the preset scene information can be a specific scene pre-set by the user according to the needs of the video content. For example, for a live broadcast of a game, a "scoring" scene, a "attack and defense conversion" scene, etc. can be set; for a film and television drama video, a "fighting action" scene, a "chess game" scene, etc. can be set. According to the preset scene information, the content information is matched and screened;

[0054] The preset target object can be a specific person or role that the user pre-sets according to the video content. For example, for a live broadcast of a game, the focus can be set to be on a specific athlete; for a film or TV series video, the focus can be set to be on the heroine or hero or a specific role, etc. The content information is matched and filtered according to the preset target object;

[0055] It is understandable that in actual applications, the scene information and the target object can also be matched at the same time, so as to filter the content information that meets the scene information and the target object. For example, filter the content information of the main attacker in the scoring scene of the volleyball game, filter the content information of the player in the chess scene, filter the content of the racing driver in the racing scene, etc.

[0056] Step 201c: retain the video clips that meet the preset scene information and / or target object to obtain the video segment content.

[0057] Through the above process, the video clips corresponding to the preset scene information and / or content information of the target object are retained, which can more accurately focus on the relevant video clips of the key characters, objects or scenes that the user is more concerned about, so that in subsequent processing, a supplementary description of the key characters, objects or scenes can be obtained.

[0058] In practical applications, in the process of determining the video segment contents in the target video based on the content information of the video segments in the target video in step 201, since the target video is segmented and combined, it is possible that the video contents of some segments are not continuous, which may affect the subsequent video description. In order to avoid such problems, the video segment combination process can be further optimized. In some optional implementations of the disclosed embodiments, such as Figure 4 As shown, step 201 mainly includes:

[0059] Step 201h: Perform content recognition and content understanding on the video clip in the target video to obtain content information of the video clip.

[0060] Step 201h is consistent with step 201a. For the same contents, please refer to the corresponding parts of the previous implementation method, which will not be repeated here.

[0061] Step 201i: Calculate a first similarity between two adjacent video segments based on the content information.

[0062] In this embodiment, two adjacent video segments refer to two video segments that are continuous in time, and whether the contents of the two video segments are continuous can be determined by the similarity between the two video segments.

[0063] Step 201j: splicing adjacent video clips whose first similarity is not less than a first preset threshold to obtain video segment content.

[0064] In practical applications, the process of judging continuity is mainly applied to videos that do not require high latency, such as film and television drama clips and video playbacks. For such videos, videos of a certain length (for example, 10 minutes, 20 minutes or 1 hour) can be obtained in advance for processing, so as to judge whether the similarity between two adjacent video clips is not less than a first preset threshold (for example, 80, 85 or 90, which can be set according to actual needs). If the similarity between the two video clips is not less than the first preset threshold, the two video clips are spliced ​​and combined into a video segment content; if the similarity between the two video clips is lower than the first preset threshold, it indicates that the contents of the two video clips are discontinuous, and the two video clips are separated and combined in different video segment contents, and when combined, they can also be combined with other adjacent video clips whose similarity is not less than the first preset threshold into a video segment content. For example, there are five video clips, including: video clip A, video clip B, video clip C, video clip D, and video clip E. Assume that the preset threshold is 85, wherein the similarity between video clip A and video clip B is 80 (lower than the preset threshold 85), the similarity between video clip B and video clip C is 90 (higher than the preset threshold 85), the similarity between video clip C and video clip D is 70 (lower than the preset threshold 85), and the similarity between video clip D and video clip E is 87 (higher than the preset threshold 85). Therefore, video clip A is combined into a video segment content M1, video clip B and video clip C are combined into a video segment content M2, and video clip D and video clip E are combined into a video segment content M3.

[0065] For live videos that have high latency requirements, it can be assumed that each live video segment is discontinuous, so that each live video segment can be directly treated independently as a video segment content.

[0066] Through the above process, video clips with large video content similarity are spliced ​​and combined, thereby ensuring high continuity in the formed video segment content. On the one hand, it can ensure the accuracy of subsequent video segment content description, and on the other hand, it can also improve user viewing experience.

[0067] In some optional implementations of the embodiments of the present disclosure, such as Figure 5 As shown, step 203 mainly includes:

[0068] Step 203a: Use the large language model in combination with the video description text of a video segment content adjacent to the video segment content to analyze the text information of the video segment content and obtain prompt text information of the video segment content.

[0069] In this embodiment, the execution subject can analyze the text information of the video segment content based on the preset conditional rules, using the large language model in combination with the video description text of a video segment content adjacent to the video segment content. Among them, the preset conditional rule refers to the conditional rule used to ask and answer questions to the large language model. Specifically, the large language model can be a question-answering system model, and the corresponding conditional rule is a question for the question-answering system model to answer. For example, the conditional rule is "What is the scene of the video? What is the current subtitle? What does the video mainly express?" For this conditional rule, the video description text of a video segment content adjacent to the video segment content is obtained, so as to extract the description content related to the previous scene xxxx, and extract the current subtitle to obtain the scene, character, event and other information to be described by the video segment content, thereby generating the prompt text information of the video segment content, such as "The previous scene is xxxx, the current subtitle is xxxx, what does the video mainly express?".

[0070] Step 203b: Generate video description text of the video segment content based on the prompt text information using the video understanding model.

[0071] In this embodiment, the execution entity uses an existing video understanding model in the related field (for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a 3D convolutional neural network (3D CNN), a Transformer model, etc.) to generate a video description text of the video segment content based on the prompt text information.

[0072] In some optional implementations of this embodiment, such as Figure 6 As shown, the process of fusion processing in step 204 mainly includes:

[0073] Step 204a: Semantically understand the text information and the video description text respectively to obtain the text content and the description text content.

[0074] In this embodiment, the execution entity uses existing semantic understanding models in related fields (for example, convolutional neural networks (CNN), recurrent neural networks (RNN), 3D convolutional neural networks (3D CNN), Transformer models, etc.) to perform semantic understanding on the text information and the video description text, respectively, thereby obtaining the text content of the text information and the descriptive text content of the video description text.

[0075] Step 204b: Calculate the second similarity between each sentence in the text content and each sentence in the description text content.

[0076] In this embodiment, the similarity between each sentence in the text content and each sentence in the description text content is used as the basis for subsequent fusion processing. Therefore, the execution body calculates the second similarity between each sentence in the text content and each sentence in the description text content respectively.

[0077] Step 204c: Process the description text content based on the second similarity, and integrate the text content and the processed description text content to obtain the video content text.

[0078] After calculating the second similarity, the execution entity can process the description text content based on the second similarity. Specifically, the sentences in the description text content whose second similarity is not less than the second preset threshold are deleted, aiming to eliminate the description text content with the same or similar semantics as the text content to avoid duplication of the description content.

[0079] Then, the text content and the processed description text content are integrated. Specifically, according to the order of the retained sentences and the deleted sentences in the description text content, the sentences in the text content are correspondingly inserted into the positions of the deleted sentences, thereby obtaining the video content text.

[0080] Please refer to Figure 7 , Figure 7 A flowchart of another video content text generation method provided by an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0081] Step 301: Determine the video segment content in the target video based on the content information of the video clip in the target video.

[0082] Step 302: Convert the voice information in the video segment content into text information.

[0083] Step 303: Generate video description text based on the text information and the video segment content.

[0084] Step 304: The text information and the video description text are merged to obtain the video content text.

[0085] The above steps 301-304 are similar to Figure 2 Steps 201-204 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0086] Step 305: Generate voice playback information based on the video content text.

[0087] Step 306: Play the voice message in combination with the text message.

[0088] In practical applications, the video content text generated for the target video is a kind of text information, which can be displayed by a fixed or mobile display device. The display of the mobile display device can improve convenience to a certain extent, so that the user can watch the relevant description content of the video at different locations, rather than just sitting in front of the display device (such as a TV) to watch. However, the method of displaying only by text still has certain limitations, and the user needs to face the screen of the display device specifically to view the corresponding content. Based on this, in the present embodiment, a further implementation method is proposed, which can generate voice playback information based on the video content text, and play the voice playback information in combination with the text information of the target video, so that the user is not limited to obtaining the corresponding information by watching the video content text displayed on the screen, but can directly obtain the description content for the target video by listening to the voice playback, which expands the way and method for the user to receive information, and can also help the user better understand the background and atmosphere of the video content.

[0089] In some optional implementations of the embodiments of the present disclosure, step 306, the process of playing the voice playback information in combination with the text information, mainly includes:

[0090] Step 1: Determine the playback position of the voice playback information in combination with the audio playback information corresponding to the text information.

[0091] The voice playback information is generated based on the video content text, so the corresponding voice playback information also corresponds to the positional relationship between the video content text and the text information. This processing process is also to make the voice playback information correspond to the audio playback information of the original video corresponding to the text information, and play at the position associated with the audio playback information. In specific implementation, the execution subject can record timestamps for the audio playback information and the voice playback information respectively, and determine the playback position of the voice playback information by the timestamps of the two.

[0092] Step 2: Based on the playback position, during the playback of the target video, the voice playback information is played.

[0093] After determining the playback position of the voice playback information, the voice playback information can be played in addition to the audio playback information of the original video during the playback of the target video. Thus, the content that improves and supplements the description of the target video can be played out through voice broadcast for users to listen to, thereby expanding the ways and means for users to receive information and helping users better understand the background and atmosphere of the video content.

[0094] In actual applications, the voice playback information played by the execution subject can also output different language versions according to the needs of the user. For example, the user can pre-select or set the target language of the voice playback they want to hear to be Chinese, English, or Mandarin, a local dialect, etc. After the execution subject determines that the audio playback information of the target video is not the target language selected or set by the user, it can further translate the video content text, convert it into the target language selected or set by the user, and then perform voice broadcast. Through this process, the user can receive the description content of the video according to the language selected or set by himself, better understand the video content, so as to better match the user's usage needs and improve the user experience.

[0095] In practical applications, the video content text generation method provided by the embodiment of the present disclosure can be widely used in the use scenarios of smart homes. For example, it can be applied to the smart home ecosystem and work in coordination with other smart home devices in the ecosystem. For example, when the camera detects that the user has left the TV screen, the audio description function is automatically turned on, the smart home devices are linked, and the audio is automatically transferred to the voice playback device near the user (such as a Bluetooth speaker, etc.) to provide the user with voice broadcast, providing the user with a more intelligent and convenient viewing experience. In addition, the audio description function of the video can also be provided to the user as an optional function to meet the viewing needs of visually impaired users and other users with special needs.

[0096] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 8 The scene example shown is an intelligent audio description scene for film and television content.

[0097] In this scenario, when users watch TV content, the execution entity can efficiently retrieve information such as the character image relationship and series introduction of the series. Combined with the development of the plot content, it can provide additional descriptions of the characters that users choose to follow. At the same time, it can also provide intelligent translation for foreign language dialogues so that non-native users can understand the video content.

[0098] The specific implementation process includes:

[0099] Step 1: For the video clips of the film and television content, content recognition and content understanding are performed on the video clips according to the scene information such as the person of interest (Moon Dong-eun) selected by the user and the plot introduction. The video clips are subjected to continuity detection based on the information obtained from content recognition and content understanding, and continuous video clips are spliced ​​to obtain video segment content of continuous content;

[0100] Step 2: Extract the audio content of the video and obtain the subtitle text [(JooYeo-jeong) "Define Go in one sentence: which opponent occupies more territory...", (Moon Dong-eun) "I like it"], and then combine the previous scene of the plot (Moon Dong-eun takes a car to the park) and other prompt words "What does this video mainly express?" as prompt word text to understand the video content and obtain the video description text (When Moon Dong-eun heard JooYeo-jeong's definition of Go, she thought it was very interesting and couldn't help but smile slightly, indicating that she liked it);

[0101] Step 3: Based on the similarity calculation, filter out the sentences with synonyms in the video description text and the subtitles ("When Moon Dong-eun heard JooYeo-jeong's definition of Go", "Expressed his love"), and then merge them to get the video content text (Moon Dong-eun found it very interesting and couldn't help but smile). Based on the video content text, insert it between the subtitles "Define Go in one sentence: Who occupies more territory between opponents..." and "I like it" for voice playback.

[0102] If the user turns on the translation function, the video content text can be translated and then played in voice after translation.

[0103] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Fig. 9 The scene example shown is a scene for describing the focus of characters in a live broadcast of competitive events.

[0104] In this scenario, when users are watching competitive events, the executive entity can efficiently retrieve the rules of the event, past tactical combinations, and combine the player habits and actual game conditions to provide professional supplementary descriptions of the players that users choose to track, as well as match point reminders and ambient sounds.

[0105] The specific implementation process includes:

[0106] Step 1: for the video clips of the live content, the video clips are subject to content recognition and content understanding based on the scene information such as the current event’s focus figures, competition rules, venue information, and event progress. The video clips are processed based on the information obtained from content recognition and content understanding, and video clips that do not conform to the corresponding theme are filtered out. The filtered video clips are combined to obtain video segment content;

[0107] Step 2: Extract the audio content of the video and obtain the subtitle text [(commentator) "Meng XX passed the first pass to Li XX", (commentator) "Another small diagonal shot"], and then combine it with the previous scene of the live broadcast (Team A saved a ball and created an offensive opportunity for Li XX) and other prompts "What does the video mainly express?", "What is the score on the field, has it reached the match point?" as prompts to understand the video content and obtain the video description text (Li XX quickly jumped up, jumped high, and slammed the ball into the opponent's court. Because Li XX hit a small diagonal shot, the opponent's player failed to effectively block the ball, and Team A successfully scored);

[0108] Step 3: Based on the similarity calculation, filter the sentences with synonyms in the video description text and subtitles ("Because Li XX hit a small diagonal line"), and then merge them to obtain the video content text (Li XX quickly jumped up, leaped high, and slammed the ball into the opponent's court. The opponent's players failed to block the ball effectively, and Team A successfully scored). Based on the video content text, insert the voice after the commentator's voice broadcast "Another small diagonal line was hit" for voice playback.

[0109] In addition, for the scene information pre-set by the user that includes information such as "match point", in response to the prompt word "What is the score on the field, have we reached the match point?", the execution entity can further obtain the video content text "Currently Team B is playing against Team A, the score is 22 to 21, and we have reached the match point" in combination with the video content, and play it by voice.

[0110] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Fig.10 The scene example shown is an intelligent audio description scene for action content.

[0111] In this scenario, when the user is practicing dance / fitness / martial arts / yoga, the executing subject can efficiently retrieve relevant knowledge, turn on audio description, identify the corresponding character's body movements, and generate concise action descriptions for the user, so that the user does not need to follow the screen with his eyes and can focus more on adjusting his own movements.

[0112] The specific implementation process includes:

[0113] Step 1: For the video clips of the dance video, based on the scene information such as characters, venues, and dance types, content recognition and content understanding are performed on the video clips. Based on the information obtained from content recognition and content understanding, the video clips are subjected to continuity detection. After splicing the continuous video clips, the video segment content of the continuous content is obtained;

[0114] Step 2: Extract the audio content of the video to obtain the subtitle text [(Explanation) "First practice the pique step"], and then combine it with other prompt words "What does this video mainly express?" as prompt words to understand the video content and obtain the video description text (Practice the pique step, step forward with your left foot, 80% of the center of gravity falls on the front foot, and 20% falls on the back foot);

[0115] Step 3: Based on the similarity calculation, filter the sentences with synonyms in the video description text and subtitles ("Practice the pique step"), and then merge them to obtain the video content text (take a step forward with your left foot, 80% of your weight falls on the front foot and 20% falls on the back foot). Based on the video content text, play the voice after the explanation voice broadcast "Practice the pique step".

[0116] Further references Fig.11 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video content text generation device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0117] like Fig.11 As shown, the video content text generation device 700 of this embodiment may include: a video segment content generation module 701, a text information generation module 702, a video description text generation module 703 and a video content text generation module 704. Among them, the video segment content generation module 701 is configured to determine the video segment content in the target video based on the content information of the video clip in the target video. The text information generation module 702 is configured to convert the voice information in the video segment content into text information. The video description text generation module 703 is configured to generate a video description text based on the text information and the video segment content. The video content text generation module 704 is configured to fuse the text information and the video description text to obtain the video content text.

[0118] In this embodiment, in the video content text generation device 700, the specific processing of the video segment content generation module 701, the text information generation module 702, the video description text generation module 703 and the video content text generation module 704 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0119] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The video content text generation device provided by this embodiment describes the processed video content by identifying, grouping and other processing of the video content, analyzes and summarizes the video content in combination with the description content and the text information of the video, thereby improving and supplementing the video content, so that the user can maintain an immersive experience while receiving more details in the video, and can have a deeper understanding and feeling of the video content.

[0120] Further references Fig.12 As an implementation of the methods shown in the above figures, the present disclosure provides another embodiment of a video content text generation device, which is similar to Figure 7 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Fig.12 As shown, the video content text generation device 800 of this embodiment may include: a video segment content generation module 801, a text information generation module 802, a video description text generation module 803, a video content text generation module 804, a voice playback information generation module 805 and a voice playback module 806. Among them, the video segment content generation module 801 is configured to determine the video segment content in the target video based on the content information of the video clip in the target video. The text information generation module 802 is configured to convert the voice information in the video segment content into text information. The video description text generation module 803 is configured to generate a video description text based on the text information and the video segment content. The video content text generation module 804 is configured to fuse the text information and the video description text to obtain the video content text. The voice playback information generation module 805 is configured to generate voice playback information based on the video content text. The voice playback module 806 is configured to play the voice playback information in combination with the text information.

[0122] In this embodiment, in the video content text generation device 800, the specific processing of the video segment content generation module 801, the text information generation module 802, the video description text generation module 803, the video content text generation module 804, the voice playback information generation module 805 and the voice playback module 806 and the technical effects thereof can be referred to respectively. Figure 7 The relevant descriptions of steps 301 - 306 in the corresponding embodiment are not repeated here.

[0123] This embodiment exists as a device embodiment corresponding to the above-mentioned method embodiment. The video content text generation device provided in this embodiment describes the processed video content through the identification, grouping and other processing of the video content, analyzes and summarizes the video content in combination with the description content and the text information of the video, and realizes the improvement and supplement of the video content, so that the user can maintain the immersive experience while receiving more details in the video, and can have a deeper understanding and feeling of the video content. In addition, voice playback information can be generated based on the video content text, and the voice playback information can be played in combination with the text information of the target video. It allows users to not only obtain the corresponding information by watching the video content text displayed on the screen, but can directly obtain the description content for the target video by listening to the voice playback, which expands the ways and means for users to receive information, and can also help users better understand the background and atmosphere of the video content.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the video content text generation method described in any of the above embodiments can be implemented when the at least one processor executes.

[0125] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the video content text generation method described in any of the above embodiments when executed.

[0126] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the video content text generation method described in any of the above embodiments.

[0127] Fig.13 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Fig.13As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0129] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a video content text generation method. For example, in some embodiments, the video content text generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or a communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the video content text generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the video content text generation method by any other appropriate means (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating text from video content, comprising: Determining the video segment content in the target video based on the content information of the video clip in the target video; Converting the voice information in the video segment content into text information; Generate a video description text based on the text information and the video segment content; The text information and the video description text are fused to obtain the video content text.

2. The video content text generation method according to claim 1, wherein: The determining of the video segment content in the target video based on the content information of the video clip in the target video includes: Performing content recognition and content understanding on the video clip in the target video to obtain content information of the video clip; Matching the content information with preset scene information and / or target object; The video segments that meet the preset scene information and / or target object are retained to obtain the video segment content.

3. The video content text generation method according to claim 1, wherein: The determining of the video segment content in the target video based on the content information of the video clip in the target video includes: Performing content recognition and content understanding on the video clip in the target video to obtain content information of the video clip; Calculating a first similarity between two adjacent video segments based on the content information; Adjacent video clips whose first similarity is not less than a first preset threshold are spliced ​​to obtain the video segment content.

4. The video content text generation method according to claim 1, wherein: The generating of the video description text based on the text information and the video segment content includes: Using a large language model in combination with a video description text of a video segment content adjacent to the video segment content, analyzing the text information of the video segment content to obtain prompt text information of the video segment content; A video description text of the video segment content is generated based on the prompt text information using a video understanding model.

5. The video content text generation method according to claim 1, wherein: The fusing of the text information and the video description text to obtain the video content text includes: Semantically understand the text information and the video description text respectively to obtain text content and description text content; Calculating a second similarity between each sentence in the text content and each sentence in the description text content; The description text content is processed based on the second similarity, and the text content and the processed description text content are integrated to obtain the video content text.

6. The video content text generation method according to claim 5, wherein: The step of processing the description text content based on the second similarity and integrating the text content and the processed description text content to obtain the video content text includes: Deleting sentences in the description text content whose second similarity is not less than a second preset threshold; According to the order relationship between the retained sentences and the deleted sentences in the description text content, the sentences in the text content are correspondingly inserted into the positions of the deleted sentences to obtain the video content text.

7. The video content text generation method according to any one of claims 1 to 6, wherein: Also includes: Generate voice playback information based on the video content text; The voice playback information is played in combination with the text information.

8. The video content text generation method according to claim 7, wherein: The playing of the voice playback information in combination with the text information includes: Determine the playback position of the voice playback information in combination with the audio playback information corresponding to the text information; Based on the playback position, the voice playback information is played during the playback of the target video.

9. A video content text generation device, comprising: A video segment content generation module is configured to determine the video segment content in the target video based on the content information of the video clip in the target video; A text information generation module, configured to convert the voice information in the video segment content into text information; A video description text generation module, configured to generate a video description text based on the text information and the video segment content; The video content text generation module is configured to fuse the text information and the video description text to obtain the video content text.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video content text generation method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the video content text generation method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method for generating text from video content according to any one of claims 1 to 8 are implemented.

Images