Video understanding processing method and device, equipment and storage medium

Abstract

The invention provides a video understanding processing method and device, equipment and a storage medium, and belongs to the technical field of video understanding. The method comprises the following steps: after any to-be-analyzed video is acquired, extracting audio data of the to-be-analyzed video, and determining text content of the to-be-analyzed video; the method comprises the following steps: segmenting a to-be-analyzed video and text content of the to-be-analyzed video by calling a large language model to obtain a plurality of text paragraphs and a plurality of video segments, and generating a text tag corresponding to any text paragraph and an image tag corresponding to a key frame in each video segment; taking the text tag and the video tag as video tags of a to-be-analyzed video, calling a large language model, and generating a video title of the to-be-analyzed video according to the video tags; and storing the text content, the video tag, the key frame and the video title of the to-be-analyzed video as an analysis result of the to-be-analyzed video. The invention aims to improve the video understanding efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of video understanding, and specifically, to a processing method, apparatus, device and storage medium for video understanding. Background Art

[0002] With the development of artificial intelligence technology, a variety of functions can be realized based on artificial intelligence. For example, video understanding and retrieval is a basic research in the field of artificial intelligence. Video understanding mainly analyzes the temporal events in the video, including detection and classification tasks. With the explosive development of short videos and self-media, video understanding and retrieval technology are becoming increasingly important for the management of massive video data that is continuously generated. At the same time, video understanding and retrieval technology can also provide reliable prior knowledge and rich data support for various downstream tasks.

[0003] Currently, there are two main types of video understanding and retrieval technologies.

[0004] One is the video understanding and retrieval technology based on deep neural networks. However, the traditional video understanding and retrieval technology based on deep neural networks relies on a series of sub-model modules, which leads to excessively high costs. In addition, sub-models of different modalities need to be trained based on different training samples, which is costly and difficult. In addition, it strongly relies on serialized model groups for prediction, which leads to increasing error transmission and ultimately causes deviations in video understanding.

[0005] The other is the video understanding and retrieval technology based on a multimodal large language model. The emerging video understanding and retrieval technology based on a multimodal large language model directly takes the video as input, and uses the method of prompt learning to make the multimodal large language model output the output of the specified modal combination to represent the understanding result of the video, and then retrieves it through the retrieval module. However, because the technology strongly relies on the multimodal large language model, it has high requirements for computing resources and the production cost is too high in the understanding of massive videos. In addition, because the knowledge and capabilities emerging from the multimodal large language model are obtained through pre-training based on massive training data, it also has the problem of large language model hallucination, which is easy to produce answers out of thin air, and ultimately leads to deviations in the understanding of video content. Summary of the invention

[0006] In order to solve the above problems, the embodiments of the present application provide a video understanding processing method, apparatus, device and storage medium.

[0007] In a first aspect, an embodiment of the present application provides a method for processing video understanding, the method comprising: After obtaining any video to be analyzed, extracting audio data of the video to be analyzed, and determining text content of the video to be analyzed; By calling the large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated; Taking all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, calling a large language model, and generating a video title of the video to be analyzed according to the video labels; The text content, video tags, key frames and video titles of the video to be analyzed are stored as analysis results of the video to be analyzed.

[0008] Optionally, extracting audio data of the video to be analyzed and determining text content of the video to be analyzed includes: Extracting audio data of the video to be analyzed; The text content of the video to be analyzed is generated according to the audio data of the video to be analyzed through a speech recognition model.

[0009] Optionally, by calling a large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated, including: By calling the large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments; By calling the large language model, the multiple text paragraphs are respectively analyzed to generate a paragraph summary and a text label of any text paragraph, and determine a key frame extraction range in the video segment corresponding to the text paragraph; Extracting key frames in the video segment according to the key frame extraction range; Through the preset image classification model, the image label of any video segment is generated according to the key frames in the video segment.

[0010] Optionally, the method further comprises: Extract frames from the video to be analyzed according to a first frame extraction rule to obtain a first frame extraction set, wherein the first frame extraction set includes a plurality of video images; Optical character recognition is performed on multiple video images in the first frame extraction set to obtain image text results.

[0011] Optionally, the method further comprises: Extract frames from the video to be analyzed according to a second frame extraction rule to obtain a second frame extraction set, wherein the second frame extraction set includes multiple frames of video images; Face recognition is performed on the multiple video images in the second frame extraction set respectively, and based on a preset face database, character information included in the multiple video images in the second frame extraction set is determined.

[0012] Optionally, storing the text content, video tags, key frames and video titles of the video to be analyzed as analysis results of the video to be analyzed includes: After sorting the text content, video tags, key frames, video titles, image text results and character information of the video to be analyzed, a vectorized analysis result of the video to be analyzed is generated and stored.

[0013] Optionally, the method further comprises: In response to the multimodal search operation, obtaining a text query and / or an image query of a video to be retrieved; Vectorizing the text query and / or image query of the video to be retrieved; According to the vectorized text query and / or image query, the analysis result of the video to be retrieved is searched and returned.

[0014] In a second aspect, an embodiment of the present application provides a video understanding processing device, the device comprising: The audio analysis module is used to extract the audio data of any video to be analyzed and determine the text content of the video to be analyzed after obtaining the video to be analyzed; A label generation module is used to segment the video to be analyzed and the text content of the video to be analyzed by calling a large language model to obtain multiple text paragraphs and multiple video segments, and generate a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment; A title generation module, used to use all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, call a large language model, and generate a video title of the video to be analyzed according to the video labels; The analysis result generating module is used to store the text content, video tags, key frames and video titles of the video to be analyzed as the analysis result of the video to be analyzed.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the video understanding processing method described in the first aspect of the embodiment is performed.

[0016] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the video understanding processing method described in the first aspect of the embodiment is performed.

[0017] Beneficial effects: After obtaining any video to be analyzed, extract the audio data of the video to be analyzed, and determine the text content of the video to be analyzed; by calling a large language model, segment the video to be analyzed and the text content of the video to be analyzed to obtain multiple text paragraphs and multiple video segments, and generate text labels corresponding to any text paragraph and image labels corresponding to key frames in each video segment; use all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, call the large language model, and generate a video title of the video to be analyzed according to the video labels; store the text content, video labels, key frames and video title of the video to be analyzed as the analysis results of the video to be analyzed.

[0018] This method uses a large language model in combination with small models such as a speech recognition model and an image classification model to analyze the video. This method improves the efficiency of video understanding and analysis while occupying fewer resources. The video and its text content can be synchronously segmented based solely on the assistance of the large language model, thereby efficiently refining the understanding of the video content and supporting multimodal retrieval of text and images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 is a flowchart of the steps of a video understanding processing method proposed in an embodiment of the present application; Figure 2 is a schematic diagram of a video understanding processing method proposed in an embodiment of the present application; Figure 3 It is a functional module diagram of a video understanding processing device provided by an embodiment of the present application; Figure 4 is a schematic diagram of a computer device proposed in one embodiment of the present application; Figure 5 It is a schematic diagram of a non-volatile readable storage medium proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] LLM: Large Language Models, which are trained with large amounts of data and billions of parameters to generate human-like text or answer natural language questions; large language models are widely used in natural language processing, text generation, and intelligent dialogue. ASR: Automatic Speech Recognition, speech recognition technology, refers to the technology that converts the vocabulary content in human speech into text content; OCR: Optical Character Recognition, is a technology that uses computers to automatically recognize and analyze text in images and convert them into text.

[0023] With the development of artificial intelligence technology, a variety of functions can be realized based on artificial intelligence. For example, video understanding and retrieval is a basic research in the field of artificial intelligence. It mainly uses deep neural network models to automatically and efficiently parse videos. Among them, video understanding mainly parses the temporal events in the video, including tasks such as detection and classification.

[0024] With the explosive development of short videos and self-media, video understanding and retrieval technologies are becoming increasingly important for managing the massive amount of video data that is continuously generated. At the same time, video understanding and retrieval technologies can also provide reliable prior knowledge and rich data support for various downstream tasks.

[0025] There are two main types of video understanding and retrieval technologies: The first type: video understanding and retrieval technology based on deep neural networks; traditional video understanding and retrieval technology based on deep neural networks relies on a series of sub-model modules, and requires different models to make a series of predictions on different data and different targets, so as to achieve video understanding and retrieval.

[0026] The first type: video understanding and retrieval technology based on a multimodal large language model; the emerging video understanding and retrieval technology based on a multimodal large language model directly takes the video as input, and uses the prompt learning method to make the multimodal large language model output the output of the specified modal combination to represent the understanding result of the video, and then retrieves it through the retrieval module.

[0027] However, both of the above two video understanding and detection technologies have their own problems: For video understanding and retrieval technology based on deep neural networks, first of all, a series of models of various modules are needed, which are combined with different objectives. Therefore, a variety of models are needed to understand the content, resulting in excessively high task costs; secondly, traditional video content understanding requires sub-models of various modalities. Different models of the same modality often require different training data for task training, which makes them overly dependent on multiple training data, resulting in increased task cycles and difficulty; finally, since traditional video content understanding based on multiple deep models strongly relies on serialized model groups for prediction, this will lead to increasing error transmission and ultimately lead to deviations in video understanding.

[0028] The advantage of video understanding and retrieval technology based on multimodal large language models is that it only requires promt adjustment and only requires one or very few inputs to obtain video understanding content. However, the disadvantages are also obvious. First, the technology that strongly relies on multimodal large language models has high requirements for computing resources and is too costly to produce in the context of massive video understanding. Secondly, for vertical video content, multimodal large language models still need to be fine-tuned, which makes the technical and computing costs high. Without fine-tuning, the vertical effect is poor. Finally, since the knowledge and capabilities emerging from the multimodal large language model are obtained through pre-training based on massive training data, it also has the problem of large language model hallucination, which makes it easy to fabricate answers out of thin air, ultimately leading to deviations in the understanding of video content.

[0029] Therefore, the embodiment of the present application provides a processing method for video understanding, which can utilize a pure language large language model and a small number of commonly used multimodal model combinations to effectively improve and solve problems such as strong dependence on serialization models, insufficient computing resources, large language model hallucinations, and invalid answers.

[0030] Reference Figure 1 , shows a flowchart of a method for processing video understanding in an embodiment of the present application, and the method may specifically include the following steps: S101: After obtaining any video to be analyzed, extract the audio data of the video to be analyzed and determine the text content of the video to be analyzed.

[0031] The video to be analyzed is a video with any video understanding and analysis requirements, and for example, may be any video uploaded by a user.

[0032] After obtaining any video to be analyzed, firstly, the audio data of the video to be analyzed is extracted, and then the text content of the video to be analyzed is determined according to the audio data of the video to be analyzed.

[0033] In a feasible implementation manner, the process of extracting the audio data of the video to be analyzed and determining the text content of the video to be analyzed may include: First, the audio data of the video to be analyzed is extracted.

[0034] Then, the text content of the video to be analyzed is generated according to the audio data of the video to be analyzed through the speech recognition model.

[0035] The speech recognition model is based on the speech recognition technology ASR, which converts the speech or dialogue content in the audio data of the video to be analyzed into text content.

[0036] In the actual implementation process, any speech recognition model that can convert audio data into text content can be used, and this embodiment does not impose any limitation.

[0037] S102: By calling a large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated.

[0038] In this embodiment, the video to be analyzed is segmented based on a large language model, including dividing the video to be analyzed into multiple video segments, and dividing the text content of the video to be analyzed into multiple text paragraphs, and the video segments and the text paragraphs correspond one to one. For example, the text paragraph of the video segment from time a to time b is the text paragraph from time a to time b.

[0039] Then, the text labels corresponding to any text paragraphs in the video to be analyzed and the image labels corresponding to the key frames in each video segment can be used to generate labels for the video to be analyzed based on the audio content and video content contained in the video to be analyzed. The obtained labels can better represent the content in the video to be analyzed and are more authentic.

[0040] In a feasible implementation manner, by calling a large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated, including the following steps: A1: By calling a large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments.

[0041] That is, the powerful natural language understanding function of the large language model can be used to segment the video to be analyzed.

[0042] Specifically, the text content of the video to be analyzed can be input into the large language model, and segmentation prompt words can be input. The segmentation prompt words are used to instruct the large language model to segment the text content of the video to be analyzed. The development of the current large language model has gradually matured. The large language model has been directly applied after being trained with massive natural languages.

[0043] For example, the segmentation prompt may be "If you are a language expert, and there is a time-series text content "XXXXX...XXXX", please segment this time-series text content."

[0044] In the actual implementation process, actual application scenarios of video analysis and understanding can also be considered, including short video scenarios, school surveillance video scenarios, traffic video surveillance video scenarios, etc. The requirements for segmentation may be different in different application scenarios. Therefore, fine-tuning prompts can be added to the segmentation prompts based on the actual application scenarios. For example, the segmentation prompts can also be: "If you are a language expert, please learn the segmentation result X of the time-series text content A "XXXXX", and then segment the following time-series text content "XXXXX...XXXX"."

[0045] After segmenting the video to be analyzed using the auxiliary segmentation function of the large language model, multiple text paragraphs of the text content of the video to be analyzed can be obtained. For example, the text content of 0-5 minutes is taken as text paragraph 1, the text content of 5-9 minutes is taken as text paragraph 2, and so on.

[0046] According to the division results of multiple text paragraphs of the text content of the video to be analyzed, the video to be analyzed can be divided into multiple video segments accordingly. For example, the video of 0-5 minutes is taken as video segment 1, the video of 5-9 minutes is taken as video segment 2, and so on.

[0047] A2: By calling the large language model, paragraph analysis is performed on the multiple text paragraphs respectively, a paragraph summary and text label of any text paragraph are generated, and a key frame extraction range in the video segment corresponding to the text paragraph is determined.

[0048] Then, the auxiliary summarization function of the large language model can be provided to perform paragraph analysis on each text paragraph, including generating a paragraph summary for each text paragraph and determining the key frame extraction range in the video segment corresponding to any text paragraph.

[0049] Specifically, all text paragraphs of the video to be analyzed can be input into the large language model, as well as paragraph summaries and paragraph prompt words for key text time periods. The paragraph prompt words are used to instruct the large language model to generate paragraph summaries for all text paragraphs of the video to be analyzed, and to determine the key text time periods in each text paragraph.

[0050] For example, the paragraph prompt words can be "If you are a language expert, there are multiple paragraphs of time-sequential text content "XXX"... "XXXXXXX", please generate a paragraph summary for each paragraph of text content, generate labels for each paragraph, and extract which sentence at which moment in each paragraph of text content is the most important."

[0051] In the actual implementation process, fine-tuning prompts can be added to the paragraph prompts in combination with actual application scenarios. For example, the paragraph prompts can also be: "If you are a language expert, please learn the paragraph summary 1 of the text content 1 "XXXXX" and the time and labels of important content. Then, for the following multiple time-sequence text contents "XXX"... "XXXXXXX", please generate paragraph summaries for each paragraph of text content, generate labels for each paragraph, and extract which sentence at which moment in each paragraph of text content is the most important."

[0052] Based on the powerful auxiliary summarization function of the large language model, we can obtain the paragraph summary of each text paragraph of the video to be analyzed and the label of each text paragraph, and determine the key text time period in any text paragraph. For example, the large language model outputs 0-5 minutes of text paragraph 1, and 4 minutes 21-4 minutes 30 seconds are the most important words, so 4 minutes 21-4 minutes 30 seconds can be used as the key text time period.

[0053] The content of the label can be set according to the needs of the user. For example, different users have different labels. If it is a video customer, the labels may include sports, beauty, food, and advertising, etc. If it is a school customer, the labels may include: classes, fighting, quarrels, etc. In the process of the large language model generating text labels, you can also tell the large language model a preset label set, so that the large language model can select the most suitable label and the number of labels in the preset label set according to each text paragraph and the paragraph summary of each text paragraph.

[0054] A3: Extract key frames from the video segments according to the key frame extraction range.

[0055] Then, according to the key text time period, the key frame extraction range in the video segment corresponding to the text paragraph is determined, such as taking 4 minutes 21-4 minutes 30 seconds as the key frame extraction range, and extracting the key frames in 4 minutes 21-4 minutes 30 seconds in the video segment of 0-5 minutes.

[0056] In a feasible implementation, in addition to extracting key frames according to the key text time period in each text paragraph based on the large language model, the key frames can also be extracted based on the information entropy in each frame of video in each video segment. For example, when the information entropy in a frame of video image is greater than the information threshold, it indicates that the frame of video image includes more information, and the frame can be extracted as a key frame.

[0057] In addition, it is also possible to analyze whether there is a scene transition between adjacent videos. If the time t1-t3 is scene 1 and the time t4-t6 is scene 2, a key frame can be extracted for each of the two scenes.

[0058] In the actual implementation process, the key frames of the video to be analyzed may also be extracted in other ways, so as to more comprehensively cover the video content of the video to be analyzed.

[0059] A4: Generate an image label for any video segment based on the key frames in the video segment using a preset image classification model.

[0060] The extracted key frames are input into a preset image classification model. The image classification model may be an existing image classification model. The image classification model may recognize the key frames, identify the objects and behaviors in the key frames, and then generate image labels corresponding to the key frames according to the classification results of the image classification model.

[0061] For example, if the key frame is identified as any kind of food, the image label generated corresponding to the key frame is food; if the action in the key frame is identified as fighting, the image label generated corresponding to the key frame is fighting.

[0062] The mapping relationship between the classification results of the image classification model and the image labels corresponding to the key frames can be mapped according to the actual user's label configuration. Different users may set different labels according to actual business needs. For example, when food 1 is also identified, the label corresponding to user 1 is gourmet food, and the label corresponding to user 2 is vegetables.

[0063] S103: taking all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, calling a large language model, and generating a video title of the video to be analyzed according to the video labels.

[0064] Then, all text labels and all video labels of the video to be analyzed are used as the video labels of the video to be analyzed. Through the video title generation auxiliary function of the large language model, a more authentic video title of the video to be analyzed can be generated according to the video labels.

[0065] Specifically, the video tag of the video to be analyzed may be input into the large language model, and a title generation prompt word may be input, where the title generation prompt word is used to instruct the large language model to generate a video title according to the video tag of the video to be analyzed.

[0066] For example, the title generation prompt may be "If you are a language expert, now generate a title based on the following tag words "XXX" ... "XX"."

[0067] In the actual implementation process, fine-tuning prompts can be added to the title generation prompts in combination with actual application scenarios. For example, the title generation prompts can also be: "If you are a language expert, please learn the titles generated based on the label vocabulary "XX"... "XX", and then generate a title based on the detailed label vocabulary "XXX"... "XX"."

[0068] User-generated videos are customized video titles or video descriptions that are usually inconsistent with the actual content of the video. This method generates a more realistic video title based on the audio content and video image content in the video, so that the video title can be more in line with the video content.

[0069] S104: storing the text content, video tags, key frames and video title of the video to be analyzed as the analysis result of the video to be analyzed, for example, in a video database.

[0070] Finally, the text content, video tags, key frames and video titles of the video to be analyzed are stored as the analysis results of the video to be analyzed. Specifically, the text content, video tags, key frames and video titles can be sorted and then vectorized.

[0071] The fields for sorting text content, video tags, key frames, and video titles after vectorization can be defined according to the needs of actual applications, and this embodiment does not impose any restrictions.

[0072] After vectorization, storage is convenient for subsequent query. If the user wants to query a tag, the tag is vectorized and matched with the vectorization analysis result of any video in the video database. Vectorization can make retrieval easier. For example, although the texts of "food" and "food" are different, the vectorization results of the two are similar, so more relevant video results can be matched.

[0073] In a feasible implementation manner, after obtaining the video to be analyzed, the method further includes: The video to be analyzed is framed according to a first frame extraction rule to obtain a first frame extraction set, wherein the first frame extraction set includes multiple video images; optical character recognition is performed on the multiple video images in the first frame extraction set to obtain image text results.

[0074] Among them, the first frame extraction rule includes obtaining a first frame extraction frequency configured by the user, and then extracting frames for the video to be analyzed according to the first frame extraction frequency to obtain a first frame extraction set; the first frame extraction rule also includes extracting frames according to whether the information entropy in the image frame of the video is greater than the information threshold to obtain a first frame extraction set. In the actual implementation process, the first frame extraction rule can also be defined according to the needs of the actual application.

[0075] Optical character recognition, i.e., OCR recognition, is performed on the multiple video images in the first frame extraction set, so that the text contained in each of the multiple video images in the first frame extraction set can be recognized, and the text contained in all the video images is used as the image text result.

[0076] Reference Figure 2 , showing a schematic diagram of a processing method for video understanding provided by an embodiment of the present application. In a feasible implementation manner, after obtaining any video to be analyzed, in addition to extracting the audio data of the video to be analyzed, the text content of the video to be analyzed is determined; by calling a large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated; all text labels and all video labels of the video to be analyzed are used as video labels of the video to be analyzed, and the large language model is called to generate a video title of the video to be analyzed according to the video labels.

[0077] After obtaining the video to be analyzed, the method further includes: The video to be analyzed is framed according to a second frame extraction rule to obtain a second frame extraction set, wherein the second frame extraction set includes multiple frames of video images; face recognition is performed on the multiple video images in the second frame extraction set respectively, and based on a preset face database, the character information included in the multiple video images in the second frame extraction set is determined.

[0078] Among them, the second frame extraction rule includes obtaining a second frame extraction frequency configured by the user, and then extracting frames for the video to be analyzed according to the second frame extraction frequency to obtain a second frame extraction set; the second frame extraction rule also includes extracting frames according to whether the information entropy in the image frame of the video is greater than the information threshold to obtain a second frame extraction set, or detecting an image containing a face as the extracted video image. In the actual implementation process, the second frame extraction rule can also be defined according to the needs of the actual application.

[0079] Face recognition is performed on multiple video images in the second frame set respectively, and the character information included in the multiple video images in the second frame set is determined based on the user-defined face database. In the actual implementation process, the user can upload different character information as needed.

[0080] Then, after sorting the text content, video tags, key frames, video titles, image text results and character information of the video to be analyzed, a vectorized analysis result of the video to be analyzed is generated and stored, so that the analysis result of the video to be analyzed is richer.

[0081] In response to a multimodal search operation, a text query and / or an image query of a video to be retrieved is obtained; the text query and / or the image query of the video to be retrieved is vectorized; and based on the vectorized text query and / or the image query, an analysis result of the video to be retrieved is searched and returned.

[0082] Storing data after vectorization facilitates subsequent queries, supporting not only text queries but also image queries.

[0083] In the video understanding processing method provided in this embodiment, the auxiliary function of the large language model is used to segment and generate paragraph summaries, text labels, and video titles of text paragraphs, and then the video is analyzed with small models such as speech recognition models and image classification models. It improves the efficiency of video understanding and analysis while occupying fewer resources. Only based on the assistance of the large language model, the video and the text content of the video can be synchronously segmented, thereby efficiently refining the understanding of the video content, and supporting multimodal retrieval of text and images, which has high business production applicability.

[0084] This method is only assisted by a large language model, and uses a new method to synchronously segment the video, which can efficiently refine the video content understanding information. At the same time, only by using a large language model, it can realize the label serialization of the video, and generate more realistic video labels and video titles according to the audio content and video image content in the video. Moreover, the labels can have time sequence and can quickly locate the labeled video.

[0085] Reference Figure 3 , shows a functional module diagram of a video understanding processing device provided in an embodiment of the present application, the device comprising: The audio analysis module 100 is used to extract the audio data of any video to be analyzed and determine the text content of the video to be analyzed after obtaining the video to be analyzed; The label generation module 200 is used to segment the video to be analyzed and the text content of the video to be analyzed by calling the large language model to obtain multiple text paragraphs and multiple video segments, and generate a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment; A title generation module 300 is used to use all text tags and all video tags of the video to be analyzed as video tags of the video to be analyzed, call a large language model, and generate a video title of the video to be analyzed according to the video tags; The analysis result generating module 400 is used to store the text content, video tags, key frames and video titles of the video to be analyzed as the analysis result of the video to be analyzed.

[0086] Optionally, the audio analysis module includes: An extraction unit, used to extract audio data of the video to be analyzed; The speech recognition unit is used to generate text content of the video to be analyzed based on the audio data of the video to be analyzed through a speech recognition model.

[0087] Optionally, the label generation module includes: A segmentation unit, configured to segment the video to be analyzed and the text content of the video to be analyzed by calling a large language model to obtain a plurality of text paragraphs and a plurality of video segments; A text label generating unit, configured to perform paragraph analysis on the plurality of text paragraphs respectively by calling a large language model, generate a paragraph summary and a text label for any text paragraph, and determine a key frame extraction range in a video segment corresponding to the text paragraph; A key frame extraction unit, configured to extract key frames from the video segment according to the key frame extraction range; The image label generation unit is used to generate an image label for a video segment according to a key frame in any video segment through a preset image classification model.

[0088] Optionally, the device further comprises an image text generation module, which is used to: Extract frames from the video to be analyzed according to a first frame extraction rule to obtain a first frame extraction set, wherein the first frame extraction set includes a plurality of video images; Optical character recognition is performed on multiple video images in the first frame extraction set to obtain image text results.

[0089] Optionally, the device further includes a character information recognition module, which is used to: Extract frames from the video to be analyzed according to a second frame extraction rule to obtain a second frame extraction set, wherein the second frame extraction set includes multiple frames of video images; Face recognition is performed on the multiple video images in the second frame extraction set respectively, and based on a preset face database, character information included in the multiple video images in the second frame extraction set is determined.

[0090] Optionally, the analysis result generating module includes: The analysis result generating unit is used to sort the text content, video tags, key frames, video titles, image text results and character information of the video to be analyzed, and then generate a vectorized analysis result of the video to be analyzed for storage.

[0091] Optionally, the device further comprises a retrieval module, configured to: In response to the multimodal search operation, obtaining a text query and / or an image query of a video to be retrieved; Vectorizing the text query and / or image query of the video to be retrieved; According to the vectorized text query and / or image query, the analysis result of the video to be retrieved is searched and returned.

[0092] Reference Figure 4 , shows a schematic diagram of a computer device provided in an embodiment of the present application, including: at least one processor 401, and a memory 402, wherein the memory 402 stores a computer program that can be run on the processor 401, wherein the processor 401 executes the video understanding processing method described in the embodiment when executing the computer program.

[0093] Reference Figure 5 , shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 500 stores a computer program 501, wherein the computer program 501, when executed by a processor, performs the video understanding processing method described in the embodiment.

[0094] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0095] It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0096] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0101] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video understanding processing method, characterized in that: The method comprises: After obtaining any video to be analyzed, extracting audio data of the video to be analyzed, and determining text content of the video to be analyzed; By calling the large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated; Taking all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, calling a large language model, and generating a video title of the video to be analyzed according to the video labels; The text content, video tags, key frames and video titles of the video to be analyzed are stored as analysis results of the video to be analyzed.

2. The method according to claim 1, characterized in that Extracting audio data of the video to be analyzed and determining text content of the video to be analyzed includes: Extracting audio data of the video to be analyzed; The text content of the video to be analyzed is generated according to the audio data of the video to be analyzed through a speech recognition model.

3. The method according to claim 1, characterized in that By calling the large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments, and a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment are generated, including: By calling the large language model, the video to be analyzed and the text content of the video to be analyzed are segmented to obtain multiple text paragraphs and multiple video segments; By calling the large language model, the multiple text paragraphs are respectively analyzed to generate a paragraph summary and a text label of any text paragraph, and determine a key frame extraction range in the video segment corresponding to the text paragraph; Extracting key frames in the video segment according to the key frame extraction range; Through the preset image classification model, the image label of any video segment is generated according to the key frames in the video segment.

4. The method according to claim 1, characterized in that: The method further comprises: Extract frames from the video to be analyzed according to a first frame extraction rule to obtain a first frame extraction set, wherein the first frame extraction set includes a plurality of video images; Optical character recognition is performed on multiple video images in the first frame extraction set to obtain image text results.

5. The method according to claim 4, characterized in that The method further comprises: Extract frames from the video to be analyzed according to a second frame extraction rule to obtain a second frame extraction set, wherein the second frame extraction set includes multiple frames of video images; Face recognition is performed on the multiple video images in the second frame extraction set respectively, and based on a preset face database, character information included in the multiple video images in the second frame extraction set is determined.

6. The method according to claim 5, characterized in that The text content, video tags, key frames and video titles of the video to be analyzed are stored as analysis results of the video to be analyzed, including: After sorting the text content, video tags, key frames, video titles, image text results and character information of the video to be analyzed, a vectorized analysis result of the video to be analyzed is generated and stored.

7. The method according to claim 6, characterized in that The method further comprises: In response to the multimodal search operation, obtaining a text query and / or an image query of a video to be retrieved; Vectorizing the text query and / or image query of the video to be retrieved; According to the vectorized text query and / or image query, the analysis result of the video to be retrieved is searched and returned.

8. A video understanding processing device, characterized in that: The device comprises: An audio analysis module, used to extract the audio data of any video to be analyzed and determine the text content of the video to be analyzed after obtaining the video to be analyzed; A label generation module is used to segment the video to be analyzed and the text content of the video to be analyzed by calling a large language model to obtain multiple text paragraphs and multiple video segments, and generate a text label corresponding to any text paragraph and an image label corresponding to a key frame in each video segment; A title generation module, used to use all text labels and all video labels of the video to be analyzed as video labels of the video to be analyzed, call a large language model, and generate a video title of the video to be analyzed according to the video labels; The analysis result generating module is used to store the text content, video tags, key frames and video titles of the video to be analyzed as the analysis result of the video to be analyzed.

9. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the video understanding processing method described in any one of claims 1-7 is executed.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the video understanding processing method according to any one of claims 1 to 7.

Images