A video content analysis method based on self-supervised learning and related device

Abstract

The present application provides a video content analysis method and related devices based on self-supervised learning, which relates to the field of software technology, and pre-trains different models in a self-supervised learning manner. The target video to be processed is segmented to obtain a first video clip corresponding to the first speaker; the audio in the first video clip is converted into a first audio text through an audio detection model, and the first subtitle text in the first video clip is extracted through a subtitle detection model and a text recognition model, and the visual vector representation of the first video clip is extracted through a video encoder; prompt words are constructed with the first audio text and the first subtitle text, and the prompt words are segmented to obtain a text vector representation of the first video clip; the text vector representation and the visual vector representation are spliced ​​and sent to a large language model to obtain the video content analysis result output by the large language model. The present application improves the generalization of the model through unsupervised learning to achieve different video content analysis tasks.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a video content analysis method based on self-supervised learning and related devices. Background Art

[0002] In the field of video content analysis, traditional methods usually involve selecting specific tasks and using manually labeled data for supervised training for these tasks. This method not only consumes a lot of human resources for data annotation, but also requires customizing specific models for each task, which limits the generalization ability of the model and makes it difficult to adapt to different application scenarios. Summary of the invention

[0003] In view of the above problems, this application provides a video content analysis method and related devices based on self-supervised learning to achieve the purpose of implementing different video content analysis tasks. The specific scheme is as follows:

[0004] The first aspect of the present application provides a video content analysis method based on self-supervised learning, and the video content analysis method based on self-supervised learning comprises:

[0005] Acquire a target video to be processed, and segment the target video to obtain a first video segment corresponding to a first speaker;

[0006] Converting the audio in the first video clip into a first audio text through an audio detection model, extracting the first subtitle text in the first video clip through a subtitle detection model and a text recognition model, and extracting a visual vector representation of the first video clip through a video encoder; wherein the audio detection model, the subtitle detection model, the text recognition model and the video encoder are pre-trained in a self-supervised learning manner;

[0007] Constructing prompt words with the first audio text and the first subtitle text, and obtaining a text vector representation of the first video clip by segmenting the prompt words;

[0008] The text vector representation and the visual vector representation are concatenated and sent to a large language model to obtain a video content analysis result output by the large language model.

[0009] In a possible implementation, segmenting the target video to obtain a first video segment corresponding to the first speaker includes:

[0010] The target video is sequentially subjected to voice activity detection and speaker segmentation to obtain the first video segment.

[0011] In a possible implementation, extracting the first subtitle text in the first video clip by using a subtitle detection model and a text recognition model includes:

[0012] Extracting video frames from the first video clip;

[0013] Detecting a first subtitle bounding box in the video frame by the subtitle detection model;

[0014] Cropping the video frame according to the first subtitle boundary box;

[0015] The first subtitle text is extracted from the cropped result of the video frame by using the text recognition model.

[0016] In a possible implementation, the process of pre-training the audio detection model, the subtitle detection model, the text recognition model, and the video encoder in a self-supervised learning manner includes:

[0017] Build a video data pool by collecting public videos;

[0018] For each video in the video data pool, the video is segmented to obtain a second video segment corresponding to a second speaker; audio is extracted from the second video segment to obtain a second audio text, and at the same time, OCR recognition is performed on the second video segment to obtain a second subtitle boundary box and a second subtitle text; if the second audio text matches the second subtitle text, a video understanding text of the second video segment is extracted through a multimodal large model;

[0019] The second video segment, the audio in the second video segment, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pseudo-label dataset to achieve:

[0020] The audio detection model is obtained by training with the audio in the second video clip and the second audio text; the subtitle detection model is obtained by training with the second video clip and the second subtitle bounding box; the text recognition model is obtained by training with the second video clip and the second subtitle text; and the video encoder is obtained by training with the video understanding text in a manner that the weights of the large language model are fixed.

[0021] In one possible implementation, the video encoder includes a video frame sampling component, an encoder, and a linear layer.

[0022] In a possible implementation, the process of pre-training the audio detection model, the subtitle detection model, the text recognition model, and the video encoder in a self-supervised learning manner further includes:

[0023] If the second audio text matches the second subtitle text, the second video segment is classified into a high-quality data pool.

[0024] A second aspect of the present application provides a video content analysis device based on self-supervised learning, the video content analysis device based on self-supervised learning comprising:

[0025] A self-supervised learning module, which is used to pre-train an audio detection model, a subtitle detection model, a text recognition model, and a video encoder in a self-supervised learning manner;

[0026] The video content analysis module is used to obtain a target video to be processed, and to perform video segmentation on the target video to obtain a first video segment corresponding to a first speaker; convert the audio in the first video segment into a first audio text through the audio detection model, extract the first subtitle text in the first video segment through the subtitle detection model and the text recognition model, and extract the visual vector representation of the first video segment through the video encoder; construct a prompt word with the first audio text and the first subtitle text, and obtain a text vector representation of the first video segment by performing word segmentation on the prompt word; and splice the text vector representation and the visual vector representation and send them to a large language model to obtain a video content analysis result output by the large language model.

[0027] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the video content analysis method based on self-supervised learning according to the first aspect or any implementation method of the first aspect.

[0028] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0029] The memory is used to store computer programs;

[0030] The processor is used to execute the computer program so that the electronic device can implement the video content analysis method based on self-supervised learning according to the above-mentioned first aspect or any implementation method of the first aspect.

[0031] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the video content analysis method based on self-supervised learning according to the above-mentioned first aspect or any implementation method of the first aspect.

[0032] By means of the above technical scheme, the present application provides a video content analysis method and related devices based on self-supervised learning, which are pre-trained in a self-supervised learning manner to obtain an audio detection model, a subtitle detection model, a text recognition model and a video encoder. Obtain the target video to be processed, and segment the target video to obtain the first video clip corresponding to the first speaker; convert the audio in the first video clip into a first audio text through the audio detection model, extract the first subtitle text in the first video clip through the subtitle detection model and the text recognition model, and extract the visual vector representation of the first video clip through the video encoder; construct prompt words with the first audio text and the first subtitle text, and obtain the text vector representation of the first video clip by segmenting the prompt words; splice the text vector representation and the visual vector representation and send them to the large language model to obtain the video content analysis result output by the large language model. The present application improves the generalization of the model through unsupervised learning, can realize different video content analysis tasks, and by integrating the text information and visual information of the video into the large language model, it can efficiently complete the video content analysis task at a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0034] Figure 1 A flowchart of a video content analysis method based on self-supervised learning provided in an embodiment of the present application;

[0035] Figure 2 A partial flow chart of a video content analysis method based on self-supervised learning provided in an embodiment of the present application;

[0036] Figure 3 Another schematic diagram of a process of a video content analysis method based on self-supervised learning provided in an embodiment of the present application;

[0037] Figure 4 An example of a prompt word provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of a video content analysis device based on self-supervised learning provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0041] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0042] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0043] Self-supervised learning: It is an unsupervised learning method that does not require external labeled data to learn the feature representation of the data. Self-supervised learning trains the model by generating pseudo-labels from the data itself. These pseudo-labels are derived from the structure or attributes of the data itself, rather than being labeled by human experts.

[0044] OCR (Optical Character Recognition): is a technology that converts text in an image into machine-encoded text, usually consisting of text detection technology and text recognition technology.

[0045] Subtitle area detection: Detect the subtitle area in the image and present it in the form of a bounding box.

[0046] Voice activity detection: Voice activity detection is a speech recognition technology that distinguishes between speech and non-speech parts and removes silence and background noise.

[0047] Speaker segmentation: This is a technique that divides a continuous audio signal into multiple segments, each corresponding to a specific speaker. Speaker segmentation can more effectively manage and process audio data from multiple speakers, improving the accuracy and efficiency of subsequent processing.

[0048] Speech recognition technology: Automatic Speech Recognition (ASR) is a technology that converts human speech signals into text.

[0049] Large Language Model: Large Language Model (LLM) is a type of natural language processing model based on deep learning. They can understand and generate natural language text through large-scale data training.

[0050] Existing video content analysis technologies usually use supervised learning methods, customize models for specific tasks, and rely heavily on manually labeled data. Although this method can train models with better performance for specific tasks, it also exposes some obvious shortcomings. First, this method requires a lot of manual annotation, which is not only costly, but also time-consuming and labor-intensive. Secondly, because the model is customized for a specific task, it lacks generalization and is difficult to adapt to new scenarios or scenarios with different distributions from the training data. In addition, the migration ability of supervised learning models to new tasks is limited, and data needs to be collected again and the model needs to be trained, which further increases the workload.

[0051] Self-supervised learning is an unsupervised learning method that trains the model by generating pseudo labels from the input data itself. It can learn useful representations for downstream tasks without external labeled data, thereby overcoming these shortcomings to a certain extent. Self-supervised learning does not require external labeled data, but uses the input data itself to generate pseudo labels. It has wide applications in many fields, including image processing, natural language processing, audio analysis, and recommendation systems. Based on this, in order to solve the above problems, the present application provides a video content analysis method and related devices based on self-supervised learning. The present application uses massive unlabeled video data to construct a complex data synthesis pipeline, and combines subtitle detection, text recognition, voice activity detection, speaker segmentation, speech recognition and other technologies to construct a rich text vector representation. The visual vector representation (embedding) containing the key information of the video is obtained through the video encoder, and then the text vector representation and the visual vector representation are analyzed using a large language model. An efficient video content analysis framework is built, which can realize a variety of video content analysis tasks, such as video summarization, video segmentation, and video exciting plot extraction.

[0052] See also Figure 1 , Figure 1 A flowchart of a video content analysis method based on self-supervised learning provided in an embodiment of the present application. Figure 1 As shown, a video content analysis method based on self-supervised learning provided in an embodiment of the present application may include steps S101 to S104, and these steps are described in detail below.

[0053] S101, obtaining a target video to be processed, and segmenting the target video to obtain a first video segment corresponding to a first speaker.

[0054] In the embodiment of the present application, the target video is a video to be subjected to video content analysis, and video segments (i.e., first video segments) corresponding to different speakers (i.e., first speakers) are determined by performing video segmentation. Specifically, one first speaker may correspond to one or more first video segments.

[0055] In a possible implementation, the first video clip can be obtained by voice activity detection and speaker segmentation. In this regard, an embodiment of the present application provides a video content analysis method based on self-supervised learning, wherein in step S10, "segmenting the target video to obtain the first video clip corresponding to the first speaker" can be performed using the following steps:

[0056] The target video is sequentially subjected to voice activity detection and speaker segmentation to obtain a first video segment.

[0057] Specifically, in an embodiment of the present application, video clips with voice and background clips in the target video can be determined based on open source voice activity detection, the background clips can be eliminated, and then the video clips with voice can be speaker segmented to separate the video clips of a specific speaker, thereby constructing a set of video clips corresponding to a single speaker, and the video clip set includes at least one first video clip.

[0058] S102, converting the audio in the first video clip into a first audio text through an audio detection model, extracting the first subtitle text in the first video clip through a subtitle detection model and a text recognition model, and extracting a visual vector representation of the first video clip through a video encoder; wherein the audio detection model, the subtitle detection model, the text recognition model and the video encoder are pre-trained in a self-supervised learning manner.

[0059] In an embodiment of the present application, an audio detection model, a subtitle detection model, a text recognition model, and a video encoder can be pre-trained in a self-supervised learning manner. The audio in the first video clip is input into the audio detection model to obtain the audio text (i.e., the first audio text) output by the audio detection model; the first video clip is input into the subtitle detection model to determine the corresponding subtitle bounding box through the subtitle detection model, and then the subtitle text (i.e., the first subtitle text) in the text bounding box is recognized through the text recognition model. In this regard, detailed video text information in the target video can be obtained, such as (time: 0-3 seconds, speaker 1, first audio text xxxxx, first subtitle text xxxxx), (time: 3-8 seconds, speaker 2, first audio text xxxxx, first subtitle text xxxxx).

[0060] In addition, the first video segment is input into a video encoder to obtain a visual vector representation of the first video segment output by the video encoder.

[0061] See also Figure 2 , Figure 2 A partial flow chart of a video content analysis method based on self-supervised learning provided in an embodiment of the present application. Figure 2 As shown, an embodiment of the present application provides a video content analysis method based on self-supervised learning, wherein "extracting the first subtitle text in the first video clip through a subtitle detection model and a text recognition model" in step S102 may include steps S1021 to S1024, and these steps are described in detail below.

[0062] S1021: Extract video frames from the first video segment.

[0063] In the embodiment of the present application, video frames are extracted from the first video segment in a certain manner, such as extracting at least one video frame at a fixed time interval.

[0064] S1022: Detect a first subtitle boundary box in the video frame using a subtitle detection model.

[0065] In an embodiment of the present application, a video frame is input into a subtitle detection model, and the subtitle detection model outputs a subtitle boundary box (ie, a first subtitle boundary box) in the video frame.

[0066] S1023: Crop the video frame according to the first subtitle boundary box.

[0067] In the embodiment of the present application, the video frame is cropped according to the first subtitle boundary box to obtain the area where the first subtitle boundary box is located.

[0068] S1024, extracting a first subtitle text from the cropped result of the video frame using a text recognition model.

[0069] In an embodiment of the present application, the region obtained by cropping the video frame is input into a text recognition model, and the text recognition model outputs the subtitle text within the first subtitle boundary box (ie, the first subtitle text).

[0070] In one possible implementation, a data pipeline is established to generate a pseudo-labeled dataset to complete the model training. Figure 3 , Figure 3 Another flowchart of a video content analysis method based on self-supervised learning provided in an embodiment of the present application is shown below. Figure 3 As shown, an embodiment of the present application provides a video content analysis method based on self-supervised learning, wherein the process of pre-training an audio detection model, a subtitle detection model, a text recognition model and a video encoder by self-supervised learning includes steps S201 to S203, and these steps are described in detail below.

[0071] S201, building a video data pool by collecting public videos.

[0072] In the embodiments of the present application, a massive amount of videos are collected from the Internet and public datasets to construct a video data pool.

[0073] S202, for each video in the video data pool, the video is segmented to obtain a second video segment corresponding to the second speaker; audio is extracted from the second video segment to obtain a second audio text, and at the same time, OCR recognition is performed on the second video segment to obtain a second subtitle boundary box and a second subtitle text; if the second audio text matches the second subtitle text, the video understanding text of the second video segment is extracted through the multimodal large model.

[0074] In an embodiment of the present application, for each video in a video data pool, voice activity detection and speaker segmentation are performed on the video in turn to obtain video clips (i.e., second video clips) corresponding to different speakers (i.e., second speakers). Specifically, one second speaker may correspond to one or more second video clips. Further, audio recognition technology is used to extract audio from the second video clip to obtain an audio text (i.e., second audio text). At the same time, OCR technology is used to extract text information from the second video clip, and the subtitle text (i.e., second subtitle text) is screened out through logical processing, and the corresponding subtitle boundary box (i.e., second subtitle boundary box) is determined. The second audio text is compared with the second subtitle text. If the two are the same, it is determined that the second audio text matches the second subtitle text, and the multimodal large model is used to extract the video understanding text of the second video text.

[0075] In a possible implementation, a high-quality data pool can be constructed for subsequent video content analysis tasks. In this regard, if the second audio text matches the second subtitle text, the second video clip is classified into the high-quality data pool. In practical applications, the second video clips that match the second audio text and the second subtitle text, the text length of the second audio text / second subtitle text is greater than 50, the number of video frames of the second video clip exceeds 30, and the video clarity exceeds 1080P can be classified into the high-quality data pool, otherwise, they are classified into the low-quality data pool.

[0076] S203, the second video clip, the audio in the second video clip, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pseudo-label data set to achieve:

[0077] An audio detection model is obtained by training with the audio in the second video clip and the second audio text; a subtitle detection model is obtained by training with the second video clip and the second subtitle bounding box; a text recognition model is obtained by training with the second video clip and the second subtitle text; and a video encoder is obtained by training with the video understanding text in a manner that fixes the weights of the large language model.

[0078] In an embodiment of the present application, the second video clip, the audio in the second video clip, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pair of pseudo-label data sets to perform model training based on the pseudo-label data sets. Specifically, the audio in the second video clip and the second audio text are used to train an audio detection model to accurately extract the audio text. The basic model of the audio detection model can be an open source model such as Whisper; since subtitles may not be accurately recognized by relying solely on OCR technology and logical processing, non-subtitle text areas may be mistakenly recognized as subtitle areas, so the second video clip and the second subtitle bounding box are used to train a subtitle detection model to accurately identify the subtitle bounding box. The basic model of the subtitle detection model can be an open source target detection model such as YOLOv11; the second video clip and the second subtitle text are used to train a text recognition model to accurately extract the subtitle text. The basic model of the text recognition model can be an open source text recognition model such as PaddleOCRv4.

[0079] In addition, the video understanding text in the pseudo-label dataset can also be used to train the video encoder. The massive amount of videos in the video data pool can effectively align the video encoder with the text space. The training method adopts the commonly used training method of the large language model, fixes the weight of the large language model, and only trains the weight of the video encoder. Specifically, the second video clip is input into the video encoder to obtain the visual vector representation of the second video clip output by the video encoder; the second audio text and the second subtitle text are used to construct the prompt words for video understanding, and the text vector representation of the second video clip is obtained by analyzing the prompt words; the text vector representation of the second video clip is spliced ​​with the visual vector representation and sent to the large language model to obtain the video understanding text output by the large language model; the video understanding text in the pseudo-label dataset is used as the target, and the video understanding text output by the large language model is used as the prediction result, and the weight of the video encoder is adjusted until the video understanding text output by the large language model approaches the video understanding text in the pseudo-label dataset, and the training of the video encoder is terminated.

[0080] In practical applications, a video encoder may include a video frame sampling component, an encoder, and a linear layer. Specifically, for the video input of the video encoding, the video frame sampling component may sample it, such as using key frames, average sampling, deep learning-based sampling, etc. The sampled video will be sent to the encoder and the linear layer in turn, and the linear layer will output the corresponding visual vector representation. The visual vector representation can also be called visual embedding, which contains rich visual information and can effectively make up for the shortcomings of large language models that lack visual information.

[0081] S103: construct prompt words using the first audio text and the first subtitle text, and obtain a text vector representation of the first video clip by segmenting the prompt words.

[0082] In the embodiment of the present application, based on the video content analysis task, corresponding prompt words are constructed with the first audio text and the first subtitle text, and the text vector representation of the first video clip is obtained by analyzing the prompt words.

[0083] Prompts are text or instructions that provide input to the large language model to guide the large language model to produce specific outputs. As a bridge between the large language model and video text information, prompts can help users implement their instructions. Figure 4 , Figure 4 This is an example of a prompt word provided in the embodiment of the present application. Different prompt words can be configured according to specific task requirements. By adjusting the prompt words, it can be better adapted to the large language model, thereby obtaining better results or realizing more functions.

[0084] S104, concatenating the text vector representation and the visual vector representation and sending them to the large language model to obtain the video content analysis result output by the large language model.

[0085] Video content analysis technology is a technology that uses computer vision and artificial intelligence technology to automatically or semi-automatically analyze video data. This technology can identify objects, scenes, events, etc. in the video and extract the relationships and attributes between these elements. Video content analysis includes many functions, such as video summary, video segmentation, highlights, video retrieval, etc.

[0086] In the embodiment of the present application, the text vector representation and the visual vector representation of the first video clip are spliced ​​and sent to the large language model. The large language model can be used to generate video summaries, video segments, exciting plots, video retrieval, etc.

[0087] Through the above description, a self-supervised learning video content analysis method provided in an embodiment of the present application obtains a basic model with strong generalization performance by constructing a data pipeline and self-supervised learning, and introduces visual information into the large language model through a video encoder to realize multiple video content analysis functions.

[0088] The above introduces a video content analysis method based on self-supervised learning provided in an embodiment of the present application. The following will introduce a device for executing the above-mentioned video content analysis method based on self-supervised learning.

[0089] See also Figure 5 , Figure 5 The structure diagram of a video content analysis device based on self-supervised learning provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, a video content analysis device based on self-supervised learning provided in an embodiment of the present application includes:

[0090] A self-supervised learning module 10 is used to pre-train an audio detection model, a subtitle detection model, a text recognition model and a video encoder in a self-supervised learning manner;

[0091] The video content analysis module 20 is used to obtain a target video to be processed, and to segment the target video to obtain a first video segment corresponding to a first speaker; to convert the audio in the first video segment into a first audio text through an audio detection model, to extract a first subtitle text in the first video segment through a subtitle detection model and a text recognition model, and to extract a visual vector representation of the first video segment through a video encoder; to construct a prompt word with the first audio text and the first subtitle text, and to obtain a text vector representation of the first video segment by segmenting the prompt word; and to splice the text vector representation and the visual vector representation and send them to a large language model to obtain a video content analysis result output by the large language model.

[0092] In a possible implementation, the video content analysis module 20 for segmenting the target video to obtain the first video segment corresponding to the first speaker is specifically configured to:

[0093] The target video is sequentially subjected to voice activity detection and speaker segmentation to obtain a first video segment.

[0094] In a possible implementation, the video content analysis module 20 for extracting the first subtitle text in the first video segment by using the subtitle detection model and the text recognition model is specifically configured to:

[0095] A video frame is extracted from a first video clip; a first subtitle boundary box in the video frame is detected by a subtitle detection model; the video frame is cropped according to the first subtitle boundary box; and a first subtitle text is extracted from the cropped result of the video frame by a text recognition model.

[0096] In a possible implementation, the self-supervised learning module 10 is specifically used to:

[0097] A video data pool is constructed by collecting public videos; for each video in the video data pool, the video is segmented to obtain a second video clip corresponding to a second speaker; audio is extracted from the second video clip to obtain a second audio text, and at the same time, OCR recognition is performed on the second video clip to obtain a second subtitle bounding box and a second subtitle text; if the second audio text matches the second subtitle text, the video understanding text of the second video clip is extracted through a multimodal large model; the second video clip, the audio in the second video clip, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pseudo-label data set to achieve: an audio detection model is obtained by training with the audio in the second video clip and the second audio text; a subtitle detection model is obtained by training with the second video clip and the second subtitle bounding box; a text recognition model is obtained by training with the second video clip and the second subtitle text; and a video encoder is obtained by training with the video understanding text in a manner that the weights of the large language model are fixed.

[0098] In one possible implementation, a video encoder includes a video frame sampling component, an encoder, and a linear layer.

[0099] In a possible implementation, the self-supervised learning module 10 is further used to:

[0100] If the second audio text matches the second subtitle text, the second video segment is classified into the high-quality data pool.

[0101] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned video content analysis method embodiment based on self-supervised learning, and will not be repeated here.

[0102] The present application also provides an electronic device in an embodiment. Figure 6 , Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0103] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0104] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0105] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the video content analysis methods based on self-supervised learning provided in the embodiments of the present application.

[0106] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the video content analysis methods based on self-supervised learning provided in the embodiment of the present application.

[0107] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0108] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0109] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0110] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A video content analysis method based on self-supervised learning, characterized in that: The video content analysis method based on self-supervised learning includes: Acquire a target video to be processed, and segment the target video to obtain a first video segment corresponding to a first speaker; Converting the audio in the first video clip into a first audio text through an audio detection model, extracting the first subtitle text in the first video clip through a subtitle detection model and a text recognition model, and extracting a visual vector representation of the first video clip through a video encoder; wherein the audio detection model, the subtitle detection model, the text recognition model and the video encoder are pre-trained in a self-supervised learning manner; Constructing prompt words with the first audio text and the first subtitle text, and obtaining a text vector representation of the first video clip by segmenting the prompt words; splicing the text vector representation and the visual vector representation and sending them into a large language model to obtain a video content analysis result output by the large language model; The process of pre-training the audio detection model, the subtitle detection model, the text recognition model, and the video encoder in a self-supervised learning manner includes: Build a video data pool by collecting public videos; For each video in the video data pool, the video is segmented to obtain a second video segment corresponding to a second speaker; audio is extracted from the second video segment to obtain a second audio text, and at the same time, OCR recognition is performed on the second video segment to obtain a second subtitle boundary box and a second subtitle text; if the second audio text matches the second subtitle text, a video understanding text of the second video segment is extracted through a multimodal large model; The second video segment, the audio in the second video segment, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pseudo-label dataset to achieve: The audio detection model is obtained by training with the audio in the second video clip and the second audio text; the subtitle detection model is obtained by training with the second video clip and the second subtitle bounding box; the text recognition model is obtained by training with the second video clip and the second subtitle text; and the video encoder is obtained by training with the video understanding text in a manner that the weights of the large language model are fixed.

2. The video content analysis method based on self-supervised learning according to claim 1, characterized in that: The step of segmenting the target video to obtain a first video segment corresponding to the first speaker includes: The target video is sequentially subjected to voice activity detection and speaker segmentation to obtain the first video segment.

3. The video content analysis method based on self-supervised learning according to claim 1, characterized in that: The extracting the first subtitle text in the first video clip by using a subtitle detection model and a text recognition model includes: Extracting video frames from the first video clip; Detecting a first subtitle bounding box in the video frame by the subtitle detection model; Cropping the video frame according to the first subtitle boundary box; The first subtitle text is extracted from the cropped result of the video frame by using the text recognition model.

4. The video content analysis method based on self-supervised learning according to claim 1, characterized in that: The video encoder includes a video frame sampling component, an encoder and a linear layer.

5. The video content analysis method based on self-supervised learning according to claim 1, characterized in that: The process of pre-training the audio detection model, the subtitle detection model, the text recognition model, and the video encoder in a self-supervised learning manner also includes: If the second audio text matches the second subtitle text, the second video segment is classified into a high-quality data pool.

6. A video content analysis device based on self-supervised learning, characterized in that: The video content analysis device based on self-supervised learning comprises: A self-supervised learning module, which is used to pre-train an audio detection model, a subtitle detection model, a text recognition model, and a video encoder in a self-supervised learning manner; A video content analysis module is used to obtain a target video to be processed, and to perform video segmentation on the target video to obtain a first video segment corresponding to a first speaker; convert the audio in the first video segment into a first audio text through the audio detection model, extract the first subtitle text in the first video segment through the subtitle detection model and the text recognition model, and extract the visual vector representation of the first video segment through the video encoder; construct a prompt word with the first audio text and the first subtitle text, and obtain a text vector representation of the first video segment by performing word segmentation on the prompt word; and splice the text vector representation and the visual vector representation and send them to a large language model to obtain a video content analysis result output by the large language model; The process of pre-training the audio detection model, the subtitle detection model, the text recognition model, and the video encoder in a self-supervised learning manner includes: Build a video data pool by collecting public videos; For each video in the video data pool, the video is segmented to obtain a second video segment corresponding to a second speaker; audio is extracted from the second video segment to obtain a second audio text, and at the same time, OCR recognition is performed on the second video segment to obtain a second subtitle boundary box and a second subtitle text; if the second audio text matches the second subtitle text, a video understanding text of the second video segment is extracted through a multimodal large model; The second video segment, the audio in the second video segment, the second audio text, the second subtitle bounding box, the second subtitle text and the video understanding text are combined into a pseudo-label dataset to achieve: The audio detection model is obtained by training with the audio in the second video clip and the second audio text; the subtitle detection model is obtained by training with the second video clip and the second subtitle bounding box; the text recognition model is obtained by training with the second video clip and the second subtitle text; and the video encoder is obtained by training with the video understanding text in a manner that the weights of the large language model are fixed.

7. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the video content analysis method based on self-supervised learning as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the video content analysis method based on self-supervised learning as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the video content analysis method based on self-supervised learning as described in any one of claims 1 to 5.

Images