Extraction system and method based on highlight video in automobile field

Abstract

The invention belongs to the technical field of computers, and discloses an extraction system based on a highlight video in the automobile field, and the system comprises a text semantic analysis module which is used for obtaining the semantic information of the whole video. After a video is input, multi-modal information in an image and original information corresponding to the image and voice are extracted by framing the video, and then a text semantic vector is obtained by converting related titles, texts, voice and subtitles in the video; through a general classification model, a general quality scoring model, a general face detection and recognition model and a fine-grained detection model in the automobile field, image semantic vectors of the whole video can be formed based on the characteristics of the automobile field, and finally audio data in the video are obtained, feature vectors of the audio data are extracted and merged with the image vectors, and the video quality is improved. The semantic vector of the video is obtained, and a Cross Attention structure is utilized to enable the semantic vector of the text to guide the semantic features of the video for prediction.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a system and method for extracting highlight videos based on the automotive field. Background Art

[0002] Highlight videos refer to those particularly exciting and eye-catching clips in a video. Compared with medium or short videos, long videos require viewers to invest more time and patience to screen the content and gradually delve into the plot. In order to optimize the user's viewing experience and improve their efficiency in discovering and screening long videos, the function of automatically playing the highlight moments of long videos or directly displaying the highlight clips has been introduced. This initiative aims to enable users to quickly capture the core charm of long videos in a short period of time, and have a preliminary understanding and interest in the video content without watching it in its entirety. In this way, users can browse and select long videos of their interest more conveniently, thereby promoting the dissemination and consumption of long video content, effectively improving the exposure and attractiveness of long videos, and achieving the purpose of attracting traffic to long video content.

[0003] At present, the highlight video detection algorithm mainly extracts highlight clips from themes such as movies, variety shows, sports events or games. The characteristics of these videos are that there is no clear theme content and the scenes change a lot, so the range of extractable content is large. In terms of the existing highlight video extraction algorithm technology, the video is first segmented by frame, and the image, voice and text are input into the corresponding general CNN model to obtain the corresponding vectors. These vectors are then input into the fully connected network for the task, and each frame is scored and predicted. The score is generally used to evaluate the actual effect of the model. Finally, according to the actual task and the previous score, the corresponding strategy is used to detect the highlight clips. Although this scheme can detect the highlight clips of the video, the disadvantage of this scheme is that the recognition effect is poor for a single scene or a video with little change in the scene content, and the theme information of the video cannot be extracted. Secondly, the CNN classification model is used to obtain the semantic features of the video, which will be interfered by other background information and affect the recognition results, so it needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a system and method for extracting highlight videos based on the automotive field, so as to solve the problems raised in the above-mentioned background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a system for extracting highlight video based on the automotive field, comprising:

[0006] Text semantic analysis module: that is, obtaining the semantic information of the entire video;

[0007] Among them, the text semantic analysis module includes:

[0008] General classification model: used to obtain basic features such as video edges and geometry;

[0009] General quality scoring model: used to evaluate the quality of the video;

[0010] General face detection and recognition model: used to detect and recognize faces appearing in videos;

[0011] Fine-grained detection model in the automotive field: used to identify the shooting angle of the car in the image and obtain the middle layer of the neural network corresponding to the detection box as its semantic vector.

[0012] Preferably, the sources of video semantic information acquired by the text semantic analysis module are: video-related title and text information, voice in the video, and subtitles in the video. These three text information are merged together, and the semantics to be expressed by the entire video is obtained through a trained language model. The output result here is a vector of a specified length.

[0013] Preferably, the general classification model, the general quality scoring model, the general face detection and recognition model, and the fine-grained detection model in the automotive field are merged together to form an image semantic vector of the entire video, and the audio data in the video is obtained, its feature vector is extracted, and merged with the image vector to obtain the semantic vector of the video.

[0014] Preferably, the face detection and recognition model needs to pay attention to the accuracy and robustness of the model, so as to ensure that the model can accurately detect and recognize faces in different environments.

[0015] The specific steps of the extraction method of highlight video based on the automotive field are as follows:

[0016] Step 1: Video Input

[0017] First, receive a video file as input;

[0018] Step 2: Video Frame Division

[0019] Next, the input video is frame-processed, that is, the video is decomposed into a series of static images;

[0020] Step 3:

[0021] 3.1: Text content extraction

[0022] For each decomposed frame, the system extracts the text content, including subtitles, tags, or other text information that appears in the video;

[0023] 3.2: Extracting single-frame multimodal raw information

[0024] While extracting text content, the system also collects multimodal information and the original information of the corresponding image and speech for each frame. These multimodal information and original information together constitute a comprehensive description of each frame;

[0025] Step 4:

[0026] 4.1: Analyze the entire video semantics

[0027] 4.2: Extracting multimodal semantic information

[0028] Step 5: Model Scoring

[0029] Use text semantic vectors to guide video semantic vectors for scoring, and use the Cross Attention structure to allow text information to guide video semantic features for prediction;

[0030] Step 6: Export highlight clips

[0031] Output highlight clips based on business characteristics.

[0032] Preferably, when inputting a video in step 1, it is necessary to confirm that the received video file format can be supported. When the video format cannot be identified, the video format needs to be converted.

[0033] Preferably, the video framing described in step 2 needs to select a suitable framing method to ensure that key information is not lost due to frame segmentation, and the impact of the frame rate on subsequent processing needs to be considered, such as too high a frame rate will increase processing difficulty and computational complexity.

[0034] Preferably, the video frame input by the Cross Attention structure in step 5 generally takes the features of the current frame and the previous and next 3 frames, a total of 7 frames, as input, wherein the video semantic features are used as the main information for prediction, and the text semantic features are used as auxiliary features.

[0035] Preferably, before generating the final highlight clip in step 6, adjustment or optimization needs to be performed according to specific business characteristics or needs. For example, some businesses may pay more attention to the creativity of the video, while other businesses may pay more attention to the efficiency of information transmission.

[0036] The beneficial effects of the present invention are as follows:

[0037] After the video is input, the video is divided into frames to extract the multimodal information in the image and the original information of the corresponding image and voice. Then, the relevant title and text as well as the voice and subtitles in the video are converted to obtain the text semantic vector. Then, through the general classification model, general quality scoring model, general face detection and recognition model and fine-grained detection model in the automotive field, the image semantic vector of the entire video can be formed based on the characteristics of the automotive field. Finally, the audio data in the video is obtained, its feature vector is extracted, and it is merged with the image vector to obtain the semantic vector of the video. The Cross Attention structure is used to make the text semantic vector guide the prediction of the video semantic features, so that the highlight clips in the video can be accurately predicted, thereby improving the accuracy of the video highlight clip prediction. At the same time, the highlight clips are extracted and output according to the business characteristics and user preferences, realizing personalized recommendations and improving the accuracy of recommendations and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0039] Figure 2 It is a schematic diagram of the image processing flow of the present invention;

[0040] Figure 3 This is a diagram of the Cross Attention structure of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figures 1 to 3 As shown, an embodiment of the present invention provides a system for extracting highlight videos based on the automotive field, including:

[0043] Text semantic analysis module: that is, obtaining the semantic information of the entire video;

[0044] Among them, the text semantic analysis module includes:

[0045] General classification model: used to obtain basic features such as video edges and geometry;

[0046] General quality scoring model: used to evaluate the quality of the video;

[0047] General face detection and recognition model: used to detect and recognize faces appearing in videos;

[0048] Fine-grained detection model in the automotive field: used to identify the shooting angle of the car in the image and obtain the middle layer of the neural network corresponding to the detection box as its semantic vector.

[0049] By using a general classification model, the content in the video can be preliminarily classified or identified, so as to identify the basic elements in the video, such as objects, scenes, etc., providing a basis for subsequent more refined processing. The quality of the video frame is evaluated by adopting a quality scoring model, and the face detection and recognition model can detect and recognize faces in the video in real time, providing an additional dimension for video analysis and processing. Finally, the fine-grained detection model in the automotive field can extract the detection frame of the car from the video frame and obtain the middle layer of the neural network corresponding to the detection frame as its semantic vector. This semantic vector contains rich information about the car, such as car model, color, posture, etc., which provides an important basis for subsequent analysis and processing.

[0050] Among them, the sources of video semantic information obtained by the text semantic analysis module are: video-related title and text information, voice in the video, and subtitles in the video. These three text information are merged together, and the semantics to be expressed by the entire video is obtained through the trained language model. The output here is a vector of a specified length.

[0051] The speech in the video can be converted into text through ASR, and the subtitles in the video can be converted into text through OCR.

[0052] Among them, the general classification model, the general quality scoring model, the general face detection and recognition model, and the fine-grained detection model in the automotive field are merged together to form the image semantic vector of the entire video. The audio data in the video is obtained, its feature vector is extracted, and merged with the image vector to obtain the semantic vector of the video.

[0053] By extracting the image semantic vector of the video and the feature vector of the audio data, the semantic vector of the entire video is obtained. The semantic vector of the video not only helps integrate the overall content of the video, but also guides the scoring process and ultimately optimizes the output results.

[0054] Among them, the face detection and recognition model needs to pay attention to the accuracy and robustness of the model to ensure that the model can accurately detect and recognize faces in different environments.

[0055] In practical applications, we may encounter various complex face situations, such as occlusion, side face, lighting changes, etc. Therefore, the accuracy and robustness of face detection and recognition models are very important.

[0056] The specific steps of the extraction method of highlight video based on the automotive field are as follows:

[0057] Step 1: Video Input

[0058] First, receive a video file as input;

[0059] Step 2: Video Frame Division

[0060] Next, the input video is frame-processed, that is, the video is decomposed into a series of static images;

[0061] Step 3:

[0062] 3.1: Text content extraction

[0063] For each decomposed frame, the system extracts the text content, including subtitles, tags, or other text information that appears in the video;

[0064] 3.2: Extracting single-frame multimodal raw information

[0065] While extracting text content, the system also collects multimodal information and the original information of the corresponding image and speech for each frame. These multimodal information and original information together constitute a comprehensive description of each frame;

[0066] Step 4:

[0067] 4.1: Analyze the entire video semantics

[0068] 4.2: Extracting multimodal semantic information

[0069] Step 5: Model Scoring

[0070] Use text semantic vectors to guide video semantic vectors for scoring, and use the Cross Attention structure to allow text information to guide video semantic features for prediction;

[0071] Step 6: Export highlight clips

[0072] Output highlight clips based on business characteristics.

[0073] After the video is input, the video is divided into frames to extract the multimodal information in the image and the original information of the corresponding image and speech. Then the semantic content of the entire video is analyzed to understand the theme of the video, etc. Then, the CrossAttention structure is used to use the text semantic vector to guide the video semantic vector for prediction, so that the highlight segments in the video can be accurately predicted, improving the prediction accuracy of the visual highlight segments.

[0074] Among them, in step 1, when inputting the video, it is necessary to confirm that the received video file format can be supported. When the video format cannot be recognized, the video format needs to be converted.

[0075] By converting the video format when inputting the video, we can ensure that the video can be recognized, so that the subsequent processes can proceed stably.

[0076] Among them, the video framing in step 2 needs to select an appropriate framing method to ensure that key information is not lost due to frame segmentation, and the impact of the frame rate on subsequent processing needs to be considered. For example, too high a frame rate will increase the processing difficulty and calculation amount.

[0077] By decomposing the video into multiple individual frames, the continuous video content can be converted into a discrete image sequence, which provides a basis for subsequent processing.

[0078] Among them, the video frame input by the Cross Attention structure in step 5 generally takes the features of the current frame and the previous and next 3 frames, a total of 7 frames, as input, among which the video semantic features are used as the main information for prediction, and the text semantic features are used as auxiliary features.

[0079] The more similar the content expressed by the video feature is to the text feature, the greater the possibility of predicting it as a highlight frame.

[0080] Among them, before generating the final highlight clip in step six, it is necessary to adjust or optimize according to specific business characteristics or needs. For example, some businesses may pay more attention to the creativity of the video, while other businesses may pay more attention to the efficiency of information transmission.

[0081] By generating highlight clips with different focus points according to the characteristics of the business, the highlight clips can be more in line with business needs.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0083] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A system for extracting highlight videos in the automotive field, characterized by ,include: Text semantic analysis module: that is, obtaining the semantic information of the entire video; Among them, the text semantic analysis module includes: General classification model: used to obtain basic features such as video edges and geometry; General quality scoring model: used to evaluate the quality of the video; General face detection and recognition model: used to detect and recognize faces appearing in videos; Fine-grained detection model in the automotive field: used to identify the shooting angle of the car in the image and obtain the middle layer of the neural network corresponding to the detection box as its semantic vector.

2. The system for extracting highlight video based on the automotive field according to claim 1, characterized in that: The sources of video semantic information obtained by the text semantic analysis module are: video-related title and text information, voice in the video, and subtitles in the video. These three text information are merged together, and the semantics to be expressed by the entire video is obtained through the trained language model. The output here is a vector of a specified length.

3. The system for extracting highlight video based on the automotive field according to claim 1, characterized in that: The general classification model, the general quality scoring model, the general face detection and recognition model, and the fine-grained detection model in the automotive field are combined together to form an image semantic vector of the entire video, and the audio data in the video is obtained to extract its feature vector. Combined with the image vector, we get the semantic vector of the video.

4. The system for extracting highlight video based on the automotive field according to claim 1, characterized in that: The face detection and recognition model needs to pay attention to the accuracy and robustness of the model to ensure that the model can accurately detect and recognize faces in different environments.

5. A method for extracting highlight video based on the automotive field, characterized in that , the specific steps are as follows: Step 1: Video Input First, receive a video file as input; Step 2: Video Frame Division Next, the input video is frame-processed, that is, the video is decomposed into a series of static images; Step 3: 3.1: Text content extraction For each decomposed frame, the system extracts the text content, including subtitles, tags, or other text information that appears in the video; 3.2: Extracting single-frame multimodal raw information While extracting text content, the system also collects multimodal information and the original information of the corresponding image and speech for each frame. These multimodal information and original information together constitute a comprehensive description of each frame; Step 4: 4.1: Analyze the entire video semantics 4.2: Extracting multimodal semantic information Step 5: Model Scoring Use text semantic vectors to guide video semantic vectors for scoring, and use the Cross Attention structure to allow text information to guide video semantic features for prediction; Step 6: Export highlight clips Output highlight clips based on business characteristics.

6. The method for extracting highlight video based on the automotive field according to claim 5, characterized in that: When inputting a video as described in step 1, it is necessary to confirm that the received video file format can be supported. When the video format cannot be recognized, the video format needs to be converted.

7. The method for extracting highlight video based on the automotive field according to claim 5, characterized in that: The video framing described in step 2 requires selecting an appropriate framing method to ensure that key information is not lost due to frame segmentation, and the impact of the frame rate on subsequent processing needs to be considered. For example, too high a frame rate will increase the processing difficulty and computational complexity.

8. The method for extracting highlight video based on the automotive field according to claim 5, characterized in that: The video frame input by the Cross Attention structure in step 5 generally takes the features of the current frame and the previous and next 3 frames, a total of 7 frames, as input, among which the video semantic features are used as the main information for prediction, and the text semantic features are used as auxiliary features.

9. The method for extracting highlight video based on the automotive field according to claim 5, characterized in that: Before generating the final highlight clip as described in step 6, it is necessary to adjust or optimize according to specific business characteristics or needs. For example, some businesses may pay more attention to the creativity of the video, while other businesses may pay more attention to the efficiency of information transmission.

Images