Method and system for automatic generation of multi-labels of video frames based on neural network model

Abstract

The present invention belongs to the field of video label generation, and in particular to a method and system for automatically generating multiple labels of video frames based on a neural network model, wherein the method comprises: obtaining a target video file to be processed; extracting video frames from the target video file according to a preset video frame extraction interval value, and generating extraction records; performing preprocessing according to the extracted video frames, and transmitting the preprocessed video frames to a neural network model to extract multi-dimensional frame-level features; obtaining category confidence by passing the extracted multi-dimensional frame-level features through a fully connected layer in the neural network model, and generating a multi-label set according to the category confidence; and returning the multi-label set and extraction record of the target video file. The present invention can solve the problem of multi-label classification generation of video frames.

Description

Technical Field

[0001] The present invention belongs to the field of video label generation, and in particular relates to a method and system for automatically generating multiple labels for video frames based on a neural network model. Background Art

[0002] With the rise and development of short video formats, the number of frequently active users of short videos continues to rise, and the usage time of users continues to grow. Therefore, it becomes particularly important to automatically generate tags based on the content of short videos to achieve more accurate video recommendations. Automatically generating tags based on the content of short videos belongs to the field of multi-label classification of video frames.

[0003] In the existing computer vision field, image binary classification and multi-classification methods are mainly used, but these methods are mainly used for single-label classification generation, while the labels in video frames usually belong to multiple different categories. When using image binary classification and multi-classification methods to identify labels of scenes in video frames, labels with low presence often appear, that is, the label appears less frequently in the video frame or is not obvious due to occlusion of the scene in the video frame. Image binary classification and multi-classification methods often feed back such labels with low presence to users, resulting in the problem of numerous and inaccurate video frame label sets. Summary of the invention

[0004] The technical problem solved by the present invention is to provide a method and system for automatically generating multiple labels for video frames based on a neural network model, so as to solve the problem in the prior art that there are too many types of labels and they are inaccurate in the generation set of multiple label classification for video frames.

[0005] The first basic solution provided by the present invention is a method for automatically generating multiple labels of video frames based on a neural network model, comprising:

[0006] Get the target video file to be processed;

[0007] Extracting video frames from the target video file according to a preset video frame extraction interval value, and generating an extraction record;

[0008] Preprocessing is performed according to the extracted video frames, and the preprocessed video frames are transmitted to the neural network model to extract multi-dimensional frame-level features;

[0009] The extracted multi-dimensional frame-level features are passed through the fully connected layer in the neural network model to obtain category confidence, and a multi-label set is generated based on the category confidence;

[0010] Returns the multi-label set and extracted records of the target video file.

[0011] The principle and advantage of the basic solution 1 provided by the present invention are: in the related art, facing the problem of multi-label classification generation of video frames, the existing image binary classification method and multi-classification method often feed back labels with low presence to users, resulting in the problem of numerous and inaccurate video frame label sets.

[0012] Therefore, in order to solve the problems existing in the related art, the present invention first obtains the target video file that needs to be generated by multi-label classification, extracts video frames from the obtained target video file according to a preset video frame extraction interval value, and pre-processes the extracted video frames, and then transmits them to the trained neural network model to extract multi-dimensional frame-level features, and then transmits the extracted multi-dimensional frame-level features to the fully connected layer in the neural network model, extracts the category confidence, and compares the extracted category confidence with the preset label threshold to generate a corresponding label set, and finally returns the output multi-label set and the record of the extracted video frame. Therefore, the advantages of the present invention are as follows: (1) multi-label category generation is performed through the neural network model to realize the autonomous learning function, and the label set of the video frame can be continuously enriched; (2) in the face of complex video scenes, the neural network model is used for recognition, the category confidence is output, and it is compared with the preset threshold of the label to determine the label type, which can effectively solve the problem of the large variety and inaccuracy of labels in the existing technology for the multi-label classification generation set of video frames.

[0013] Further, extracting video frames from the target video file according to a preset video frame extraction interval value and generating extraction records includes:

[0014] Check whether the target video file is in a valid video format. If the video format is invalid, re-acquire the target video file to be processed;

[0015] Set the video frame interval value of the target video file according to the user's subjective consciousness;

[0016] Extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time.

[0017] Beneficial effects: Check whether the target video file is in a valid format, set the video frame extraction interval value according to the user's main consciousness, avoid video frame extraction errors, and record the frame number and time of the extracted video frame to facilitate users to query the distribution of video frame labels later.

[0018] Further, the preprocessing is performed according to the extracted video frames, and the preprocessed video frames are transmitted to the neural network model to extract multi-dimensional frame-level features, including:

[0019] Build and train an attention-driven neural network model and load N category labels;

[0020] The extracted video frames are cropped, compressed and normalized;

[0021] The processed video frames are transferred to the trained neural network model to extract multi-dimensional frame-level features.

[0022] Beneficial effects: By constructing and training an attention-driven neural network model, the preprocessed video frames are transferred to the neural network model, and then the multi-dimensional frame-level features of the video frames are extracted, which makes it easier to obtain category confidence based on the frame-level features, thereby generating labels for the video frames.

[0023] Furthermore, the step of obtaining the category confidence by passing the extracted multi-dimensional frame-level features through a fully connected layer in a neural network model, and generating a multi-label set according to the category confidence comprises:

[0024] Connect the multi-dimensional frame-level features to the fully connected layer in the neural network model to obtain Y category confidences, compare the Y category confidences with the preset label threshold, and obtain the corresponding labels whose category confidences are greater than the preset label threshold;

[0025] Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set.

[0026] Beneficial effect: The category confidence is obtained through frame-level features to obtain labels for video frames, and the labels of video frames are generated into a multi-label set, which is convenient for users to obtain and view the multi-label situation of the target video.

[0027] The second basic solution provided by the present invention is a video frame multi-label automatic generation system based on a neural network model, comprising:

[0028] Video acquisition module: used to obtain the target video file to be processed;

[0029] Video frame extraction module: used to extract video frames from the target video file according to a preset video frame extraction interval value and generate extraction records;

[0030] Preprocessing module: preprocess the extracted video frames;

[0031] Frame-level feature extraction module: transfers the preprocessed video frames to the neural network model to extract multi-dimensional frame-level features;

[0032] Confidence acquisition module: used to obtain category confidence by passing the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model, and generate a multi-label set based on the category confidence;

[0033] Return module: Returns the multi-label set and extraction records of the target video file.

[0034] The principles and advantages of the basic solution 2 provided by the present invention are: in the related technology, facing the problem of multi-label classification generation of video frames, the existing image binary classification method and multi-classification method cannot be effectively applied to the problem of multi-label classification generation of video frames. The main reason is that the video scenes are complex, the categories are numerous, and different video label scenes occlude and affect each other.

[0035] Therefore, in order to solve the problems existing in the related art, the present invention first obtains the target video file that needs to be multi-label classified and generated through the video acquisition module, extracts the video frame of the acquired target video file through the video frame extraction module according to the preset video frame extraction interval value, and preprocesses the extracted video frame through the preprocessing module, and then transmits it to the trained neural network model through the frame-level feature extraction module to extract the multi-dimensional frame-level features, and then transmits the extracted multi-dimensional frame-level features to the fully connected layer in the neural network model through the confidence acquisition module, extracts the category confidence, and compares the extracted category confidence with the preset label threshold to generate a corresponding label set, and finally returns the output multi-label set and the record of the extracted video frame through the return module. Therefore, the advantages of the present invention are: (1) multi-label category generation is performed through the neural network model to realize the autonomous learning function, and the label set of the video frame can be continuously enriched; (2) in the face of complex video scenes, the neural network model is used for recognition, the category confidence is output, and the label type is judged, which can effectively solve the problem of multi-label classification generation of video frames.

[0036] Furthermore, the video frame extraction module is preset with a video frame extraction interval value, and the video frame extraction module includes a judgment module and a recording module. The judgment module is used to judge whether the target video file is a valid video format, and the recording module is used to extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time.

[0037] Beneficial effects: The judgment module checks whether the target video file is in a valid format, and the video frame extraction module sets the video frame extraction interval value according to the user's main consciousness, which can avoid the occurrence of video frame extraction errors. The recording module records the frame number and time of the extracted video frame, which is convenient for users to query the distribution of video frame labels later.

[0038] Furthermore, the preprocessing includes cropping, compression and normalization; the frame-level feature extraction module includes a neural network modeling module and an extraction module, the neural network modeling module is used to build and train an attention-driven neural network model and load N category labels, and the extraction module is used to transfer the processed video frames to the trained neural network model to extract multi-dimensional frame-level features.

[0039] Beneficial effects: An attention-driven neural network model is constructed and trained through the neural network modeling module, and the preprocessed video frames are transferred to the neural network model through the extraction module, thereby extracting the multi-dimensional frame-level features of the video frames, making it easier to obtain category confidence based on the frame-level features, thereby generating labels for the video frames.

[0040] Furthermore, the confidence acquisition module obtains the category confidence by passing the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model, and generates a multi-label set according to the category confidence as follows:

[0041] Connect the multi-dimensional frame-level features to the fully connected layer in the neural network model to obtain Y category confidences, compare the Y category confidences with the preset label threshold, and obtain the corresponding labels whose category confidences are greater than the preset label threshold;

[0042] Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set.

[0043] Beneficial effect: The category confidence is obtained through frame-level features to obtain labels for video frames, and the labels of video frames are generated into a multi-label set, which is convenient for users to obtain and view the multi-label situation of the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of an embodiment of the present invention;

[0045] Figure 2 It is a principle block diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. At the same time, although a logical order is shown in the embodiments, in some cases, the steps shown or described may be performed in an order different from that shown herein.

[0047] The following is further described in detail through specific implementation methods:

[0048] The embodiment is basically as shown in the attached Figure 1 As shown: A method for automatically generating multiple labels for video frames based on a neural network model, comprising:

[0049] Step 1: Get the target video file to be processed.

[0050] In this embodiment, it is mainly used to obtain video files to be generated by multi-label classification. The labels contained in the videos or movies we browse daily may reach hundreds of thousands or even millions. If the video label classification is generated by image classification method, on the one hand, the time cost and hardware load are extremely high. On the other hand, the video scenes are complicated, and the scenes occlude and affect each other, which is not conducive to accurate label generation. Therefore, the purpose of the present invention is to solve the problem that multi-label classification of video frames is difficult to generate.

[0051] Step 2: Extract video frames from the target video file according to a preset video frame extraction interval value, and generate extraction records.

[0052] In step 2, it also includes:

[0053] Check whether the target video file is in a valid video format. If the video format is invalid, re-acquire the target video file to be processed;

[0054] Set the video frame interval value of the target video file according to the user's subjective consciousness;

[0055] Extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time.

[0056] In this embodiment, according to the acquired target video file, on the one hand, a video format check is required, and on the other hand, frame-level division is required. First, the video format is checked, and the video formats include MPEG, AVI, 3GP, MP4 and other formats. In this embodiment, the MPEG video format is selected. If the video format of the acquired target video file is not MPEG, the target video file of the correct video format is re-acquired, or the video format is converted through a format conversion tool, which is not described in detail in this embodiment; for frame-level division, the video frame extraction interval value is mainly set based on the user's main consciousness. In this embodiment, the video frame extraction interval value is 5 frames, and a complete video is processed by extracting frames every 5 frames. The part below 5 frames is directly obtained according to the actual number of frames. After the video frame extraction is completed, the corresponding video frame number and time are recorded, so that the user can view the details after the multi-label classification of the video frame is generated.

[0057] Step 3: Preprocess the extracted video frames and transfer the preprocessed video frames to the neural network model to extract multi-dimensional frame-level features.

[0058] Among them, step three includes:

[0059] Build and train an attention-driven neural network model and load N category labels;

[0060] The extracted video frames are cropped, compressed and normalized;

[0061] The processed video frames are transferred to the trained neural network model to extract multi-dimensional frame-level features.

[0062] In this embodiment, an attention-driven neural network model is first constructed and trained. In this embodiment, an attention-driven dynamic graph convolutional network model is selected, which includes a semantic attention module and a dynamic GCN module. The semantic attention module is provided with a content-aware category representation V. The semantic attention module receives a set of dynamic feature maps with labeled samples. Each category representation V describes the content related to a specific label from the dynamic feature map. Specifically, the semantic attention module first calculates the activation map M of a specific category, and then uses M to convert the converted feature map X into a content-aware category representation V; the dynamic GCN module receives the content-aware category representation V in the semantic attention module, and adaptively transforms the coherent correlation of V, and finally classifies the labels.

[0063] Therefore, after building the attention-driven dynamic graph convolutional network model, the extracted video frames are preprocessed, including cropping, compression and normalization. First, the video frames are cropped to cut out invalid video frames, such as the black screen part. Then the cropped video frames are compressed and the compressed video frames are normalized. Specifically, the peak value of the audio data in the video frame is standardized, the maximum audio value is adjusted to a specific size, and the remaining values ​​are increased or decreased accordingly to keep the audio size consistent.

[0064] Finally, the preprocessed video frames are transferred to the trained dynamic graph convolutional network model for multi-dimensional frame-level feature extraction. Specifically, the video frames that have been processed and preprocessed are firstly subjected to a Hamming window to smooth the edges of each frame of the video, that is:

[0065]

[0066] Next, perform FFT processing ω[H i ]Get the frame level signal P i :

[0067]

[0068] Among them, N size represents the length of FFT analysis, ω[H i ] represents the signal of the i-th frame.

[0069] Finally, the logarithmic operation is performed on the video frame:

[0070] x i =log P i

[0071] That is, the video frame can be extracted into features x = (x1, ...x i ) T .

[0072] Step 3: Pass the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model to obtain category confidence, and generate a multi-label set based on the category confidence.

[0073] Among them, step three includes:

[0074] Connect the multi-dimensional frame-level features to the fully connected layer in the neural network model to obtain Y category confidences, compare the Y category confidences with the preset label threshold, and obtain the corresponding labels whose category confidences are greater than the preset label threshold;

[0075] Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set.

[0076] In this embodiment, the extracted frame-level features are connected to the fully connected layer of the dynamic graph convolutional network model to obtain the category confidence, that is, the number of output values ​​of the last layer of the fully connected layer is the number of categories, and the output value is called logits, as shown in the following formula:

[0077] logits=(θ i ), 0≤i≤N

[0078] Where N is the number of category labels;

[0079] After obtaining logits, the Sigmoid activation function is used to calculate the activation value. The activation value obtained by the Sigmoid activation function is regarded as the confidence of N categories. The Sigmoid activation function is shown as follows:

[0080]

[0081] After obtaining the confidence of all categories through the above formula, a threshold τ is set to select the labels of the categories corresponding to all f(x) greater than the threshold τ; in other embodiments of the present embodiment, it is also determined whether position i has a label by comparison, specifically, the confidence of all positions is sorted from large to small, and then the labels of the categories corresponding to the first K f(x) are selected.

[0082] Step 4: Return the multi-label set and extraction records of the target video file.

[0083] like Figure 2 As shown, in another embodiment of this embodiment, a video frame multi-label automatic generation system based on a neural network model is also included, including:

[0084] Video acquisition module: used to obtain the target video file to be processed;

[0085] Video frame extraction module: used to extract video frames from the target video file according to a preset video frame extraction interval value, and generate extraction records; the video frame extraction module is preset with a video frame extraction interval value, and the video frame extraction module includes a judgment module and a recording module, the judgment module is used to judge whether the target video file is a valid video format, and the recording module is used to extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time;

[0086] Preprocessing module: preprocess the extracted video frames; preprocessing includes cropping, compression and normalization;

[0087] Frame-level feature extraction module: transfers the preprocessed video frames to the neural network model to extract multi-dimensional frame-level features; the frame-level feature extraction module includes a neural network modeling module and an extraction module. The neural network modeling module is used to build and train an attention-driven neural network model and load N category labels. The extraction module is used to transfer the processed video frames to the trained neural network model to extract multi-dimensional frame-level features.

[0088] Confidence acquisition module: used to obtain category confidence by passing the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model, and generate a multi-label set based on the category confidence; the confidence acquisition module passes the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model to obtain category confidence, and generates a multi-label set based on the category confidence. Specifically:

[0089] Connect the multi-dimensional frame-level features to the fully connected layer in the neural network model to obtain Y category confidences, compare the Y category confidences with the preset label threshold, and obtain the corresponding labels whose category confidences are greater than the preset label threshold;

[0090] Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set.

[0091] Return module: Returns the multi-label set and extraction records of the target video file.

[0092] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for automatically generating multiple labels for video frames based on a neural network model, characterized in that: include: Get the target video file to be processed; Extracting video frames from the target video file according to a preset video frame extraction interval value, and generating an extraction record; Preprocessing is performed according to the extracted video frames, and the preprocessed video frames are transmitted to the neural network model to extract multi-dimensional frame-level features; Specifically: Build and train an attention-driven neural network model and load N category labels; The extracted video frames are cropped, compressed and normalized; The processed video frames are transmitted to the trained neural network model to extract multi-dimensional frame-level features; wherein, the processed video frames are transmitted to the trained neural network model to extract multi-dimensional frame-level features as follows: The decimated and preprocessed video frames are smoothed using a Hamming window, that is: Next, perform FFT processing ω[H i ]Get the frame level signal P i : Among them, N size represents the length of FFT analysis, ω[H i ] represents the signal of the i-th frame; Take the logarithm of the video frame: x i =logP i The video frame is extracted into features x = (x1, ...x i ) T ; The extracted multi-dimensional frame-level features are passed through the fully connected layer in the neural network model to obtain the category confidence, and a multi-label set is generated based on the category confidence; specifically: The multi-dimensional frame-level features are connected to the fully connected layer in the neural network model to obtain Y category confidences, which are compared with the preset label thresholds to obtain the corresponding labels whose category confidences are greater than the preset label thresholds; the number of output values ​​of the last layer of the fully connected layer is the number of categories, which is called logits, as shown in the following formula: Where N is the number of category labels; After obtaining logits, the Sigmoid activation function is used to calculate the activation value. The activation value obtained by the Sigmoid activation function is regarded as the confidence of N categories. The Sigmoid activation function is shown as follows: After obtaining the confidence of all categories through the above formula, by setting the threshold τ, select the labels of the categories corresponding to all f(x) greater than the threshold τ; Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set; Returns the multi-label set and extracted records of the target video file.

2. The method for automatically generating multiple labels for video frames based on a neural network model according to claim 1, characterized in that: The step of extracting video frames from the target video file according to a preset video frame extraction interval value and generating an extraction record comprises: Check whether the target video file is in a valid video format. If the video format is invalid, re-acquire the target video file to be processed; Set the video frame interval value of the target video file according to the user's subjective consciousness; Extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time.

3. A video frame multi-label automatic generation system based on a neural network model, characterized by: include: Video acquisition module: used to obtain the target video file to be processed; Video frame extraction module: used to extract video frames from the target video file according to a preset video frame extraction interval value and generate extraction records; Preprocessing module: preprocess the extracted video frames; Frame-level feature extraction module: transfers the preprocessed video frames to the neural network model to extract multi-dimensional frame-level features; Confidence acquisition module: used to obtain category confidence by passing the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model, and generate a multi-label set based on the category confidence; Return module: returns the multi-label set and extraction records of the target video file; The preprocessing includes cropping, compression and normalization processing; the frame-level feature extraction module includes a neural network modeling module and an extraction module, the neural network modeling module is used to build and train an attention-driven neural network model and load N category labels, and the extraction module is used to transmit the processed video frame to the trained neural network model to extract multi-dimensional frame-level features; wherein, the processed video frame is transmitted to the trained neural network model to extract multi-dimensional frame-level features specifically as follows: The decimated and preprocessed video frames are smoothed using a Hamming window, that is: Next, perform FFT processing ω[H i ]Get the frame level signal P i : Among them, N size represents the length of FFT analysis, ω[H i ] represents the signal of the i-th frame; Take the logarithm of the video frame: x i =logP i The video frame is extracted into features x = (x1, ...x i ) T ; The confidence acquisition module obtains the category confidence by passing the extracted multi-dimensional frame-level features through the fully connected layer in the neural network model, and generates a multi-label set according to the category confidence as follows: The multi-dimensional frame-level features are connected to the fully connected layer in the neural network model to obtain Y category confidences, which are compared with the preset label thresholds to obtain the corresponding labels whose category confidences are greater than the preset label thresholds; the number of output values ​​of the last layer of the fully connected layer is the number of categories, which is called logits, as shown in the following formula: Where N is the number of category labels; After obtaining logits, the Sigmoid activation function is used to calculate the activation value. The activation value obtained by the Sigmoid activation function is regarded as the confidence of N categories. The Sigmoid activation function is shown as follows: After obtaining the confidence of all categories through the above formula, by setting the threshold τ, select the labels of the categories corresponding to all f(x) greater than the threshold τ; Repeatedly obtain the labels corresponding to the category confidences of the multi-dimensional frame-level features in the video frames of the target video file, and generate a multi-label set.

4. The video frame multi-label automatic generation system based on the neural network model according to claim 3 is characterized by: The video frame extraction module is preset with a video frame extraction interval value, and the video frame extraction module includes a judgment module and a recording module. The judgment module is used to judge whether the target video file is a valid video format, and the recording module is used to extract video frames from the target video file in a valid format according to the set video frame extraction interval value, and record the corresponding frame number and time.

Images