Model training method, video classification method, device and readable storage medium

Abstract

The present application discloses a model training method, a video classification method, an apparatus and a computer-readable storage medium. The training method comprises: obtaining a training video segment; performing audio stream extraction processing on the training video segment to obtain a training audio segment; generating a training spectrogram corresponding to the training audio segment; determining a number of training video frames in the training video segment; determining a number of sampling time intervals based on the sampling time points corresponding to each training video frame, and selecting a number of training spectrum segments in the training spectrogram according to each sampling time interval; combining each training video frame and the corresponding training spectrum segment into a training data group, and generating training data using each training data group; inputting the training data into an initial model with a time migration module to obtain a prediction result, and adjusting the parameters of the initial model using the prediction result; if it is detected that the training completion condition is met, it is determined to be a video classification model; the video classification model has high accuracy.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a model training method, a video classification method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the current era of video popularity, automatically generating personalized videos from user-generated audio can increase dissemination rates and the commercial value of products. Materials that can be widely used in video generation must meet the condition that the video content is unrelated to the background music, such as some scenery clips and scene plot clips. In addition, large-scale application requires the establishment of a large material library, and manual material screening is unrealistic. Related technologies typically perform frame extraction and feature extraction on video and audio separately, and then determine the correlation between video and audio by fusing the features of video and audio frames, and then determine whether the video can be used as material for video generation. However, this method has poor judgment accuracy, resulting in poor quality of the established material library. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide model training, video classification method, electronic device and computer-readable storage medium, so as to improve the classification accuracy of the video classification model.

[0004] To solve the above technical problems, in the first aspect, the present application provides a model training method, comprising:

[0005] Get the training video segment;

[0006] Performing audio stream extraction processing on the training video segment to obtain a training audio segment;

[0007] Generating a training spectrogram corresponding to the training audio segment;

[0008] Determining a plurality of training video frames in the training video segment;

[0009] Determining a plurality of sampling time intervals based on sampling time points corresponding to each of the training video frames, and selecting a plurality of training spectrogram segments in the training spectrogram according to each of the sampling time intervals;

[0010] Combining each of the training video frames and the corresponding training spectrogram segments into a training data group, and generating training data using each of the training data groups;

[0011] Inputting the training data into an initial model with a time migration module to obtain a prediction result, and using the prediction result to adjust parameters of the initial model;

[0012] If it is detected that the training completion condition is met, the initial model after parameter adjustment is determined as the video classification model.

[0013] Optionally, the initial model includes several convolution modules, and the time migration module is arranged before at least one of the convolution modules; wherein, the time migration module is outside the residual structure to which the convolution module belongs, or the time migration module is within the residual structure to which the convolution module belongs.

[0014] Optionally, inputting the training data into an initial model having a time migration module to obtain a prediction result includes:

[0015] Inputting the training data into the initial convolution module of the initial model to obtain an initial feature map;

[0016] Inputting the initial feature map into a feature extraction network to obtain a target feature map; wherein the feature extraction network includes a plurality of feature extraction modules, the feature extraction modules including the time migration module and the convolution module, the time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map, and the convolution module performs a convolution process on the offset feature map to obtain an output feature map; the output feature map of the last feature extraction module is the target feature map;

[0017] The target feature map is input into the fully connected layer to obtain the prediction result.

[0018] Optionally, if the time shift module is outside the residual structure to which the convolution module belongs, the method includes:

[0019] The offset feature map is input into a shortcut connection path in the residual structure to which the convolution module belongs.

[0020] Optionally, it also includes:

[0021] Obtaining training labels corresponding to the training video segment and the training audio segment;

[0022] The use of the prediction results to adjust the parameters of the initial model includes:

[0023] Obtaining a loss value using the training label and the prediction result;

[0024] Parameters of the initial model are adjusted based on the loss value.

[0025] Optionally, obtaining the training video segment and the corresponding training audio segment includes:

[0026] Get the training video and corresponding training audio;

[0027] Inputting the training video into a pre-trained shot boundary detection model to obtain boundary detection results corresponding to each video frame in the training video;

[0028] Segment the training video at the position of the target video frame where the boundary detection result is hit, to obtain the training video segment;

[0029] Based on the segmentation of the training video, the training audio is segmented to obtain the training audio segments.

[0030] Optionally, determining a plurality of sampling time intervals based on the sampling time points corresponding to the respective training video frames includes:

[0031] Taking the target sampling time point as a reference, extending a first preset length to the left on the time axis to obtain a first time coordinate, and extending a second preset length to the right to obtain a second time coordinate; the target sampling time point is any one of the sampling time points;

[0032] The first time coordinate is determined as the interval start point, and the second time coordinate is determined as the interval end point, to obtain the sampling time interval corresponding to the target sampling time point.

[0033] Optionally, the initial model includes a feature exchange module, which is arranged before at least one of the convolution modules; the feature exchange module is used to exchange the positions of several groups of video data and audio data in the input data corresponding to the same training data group.

[0034] In a second aspect, the present application provides a video classification method, comprising:

[0035] Get the video segment to be classified;

[0036] Performing audio stream extraction processing on the video segment to be classified to obtain an audio segment to be classified;

[0037] Generating a spectrogram to be classified corresponding to the audio segment to be classified;

[0038] Determining a plurality of video frames to be classified in the video segment to be classified;

[0039] Determining a plurality of sampling time intervals based on sampling time points corresponding to the respective video frames to be classified, and selecting a plurality of spectrum segments to be classified in the spectrogram to be classified according to the respective sampling time intervals;

[0040] Combining the to-be-classified video frames and the corresponding to-be-classified spectrum segments into to-be-classified data groups, and generating to-be-classified data using each of the to-be-classified data groups;

[0041] The data to be classified is input into a video classification model with a time migration module to obtain a classification result; the video classification model is obtained using the above-mentioned model training method.

[0042] Optionally, the video classification model includes a convolution module, and inputting the data to be classified into the video classification model having a time shift module to obtain a classification result includes:

[0043] Inputting the training data into the initial convolution module of the video classification model to obtain a preprocessed feature map;

[0044] Inputting the preprocessed feature map into a feature extraction network to obtain a post-processed feature map; wherein the feature extraction network includes a plurality of feature extraction modules, and the feature extraction modules include the time migration module and the convolution module. The time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map, and the convolution module performs a convolution process on the offset feature map to obtain an output feature map; the output feature map of the last feature extraction module is the post-processed feature map;

[0045] The post-processing feature map is input into a fully connected layer to obtain the classification result.

[0046] Optionally, the feature extraction module includes a feature exchange module, which is arranged before the convolution module or the time migration module; the feature exchange module is used to exchange the positions of several groups of video data and audio data in the input data corresponding to the same group of data to be classified.

[0047] Optionally, obtaining the video segment to be classified and the corresponding audio segment to be classified includes:

[0048] Get the video to be classified and the corresponding audio to be classified;

[0049] Inputting the video to be classified into a pre-trained shot boundary detection model to obtain boundary detection results corresponding to each video frame in the video to be classified;

[0050] Segment the video to be classified at a position of the target video frame where the boundary detection result is a hit, to obtain the video segment to be classified;

[0051] Based on the segmentation of the video to be classified, the audio to be classified is segmented to obtain the audio segments to be classified.

[0052] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein:

[0053] The memory is used to store computer programs;

[0054] The processor is used to execute the computer program to implement the above-mentioned model training method and / or the above-mentioned video classification method.

[0055] In a fourth aspect, the present application also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned model training method and / or the above-mentioned video classification method.

[0056] The model training method provided in the present application obtains a training video segment; performs audio stream extraction processing on the training video segment to obtain a training audio segment; generates a training spectrogram corresponding to the training audio segment; determines a number of training video frames in the training video segment; determines a number of sampling time intervals based on the sampling time points corresponding to each training video frame, and selects a number of training spectrum segments in the training spectrogram according to each sampling time interval; combines each training video frame and the corresponding training spectrum segment into a training data group, and generates training data using each training data group; inputs the training data into an initial model with a time migration module to obtain a prediction result, and uses the prediction result to adjust the parameters of the initial model; if it is detected that the training completion conditions are met, the initial model after parameter adjustment is determined as the video classification model.

[0057] As can be seen, after obtaining a training video segment, this method extracts the corresponding audio separately to generate a cloud-linked audio segment. Training data is then generated using the training video and audio segments. However, since the correlation between audio and video typically varies over time, the temporal correlation of the audio and video content is ignored. However, the correlation between shorter video and audio segments varies less, so the training data obtained from this processing yields a higher accuracy model. By generating a training spectrogram corresponding to the training audio segment, the audio features can be represented in an image format. Through sampling, the local states of several training video frames are used to represent the global state of the training video segment. The training spectrogram is sampled based on the corresponding sampling time intervals. The resulting training spectrogram segments can represent the audio features for a period of time before and after the training video frame. By combining the training video frames and the corresponding training spectrogram segments into a training data set and then using this training data set to compose the training data, the training data integrates audio and video features, emphasizing the temporal correlation between audio and video. The initial model includes a time migration module that migrates features across time, enabling information exchange between features at different time points, allowing the model to better learn the correlation between audio and video. The video classification model after training has high classification accuracy, and the quality of the material library constructed by using it for video classification is high.

[0058] In addition, the present application also provides a video classification method, an electronic device, and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0060] Figure 1 A schematic diagram of a hardware composition framework applicable to a model training method provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a hardware composition framework applicable to another model training method provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of available and unavailable video segments provided in an embodiment of the present application;

[0064] Figure 5 A flow chart of a model training method provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of a specific shot boundary detection model structure provided in an embodiment of the present application;

[0066] Figure 7 An audio signal and corresponding spectrogram provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of a specific training data set provided in an embodiment of the present application;

[0068] Figure 9 This is a working effect diagram of the time migration module provided in the embodiment of the present application;

[0069] Figure 10 Two methods of inserting time migration modules are provided in the embodiments of this application;

[0070] Figure 11 A specific video classification flowchart provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0072] For ease of understanding, the hardware composition framework used in the model training method and / or the video classification method provided in the embodiment of the present application is first introduced. Figure 1 , Figure 1 Schematic diagram of a hardware composition framework applicable to a model training method provided in an embodiment of the present application. The electronic device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input / information output (I / O) interface 104, and a communication component 105.

[0073] Among them, the processor 101 is used to control the overall operation of the electronic device 100 to complete the model training method and / or all or part of the steps in the video classification method; the memory 102 is used to store various types of data to support the operation of the electronic device 100. These data may include, for example, instructions for any application or method operating on the electronic device 100, as well as application-related data. The memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. In this embodiment, the memory 102 stores at least one or more programs and / or data for implementing the following functions:

[0074] Get the training video segment;

[0075] Performing audio stream extraction processing on the training video segment to obtain a training audio segment;

[0076] Generating a training spectrogram corresponding to the training audio segment;

[0077] Determining a plurality of training video frames in the training video segment;

[0078] Determining a plurality of sampling time intervals based on sampling time points corresponding to each of the training video frames, and selecting a plurality of training spectrogram segments in the training spectrogram according to each of the sampling time intervals;

[0079] Combining each of the training video frames and the corresponding training spectrogram segments into a training data group, and generating training data using each of the training data groups;

[0080] Inputting the training data into an initial model with a time migration module to obtain a prediction result, and using the prediction result to adjust parameters of the initial model;

[0081] If it is detected that the training completion condition is met, the initial model after parameter adjustment is determined as the video classification model.

[0082] and / or,

[0083] Get the video segment to be classified;

[0084] Performing audio stream extraction processing on the video segment to be classified to obtain an audio segment to be classified;

[0085] Generating a spectrogram to be classified corresponding to the audio segment to be classified;

[0086] Determining a plurality of video frames to be classified in the video segment to be classified;

[0087] Determining a plurality of sampling time intervals based on sampling time points corresponding to the respective video frames to be classified, and selecting a plurality of spectrum segments to be classified in the spectrogram to be classified according to the respective sampling time intervals;

[0088] Combining the to-be-classified video frames and the corresponding to-be-classified spectrum segments into to-be-classified data groups, and generating to-be-classified data using each of the to-be-classified data groups;

[0089] The data to be classified is input into a video classification model with a time migration module to obtain a classification result; the video classification model is obtained using the above-mentioned model training method.

[0090] The multimedia component 103 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 102 or sent through the communication component 105. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 104 provides an interface between the processor 101 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 105 is used for wired or wireless communication between the electronic device 100 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 105 may include: a Wi-Fi component, a Bluetooth component, an NFC component.

[0091] The electronic device 100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the model training method.

[0092] certainly, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device 100 may include Figure 1 More or fewer components than shown, or combinations of certain components.

[0093] It is understandable that the number of electronic devices is not limited in the embodiments of the present application, and it can be that multiple electronic devices work together to complete the model training method and / or the video classification method. In a possible implementation, please refer to Figure 2 , Figure 2 Schematic diagram of the hardware composition framework applicable to another model training method provided in the embodiment of the present application. Figure 2It can be seen that the hardware composition framework may include: a first electronic device 11 and a second electronic device 12 , which are connected via a network 13 .

[0094] In the embodiment of the present application, the hardware structure of the first electronic device 11 and the second electronic device 12 can refer to Figure 1 In other words, the electronic device 100 in this embodiment has two electronic devices 100, and the two devices exchange data. Furthermore, the embodiment of the present application does not limit the form of the network 13, that is, the network 13 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0095] Among them, the first electronic device 11 and the second electronic device 12 can be the same electronic device, such as the first electronic device 11 and the second electronic device 12 are both servers; or they can be different types of electronic devices, for example, the first electronic device 11 can be a smartphone or other smart terminal, and the second electronic device 12 can be a server. In one possible implementation, a server with strong computing power can be used as the second electronic device 12 to improve data processing efficiency and reliability, thereby improving the processing efficiency of model training. At the same time, a smartphone with low cost and wide application range is used as the first electronic device 11 to realize the interaction between the second electronic device 12 and the user. It can be understood that the interaction process can be: the smartphone obtains the training video segment and sends it to the second electronic device 12, and the second electronic device 12 performs subsequent processing on it to obtain a video classification model; or, the smartphone obtains the video segment to be classified and sends it to the server, the server uses the video classification model to obtain the classification result, and feeds the classification result back to the smartphone.

[0096] Please refer to Figure 3 , Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of the present application. After obtaining a video classification model using the model training method provided by this application, the existing complete video library is subjected to the video classification processing provided by this application, and the following is obtained: Figure 3 The available clip library is shown. Users can initiate a request for audio and video synthesis through the client and send their recorded pure audio to the server. Based on the available clip library, the server selects clips that match the pure audio, uses them to construct the corresponding video, and synthesizes the video and pure audio to generate the audio and video, which is then fed back to the client.

[0097] The video classification processing in this application refers to the process of determining whether a video can be reused to generate the above-mentioned video. The video that can be reused has certain characteristics, that is, it has low correlation with the original corresponding audio. For videos such as song MVs, there are usually two types of video segments, namely, singer singing segments and non-singing segments such as plots and scenery. For example, for the official MV of "Confession Balloon", the complete MV includes multiple transition segments. Please refer to Figure 4 , Figure 4 A schematic diagram of available and unavailable video segments provided in an embodiment of the present application. Among them, the singer's singing clip cannot be reused because the singer's mouth shape, movements, etc. are strongly related to the audio. The non-singing clip is only used to express the emotions that the song wants to express, and has a weak correlation with the audio, so it can be reused. Figure 4 As shown, the four pictures in each row are the first frame and the last frame of two video clips, respectively. The video clips with black frames (the first one in the second row and the two in the third row) are unusable clips and cannot be reused. The video clips without black frames are usable clips.

[0098] Accurately identifying whether a video is usable can reduce unusable segments that are mistakenly identified as usable segments in an available segment library (ie, a material library), thereby improving the quality of the material library.

[0099] Based on the above description, please refer to Figure 5 , Figure 5 A flow chart of a model training method provided in an embodiment of the present application. The method in this embodiment includes:

[0100] S101: Obtain a training video segment.

[0101] S102: Perform audio stream extraction processing on the training video segment to obtain a training audio segment.

[0102] A training video segment refers to a video segment with continuous, uninterrupted shots. Since the correlation between a video with continuous shots and its corresponding audio is relatively stable over time, the model trained with data consisting of training video segments and corresponding training audio segments is more accurate in judging correlations. Secondly, in a long video composed of multiple video segments, some video segments may be reusable. If the entire long video is judged as not reusable, the reusable segments are wasted. Therefore, in this application, the objects of the training process and the subsequent classification process are both video segments and audio segments.

[0103] After obtaining the training video segment, the corresponding audio data, i.e., the audio stream, is extracted from it to obtain the training audio segment. The specific extraction method is not limited. For example, the training video segment can be decoded and the audio decoded data in the decoded data can be encapsulated in an audio file format to obtain the training audio segment.

[0104] In one embodiment, the training video segments are prepared in advance and directly retrieved and used during model training. In another embodiment, the long video and the corresponding audio are segmented during training to obtain the training video segments. In this case, the steps performed include:

[0105] Step 11: Get the training video.

[0106] Step 12: Input the training video into the pre-trained shot boundary detection model to obtain the boundary detection results corresponding to each video frame in the training video.

[0107] Step 13: Segment the training video at the position of the target video frame where the boundary detection result hits, to obtain a training video segment.

[0108] The training video is a long video with discontinuous shots. The training audio is the audio corresponding to the training video. For example, the training audio is a song and the training video is the complete music video of the song. The shot boundary detection model is used to detect the shot boundary in the training video. In one embodiment, the shot boundary detection model is a TransNet network. Please refer to Figure 6 , Figure 6 A schematic diagram of a specific shot boundary detection model structure provided for an embodiment of the present application. The input of the model is an image frame sequence of length N, and the size of the image frame can be adjusted to 48×27. The image frame sequence will pass through 4 Dilated 3D CNN structures with different dilation rates, and then the output features will be concat (connected). This entire module (block) is called a DDCNN cell (Depthwise Disout Convolutional NeuralNetwork cell, deep discrete convolutional neural network unit). A max pooling network layer is added after it, which is the SDDCNN block. Such DDCNN cells and SDDCNN blocks will be stacked multiple times, and the depth can be set as needed. Finally, after two fully connected layers (Dense D layer and Dense 2 layer) and a softmax activation function layer, an N×2 vector is output. This vector is the boundary detection result, and each element of the vector indicates whether the corresponding frame in the input image frame sequence is a shot boundary.

[0109] If the boundary detection result shows that a certain video frame is a shot boundary, it is determined as the target video frame, and the training video is segmented at the position of the target video frame to obtain a training video segment. The target video frame may be the last frame of a continuous shot, or it may be the first frame.

[0110] S103: Generate a training spectrogram corresponding to the training audio segment.

[0111] A spectrogram is a graph of the speech spectrum. Its horizontal axis represents time, its vertical axis represents frequency, and the coordinate value represents the energy of the speech data. Because a spectrogram uses a two-dimensional plane to represent three-dimensional information, the energy value of the speech data is represented by color. Generally, darker colors indicate stronger speech energy at that point. While the specific method for generating a spectrogram is not limited, in one embodiment, for a speech signal x(t), the following steps can be used to generate a spectrogram:

[0112] 1) Frame the image and obtain x(m,n), where m is the number of frames and n is the frame length;

[0113] 2) Perform FFT transformation on each frame to obtain X(m,n);

[0114] 3) Draw the periodogram Y(m,n), where Y(m,n) = X(m,n)*X(m,n)T;

[0115] 4) Take 10log10(Y(m,n)), transform m into the horizontal axis scale M according to time, and n into the vertical axis scale N according to frequency;

[0116] 5) Draw a two-dimensional graph according to (M,N,10log10(Y(m,n))) to obtain the spectrogram.

[0117] Please refer to Figure 7 , Figure 7 An audio signal and corresponding spectrogram provided in an embodiment of the present application. The upper half is the audio signal, and the lower half is the spectrogram.

[0118] S104: Determine a number of training video frames in the training video segment.

[0119] Training video frames are video frames selected from the training video segment for generating training data. These frames can be randomly selected from the training video segment and represent the temporal changes in the training video segment. The number of training video frames is not limited and can be T, where T is a positive integer.

[0120] It is understandable that step S103 and step S104 can be executed in parallel to increase the model training speed.

[0121] S105: Determine a plurality of sampling time intervals based on the sampling time points corresponding to the respective training video frames, and select a plurality of training spectrogram segments from the training spectrogram according to the respective sampling time intervals.

[0122] After determining the training video frames, the corresponding sampling time intervals are constructed based on the sampling time points corresponding to each training video frame. Since each sampling time point is only the data of a moment in the training spectrogram and is represented as a line, it is impossible to use only the training spectrogram data at the sampling time point to represent the changes in the audio over time. By determining the sampling time interval based on the sampling time point, the portion of the training spectrogram that is in the sampling time interval can be determined as the training spectrogram segment corresponding to the sampling time point. Since the sampling time point corresponds to the training video frame, a one-to-one correspondence can be established between the training spectrogram segment and the training video frame.

[0123] The process of determining the sampling time interval includes:

[0124] Step 21: Taking the target sampling time point as a reference, extend the time axis to the left by a first preset length to obtain a first time coordinate, and extend the time axis to the right by a second preset length to obtain a second time coordinate.

[0125] Step 22: Determine the first time coordinate as the interval start point, and determine the second time coordinate as the interval end point, to obtain a sampling time interval corresponding to the target sampling time point.

[0126] The target sampling time point can be any sampling time point. Based on the target sampling time point, the time axis is extended in two directions to obtain two time coordinates. The first preset length and the second preset length are not limited in size and can be the same or different, for example, both can be 1 second. After the extension is completed, the smaller time coordinate is used as the starting point of the interval, and the larger time coordinate is used as the end point of the interval. The middle part is the sampling time interval.

[0127] It should be noted that, because the training spectrogram segments need to be extended on the time axis when generating them, the sampling time point should not be too close to the starting and ending points of the training video segments to prevent insufficient training spectrogram data from being extended. For example, if the first preset length and the second residual length are both 1 second, the sampling time point should be after 1 second and before 1 second from the last moment of the training video segment.

[0128] S106: Combining each training video frame and the corresponding training spectrum segment into a training data group, and generating training data using each training data group.

[0129] If the number of training video frames is T, then the number of training spectrogram segments is also T. Each training video frame and its corresponding training spectrogram segment have the same sampling time point and can be determined as a training data set. It is understood that before being combined into a training data set, certain preprocessing operations such as resizing and cropping can be performed. For example, the short side of the training video frame can be resized to 256 and then cropped to 224×224 in the center. The training spectrogram segment is cropped to 224×224.

[0130] Please refer to Figure 8 , Figure 8 A specific training data set diagram provided in the embodiment of the present application. 1 , v 2 ,…,v t Represents the preprocessed video frame, with a 1 , a 2 ,…,a t Represents the preprocessed audio image (i.e., the training spectrogram segment), where t is an integer from 1 to T.

[0131] After all the training data groups are obtained, they are sorted and combined in chronological order to obtain complete training data.

[0132] S107: Input the training data into the initial model with the time migration module to obtain a prediction result, and use the prediction result to adjust the parameters of the initial model.

[0133] The temporal shift module, TSM, moves the feature maps corresponding to the video frames in the time dimension, prompting the two-dimensional convolutional neural network (2D CNN) to comprehensively utilize the information of multiple video frames for classification. In this application, the temporal shift module will migrate in units of sampling time points. Therefore, in this application, not only the features corresponding to the video frames are migrated in the time dimension, but also the audio features, because the same training data group corresponds to the same sampling time point. The initial model refers to a convolutional neural network that has not been sufficiently trained, which can specifically be a resnet50 model.

[0134] Please refer to Figure 9 , Figure 9 This is a working effect diagram of the time migration module provided in the embodiment of the present application. It can be seen that the traditional 2D CNN model extracts features from images of different time dimensions (i.e., different sampling time points) separately, and the feature maps obtained are independent of each other and have no information exchange. Figure 9 (2) in the figure represents the working mode of TSM, which can realize information sharing between video frames at different sampling moments by translating part of the feature map of the neural network in the time dimension. Figure 9 (3) in the present application shows the solution. Since the present application wants to train a video classification model to determine whether the video and audio are sufficiently related, the following is introduced: Figure 8 The training spectrogram segments shown are used as audio input corresponding to the training video frames. TSM is used to translate the feature maps to achieve interaction between video and audio information at different sampling time points, learn the temporal correlation between video and audio, and then subsequently determine whether the two are sufficiently correlated.

[0135] The initial model includes several convolutional modules, and the time shift module is set before at least one convolutional module; wherein the time shift module is outside the residual structure to which the convolutional module belongs, or the time shift module is inside the residual structure to which the convolutional module belongs. Please refer to Figure 10 , Figure 10 The two insertion methods of the time migration module provided in the embodiments of the present application correspond to two situations: the time migration module is outside the residual structure to which the convolution module belongs, and the time migration module is inside the residual structure to which the convolution module belongs.

[0136] The upper method is called in-place shift, while the lower method is called residual shift. In-place shift places the temporal shift module outside the residual structure to which the convolutional module belongs, while residual shift places it within the residual structure to which the convolutional module belongs. A potential disadvantage of in-place shift is the loss of spatial feature learning: after the shift, some information stored in the shifted channel is lost to the current frame. To address this issue, the temporal shift module can be placed within the residual branch of the residual structure, rather than inserted outside it. This allows the information in the shifted channel to remain available through identity mapping. This shift is denoted as residual shift. Experiments show that residual-based temporal shifting performs better than position-based temporal shifting and is less sensitive to scale. Furthermore, it learns temporal features without affecting spatial features.

[0137] Based on this, the training data is input into the initial model with the time migration module, and the process of obtaining the prediction result may include:

[0138] Step 31: Input the training data into the initial convolution module of the initial model to obtain the initial feature map;

[0139] Step 32: Input the initial feature map into the feature extraction network to obtain the target feature map.

[0140] Step 33: Input the target feature map into the fully connected layer to obtain the prediction result.

[0141] The feature extraction network includes several feature extraction modules, which include a time migration module and a convolution module. The time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map. The offset process based on empty boundary filling is as follows: Figure 9 The offset processing shown here records the missing locations after the offset by adding zero data. Each feature map after the addition is the offset feature map. The convolution module performs convolution on the offset feature map to obtain the output feature map. The output feature map is used as input to the next feature extraction module or as output of the entire feature extraction network. In other words, the output feature map of the last feature extraction module is the target feature map.

[0142] It is understandable that if the time migration module is outside the residual structure to which the convolution module belongs, then in addition to inputting the offset feature map into the convolution module, it is also necessary to input the offset feature map into the shortcut connection path in the residual structure to which the convolution module belongs. Figure 10 The above insertion method shows the curved path from before conv (convolution module) to after conv.

[0143] Furthermore, in order to further promote the initial network to learn the correlation between audio and video, the initial model of the present application further includes a feature exchange module, which is set before at least one convolution module and is used to exchange the positions of several groups of video data and audio data corresponding to the same training data group in the input data. Figure 9 , Figure 9 (3) The solid line blocks represent video data (i.e., video features), and the dotted line blocks represent audio data (i.e., audio features). Through the feature exchange module, the audio data and video data corresponding to the same sampling time point can be exchanged. For example, Figure 9 In (3), the video features in the third column and the fifth row and the audio features in the third column and the sixth row are swapped.

[0144] Before adjusting parameters, you need to obtain the training labels corresponding to the training video and audio segments. After obtaining the prediction results, you can use them together with the training labels to generate a loss value, which is then used to adjust the parameters of the initial model and continue iterative training. The training label indicates whether the training video segment can be reused. The specific form is not limited. For example, 0 can indicate that it cannot be reused, while 1 can indicate that it can be reused.

[0145] S108: If it is detected that the training completion condition is met, the initial model after parameter adjustment is determined as the video classification model.

[0146] The training completion condition refers to a condition indicating that the initial model has been fully trained. The specific content is not limited. For example, it can be an upper limit condition for the number of training rounds or a lower limit condition for the recognition accuracy.

[0147] The model training method provided in the embodiments of the present application is applied to generate training data using training video segments and training audio segments. Since the correlation between audio and video typically changes over time, the temporal correlation between audio and video content is ignored. However, the correlation between shorter video and audio segments changes less, so the model trained with the training data obtained through processing has a higher accuracy rate. By generating a training spectrogram corresponding to the training audio segment, the characteristics of the audio can be represented in the form of an image. Through sampling, the local state of several training video frames is used to represent the global state of the training video segment, and the training spectrogram is sampled based on the corresponding sampling time interval. The resulting training spectrogram segment can represent the characteristics of the audio for a period of time before and after the training video frame. By combining the training video frame and the corresponding training spectrogram segment into a training data group and using the training data group to form training data, the characteristics of audio and video can be integrated into the training data, emphasizing the temporal correlation between audio and video. The initial model has a time migration module that can migrate features in the time dimension, realizing information interaction between features at different time points, so that the model can better learn the correlation between audio and video. The video classification model after training has high classification accuracy, and the quality of the material library constructed by using it for video classification is high.

[0148] Based on the above embodiment, after obtaining the video classification model, it can be used to classify existing videos to obtain reusable material clips. Figure 11 , Figure 11 A specific video classification flow chart provided for an embodiment of the present application. After obtaining the complete video, it is segmented based on a neural network, and a corresponding speech spectrum (i.e., spectrogram) is generated for its audio. At the same time, the segmented video is framed, and samples are taken in the spectrogram based on the sampling time point of the frame extraction to complete the alignment and splicing of audio and video information to obtain training data. The training data is input into a two-dimensional classification network based on TSM (i.e., a video classification model) for classification, and the video segment with the classification result as available is determined as an available material segment. The whole process can specifically include the following steps:

[0149] S201: Obtain a video segment to be classified.

[0150] S202: Perform audio stream extraction processing on the video segment to be classified to obtain an audio segment to be classified.

[0151] S203: Generate a spectrogram to be classified corresponding to the audio segment to be classified.

[0152] S204: Determine a number of video frames to be classified in the video segment to be classified.

[0153] S205: Determine a plurality of sampling time intervals based on the sampling time points corresponding to the respective video frames to be classified, and select a plurality of spectrum segments to be classified from the spectrogram to be classified according to the respective sampling time intervals.

[0154] S206: Combining the video frames to be classified and the corresponding spectrum segments to be classified into data groups to be classified, and generating data to be classified using each data group to be classified.

[0155] S207: Input the data to be classified into a video classification model with a time migration module to obtain a classification result.

[0156] The video classification model is obtained using the above-mentioned model training method. The specific contents of each of the above steps can refer to the corresponding steps in the above-mentioned model training process, the difference being that different data are processed.

[0157] Furthermore, the method for obtaining the video and audio segments to be classified can be the same as that for the training video and audio segments. Specifically, the video to be classified is first obtained and fed into a pretrained shot boundary detection model to obtain boundary detection results for each frame in the video to be classified. The video to be classified is then segmented at the locations of the target video frames where the boundary detection results hit, obtaining video segments to be classified. Audio stream extraction is then performed on the video segments to obtain audio segments to be classified.

[0158] Furthermore, the video classification model includes a convolution module, which inputs the data to be classified into the video classification model with a time migration module to obtain the classification results, including:

[0159] Step 41: Input the training data into the initial convolution module of the video classification model to obtain a preprocessed feature map;

[0160] Step 42: Input the pre-processed feature map into the feature extraction network to obtain a post-processed feature map.

[0161] Step 43: Input the post-processed feature map into the fully connected layer to obtain the classification result.

[0162] The feature extraction network includes several feature extraction modules, including a time migration module and a convolution module. The time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map, and the convolution module performs a convolution process on the offset feature map to obtain an output feature map. The output feature map of the last feature extraction module is a post-processing feature map. The pre-processing feature map in this embodiment corresponds to the initial feature map in the above-mentioned training process, and the post-processing feature map corresponds to the target feature map in the above-mentioned training process.

[0163] Preferably, the feature extraction module includes a feature exchange module, which is arranged before the convolution module or the time migration module and is used to exchange the positions of several groups of video data and audio data corresponding to the same data group to be classified in the input data.

[0164] The computer-readable storage medium provided in the embodiments of the present application is introduced below. The computer-readable storage medium described below and the model training method described above can be referenced to each other.

[0165] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned model training method are implemented.

[0166] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0167] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0168] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0169] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0170] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0171] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A model training method, characterized in that: include: Get the training video segment; Performing audio stream extraction processing on the training video segment to obtain a training audio segment; Generating a training spectrogram corresponding to the training audio segment; Determining a plurality of training video frames in the training video segment; Determining a plurality of sampling time intervals based on sampling time points corresponding to each of the training video frames, and selecting a plurality of training spectrogram segments in the training spectrogram according to each of the sampling time intervals; Combining each of the training video frames and the corresponding training spectrogram segments into a training data group, and generating training data using each of the training data groups; Inputting the training data into an initial model with a time migration module to obtain a prediction result, and using the prediction result to adjust parameters of the initial model; If it is detected that the training completion condition is met, the initial model after parameter adjustment is determined as a video classification model, so that the video classification model determines whether the video can be reused based on the strength of the correlation between the video and the corresponding audio.

2. The model training method according to claim 1, characterized in that The initial model includes several convolution modules, and the time migration module is arranged before at least one of the convolution modules; wherein, the time migration module is outside the residual structure to which the convolution module belongs, or the time migration module is inside the residual structure to which the convolution module belongs.

3. The model training method according to claim 2, characterized in that Inputting the training data into the initial model with the time migration module to obtain a prediction result includes: Inputting the training data into the initial convolution module of the initial model to obtain an initial feature map; Inputting the initial feature map into a feature extraction network to obtain a target feature map; wherein the feature extraction network includes a plurality of feature extraction modules, the feature extraction modules including the time migration module and the convolution module, the time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map, and the convolution module performs a convolution process on the offset feature map to obtain an output feature map; the output feature map of the last feature extraction module is the target feature map; The target feature map is input into the fully connected layer to obtain the prediction result.

4. The model training method according to claim 3, characterized in that If the time shift module is outside the residual structure to which the convolution module belongs, the method includes: The offset feature map is input into a shortcut connection path in the residual structure to which the convolution module belongs.

5. The model training method according to claim 1, characterized in that Also includes: Obtaining training labels corresponding to the training video segment and the training audio segment; The use of the prediction results to adjust the parameters of the initial model includes: Obtaining a loss value using the training label and the prediction result; Parameters of the initial model are adjusted based on the loss value.

6. The model training method according to claim 1, characterized in that The obtaining of the training video segment includes: Get training videos; Inputting the training video into a pre-trained shot boundary detection model to obtain boundary detection results corresponding to each video frame in the training video; The training video is segmented at a position of the target video frame where the boundary detection result is a hit, to obtain the training video segment.

7. The model training method according to claim 1, characterized in that The determining of a plurality of sampling time intervals based on the sampling time points corresponding to the respective training video frames includes: Taking the target sampling time point as a reference, extending a first preset length to the left on the time axis to obtain a first time coordinate, and extending a second preset length to the right to obtain a second time coordinate; the target sampling time point is any one of the sampling time points; The first time coordinate is determined as the interval start point, and the second time coordinate is determined as the interval end point, to obtain the sampling time interval corresponding to the target sampling time point.

8. The model training method according to claim 2, characterized in that The initial model includes a feature exchange module, which is arranged before at least one of the convolution modules; the feature exchange module is used to exchange the positions of several groups of video data and audio data in the input data corresponding to the same training data group.

9. A video classification method, characterized in that: include: Get the video segment to be classified; Performing audio stream extraction processing on the video segment to be classified to obtain an audio segment to be classified; Generating a spectrogram to be classified corresponding to the audio segment to be classified; Determining a plurality of video frames to be classified in the video segment to be classified; Determining a plurality of sampling time intervals based on sampling time points corresponding to the respective video frames to be classified, and selecting a plurality of spectrum segments to be classified in the spectrogram to be classified according to the respective sampling time intervals; Combining the to-be-classified video frames and the corresponding to-be-classified spectrum segments into to-be-classified data groups, and generating to-be-classified data using each of the to-be-classified data groups; Inputting the data to be classified into a video classification model with a time migration module to obtain a classification result; The video classification model is obtained using the model training method described in any one of claims 1 to 8, and the video classification model is used to determine whether a video can be reused based on the strength of the correlation between the video and the corresponding audio.

10. The video classification method according to claim 9, characterized in that: The video classification model includes a convolution module, and the inputting of the data to be classified into the video classification model with a time migration module to obtain a classification result includes: Inputting the training data into the initial convolution module of the video classification model to obtain a preprocessed feature map; Inputting the preprocessed feature map into a feature extraction network to obtain a post-processed feature map; wherein the feature extraction network includes a plurality of feature extraction modules, and the feature extraction modules include the time migration module and the convolution module. The time migration module performs an offset process based on empty boundary filling on the input feature map to obtain an offset feature map, and the convolution module performs a convolution process on the offset feature map to obtain an output feature map; the output feature map of the last feature extraction module is the post-processed feature map; The post-processing feature map is input into a fully connected layer to obtain the classification result.

11. The video classification method according to claim 10, characterized in that: The feature extraction module includes a feature exchange module, which is arranged before the convolution module or the time migration module; the feature exchange module is used to exchange the positions of several groups of video data and audio data in the input data corresponding to the same group of data to be classified.

12. The video classification method according to claim 9, characterized in that: The obtaining of the video segment to be classified includes: Get the video to be classified; Inputting the video to be classified into a pre-trained shot boundary detection model to obtain boundary detection results corresponding to each video frame in the video to be classified; The video to be classified is segmented at a position of the target video frame where the boundary detection result is a hit, to obtain the video segment to be classified.

13. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the model training method according to any one of claims 1 to 8, and / or the video classification method according to any one of claims 9 to 12.

14. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the model training method according to any one of claims 1 to 8, and / or the video classification method according to any one of claims 9 to 12.

Images