Video recognition model training method, video recognition method, device, and storage medium

Abstract

The present application relates to the technical field of computer vision, and provides a video recognition model training method, a video recognition method, equipment and a storage medium, the video recognition model training method comprises: acquiring a sample video and a sample sound corresponding to the sample video; inputting the sample video into a video extraction model to obtain first video features output by the video extraction model; inputting the sample sound into a sound extraction model to obtain first sound features output by the sound extraction model; performing feature matching on the first sound features and the first video features, performing model training based on the features after feature matching, and obtaining a video recognition model. The present application performs feature matching on the first sound features and the first video features, then performs model training based on the features after feature matching, thereby improving the model training effect, improving the robustness of the video recognition model, and finally improving the accuracy of video recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video recognition model training method, a video recognition method, a device, and a storage medium. Background Technology

[0002] With the rapid development of computer vision technology, its applications are becoming increasingly widespread. For video recognition tasks, it is necessary to perform video recognition on the video to be recognized in order to obtain the recognition results.

[0003] For example, repetitive action counting is a traditional field in computer vision, with wide applications such as analyzing motion videos and counting fitness movements. It is an important branch of video understanding. Traditional techniques include using Fourier analysis, wavelet transform, and convolutional neural networks for repetitive action counting, but the final video counting and recognition results always suffer from low accuracy. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a video recognition model training method that performs feature matching between a first audio feature and a first video feature, and then trains the model based on the feature-matched features, thereby improving the model training effect, enhancing the robustness of the video recognition model, and ultimately improving the accuracy of video recognition.

[0005] The present invention also provides a video recognition device, a video recognition method, an electronic device, and a storage medium.

[0006] A video recognition model training method according to a first aspect of the present invention includes:

[0007] Acquire sample videos and corresponding sample audio.

[0008] The sample video is input into the video extraction model to obtain the first video feature output by the video extraction model;

[0009] The sample sound is input into the sound extraction model to obtain the first sound feature output by the sound extraction model;

[0010] The first sound feature is matched with the first video feature, and the model is trained based on the matched features to obtain the video recognition model.

[0011] According to the video recognition model training method of the present invention, a sample video is input into a video extraction model to obtain the first video feature output by the video extraction model, and a sample sound is input into a sound extraction model to obtain the first sound feature output by the sound extraction model. The first sound feature is then matched with the first video feature, and the model is trained based on the matched feature, thereby improving the model training effect, improving the robustness of the video recognition model, and ultimately improving the accuracy of video recognition.

[0012] According to an embodiment of the present invention, the feature matching of the first audio feature and the first video feature includes:

[0013] Based on global average pooling, the first audio feature and the first video feature are dimensionally aligned to obtain the second audio feature and the second video feature.

[0014] According to one embodiment of the present invention, the step of dimensionally aligning the first audio feature and the first video feature based on global average pooling to obtain a second audio feature and a second video feature further includes:

[0015] The second sound feature is subjected to spectral transformation analysis to obtain the sound signal channel;

[0016] Perform spectral transformation analysis on the second video feature to obtain the video signal channel;

[0017] Based on knowledge distillation, the audio signal channel and the video signal channel in the time dimension are aligned so that the second audio feature is enhanced into the second video feature.

[0018] According to one embodiment of the present invention, the audio signal channel includes a high-frequency signal channel and a low-frequency signal channel, and the video signal channel includes a high-frequency signal channel and a low-frequency signal channel;

[0019] The method of aligning the audio signal channel with the video signal channel in the time dimension based on knowledge distillation includes:

[0020] Based on knowledge distillation, the high-frequency and low-frequency signals of the audio signal channel are aligned with the high-frequency and low-frequency signals of the video signal channel, respectively.

[0021] According to one embodiment of the present invention, the knowledge distillation method is a knowledge distillation optimization method based on the mean absolute value error loss function.

[0022] According to one embodiment of the present invention, the video recognition model is used to perform video recognition on the video to be recognized to obtain a video recognition result. The video recognition result is obtained by performing video recognition based on the classification layer of the video recognition model, or the video recognition result is obtained by performing video recognition based on the regression layer of the video recognition model.

[0023] According to one embodiment of the present invention, the video recognition model is used for at least one of repetitive action counting recognition, action recognition, and video segmentation.

[0024] A video recognition method according to a second aspect of the present invention includes:

[0025] Obtain the video to be recognized;

[0026] The video to be identified is input into the video recognition model to obtain the video recognition result output by the video recognition model;

[0027] The video recognition model is trained using any of the video recognition model training methods described above.

[0028] According to the video recognition method of the present invention, during the inference process, only the video to be recognized needs to be input into the video recognition model to perform video recognition. Thus, only audio data is introduced during the training process, so that during the inference process, only the video to be recognized needs to be input into the video recognition model to accurately obtain the video recognition result. This ensures the accuracy of video recognition while improving the efficiency and applicability of video recognition.

[0029] An electronic device according to a third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements either the video recognition model training method described above or the video recognition method described above.

[0030] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the video recognition model training method as described above, or implements the video recognition method as described above.

[0031] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0032] During training, sample videos are input into the video extraction model to obtain the first video feature output by the video extraction model, and sample audio is input into the audio extraction model to obtain the first audio feature output by the audio extraction model. The first audio feature is then matched with the first video feature, and the model is trained based on the matched features, thereby improving the model training effect, enhancing the robustness of the video recognition model, and ultimately improving the accuracy of video recognition.

[0033] During the inference process, only the video to be recognized needs to be input into the video recognition model to perform video recognition. Thus, only audio data is introduced during the training process. During the inference process, only the video to be recognized needs to be input into the video recognition model to accurately obtain the video recognition result. This ensures the accuracy of video recognition while improving the efficiency and applicability of video recognition.

[0034] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the video recognition model training method provided in this embodiment of the invention;

[0037] Figure 2 This is a flowchart illustrating the video recognition method provided in an embodiment of the present invention;

[0038] Figure 3 This is the second flowchart illustrating the video recognition model training method provided in this embodiment of the invention;

[0039] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] The present invention proposes the following embodiments. The video recognition model training method and video recognition method of the present invention are described below with reference to the accompanying drawings.

[0042] The execution entity for this video recognition model training method can be a server, a home service robot, or a user's terminal, including but not limited to mobile phones, tablets, PCs, in-vehicle terminals, and smart home appliances.

[0043] Figure 1 This is one of the flowcharts illustrating the video recognition model training method provided in this embodiment of the invention, such as... Figure 1 As shown, the training of this video recognition model includes:

[0044] Step 110: Obtain the sample video and the sample audio corresponding to the sample video.

[0045] Here, the sample video and sample audio are the training samples to be used in training. The sample audio was acquired along with the sample video. The sample video consists of multiple frames.

[0046] For example, if the sample video is an action video, the trained video recognition model can count repeated actions in the video to be recognized to obtain a repeated action count recognition result, or the trained video recognition model can perform action recognition on the video to be recognized to obtain an action recognition result; or if the sample video is a video to be segmented, the trained video recognition model can perform video segmentation on the video to be recognized to obtain a video segmentation result.

[0047] Step 120: Input the sample video into the video extraction model to obtain the first video feature output by the video extraction model.

[0048] Specifically, the sample video is input into the feature extraction layer of the video extraction model to obtain the first video feature output by the feature extraction layer. The first video feature includes multiple image features, and the number of image features is consistent with the number of frames in the sample video. That is, each frame of the sample video is input into the feature extraction layer to obtain the image features of multiple frames output by the feature extraction layer.

[0049] The feature extraction layer is used to extract features from the video data. The specific structure of this feature extraction layer can be set according to actual needs, such as a 3D convolutional network layer; however, this embodiment of the invention does not impose specific limitations on it.

[0050] Step 130: Input the sample sound into the sound extraction model to obtain the first sound feature output by the sound extraction model.

[0051] Specifically, the sample sound is input into the feature extraction layer of the sound extraction model to obtain the first sound feature output by the feature extraction layer. This feature extraction layer is used to extract features from the sound data. The specific structure of this feature extraction layer can be set according to actual needs, for example, a 2D convolutional network layer; this embodiment of the invention does not specifically limit this.

[0052] In one embodiment, the encoded features of the sample sound are input into the feature extraction layer to obtain the first sound feature output by the feature extraction layer. The encoded features are obtained by time-frequency encoding of the sample sound; that is, the encoded features are time-frequency features, or spectral features.

[0053] Step 140: Perform feature matching between the first sound feature and the first video feature, and train the model based on the feature matching to obtain the video recognition model.

[0054] Specifically, the video recognition model is trained based on the features obtained from feature matching. This video recognition model can include the video extraction model and the video recognition result prediction layer described above. Furthermore, the sound extraction model can also be trained based on the features obtained from feature matching.

[0055] In one embodiment, the feature matching method between the first audio feature and the first video feature is as follows: aligning the first audio feature and the first video feature along the temporal dimension. Specifically, pooling is used to align the first audio feature and the first video feature along the temporal dimension. For example, global average pooling is used to align the first audio feature and the first video feature along the temporal dimension.

[0056] It should be noted that, considering that the sample audio and sample video may differ in the time dimension, the first audio feature and the first video feature are aligned in the time dimension. The model is then trained based on the alignment result, thereby improving the model training effect and ultimately improving the accuracy of video recognition.

[0057] The video recognition method provided in this invention inputs a sample video into a video extraction model to obtain the first video feature output by the video extraction model, and inputs a sample audio into an audio extraction model to obtain the first audio feature output by the audio extraction model. The first audio feature is then matched with the first video feature, and the model is trained based on the matched feature, thereby improving the model training effect, enhancing the robustness of the video recognition model, and ultimately improving the accuracy of video recognition.

[0058] Based on the above embodiments, in this method, step 140, which involves feature matching between the first audio feature and the first video feature, includes:

[0059] Based on global average pooling, the first audio feature and the first video feature are dimensionally aligned to obtain the second audio feature and the second video feature.

[0060] Accordingly, a video recognition model is obtained by training the model based on the second sound features and the second video features.

[0061] The video recognition method provided in this invention takes into account that sample audio and sample video may differ in dimensionality. Based on global average pooling, the first audio feature and the first video feature are dimensionally aligned to obtain the first audio feature and the first video feature under the same dimension. Then, the model is trained based on the alignment result, thereby improving the model training effect and ultimately improving the accuracy of video recognition.

[0062] Based on any of the above embodiments, in this method, after dimensionally aligning the first audio feature and the first video feature using a global average pooling method to obtain the second audio feature and the second video feature, the method further includes:

[0063] The second sound feature is subjected to spectral transformation analysis to obtain the sound signal channel;

[0064] Perform spectral transformation analysis on the second video feature to obtain the video signal channel;

[0065] Based on knowledge distillation, the audio signal channel and the video signal channel in the time dimension are aligned so that the second audio feature is enhanced into the second video feature.

[0066] Here, spectral transformation analysis may include, but is not limited to, Fourier transform analysis or wavelet transform analysis, etc.

[0067] In some embodiments, the audio signal is introduced into the video action counting task through knowledge distillation, and the high-frequency region of the audio signal is input into the convolutional neural network as a feature distillation scheme through wavelet transform features.

[0068] The video recognition method provided in this invention is based on knowledge distillation, which aligns the audio signal channel and the video signal channel in the time dimension to enhance the second audio feature into a second video feature. Then, the model is trained based on the enhanced second video feature, thereby improving the model training effect and ultimately improving the accuracy of video recognition.

[0069] Based on any of the above embodiments, in this method, the audio signal channel includes a high-frequency signal channel and a low-frequency signal channel, and the video signal channel includes a high-frequency signal channel and a low-frequency signal channel.

[0070] The method of aligning the audio signal channel with the video signal channel in the time dimension based on knowledge distillation includes:

[0071] Based on knowledge distillation, the high-frequency and low-frequency signals of the audio signal channel are aligned with the high-frequency and low-frequency signals of the video signal channel, respectively.

[0072] For ease of understanding, for example, an audio signal channel includes features of four spectral channels, such as the first channel feature, the second channel feature, the third channel feature, and the fourth channel feature; a video signal channel includes features of four spectral channels, such as the fifth channel feature, the sixth channel feature, the seventh channel feature, and the eighth channel feature. Assuming that the first channel feature is a low-frequency signal channel, the second, third, and fourth channel features are high-frequency signal channels, the fifth channel feature is a low-frequency signal channel, and the sixth, seventh, and eighth channel features are high-frequency signal channels, then the first channel feature is aligned with the fifth channel feature, the second channel feature is aligned with the sixth channel feature, the third channel feature is aligned with the seventh channel feature, and the fourth channel feature is aligned with the eighth channel feature. Furthermore, the first channel feature may also include a first sub-channel feature, a second sub-channel feature, a third sub-channel feature, and a fourth sub-channel feature, and the fifth channel feature may also include a fifth sub-channel feature, a sixth sub-channel feature, a seventh sub-channel feature, and an eighth sub-channel feature; assuming that the first sub-channel feature is a low-frequency signal channel, the second, third, and fourth sub-channel features are high-frequency signal channels, the fifth sub-channel feature is a low-frequency signal channel, and the sixth, seventh, and eighth sub-channel features are high-frequency signal channels, then the first sub-channel feature is aligned with the fifth sub-channel feature, the second sub-channel feature is aligned with the sixth sub-channel feature, the third sub-channel feature is aligned with the seventh sub-channel feature, and the fourth sub-channel feature is aligned with the eighth sub-channel feature.

[0073] The video recognition method provided in this invention is based on knowledge distillation. It aligns multiple spectral signal channels of the audio signal channel and multiple spectral signal channels of the video signal channel to enhance the second audio feature into a second video feature. Then, it trains the model based on the enhanced second video feature, thereby improving the model training effect and ultimately improving the accuracy of video recognition.

[0074] Based on any of the above embodiments, in this method, the knowledge distillation method is a knowledge distillation optimization method based on the mean absolute value error loss function.

[0075] Here, the mean absolute value error loss function is the distillation loss function of the knowledge distillation method.

[0076] Based on any of the above embodiments, in this method, the video recognition model is used to perform video recognition on the video to be recognized to obtain a video recognition result. The video recognition result is obtained by performing video recognition based on the classification layer of the video recognition model, or the video recognition result is obtained by performing video recognition based on the regression layer of the video recognition model.

[0077] Here, the video to be identified is the video for which video recognition is required. For example, the video to be identified is an action video, which is identified by counting the repeated actions to obtain the repeated action count recognition result, or by performing action recognition on the video to obtain the action recognition result; or, for example, the video to be identified is a video to be segmented, which is identified by performing video segmentation on the video to obtain the video segmentation result. The video to be identified includes multiple frames of images.

[0078] Here, the video recognition results may include, but are not limited to: repeat action counting recognition results, action recognition results, video segmentation results, etc. It is understood that the embodiments of the present invention can be applied to various video understanding scenarios.

[0079] Specifically, the video to be recognized is input into a video recognition model to obtain the video recognition result output by the model. More specifically, the video to be recognized is input into the feature extraction layer of the video recognition model to obtain the image features output by the feature extraction layer. Based on these image features, the corresponding video recognition result is determined. The number of image features is consistent with the number of frames in the video to be recognized; that is, each frame of the video to be recognized is input into the feature extraction layer to obtain the image features of multiple frames output by the feature extraction layer.

[0080] In one specific embodiment, the image features are input into the video recognition result prediction layer of the video recognition model to obtain the video recognition result output by the video recognition result prediction layer. The video recognition result prediction layer can be a classification layer or a regression layer. For example, if the repetitive action counting task is treated as a classification task, the classification layer outputs n categories, each category corresponding to a count number, where n is a pre-set maximum count number. Alternatively, if the repetitive action counting task is treated as a regression task, the regression layer outputs a single count number.

[0081] The video recognition method provided in this embodiment of the invention provides support for determining the video recognition result through the above-described manner.

[0082] Based on any of the above embodiments, in this method, the video recognition model is used for at least one of repetitive action counting recognition, action recognition, and video segmentation.

[0083] Accordingly, the video recognition results include at least one of the following: repeat action count recognition results, action recognition results, and video segmentation results.

[0084] Among them, the repetitive action counting recognition result is the video recognition result obtained by applying the repetitive action counting task scenario in the embodiments of the present invention. For example, the analysis of sports videos and the counting of fitness movements both require the repetitive action counting recognition result.

[0085] Among them, the action recognition result is the video recognition result obtained by applying the action recognition task scenario in the embodiment of the present invention.

[0086] The video segmentation result is the video recognition result obtained by applying the video segmentation task scenario in the embodiments of the present invention.

[0087] Based on the above, the entity executing this video recognition method can be a video recognition device, a server, a home service robot, or a user's terminal, including but not limited to mobile phones, tablets, PCs, vehicle terminals, and smart home appliances.

[0088] Figure 2 This is a flowchart illustrating the video recognition method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the video recognition method includes:

[0089] Step 210: Obtain the video to be identified.

[0090] Here, the video to be identified is the video for which video recognition is required. For example, the video to be identified is an action video, which is identified by counting the repeated actions to obtain the repeated action count recognition result, or by performing action recognition on the video to obtain the action recognition result; or, for example, the video to be identified is a video to be segmented, which is identified by performing video segmentation on the video to obtain the video segmentation result. The video to be identified includes multiple frames of images.

[0091] Step 220: Input the video to be recognized into the video recognition model to obtain the video recognition result output by the video recognition model.

[0092] The video recognition model is trained by the video recognition model training method described in any of the above embodiments.

[0093] Specifically, the video to be identified is input into the feature extraction layer of the video recognition model to obtain image features output by the feature extraction layer. Based on these image features, the video recognition result corresponding to the video to be identified is determined. The number of image features is consistent with the number of frames in the video to be identified; that is, each frame of the video to be identified is input into the feature extraction layer to obtain image features of multiple frames output by the feature extraction layer.

[0094] In one embodiment, the image features are input to the video recognition result prediction layer of the video recognition model to obtain the video recognition result output by the video recognition result prediction layer. The video recognition result prediction layer can be a classification layer or a regression layer; this embodiment of the invention does not specifically limit this. For example, if the repetitive action counting task is treated as a classification task, the classification layer outputs n categories, each category corresponding to a count number, where n is a pre-set maximum count number. Alternatively, if the repetitive action counting task is treated as a regression task, the regression layer outputs a single count number.

[0095] Here, the video recognition results may include, but are not limited to: repeat action counting recognition results, action recognition results, video segmentation results, etc. It is understood that the embodiments of the present invention can be applied to various video understanding scenarios.

[0096] The video recognition model is obtained through knowledge distillation.

[0097] The video recognition method provided in this embodiment of the invention only requires inputting the video to be recognized into the video recognition model during the inference process to perform video recognition. Thus, only audio data is introduced during the training process, so that during the inference process, only the video to be recognized needs to be input into the video recognition model to accurately obtain the video recognition result. This ensures the accuracy of video recognition while improving the efficiency and applicability of video recognition.

[0098] Based on any of the above embodiments, in this method, the video recognition result includes at least one of the following: repeat action count recognition result, action recognition result, and video segmentation result.

[0099] Here, the repetitive action counting and recognition result is the video recognition result obtained by applying the repetitive action counting task scenario in the embodiments of the present invention. For example, the analysis of sports videos and the counting of fitness movements both require the repetitive action counting and recognition result.

[0100] Here, the action recognition result is the video recognition result obtained by applying the action recognition task scenario in the embodiments of the present invention.

[0101] Here, the video segmentation result is the video recognition result obtained by applying the video segmentation task scenario in the embodiments of the present invention.

[0102] To facilitate understanding of the above embodiments, a specific embodiment will be used as an example for illustration. Figure 3 This is a second schematic flowchart of the video recognition model training method provided in this embodiment of the invention, as shown below. Figure 3 As shown, the whole includes a video extraction model and a sound extraction model, which introduces sound signals into video recognition tasks (such as video action counting tasks) through knowledge distillation. This knowledge distillation method is divided into output distillation and wavelet distillation.

[0103] For the video extraction model, the first step is to obtain a video segment as input. Each video segment consists of t-dimensional images (b, c, h, w), as detailed below. Figure 3 As shown in the upper part. More specifically, the video extraction model includes a 3D CNN (3D Convolutional Neural Network) to input the video into the 3D CNN, i.e., to process the input video using the 3D CNN, for example, a 3D ResNet; then, linear layers (such as fully connected layers, FC) and logits (regression layers) are used to output the video recognition result, which is a number, i.e., the video action count is 5, and this result is also used as the output of the entire model.

[0104] The sound extraction model primarily processes the sound signal. First, the original sound file is converted into a spectrogram, and then the sound extraction model is used to extract sound features. (See details...) Figure 3 The lower half is shown. More specifically, the sound extraction model includes a 2D CNN (2D convolutional neural network) to input the spectrogram into the 2D CNN, i.e., using the 2D CNN to extract features from the input spectrogram; then, linear layers (such as fully connected layers, fc) and logits (regression layers) are used to output the video recognition result, which is a number, i.e., the video action count is 5. This result is used to determine the count of repeated actions in the video using sound.

[0105] Since the video extraction model and the sound extraction model use different signals (i.e., video and sound) to judge the count of repeated actions in the video, and the feature signals used to judge the video recognition results are quite different, the sound signal is introduced into the video recognition task through knowledge distillation. The main function of this knowledge distillation method is to match the features extracted from the sound signal with the video, so that the video processing branch can also obtain the features of the judgment result corresponding to the sound signal.

[0106] Regarding the output distillation method, if the repeated action counting task is constructed as a classification task, distillation can be performed based on the classification task, that is, the distillation loss can be CE loss (cross-entropy loss); similarly, if the repeated action counting task is constructed as a regression task, distillation can be performed based on the regression task, that is, the distillation loss can be regression class loss (such as L1 loss, L2 loss, L2 smooth loss, etc.).

[0107] For wavelet distillation, two main points are considered. First, aligning the dimensions of the features extracted by the sound extraction model and the video extraction model. Global average pooling (gap) can be used for this alignment, allowing for separate processing of sound and video features to obtain features of the same dimension. Other methods can also be used for dimension alignment. Second, the core of wavelet distillation, involves transforming the dimensionally transformed sound and video features, such as using a fast Fourier transform or a discrete wavelet transform. For example, the transformation might yield four channels: LL, LH, HL, and HH, where LL is the low-frequency channel and LH, HL, and HH are the high-frequency channels. Furthermore, LL can be further decomposed into LL2, LH2, HL2, and HH2, and so on. Here, we use signal channels across all dimensions because we hope to map the high / low frequency signals of the audio component and the high / low frequency signals of the image component in the time dimension through distillation, thereby enhancing the audio information and enabling the video feature extraction model to identify the key repetitive action intervals for counting repetitive actions.

[0108] Finally, let's take wavelet distillation as an example. Specifically, the distillation optimization loss can use L1 norm loss, then constrain the dot product of features and frequencies, and finally perform a mean operation on multiple channels to obtain the final result.

[0109] In practical applications, through the above embodiments, without introducing additional computational burden and input, the audio features of the sample audio are used to predict the video recognition results only during the training process. This introduces audio data during training, improves the performance of the video recognition model, and thus ensures the accuracy of video recognition while improving the efficiency and applicability of video recognition.

[0110] The video recognition model training method provided in this application is a video repetitive action counting method based on sound signal distillation. By introducing an additional sound signal module, processing it with a separate neural network, and then performing an ensemble operation with the image model, the repetitive action counting results are obtained. By introducing additional knowledge into the original network through distillation, not only can better video recognition results be obtained, but the computational load of the model is not increased, thus without reducing the processing speed of the model.

[0111] In the above embodiments, during the training of the video recognition model, the audio signal is introduced into the video-based repetitive action counting task through knowledge distillation. This eliminates the need to input the audio signal during model usage (prediction), significantly expanding the application scenarios of the technology. Furthermore, the audio module in the technical solution of this application only participates in training, without introducing additional computational burden or input. Introducing the audio signal into the video-based repetitive action counting task does not affect the inference speed during model usage and improves the original algorithm performance.

[0112] The video recognition device provided by the present invention is described below. The video recognition device described below and the video recognition method described above can be referred to in correspondence.

[0113] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a video recognition model training method. This method includes: acquiring a sample video and a corresponding sample audio; inputting the sample video into a video extraction model to obtain a first video feature output by the video extraction model; inputting the sample audio into an audio extraction model to obtain a first audio feature output by the audio extraction model; performing feature matching between the first audio feature and the first video feature; and training the model based on the feature-matched features to obtain a video recognition model. Alternatively, the processor 410 can call logical instructions in the memory 430 to execute a video recognition method, which includes: acquiring a video to be recognized; inputting the video to be recognized into a video recognition model to obtain a video recognition result output by the video recognition model; wherein the video recognition model is trained by the video recognition model training method described above.

[0114] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video recognition model training method provided by the above methods. The method includes: acquiring a sample video and a sample audio corresponding to the sample video; inputting the sample video into a video extraction model to obtain a first video feature output by the video extraction model; inputting the sample audio into an audio extraction model to obtain a first audio feature output by the audio extraction model; performing feature matching between the first audio feature and the first video feature; and training a model based on the feature-matched feature to obtain a video recognition model. Alternatively, the computer can execute the video recognition method provided by the above methods, which includes: acquiring a video to be recognized; inputting the video to be recognized into a video recognition model to obtain a video recognition result output by the video recognition model; wherein the video recognition model is trained by the video recognition model training method described above.

[0116] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a video recognition model training method provided by the methods described above. This method includes: acquiring a sample video and sample audio corresponding to the sample video; inputting the sample video into a video extraction model to obtain a first video feature output by the video extraction model; inputting the sample audio into an audio extraction model to obtain a first audio feature output by the audio extraction model; performing feature matching between the first audio feature and the first video feature; and training a model based on the feature-matched features to obtain a video recognition model. Alternatively, when executed by a processor, the computer program implements the video recognition method provided by the methods described above. This method includes: acquiring a video to be recognized; inputting the video to be recognized into a video recognition model to obtain a video recognition result output by the video recognition model; wherein the video recognition model is trained by the video recognition model training method described above.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0120] The above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Although the invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the invention do not depart from the spirit and scope of the invention and should be covered within the scope of the claims of the invention.

Claims

1. A video recognition model training method, characterized in that, include: Acquire sample videos and corresponding sample audio. The sample video is input into the video extraction model to obtain the first video feature output by the video extraction model; The sample sound is input into the sound extraction model to obtain the first sound feature output by the sound extraction model; The first sound feature is matched with the first video feature, and the model is trained based on the matched features to obtain a video recognition model. The step of matching the first audio feature with the first video feature includes: Based on global average pooling, the first audio feature and the first video feature are dimensionally aligned to obtain the second audio feature and the second video feature. The second sound feature is subjected to spectral transformation analysis to obtain a sound signal channel; the sound signal channel includes a high-frequency signal channel and a low-frequency signal channel. The second video feature is subjected to spectral transformation analysis to obtain the video signal channel; the video signal channel includes a high-frequency signal channel and a low-frequency signal channel. Based on knowledge distillation, the high-frequency and low-frequency signals of the audio signal channel are aligned with the high-frequency and low-frequency signals of the video signal channel, respectively, so as to enhance the second audio feature to the second video feature.

2. The video recognition model training method according to claim 1, characterized in that, The knowledge distillation method described above is an optimization approach based on the mean absolute value error loss function.

3. The video recognition model training method according to claim 1, characterized in that, The video recognition model is used to perform video recognition on the video to be recognized and obtain video recognition results. The video recognition results are obtained based on the classification layer of the video recognition model, or the video recognition results are obtained based on the regression layer of the video recognition model.

4. The video recognition model training method according to any one of claims 1 to 3, characterized in that, The video recognition model is used for at least one of repetitive action counting recognition, action recognition, and video segmentation.

5. A video recognition method, characterized in that, include: Obtain the video to be recognized; The video to be identified is input into the video recognition model to obtain the video recognition result output by the video recognition model; The video recognition model is trained by the video recognition model training method as described in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the video recognition model training method as described in any one of claims 1 to 4 or the video recognition method as described in claim 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video recognition model training method as described in any one of claims 1 to 4 or the video recognition method as described in claim 5.

Images