Optimization method, identification method, optimization device, identification device, apparatus, and medium

Abstract

The application belongs to the technical field of neural network optimization, and particularly relates to an optimization method, an identification method, an optimization device, an identification device, equipment and a medium. The application extracts initial audio features, input audio features, local appearance features, global appearance features, input appearance features and optical flow features from a training video by using a feature extraction network; determines global audio positive samples and global audio negative samples according to the global appearance features and the initial audio features; determines local audio positive samples and local audio negative samples according to the local appearance features and the initial audio features; and optimizes the feature extraction network according to the global audio feature comparison loss, the local audio feature comparison loss, the mutual distillation loss and the classification loss. The application fully realizes the utilization of multi-modal information, and further improves the accuracy of video identification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural network optimization, and particularly relates to an optimization method, an identification method, an optimization device, an identification device, equipment and a medium. BACKGROUND

[0002] In recent years, with the popularization of intelligent terminals, the development of communication and the wide application of many video platforms, the data volume of videos has greatly increased, the transmission speed has become faster, and the transmission range has become more extensive. Video content review is crucial to maintaining a healthy and green network environment. In network video content supervision, special videos are an important review point. With the progress of video technology and the development of 5G communication technology, the application of special video recognition is also more extensive, which can not only be applied to Internet video content review to determine whether the network video contains special behavior fragments, but also can be applied to intelligent monitoring, smart home and other devices to monitor abnormal situations in reality and issue alarms. As can be seen, the special video recognition task has great development space and rich application scenarios.

[0003] Existing special video recognition technologies mainly include traditional methods based on hand-crafted features and methods based on deep learning. The improvement of data volume and computing power has greatly promoted the research and application of deep learning algorithms. With the rapid development of deep learning algorithms in the field of computer vision, special video recognition based on deep learning methods has become the mainstream method.

[0004] The small size of existing special video datasets limits the training of the network, and the performance of special video recognition algorithms based on deep learning is limited. Current special video data mainly contains visual and audio modal information, and the visual information is divided into appearance information and optical flow information representing motion. The appearance information mainly includes scene information, such as bloody scenes and explosions, etc.; the optical flow information mainly includes the behavior of the main characters, such as fighting and shooting, etc.; the audio information mainly includes the screams, explosion sounds and gunshots that occur when violent behavior occurs, etc. The existing special video recognition technology does not fully utilize multi-modal information, mainly in that there is no interaction between multi-modal information in the feature extraction stage, and the correlation between multi-modal information is not fully utilized, resulting in low accuracy of the extracted multi-modal information. For example, the existing audio-visual contrast learning algorithm selects positive and negative samples within a fixed time range, which introduces noise and cannot fully utilize the information in different time dimensions. SUMMARY

[0005] To address the aforementioned technical problems, this invention proposes an optimization method, a recognition method, an optimization device, a recognition device, an apparatus, and a medium. This application utilizes a feature extraction network to extract initial audio features, input audio features, local appearance features, global appearance features, and optical flow features from a training video. The process involves: determining global positive and negative audio samples based on the global appearance features and the initial audio features; determining local positive and negative audio samples based on the local appearance features and the initial audio features; determining a global audio-visual feature contrast loss based on the global positive and negative audio samples; determining a local audio-visual feature contrast loss based on the local positive and negative audio samples; acquiring self-supervised appearance features from the training video through a self-supervised training network; determining a cross-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features; determining a classification loss based on the input appearance features, the input audio features, and the optical flow features; and optimizing the feature extraction network based on the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the cross-distillation loss, and the classification loss. It fully utilizes multimodal information, making the input appearance features, optical flow features and input audio features obtained by the optimized feature extraction network more accurate, thereby improving the accuracy of video recognition.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention includes six aspects.

[0007] In a first aspect, an optimization method is provided, comprising: extracting initial audio features, input audio features, local appearance features, global appearance features, input appearance features, and optical flow features from a training video using a feature extraction network; determining global positive audio samples and global negative audio samples based on the global appearance features and the initial audio features; determining local positive audio samples and local negative audio samples based on the local appearance features and the initial audio features; determining a global audio-visual feature contrast loss based on the global positive audio samples and the global negative audio samples; determining a local audio-visual feature contrast loss based on the local positive audio samples and the local negative audio samples; obtaining self-supervised appearance features from the training video through a self-supervised training network; determining a cross-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features; determining a classification loss based on the input appearance features, the input audio features, and the optical flow features; and optimizing the feature extraction network based on the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the cross-distillation loss, and the classification loss.

[0008] In some embodiments, the determining the global audio positive sample according to the global appearance feature and the initial audio feature comprises: obtaining a first start time of a target global appearance feature, wherein the global appearance features extracted by the feature extraction network are multiple, and the target global appearance feature is determined from the multiple global appearance features; determining a first global audio segment of a first preset time length from the initial audio feature according to the first start time; and determining the global audio positive sample corresponding to the target global appearance feature according to the first global audio segment.

[0009] In some embodiments, the determining the global audio negative sample according to the global appearance feature and the initial audio feature comprises: obtaining a second start time of each of the global appearance features except the target global appearance feature; determining a second global audio segment of a first preset time length from the initial audio feature according to each of the second start times; and determining each of the global audio negative samples according to each of the second global audio segments.

[0010] In some embodiments, the determining the local audio positive sample according to the local appearance feature and the initial audio feature comprises: obtaining a third start time of a target local appearance feature, wherein the local appearance features extracted by the feature extraction network are multiple, and the target local appearance feature is determined from the multiple local appearance features; determining a first local audio segment satisfying a first preset rule from the initial audio feature according to the third start time, wherein the first local audio segment comprises multiple first audio slices; obtaining a first dot product similarity between the target local appearance feature and the first local audio segment; determining a first attention weight of each of the first audio slices according to the first dot product similarity; and determining the local audio positive sample according to the first attention weight and the first local audio segment.

[0011] In some embodiments, the determining the local audio negative sample according to the local appearance feature and the initial audio feature comprises: obtaining a fourth start time of each of the local appearance features except the target local appearance feature; determining a second local audio segment satisfying a first preset rule from the initial audio feature according to each of the fourth start times, to obtain multiple second local audio segments; wherein each of the second local audio segments comprises multiple second audio slices; and performing the following operations on each of the second local audio segments: obtaining a second dot product similarity between the target local appearance feature and the second local audio segment; determining a second attention weight of the second local audio segment according to the second dot product similarity; and determining the local audio negative sample according to the second attention weight and the second local audio segment.

[0012] In some embodiments, the first preset rule is expressed as follows: tl =T a σ(w T TAP(r li )+b); where T a The initial audio features are the duration of the data. w and b are learnable parameters, where w represents the weights and b represents the offsets. T The weights are represented by their transpose, σ represents the activation function, TAP represents average pooling along the time dimension, and r li t represents the local appearance feature of the target, i.e., the i-th local appearance feature. l This represents the dynamic time window, which is the duration of each local audio negative sample or each local audio positive sample.

[0013] In some embodiments, the local audio-visual feature contrast loss is determined by the following formula: Among them, L lc a represents the local audio / video feature contrast loss. lp It is a local audio positive sample, a ln These are local audio negative samples, B is the training batch size, τ2 is the temperature coefficient, and m l It is the ratio of the dynamic time window to the duration of the initial audio features, r l T This represents the transpose of a local appearance feature.

[0014] In some embodiments, the global audio-visual feature contrast loss is determined by the following formula:

[0015]

[0016] Among them, L gl a represents the global audio-visual feature contrast loss. gp It is a global positive audio sample with apparent features, a gn It is a global audio negative sample of appearance features, B is the training batch size, τ2 is the temperature coefficient, and r g T This represents the transpose of the global appearance feature.

[0017] In some embodiments, determining the classification loss based on the input appearance features, the input audio features, and the optical flow features includes: determining a fusion feature based on the input appearance features, the input audio features, and the optical flow features; determining a predicted score for the training video based on the fusion feature; obtaining the true label for the training video; and determining the classification loss based on the true label and the predicted score.

[0018] In some embodiments, the determining the mutual distillation loss according to the self-supervised appearance feature, the input appearance feature, the input audio feature and the optical flow feature comprises: determining a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature; determining a first mutual distillation score according to the self-supervised appearance feature; determining a second mutual distillation score according to the fusion feature; and determining the mutual distillation loss according to the first mutual distillation score and the second mutual distillation score.

[0019] In some embodiments, the determining the fusion feature according to the input appearance feature, the input audio feature and the optical flow feature comprises: performing spatial mapping processing on the input appearance feature, the input audio feature and the optical flow feature to obtain a mapped input appearance feature, a mapped input audio feature and a mapped optical flow feature; determining a visual feature according to the mapped input appearance feature and the mapped optical flow feature; and determining the fusion feature according to the visual feature and the mapped input audio feature.

[0020] In some embodiments, the feature extraction network comprises: an audio feature extraction network, an optical flow feature extraction network and an appearance feature extraction network; and the extracting the video appearance feature, the audio feature and the optical flow feature from the training video by using the feature extraction network comprises: uniformly extracting m frames of pictures from the training video according to a second preset rule, and evenly dividing the m frames of pictures to synthesize n global video segments of a second preset time length; uniformly extracting m' frames of pictures from the training video according to a third preset rule, and evenly dividing the m' frames of pictures to synthesize n local video segments of a second preset time length; inputting the n global video segments into the appearance feature extraction network to obtain n global appearance features; determining an input appearance feature according to the n global appearance features; inputting the n local video segments into the appearance feature extraction network to obtain n local appearance features; extracting the initial audio feature from the training video by using the audio feature extraction network; determining the input audio feature according to the initial audio feature; and extracting the optical flow feature from the training video by using the optical flow feature extraction network.

[0021] In a second aspect, the present application provides a recognition method, comprising: extracting an input appearance feature, an input audio feature and an optical flow feature of a target video by using a feature extraction network; determining a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature; determining a prediction score of the target video according to the fusion feature; and recognizing the target video according to the prediction score.

[0022] In some embodiments, the feature extraction network comprises an appearance feature extraction network, an audio feature extraction network, and an optical flow feature extraction network; the input appearance feature, the input audio feature, and the optical flow feature of the target video are extracted by the feature extraction network, comprising: uniformly extracting m frames of pictures from the target video according to a second preset rule, and dividing the m frames into n global video segments of a first preset time length; inputting the n global video segments into the appearance feature extraction network to obtain n global appearance features; determining the input appearance feature according to the n global appearance features; extracting an initial audio feature from the target video by the audio feature extraction network; determining the input audio feature according to the initial audio feature; and extracting the optical flow feature from the target video by the optical flow feature extraction network.

[0023] In some embodiments, the determining the fusion feature according to the input appearance feature, the input audio feature, and the optical flow feature comprises: performing spatial mapping processing on the input appearance feature, the input audio feature, and the optical flow feature to obtain a mapped input appearance feature, a mapped input audio feature, and a mapped optical flow feature; determining a visual feature according to the mapped input appearance feature and the mapped optical flow feature; and determining the fusion feature according to the visual feature and the input audio feature.

[0024] In a third aspect, the present application provides an optimization device, characterized in that it comprises: a first execution module for extracting an initial audio feature, an input audio feature, a local appearance feature, a global appearance feature, an input appearance feature, and an optical flow feature from a target video by using a feature extraction network; a first determination module for determining a global audio positive sample and a global audio negative sample according to the global appearance feature and the initial audio feature; a second determination module for determining a local audio positive sample and a local audio negative sample according to the local appearance feature and the initial audio feature; a third determination module for determining a global audio-visual feature contrast loss according to the global audio positive sample and the global audio negative sample; a fourth determination module for determining a local audio-visual feature contrast loss according to the local audio positive sample and the local audio negative sample; a first acquisition module for acquiring a self-supervised appearance feature from the training video by using a self-supervised training network; a fifth determination module for determining a mutual distillation loss according to the self-supervised appearance feature, the input appearance feature, the input audio feature, and the optical flow feature; a sixth determination module for determining a classification loss according to the input appearance feature, the input audio feature, and the optical flow feature; and a second execution module for optimizing the feature extraction network according to the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss, and the classification loss.

[0025] In a fourth aspect, the present application provides an optimization device, comprising: a second acquisition module configured to extract input appearance features, input audio features and optical flow features of a target video by a feature extraction network; a seventh determination module configured to determine fusion features according to the input appearance features, the input audio features and the optical flow features; an eighth determination module configured to determine a prediction score of the target video according to the fusion features; and a ninth determination module configured to identify the target video according to the prediction score.

[0026] In a fifth aspect, the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to perform the method of the first aspect and / or the method of the second aspect.

[0027] In a sixth aspect, the present application provides a storage medium, which stores a computer program capable of being executed by one or more processors, and the computer program is capable of being used to implement the method of the first aspect and / or the method of the second aspect.

[0028] The present application has the following beneficial effects: the present application extracts initial audio features, input audio features, local appearance features, global appearance features, input appearance features and optical flow features from training videos by using a feature extraction network; determines global audio positive samples and global audio negative samples according to the global appearance features and the initial audio features; determines local audio positive samples and local audio negative samples according to the local appearance features and the initial audio features; determines a global audio-visual feature contrast loss according to the global audio positive samples and the global audio negative samples; determines a local audio-visual feature contrast loss according to the local audio positive samples and the local audio negative samples; obtains self-supervised appearance features from the training videos by a self-supervised training network; determines a mutual distillation loss according to the self-supervised appearance features, the input appearance features, the input audio features and the optical flow features; determines a classification loss according to the input appearance features, the input audio features and the optical flow features; and optimizes the feature extraction network according to the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss and the classification loss. The present application fully realizes the utilization of multi-modal information, so that the input appearance features, the optical flow features and the input audio features obtained by the optimized feature extraction network are more accurate, thereby improving the accuracy of video recognition. BRIEF DESCRIPTION OF DRAWINGS

[0029] The scope of the present disclosure can be better understood by reading the following detailed description of exemplary embodiments, when read in conjunction with the attached drawings, in which:

[0030] Figure 1A whole flowchart of an optimization method provided by an embodiment of the present application is shown in FIG. 1.

[0031] Figure 2 A whole flowchart of a local audio positive sample determination method provided by an embodiment of the present application is shown in FIG. 2.

[0032] Figure 3 A whole flowchart of a mutual distillation loss determination method provided by an embodiment of the present application is shown in FIG. 3.

[0033] Figure 4 A whole flowchart of a classification loss determination method provided by an embodiment of the present application is shown in FIG. 4.

[0034] Figure 5 A whole flowchart of a fusion feature determination method provided by an embodiment of the present application is shown in FIG. 5.

[0035] Figure 6 A whole logic diagram of an optimization method provided by an embodiment of the present application is shown in FIG. 6.

[0036] Figure 7 A whole flowchart of an identification provided by an embodiment of the present application is shown in FIG. 7.

[0037] Figure 8 A structure block diagram of an optimization device provided by an embodiment of the present application is shown in FIG. 8.

[0038] Figure 9 A structure block diagram of an identification device provided by an embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0039] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0041] If similar descriptions of "first\second\third" appear in the application file, the following description is added, in the following description, the terms "first\second\third" referred to only distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0043] Embodiment 1:

[0044] The existing special video recognition technology does not fully utilize multi-modal information, mainly reflected in that no interaction between multi-modal information is performed in the feature extraction stage, and the correlation between multi-modal information is not fully utilized, so that the accuracy of the extracted multi-modal information is low. For example, the existing audio-visual contrast learning algorithm selects positive and negative samples within a fixed time range, which introduces noise and cannot fully utilize information in different time dimensions.

[0045] In view of the problems in the prior art, such as Figure 1 As shown in the method for optimizing the feature extraction network, the method is applied to an electronic device, and the electronic device can be a server, a mobile terminal, a computer, a cloud platform, etc. The function realized by the device data processing provided in the embodiments of the application can be realized by calling program code by the processor of the electronic device, wherein the program code can be saved in a computer storage medium. The optimization method comprises the following steps:

[0046] Step S11: extracting initial audio features, input audio features, local appearance features, global appearance features, input appearance features and optical flow features from the training video by using a feature extraction network.

[0047] The feature extraction network comprises an appearance feature extraction network, an audio extraction network and a flow extraction network. The appearance feature extraction network can extract global appearance features and local appearance features from the training video. The audio extraction network can extract initial audio features from the training video. The flow extraction network can extract optical flow features from the training video.

[0048] Therefore, in some embodiments, the step S11 of "extracting initial audio features, input audio features, local appearance features, global appearance features, input appearance features and optical flow features from the training video by using a feature extraction network" comprises the following steps:

[0049] Step S111: uniformly extracting m frames of pictures from the training video according to a second preset rule, and evenly dividing the m frames of pictures to synthesize n global video segments of a second preset time length.

[0050] Step S112: uniformly extracting m' frames of pictures from the training video according to a third preset rule, and evenly dividing the m' frames of pictures to synthesize n local video segments of a second preset time length.

[0051] Step S113: inputting the n global video clips into the appearance feature extraction network to obtain n global appearance features.

[0052] Step S114: determining an input appearance feature according to the n global appearance features.

[0053] Step S115: inputting the n local video clips into the appearance feature extraction network to obtain n local appearance features.

[0054] Step S116: extracting an initial audio feature from the training video by using the audio feature extraction network.

[0055] Step S117: determining an input audio feature according to the initial audio feature.

[0056] Step S118: extracting the optical flow feature from the training video by using the optical flow feature extraction network.

[0057] The video to be recognized or the training video is relatively long, so the video is divided into multiple video clips, and each video clip is composed of multiple frames. For extraction of the global appearance feature, m frames of pictures are uniformly extracted from the training video by using a low sampling rate frame extraction manner, then the m frames of pictures are combined into n global video clips with a second preset time length, and then the n global video clips are input into the appearance feature extraction network, and the appearance feature extraction network outputs n global appearance features.

[0058] For extraction of the local appearance feature, m' frames of pictures are uniformly extracted from the training video by using a high sampling rate frame extraction manner, then the m' frames of pictures are combined into n local video clips with a second preset time length, and then the n local video clips are input into the appearance feature extraction network, and the appearance feature extraction network outputs n local appearance features.

[0059] For the input appearance feature, it is obtained by splicing all the global appearance features and performing average pooling along the time dimension.

[0060] In steps S111-S118, steps S111 and S112 are not limited in sequence, that is, steps S111 and S112 can be executed simultaneously, or step S112 can be executed first. Similarly, steps S113, S115, S116 and S118 do not have a fixed execution order and can be executed simultaneously.

[0061] In the present application, different sampling rates are used to extract m frames and m' frames from the training video by the opencv package of python, and then m frames are made into n global video segments of the first preset time length, and m' frames are made into n local video segments of the first preset time length. For example, the low sampling rate is set to 4 frames per second, and the high sampling rate is set to 16 frames per second. And the first preset time length can be set to 2s.

[0062] The apparent feature extraction network in the present application is R(2+1)D-18 network. Among them, the R(2+1)D model takes R3D (3DResNet) as the model basic structure, and R3D expands all 3*3 two-dimensional convolution in residual network ResNet to 3*3*3 three-dimensional convolution, and R(2+1)-D decomposes R3D three-dimensional convolution into spatial and time components. The apparent feature extraction network of the present application has two benefits under the condition of keeping the parameter amount basically consistent: on the one hand, the ReLU nonlinear layer between the two convolution layers is increased, which improves the network representation ability; on the other hand, the decomposition operation makes the network easier to be optimized during training. Therefore, the present application adopts R(2+1)D model to extract apparent features.

[0063] When local feature extraction is performed by the apparent feature extraction network, the expression is as follows:

[0064]

[0065] Among them, represents the local apparent feature extraction network, r l is the local apparent feature, Among them, T r is the length of the segment apparent sequence 32, and 512 is the output channel number of R(2+1)D-18, is the input local video segment.

[0066] When global feature extraction is performed by the apparent feature extraction network, the expression is as follows:

[0067]

[0068] Among them, represents the global apparent feature extraction network, r g is the global apparent feature, r g ∈R 512×1 , wherein 512 is the output channel number of R(2+1)D-18, represents the global video segment.

[0069] The specific calculation formula of TAP function (Temporal Average Pooling) is as follows:

[0070]

[0071] wherein x represents an input, x∈R C×M , x i ∈R C×1 , M represents a time dimension, and C represents a feature dimension.

[0072] For the global appearance feature, after global appearance feature extraction by a feature extraction network, the extracted global appearance feature is subjected to average pooling along the time dimension.

[0073] The input appearance feature F r is obtained by average pooling along the time dimension after splicing all global appearance features. r ∈R 512 ×1 , and the calculation formula is as follows:

[0074]

[0075] In the present application, if n is 5, the extracted global appearance feature can be represented as [r g1 ,…,r g5 ], and the local appearance feature can be represented as [r l1 ,…,r l5 ], wherein r gi represents the global appearance feature corresponding to the i-th video segment, and r li represents the local appearance feature corresponding to the i-th video segment, i = 1, 2, …, 5.

[0076] For the extraction of audio features, the LibROSA tool is used to extract a mel-spectrogram from the original audio waveform of the training video, and the mel-spectrogram is input into a 1D-ResNet. The network outputs an initial audio feature a, T a which is proportional to the length of the input audio. Based on the initial audio feature a, average pooling is performed along the time dimension to obtain the input audio feature F a , F a ∈R 512×1 . Wherein, 1D-ResNet represents replacing all two-dimensional convolutions in the residual network ResNet with one-dimensional convolutions.

[0077] For the extraction of optical flow features, the TV-L1 method is used to extract the dense optical flow of the input video, and the optical flow images in the vertical and horizontal directions are saved. The extracted optical flow images are input into an X3D model to extract features, and the output vector of the fifth layer is taken as the optical flow feature of the input video.

[0078] Wherein, the TV-L1 method is a classical optical flow image extraction method, and the extracted optical flow has good effect, the application adopts the TV-L1 to extract the optical flow image, which is easy to realize and has good compatibility; the X3D model is an X3D model pre-trained on the Kinetics dataset, and the application uses the X3D model with good performance on the large motion recognition dataset Kinetics to extract the optical flow feature, so that the application can more accurately capture the motion information in the input video and generate the optical flow feature F which can represent the motion in the video l , F l ∈R 2048×1 .

[0079] Since the extraction of the initial audio feature, the extraction of the local appearance feature and the extraction of the global appearance feature are separated, it is difficult to confirm the accuracy of the initial audio feature, the local appearance feature and the global appearance feature, and thus the feature extraction network cannot grow well. Therefore, in the present application, a video-audio feature contrast loss is proposed. The video-audio feature contrast loss includes a global video-audio feature contrast loss and a local video-audio feature contrast loss.

[0080] In order to obtain a more accurate video-audio feature contrast loss, in the present application, global audio positive samples and global audio negative samples corresponding to each global appearance feature need to be made.

[0081] Step S12: determining global audio positive samples and global audio negative samples according to the global appearance feature and the initial audio feature.

[0082] In some embodiments, the step S12 of "determining global audio positive samples according to the global appearance feature and the initial audio feature" comprises:

[0083] Step S121: obtaining a first starting time of a target global appearance feature, wherein the global appearance features extracted by the feature extraction network are multiple, and the target global appearance feature is determined from the multiple global appearance features.

[0084] Step S122: according to the first starting time, a first global audio segment with a first preset time length is cut from the initial audio feature.

[0085] Step S123: determining the global audio positive sample corresponding to the target global appearance feature according to the first global audio segment.

[0086] In order to distinguish, the global appearance features of the global audio positive samples and the global audio negative samples need to be made into target appearance features. Each global appearance feature in the present application has a time starting point. In order to improve the accuracy of the global audio visual feature loss, the first starting time of the target global appearance feature needs to be obtained in the present application. Taking the first starting time as the starting point, relaxing the time synchronization, cutting the audio feature slice of the first preset time length from the initial audio feature, and then splicing and average pooling along the time dimension to obtain the global audio positive sample. The first preset time length is greater than the second preset time length. The global audio positive sample can well cover the content of the global appearance feature.

[0087] In some embodiments, the step S12 "determining a global audio negative sample according to the global appearance feature and the initial audio feature" comprises:

[0088] Step S124: obtaining the second starting time of each global appearance feature except the target global appearance feature.

[0089] Step S125: cutting a second global audio segment of a first preset time length from the initial audio feature according to each second starting time.

[0090] Step S126: determining each global audio negative sample according to each second global audio segment.

[0091] Each global appearance feature corresponds to a global audio positive sample and multiple global audio negative samples. When determining the global audio negative sample, the second starting time of each global appearance feature except the target global appearance feature is obtained. Taking each second starting time as the starting time, an audio slice of a first preset time length is cut from the initial audio feature, and the audio slice obtained according to each second starting time is spliced and average-pooled along the time dimension to obtain multiple global audio negative samples of the target global appearance feature.

[0092] In the present application, the expression of splicing and pooling into a global audio positive sample is as follows:

[0093] a gp = TAP(concat(a i )), i ∈ [t, t + t g ];

[0094] Wherein, the concat function represents splicing along the time dimension, and the TAP function represents average pooling along the time dimension. t a a represents the starting point of the audio feature slice of the global audio positive sample, t ∈ [1, T g ], t aa time length representing the initial audio feature, t g a ratio of the first preset time length to the time length of the training video a a ratio of the first preset time length to the time length of the training video. gp a global audio positive sample obtained, a gp ∈R 512×1 .

[0095] Steps S121-S126 specifically introduce how to obtain the global audio positive sample and the global audio negative sample. The video and audio feature loss of the present application also needs to determine the local video and audio feature loss, so the local audio positive sample and the local video positive sample need to be determined.

[0096] Step S13: determining a local audio positive sample and a local audio negative sample according to the local appearance feature and the initial audio feature.

[0097] Therefore, in some embodiments, as shown in step S13, “determining a local audio positive sample according to the local appearance feature and the initial audio feature”, includes: Figure 2 Step S130: obtaining a third starting time of a target local appearance feature, wherein the local appearance features extracted by the feature extraction network are multiple, and the target local appearance feature is determined from the multiple local appearance features.

[0098] Step S131: according to the third starting time, a first local audio segment satisfying a first preset rule is cut from the initial audio feature, wherein the first local audio segment includes multiple first audio slices.

[0099] Step S132: obtaining a first dot product similarity of the target local appearance feature and the first local audio segment.

[0100] Step S134: determining a first attention weight of each of the first audio slices according to the first dot product similarity.

[0101] Step S134: determining the local audio positive sample according to the first attention weight and the first local audio segment.

[0102] In some embodiments, step S13 “determining a local audio negative sample according to the local appearance feature and the initial audio feature”, includes:

[0103] Step S135: obtaining a fourth starting time of each of the local appearance features except the target local appearance feature.

[0104]

[0105] ​Step S136: According to each fourth starting time, a second local audio segment satisfying a first preset rule is intercepted from the initial audio feature, and a plurality of second local audio segments are obtained, wherein each second local audio segment includes a plurality of second audio slices;

[0106] For each second local audio segment, the following operations are performed:

[0107] Step S137: A second dot product similarity of the target local appearance feature and each second local audio segment is obtained.

[0108] Step S138: A second attention weight of each second local audio segment is determined according to the second dot product similarity.

[0109] Step S139: Each local audio negative sample is determined according to each second attention weight and each second local audio segment.

[0110] In determining the local audio positive sample, a target local appearance feature also needs to be selected. Then a third starting time of the target local appearance feature is obtained. A first local audio segment satisfying a first preset rule is intercepted from the initial audio feature with the third starting time as the starting time, and the first local audio segment includes a plurality of first audio slices. The first audio slices are spliced in time sequence, and then a first dot product similarity of the first local audio segment and the target local appearance feature is calculated. The top K first audio slices with high first dot product similarity are selected. The similarity of the first audio slices in the top K range is normalized, and then a first attention weight of each first audio slice in the top K range is calculated. Each first audio slice in the top K range is weighted according to the first attention weight, and a local audio positive sample is obtained.

[0111] In determining the local audio negative sample, a fourth starting time of each local appearance feature except the target local appearance feature is obtained. A second local audio segment satisfying a first preset rule is intercepted from the initial audio feature with each fourth starting time as the starting time, and a plurality of second local audio segments are obtained, and the second local audio segment includes a plurality of second audio slices. For each second local audio segment, the following operations are performed:

[0112] The second audio slices are spliced in time sequence, and a second dot product similarity of the second local audio segment and the target local appearance feature is calculated. The top K second audio slices with high second dot product similarity are selected in the second local audio segment. The similarity of the second audio slices in the top K range is normalized, and then a second attention weight of each second audio slice in the top K range is calculated. Each second audio slice in the top K range is weighted according to the second attention weight, and a local audio negative sample is obtained.

[0113] In some embodiments, the expression of the first preset rule is as follows:

[0114] t l = T a σ(w T TAP(r li )+b);

[0115] wherein T a is a time length of the initial audio feature, w and b are learnable parameters, w represents a weight, b represents an offset, w T represents a transpose of the weight, σ represents an activation function, TAP represents average pooling along a time dimension, r li represents a target local appearance feature, i.e., an i-th local appearance feature, t l represents a dynamic time window, i.e., a time length of each local audio negative sample or each local audio positive sample.

[0116] In some embodiments, the concatenation of the first audio slice is completed by the following formula:

[0117] a l ′ p = concat(a i ), i∈[t′,t′+t l ] (1)

[0118] wherein the concat function represents concatenation along a time dimension, a l ′ p is the first local audio segment after concatenation, a i represents the first audio slice in the range of [t′,t′+t l ], t′ represents a third start time, and t l represents a time length of the local audio positive sample.

[0119] In some embodiments, the first dot product similarity is determined by the following formula:

[0120]

[0121] wherein the local appearance feature r l ∈R 512×32 , TAP(r l )∈R 512×1 , represents a transpose of the first local audio segment after concatenation, is a similarity matrix of the local appearance feature and the first audio slice in the time length t l . The element d i in d corresponds to the local appearance feature and the first audio slice a isimilarity between local audio positive samples and the first audio slice a l ]t l represents the duration of the local audio positive sample. The present application selects the first point product similarity d i top K corresponding to the first audio slice, K is a hyperparameter 32x3. The first audio slice in the top K range is not necessarily time continuous. The similarity of the first audio slice in the top K range is sent into the Softmax function for normalization to ensure that each first audio slice has comparable similarity.

[0122] In some embodiments, the first attention weight s i of each first audio slice in the top K range is obtained by the following formula:

[0123]

[0124] where τ1 is the temperature coefficient. d i corresponding to the similarity between the local appearance feature and the first audio slice a i , i∈topK. Based on the first attention weight s i , the first audio slice in the top K range is weighted, and finally the local audio positive sample a lp is obtained.

[0125] In some embodiments, each first audio slice is weighted by the following formula:

[0126] a lp =concat(s i a i ), i∈topK (4)

[0127] where the concat function represents concatenation along the time dimension, s i represents the first attention weight, and a i represents the i-th first audio slice.

[0128] In this application, the acquisition process of the local audio positive sample is described by expressions 1-4, but the same expressions 1-4 can also be used to acquire the local audio negative sample, only the parameters therein need to be replaced with corresponding parameters. Here, we will not repeat them.

[0129] Step S14: determining a global audiovisual feature contrast loss based on the global audio positive sample and the global audio negative sample.

[0130] After steps S121-S126, we obtain the global audio positive sample and the global audio negative sample. Therefore, we can determine the global audiovisual feature contrast loss based on the global appearance feature, the global audio positive sample, and the global audio negative sample.

[0131] In some embodiments, the global visual-audio feature contrast loss is determined by the following formula:

[0132]

[0133] wherein L gl represents the global visual-audio feature contrast loss, a gp is a global audio positive sample of the apparent feature, a gn is a global audio negative sample of the apparent feature, B is the batch size of training, and τ2 is a temperature coefficient, represents the transpose of the global apparent feature.

[0134] Step S15: determining a local visual-audio feature contrast loss according to the local audio positive sample and the local audio negative sample.

[0135] After steps S130-S139, the local audio positive sample and the local audio negative sample are obtained, so that the local visual-audio feature contrast loss can be obtained by the local audio positive sample and the local audio negative sample.

[0136] In some embodiments, the local audio feature contrast loss is determined by the following formula:

[0137]

[0138] wherein L lc represents the local visual-audio feature contrast loss, a lp is a local audio positive sample, a ln is a local audio negative sample, B is the batch size of training, and τ2 is a temperature coefficient, m l is the ratio of the dynamic time window to the duration of the initial audio feature, and r l T represents the transpose of the local apparent feature.

[0139] Step S16: obtaining a self-supervised apparent feature from the training video by a self-supervised training network.

[0140] However, the existing special video dataset is small in scale, so that the network cannot be fully trained. In order to solve the problem of performance optimization limited by the small-scale dataset, the present application introduces a self-supervised pre-training network by using mutual distillation to migrate the knowledge contained in the large-scale dataset.

[0141] Therefore, in the present application, m” frames of pictures are uniformly extracted from the input video by using the opencv package of python, and the m” frames of pictures are input into the self-supervised pre-training network to obtain the self-supervised apparent feature.

[0142] Step S17: Determine the inter-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features.

[0143] In some embodiments, such as Figure 3 As shown, step S17, "determining the inter-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features," includes:

[0144] Step S171: Determine the fusion features based on the input appearance features, the input audio features, and the optical flow features.

[0145] In some embodiments, such as Figure 5 As shown, step S171, "determining fusion features based on the input appearance features, the input audio features, and the optical flow features," includes:

[0146] Step S1711: Perform spatial mapping processing on the input appearance features, the input audio features, and the optical flow features to obtain mapped input appearance features, mapped input audio features, and mapped optical flow features.

[0147] Step S1712: Determine visual features based on the mapped input appearance features and the mapped streamer features.

[0148] Step S1713: Determine the fusion feature based on the visual features and the mapped input audio features.

[0149] Input appearance features F r The input audio features and optical flow features are processed through fully connected layers and dropout layers to unify the dimensions of the three features into 256 dimensions, resulting in mapped input appearance features, mapped input audio features, and mapped optical flow features. This invention uses fully connected layers to achieve spatial mapping, mapping each modal feature to the same feature space to eliminate the influence of multimodal heterogeneity, and utilizes dropout layers to deactivate some units to prevent overfitting.

[0150] In some embodiments, the fusion of input appearance features and optical flow features into visual features is achieved by the following formula:

[0151]

[0152] Where R represents the apparent mode, L represents the optical flow mode, and F... r F represents the input appearance feature. l Representing optical flow features, FC() represents a fully connected layer that reduces the feature dimension to 256 dimensions. r ) represents the mapping of input appearance features, FC(F) l ) represents the mapped optical flow characteristics. represents intra-modal enhancement on the mapped input appearance feature, represents cross-modal enhancement on the mapped input appearance feature and the mapped optical flow feature, represents intra-modal enhancement on the mapped optical flow feature, F v represents a visual feature.

[0153] In some embodiments, the visual feature and the input audio feature are determined to fuse a feature by the following formula:

[0154]

[0155] wherein V represents a visual modality, and A represents an audio modality. The fused feature F is calculated from the visual feature F v and the input audio feature F a ; FC(F a ) represents the mapped input audio feature, represents intra-modal enhancement on the mapped input audio feature, represents cross-modal enhancement on the mapped input audio feature and the visual feature.

[0156] In some embodiments, the expression of the intra-modal enhancement is as follows:

[0157]

[0158] wherein F d represents the input spatially mapped single-modal feature, d represents the modality name, conv1() represents a convolution operation with a convolution kernel size of 3, conv2() represents a convolution operation with a convolution kernel size of 5, and σ() represents a Sigmoid activation function. The output of the intra-modal feature enhancement module is calculated by the present application

[0159] The present application uses two different sizes of convolution kernels, and the receptive field sizes of the two convolutions are different, which can capture more comprehensive channel attention; the present application proposes multi-scale channel attention capture for intra-modal feature enhancement, to strengthen the internal correlation of single-modal features.

[0160] In some embodiments, the expression of the cross-modal enhancement is as follows:

[0161]

[0162] wherein the cosine function cos() calculates the similarity between the features, and the meaning is how much proportion of the input two modal features is related. After obtaining the similarity, it is multiplied to F d1 and F d2 , F d1 and F d2d1 and d2 represent two modalities, the application calculates the output of the cross-modal feature enhancement module The application captures the connection between different modalities by calculating the similarity between modalities to perform cross-modal feature enhancement.

[0163] Step S172: Determine a first mutual distillation score according to the self-supervised appearance feature.

[0164] Step S173: Determine a second mutual distillation score according to the fusion feature.

[0165] In some embodiments, the self-supervised appearance feature is sent into two fully connected layers for dimension reduction after average pooling along the time dimension, and the obtained fusion feature is sent into two fully connected layers for dimension reduction.

[0166] The application uses two fully connected layers, the first fully connected layer reduces the feature dimension to 256 dimensions, then uses a dropout layer to deactivate some units to prevent overfitting and enhance the generalization ability of the model; the second fully connected layer reduces the feature dimension to 2 dimensions. Therefore, the application obtains a 2-dimensional self-supervised appearance feature and a 2-dimensional fusion feature through the two fully connected layers.

[0167] The distribution sends the 2-dimensional self-supervised appearance feature and the 2-dimensional fusion feature into a Softmax function to obtain a first mutual distillation score of the self-supervised appearance feature and a second mutual distillation score of the fusion feature.

[0168] Specifically, the Softmax function operates as shown in the following formula.

[0169]

[0170] where z c is the input of the Softmax function, and the Softmax function can convert the two-dimensional feature into a prediction score with a range of [0, 1] and a sum of 1. T is a hyperparameter that controls the softness of the output prediction score, and as T increases, the probability distribution produced by the Softmax function becomes softer, thereby providing more information. When calculating the mutual distillation prediction score, the application uses a setting of T=5.

[0171] Step S174: Determine a mutual distillation loss according to the first mutual distillation score and the second mutual distillation score.

[0172] The application calculates the KL divergence between the first or distillation score and the second mutual distillation score as the mutual distillation loss L MD .

[0173] Step S18: determining a classification loss according to the input appearance feature, the input audio feature and the optical flow feature.

[0174] In order to further improve the accuracy of the features extracted by the feature extraction network, the classification loss needs to be used when the feature extraction network is optimized in the present application.

[0175] In some embodiments, as shown in FIG. 18, step S18 “determining a classification loss according to the input appearance feature, the input audio feature and the optical flow feature” comprises: Figure 4

[0176] Step S181: determining a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature.

[0177] The method for determining the fusion feature has been described in detail in steps S1711-S1713, and thus will not be repeated here.

[0178] Step S182: determining a prediction score of the training video according to the fusion feature.

[0179] Step S183: obtaining a real label of the training video.

[0180] Step S184: determining the classification loss according to the real label and the prediction score.

[0181] In the present embodiment, the fusion feature obtained in steps S1711-S1713 is input into a Softmax function to obtain a prediction score corresponding to the fusion feature. The cross entropy is calculated according to the real label of the training video, and the cross entropy is taken as the classification loss.

[0182] Step S19: optimizing the feature extraction network according to the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss and the classification loss.

[0183] After obtaining the local audio-visual feature contrast loss, the global audio-visual feature contrast loss, the mutual distillation loss and the classification loss, the feature extraction network can be preliminarily optimized. The sum of the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss and the classification loss can be taken as a total loss to optimize the feature extraction network.

[0184] After obtaining the audio-visual feature contrast loss, the mutual distillation loss and the classification loss, the feature extraction network can be better optimized.

[0185] ​In some embodiments, the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss and the classification loss are added together as a total loss L to optimize the feature extraction network.

[0186] The expression of the total loss L is as follows:

[0187] L m = β (L gl + L lc ) + γL MD (p sp || p m ) + L ms

[0188] Wherein the global audio-visual feature contrast loss L gl , the local audio-visual feature contrast loss L lc , the mutual distillation loss L MD and the classification loss L ms , p sp is the first mutual distillation score, p m represents the second mutual distillation score, and β and γ are proportional hyperparameters, which are set to 0.5 in the present application.

[0189] The optimization method based on mutual distillation and contrast learning is proposed in the present application, which can well optimize the feature extraction network and further improve the recognition accuracy of special video. Based on the correlation between the apparent information and the audio information, the present application introduces a dynamic window and an attention weighting mechanism to perform global-local audio-visual feature contrast learning; the optical flow information is introduced, based on the complementarity between multi-modal information, and a feature fusion network based on attention mechanism with good ability to capture the correlation between different modalities is proposed.

[0190] This application utilizes a feature extraction network to extract initial audio features, input audio features, local appearance features, global appearance features, input appearance features, and optical flow features from training videos. It then determines global positive and negative audio samples based on the global appearance features and the initial audio features; determines local positive and negative audio samples based on the local appearance features and the initial audio features; determines a global audio-visual feature contrast loss based on the global positive and negative audio samples; determines a local audio-visual feature contrast loss based on the local positive and negative audio samples; obtains self-supervised appearance features from the training videos through a self-supervised training network; determines a cross-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features; determines a classification loss based on the input appearance features, the input audio features, and the optical flow features; and optimizes the feature extraction network based on the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the cross-distillation loss, and the classification loss. It fully utilizes multimodal information, making the input appearance features, optical flow features and input audio features obtained by the optimized feature extraction network more accurate, thereby improving the accuracy of video recognition.

[0191] To better understand the ideas in this application, such as Figure 6 As shown, this application also discloses a logic diagram of the method. After obtaining the training video, initial audio features, global appearance features, local appearance features, and optical flow features are first extracted through a feature extraction network. Input audio features are determined based on the initial audio features, and input appearance features are obtained based on the global appearance features. Simultaneously, self-supervised appearance features are extracted from the training video through a self-supervised network.

[0192] Then, based on the global appearance features and the initial audio features, positive and negative samples of global audio features are obtained through relaxation time synchronization. The global audio-visual feature contrast loss is then derived from these positive and negative samples.

[0193] Based on local appearance features and initial audio features, local positive and negative audio samples are obtained through dynamic time windows and attention weighting. A local audio-visual feature contrast loss is then derived from these local positive and negative audio samples.

[0194] After spatial mapping of the input appearance features and optical flow features, visual features are obtained through intra-modal and cross-modal enhancement. Similarly, after spatial mapping of the input audio features, these features are combined with the visual features through intra-modal and cross-modal enhancement to obtain fused features.

[0195] The obtained fusion features are input into a softmax function to obtain a prediction score and a second mutual distillation score. The classification loss can be obtained by combining the prediction score with the real label of the training video.

[0196] The self-supervised appearance features are input into a softmax function to obtain a first mutual distillation score. The mutual distillation loss is the KL divergence between the first mutual distillation score and the second mutual distillation score.

[0197] Finally, the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the classification loss, and the mutual distillation loss are added to obtain a total loss, and the total loss is used to update the feature extraction network.

[0198] Since the prior art does not interact between multi-modal information in the feature extraction stage, the correlation between multi-modal information is not fully utilized, resulting in low accuracy of the extracted multi-modal information. For example, the existing audio-visual contrast learning algorithm selects positive and negative samples within a fixed time range, which introduces noise and cannot fully utilize the information in different time dimensions. Therefore, to solve the problem of noise introduced by the audio-visual contrast learning algorithm in selecting audio positive samples within a fixed length time range, the present application creatively proposes to select local audio positive samples based on a dynamic window and attention weighting, and construct a global-local audio-visual feature contrast loss, which can achieve fine-grained semantic alignment while reducing noise information caused by far-away audio segments, and improve the performance of violent video recognition.

[0199] Embodiment 2

[0200] As shown in Figure 7 The present application provides a recognition method, which is applied to an electronic device, such as a server, a mobile terminal, a computer, a cloud platform, etc. The function realized by the device data processing provided in the embodiments of the present application can be realized by calling program code by the processor of the electronic device, wherein the program code can be saved in a computer storage medium. The recognition method comprises the following steps:

[0201] Step S21: extracting input appearance features, input audio features, and optical flow features of the target video through a feature extraction network.

[0202] In some embodiments, the step S21 of "extracting input appearance features, input audio features, and optical flow features of the target video through a feature extraction network" comprises:

[0203] The feature extraction network comprises an appearance feature extraction network, an audio extraction network, and a flow extraction network. The appearance feature extraction network can extract global appearance features and local appearance features from the target video. The audio extraction network can extract initial audio features from the target video. The flow extraction network can extract optical flow features from the target video.

[0204] Step S211: uniformly extracting m frames of pictures from the target video according to a second preset rule, and synthesizing n global video clips of a first preset time length from the m frames.

[0205] Step S212: inputting the n global video clips into the apparent feature extraction network to obtain n global apparent features.

[0206] Step S213: determining an input apparent feature according to the n global apparent features.

[0207] Step S214: extracting an initial audio feature from the target video through the audio feature extraction network.

[0208] Step S215: determining an input audio feature according to the initial audio feature.

[0209] Step S216: extracting the optical flow feature from the target video through the optical flow feature extraction network.

[0210] The video to be recognized or the target video is relatively long, so the present application divides one video into a plurality of video clips, and each video clip is synthesized by a plurality of frames. For extraction of the global apparent feature, a low sampling rate frame extraction method is adopted to uniformly extract m frames of pictures from the target video, and then the m frames of pictures are synthesized into n global video clips of a second preset time length, and then the n global video clips are input into the apparent feature extraction network, and the apparent feature extraction network outputs n global apparent features.

[0211] In the present application, the opencv package of python is adopted to uniformly extract m frames of pictures from the target video at a low sampling rate, and then the m frames of pictures are made into n global video clips of a first preset time length. For example, the low sampling rate is set to 4 frames per second. The first preset time length can be set to 2s.

[0212] The apparent feature extraction network in the present application is an R(2+1)D-18 network. The R(2+1)D model takes R3D (3DResNet) as the model basic structure, R3D expands all 3x3 two-dimensional convolutions in the residual network ResNet into 3x3x3 three-dimensional convolutions, and R(2+1)-D decomposes the R3D three-dimensional convolution into spatial and temporal components. The apparent feature extraction network of the present application has two benefits under the condition of keeping the parameter amount basically consistent: on the one hand, the ReLU nonlinear layer between the two convolution layers is increased, which improves the network representation ability; on the other hand, the decomposition operation makes the network easier to be optimized during training. Therefore, the present application adopts the R(2+1)D model to extract the apparent feature.

[0213] In the global feature extraction by the appearance feature extraction network, the expression is as follows:

[0214]

[0215] wherein represents the global appearance feature extraction network, r g is the global appearance feature, r g ∈R 512×1 , wherein 512 is the output channel number of R(2+1)D-18, represents the global video segment.

[0216] The TAP function (Temporal Average Pooling) has the specific calculation formula as follows:

[0217]

[0218] wherein x represents the input, x∈R C×M , x i ∈R C×1 , M represents the time dimension, and C represents the feature dimension.

[0219] For the global appearance feature, after the global appearance feature extraction by the feature extraction network, the extracted global appearance feature is subjected to average pooling along the time dimension.

[0220] The input appearance feature F r is obtained by splicing all the global appearance features and then subjected to average pooling along the time dimension, F r ∈R 512 ×1 , and the calculation formula is as follows:

[0221]

[0222] In the present application, if n is 5, the extracted global appearance feature can be represented as [r g1 ,…,r g5 ], the local appearance feature can be represented as [r l1 ,…,r l5 ], r gi represents the global appearance feature corresponding to the i-th video segment, r li represents the local appearance feature corresponding to the i-th video segment, and i = 1, 2, …, 5.

[0223] For the extraction of the audio feature, the LibROSA tool is used to extract the mel spectrum from the original audio waveform of the target video, and the mel spectrum is sent into the 1D-ResNet, and the network outputs the initial audio feature a, T aThe input audio feature F is obtained by performing average pooling along the time dimension based on the initial audio feature a in proportion to the length of the input audio a , F a ∈R 512×1 , wherein the 1D-ResNet represents replacing all two-dimensional convolutions in the residual network ResNet with one-dimensional convolutions.

[0224] For the extraction of the optical flow feature, a dense optical flow of the input video is extracted using a TV-L1 method, and the optical flow images in the vertical direction and the horizontal direction are saved; the extracted optical flow images are input into an X3D model to extract features, and the output vector of the fifth layer is taken as the optical flow feature of the input video.

[0225] The TV-L1 method is a classical optical flow image extraction method, and the extracted optical flow has good effect, and the application adopts the TV-L1 to extract the optical flow image, which is easy to implement and has good compatibility; the X3D model is an X3D model pre-trained on the Kinetics dataset, and the application uses the X3D model with good performance on the large motion recognition dataset Kinetics to extract the optical flow feature, and by inputting the dense optical flow, the application can more accurately capture the motion information in the input video, and generate the optical flow feature F l , F l ∈R 2048×1 .

[0226] Step S22: determining a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature.

[0227] In some embodiments, the step S22 of determining a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature comprises:

[0228] Step S221: performing spatial mapping processing on the input appearance feature, the input audio feature and the optical flow feature to obtain a mapped input appearance feature, a mapped input audio feature and a mapped optical flow feature.

[0229] Step S222: determining a visual feature according to the mapped input appearance feature and the mapped optical flow feature.

[0230] Step S223: determining the fusion feature according to the visual feature and the input audio feature.

[0231] The input appearance feature F rThe input appearance feature and the optical flow feature pass through a full connection layer and a dropout layer, and dimensions of the three features are uniformly converted to 256 dimensions to obtain a mapping input appearance feature, a mapping input audio feature and a mapping optical flow feature. The application adopts the full connection layer to realize spatial mapping, maps each modal feature to the same feature space to eliminate the influence of multi-modal heterogeneity, and inactivates some units by using the dropout layer to prevent overfitting.

[0232] In some embodiments, the input appearance feature and the optical flow feature are fused into a visual feature by the following formula:

[0233]

[0234] Wherein R represents an appearance modal, L represents an optical flow modal, F r represents an input appearance feature, F l represents an optical flow feature, FC() represents a full connection layer reducing the feature dimension to 256 dimensions, FC(F r ) represents a mapping input appearance feature, FC(F l ) represents a mapping optical flow feature, represents intra-modal enhancement on the mapping input appearance feature, represents cross-modal enhancement on the mapping input appearance feature and the mapping optical flow feature, represents intra-modal enhancement on the mapping optical flow feature, F v represents a visual feature.

[0235] In some embodiments, the visual feature and the input audio feature are determined to fuse a feature by the following formula:

[0236]

[0237] Wherein V represents a visual modal, and A represents an audio modal. The fusion feature F is calculated by the visual feature F v and the input audio feature F a ; FC(F a ) represents a mapping input audio feature, represents intra-modal enhancement on the mapping input audio feature, represents cross-modal enhancement on the mapping input audio feature and the visual feature.

[0238] In some embodiments, the expression of the intra-modal enhancement is as follows:

[0239]

[0240] Wherein, F drepresents the input spatial mapped single-modal feature, d represents the modal name, conv1() represents a convolution operation with a kernel size of 3, and conv2() represents a convolution operation with a kernel size of 5. σ() represents a Sigmoid activation function, and the output of the intra-modal feature enhancement module is calculated by the present application

[0241] The present application uses two different sizes of convolution kernels, and the receptive field sizes of the two convolutions are different, which can capture more comprehensive channel attention; the present application proposes multi-scale channel attention capture for intra-modal feature enhancement to strengthen the internal correlation of single-modal features.

[0242] In some embodiments, the expression of cross-modal enhancement is as follows:

[0243]

[0244] Wherein, the cosine function cos() calculates the similarity between features, and the meaning is how much proportion of the input two modal features is related. After obtaining the similarity, it is multiplied to F d1 and F d2 , F d1 and F d2 represent two modal features input into the cross-modal feature enhancement module, d1 and d2 represent two modalities, and the output of the cross-modal feature enhancement module is calculated by the present application The present application captures the relationship between different modalities by calculating the similarity between modalities for cross-modal feature enhancement.

[0245] Step S23: determining the prediction score of the target video according to the fusion feature.

[0246] In some embodiments, the fusion feature is input into a Softmax function to obtain the prediction score corresponding to the fusion feature.

[0247] Specifically, the Softmax function operation is shown in the following formula.

[0248]

[0249] Wherein, z c is the input of the Softmax function, and the Softmax function can convert two-dimensional features into prediction scores with a range of [0, 1] and a sum of 1. T is a hyperparameter that controls the softness of the output prediction score, and as T increases, the probability distribution generated by the Softmax function becomes softer, thereby providing more information. When calculating the mutual distillation prediction score, the present application adopts a setting of T=5.

[0250] Step S24: identifying the target video according to the prediction score.

[0251] The output prediction score is expressed as: score + special class. When the score exceeds a preset threshold, it can be determined that it is a special class video. When the score is less than or less than the preset threshold, it can be determined that it is a non-special class video.

[0252] The feature extraction network in this embodiment is based on the optimized feature extraction network of embodiment 1, and cross-modal contrast is used to improve the performance of special class video recognition.

[0253] Embodiment 3

[0254] Based on the foregoing embodiment 1, the application embodiment provides an optimization device, each module included in the device and each unit included in each module can be implemented by a processor in a computer device. Of course, it can also be implemented by a specific logic circuit. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0255] As shown in Figure 8 An optimization device includes a first execution module 11, a first determination module 12, a second determination module 13, a third determination module 14, a fourth determination module 15, a first acquisition module 16, a fifth determination module 17, a sixth determination module 18, and a second execution module 19.

[0256] The first execution module 11 is configured to extract initial audio features, input audio features, local appearance features, global appearance features, input appearance features and optical flow features from the target video by using a feature extraction network. The first determination module 12 is configured to determine global audio positive samples and global audio negative samples according to the global appearance features and the initial audio features. The second determination module 13 is configured to determine local audio positive samples and local audio negative samples according to the local appearance features and the initial audio features. The third determination module 14 is configured to determine a global audio-visual feature contrast loss according to the global audio positive samples and the global audio negative samples. The fourth determination module 15 is configured to determine a local audio-visual feature contrast loss according to the local audio positive samples and the local audio negative samples. The first acquisition module 16 is configured to acquire self-supervised appearance features from the training video by using a self-supervised training network. The fifth determination module 17 is configured to determine a mutual distillation loss according to the self-supervised appearance features, the input appearance features, the input audio features and the optical flow features. The sixth determination module 18 is configured to determine a classification loss according to the input appearance features, the input audio features and the optical flow features. The second execution module 19 is configured to optimize the feature extraction network according to the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the mutual distillation loss and the classification loss.

[0257] In some embodiments, the first determination module 12 comprises a third acquisition module, a third execution module and a twenty-fourth determination module.

[0258] The third acquisition module is configured to acquire a first start time of a target global appearance feature, wherein the feature extraction network extracts a plurality of global appearance features, and the target global appearance feature is determined from the plurality of global appearance features. The third execution module is configured to cut a first global audio segment with a first preset time length from the initial audio features according to the first start time. The twenty-fourth determination module is configured to determine the global audio positive sample corresponding to the target global appearance feature according to the first global audio segment.

[0259] In some embodiments, the first determination module 12 further comprises a fourth acquisition module, a fourth execution module and a twenty-fifth determination module.

[0260] The fourth acquisition module is configured to acquire a second start time of each of the global appearance features except the target global appearance feature. The fourth execution module is configured to cut a second global audio segment with a first preset time length from the initial audio features according to each of the second start times. The twenty-fifth determination module is configured to determine each of the global audio negative samples according to each of the second global audio segments.

[0261] In some embodiments, the second determining module 13 comprises: a fifth obtaining module, a fifth performing module, a sixth obtaining module, a twenty-sixth determining module, and a tenth determining module.

[0262] The fifth obtaining module is configured to obtain a third starting time of a target local appearance feature, wherein the local appearance features extracted by the feature extraction network are multiple, and the target local appearance feature is determined from the multiple local appearance features. The fifth performing module is configured to cut a first local audio segment satisfying a first preset rule from the initial audio feature according to the third starting time, wherein the first local audio segment comprises multiple first audio slices. The sixth obtaining module is configured to obtain a first dot product similarity between the target local appearance feature and the first local audio segment. The twenty-sixth determining module is configured to determine a first attention weight of each first audio slice according to the first dot product similarity. The tenth determining module is configured to determine the local audio positive sample according to the first attention weight and the first local audio segment.

[0263] In some embodiments, the second determining module 13 further comprises: a seventh obtaining module, a sixth performing module, an eighth obtaining module, an eleventh determining module, and a twelfth determining module.

[0264] The seventh obtaining module is configured to obtain a fourth starting time of each local appearance feature except the target local appearance feature. The sixth performing module is configured to cut a second local audio segment satisfying the first preset rule from the initial audio feature according to each fourth starting time, to obtain multiple second local audio segments; wherein each second local audio segment comprises multiple second audio slices. The eighth obtaining module is configured to obtain a second dot product similarity between the target local appearance feature and the second local audio segment. The eleventh determining module is configured to determine a second attention weight of the second local audio segment according to the second dot product similarity. The twelfth determining module is configured to determine the local audio negative sample according to the second attention weight and the second local audio segment.

[0265] In some embodiments, the sixth determining module 18 comprises: a thirteenth determining module, a fourteenth determining module, a ninth obtaining module, and a fifteenth determining module.

[0266] The thirteenth determining module is configured to determine a fusion feature according to the input appearance feature, the input audio feature, and the optical flow feature. The fourteenth determining module is configured to determine a prediction score of the training video according to the fusion feature. The ninth obtaining module is configured to obtain a true label of the training video. The fifteenth determining module is configured to determine the classification loss according to the true label and the prediction score.

[0267] In some embodiments, the fifth determining module 17 comprises a sixteenth determining module, a seventeenth determining module, an eighteenth determining module and a nineteenth determining module.

[0268] The sixteenth determining module is configured to determine a fusion feature according to the input appearance feature, the input audio feature and the optical flow feature. The seventeenth determining module is configured to determine a first mutual distillation score according to the self-supervised appearance feature. The eighteenth determining module is configured to determine a second mutual distillation score according to the fusion feature. The nineteenth determining module is configured to determine a mutual distillation loss according to the first mutual distillation score and the second mutual distillation score.

[0269] In some embodiments, the thirteenth determining module or the sixteenth determining module comprises a seventh performing module, a twentieth determining module and a twenty-first determining module.

[0270] The seventh performing module is configured to perform spatial mapping processing on the input appearance feature, the input audio feature and the optical flow feature to obtain a mapped input appearance feature, a mapped input audio feature and a mapped flow light feature. The twentieth determining module is configured to determine a visual feature according to the mapped input appearance feature and the mapped flow light feature. The twenty-first determining module is configured to determine the fusion feature according to the visual feature and the mapped input audio feature.

[0271] In some embodiments, the first performing module 11 comprises an eighth performing module, a ninth performing module, a tenth performing module, a twenty-second determining module, an eleventh performing module, a twelfth performing module, a twenty-third determining module and a thirteenth performing module.

[0272] The eighth performing module is configured to uniformly extract m frames of pictures from the training video according to a second preset rule, and evenly divide the m frames of pictures to synthesize n global video segments of a second preset time length. The ninth performing module is configured to uniformly extract m' frames of pictures from the training video according to a third preset rule, and evenly divide the m' frames of pictures to synthesize n local video segments of a second preset time length. The tenth performing module is configured to input the n global video segments into the appearance feature extraction network to obtain n global appearance features. The twenty-second determining module is configured to determine an input appearance feature according to the n global appearance features. The eleventh performing module is configured to input the n local video segments into the appearance feature extraction network to obtain n local appearance features. The twelfth performing module is configured to extract the initial audio feature from the training video through the audio feature extraction network. The twenty-third determining module is configured to determine the input audio feature according to the initial audio feature. The thirteenth performing module is configured to extract the optical flow feature from the training video through the optical flow feature extraction network.

[0273] Each module in the aforementioned optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor within the device in hardware form, or stored in the memory of the processing device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the module division in this embodiment is illustrative and represents only a logical functional division; in actual implementation, other division methods may be used.

[0274] Example 4:

[0275] Based on the aforementioned Embodiment 2, this application provides an identification device. The modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0276] like Figure 9 As shown, an identification device includes: a second acquisition module 21, a seventh determination module 22, an eighth determination module 23, and a ninth determination module 24.

[0277] The second acquisition module 21 is used to extract the input appearance features, input audio features, and optical flow features of the target video through a feature extraction network. The seventh determination module 22 is used to determine the fusion features based on the input appearance features, the input audio features, and the optical flow features. The eighth determination module 23 is used to determine the prediction score of the target video based on the fusion features. The ninth determination module 24 is used to identify the target video based on the prediction score.

[0278] In some embodiments, the second acquisition module 21 includes: a seventeenth execution module, an eighteenth execution module, a twenty-ninth determination module, a nineteenth execution module, a thirtieth determination module, and a twentieth execution module.

[0279] The seventeenth execution module is configured to uniformly extract m frames of pictures from the target video according to a second preset rule, and divide the m frames into n global video clips of a first preset time length. The eighteenth execution module is configured to input the n global video clips into the appearance feature extraction network to obtain n global appearance features. The twenty-ninth determination module is configured to determine an input appearance feature according to the n global appearance features. The nineteenth execution module is configured to extract an initial audio feature from the target video by using the audio feature extraction network. The thirtieth determination module is configured to determine an input audio feature according to the initial audio feature. The twentieth execution module is configured to extract the optical flow feature from the target video by using the optical flow feature extraction network.

[0280] In some embodiments, the seventh determination module 22 includes a twenty-first execution module, a thirty-first determination module, and a thirty-second determination module.

[0281] The twenty-first execution module is configured to perform spatial mapping processing on the input appearance feature, the input audio feature, and the optical flow feature to obtain a mapped input appearance feature, a mapped input audio feature, and a mapped optical flow feature. The thirty-first determination module is configured to determine a visual feature according to the mapped input appearance feature and the mapped optical flow feature. The thirty-second determination module is configured to determine the fusion feature according to the visual feature and the input audio feature.

[0282] The modules in the above-mentioned recognition device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a device in hardware form, or stored in a memory in a processing device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is merely a logical function division. In actual implementation, another division manner can be used.

[0283] Embodiment 5:

[0284] The third aspect provides an electronic device, including a memory and a processor, and the memory stores a computer program. When the computer program is executed by the processor, the method in the first aspect and / or the method in the second aspect are executed.

[0285] Embodiment 6:

[0286] The fourth aspect provides a storage medium, which stores a computer program capable of being executed by one or more processors. The computer program can be used to implement the method in the first aspect and / or the method in the second aspect.

[0287] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. The volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0288] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily mean the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence number of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0289] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0290] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The described device embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices, or units, and can be electrical, mechanical, or in other forms.

[0291] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on multiple network units; and some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0292] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional units.

[0293] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read-only memories (ROMs), magnetic discs or optical discs, and various storage media that can store program codes.

[0294] Alternatively, the integrated units of the present application, if implemented in the form of software functional modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a controller to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROMs, magnetic discs or optical discs, and various storage media that can store program codes.

[0295] The above merely provides the implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the change or replacement within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An optimization method, characterized in that, include: The feature extraction network is used to extract initial audio features, input audio features, local appearance features, global appearance features, input appearance features, and optical flow features from the training video. The feature extraction network includes: an audio feature extraction network, an optical flow feature extraction network, and an appearance feature extraction network; the step of extracting video appearance features, audio features, and optical flow features from the training video using the feature extraction network includes: According to the second preset rule, m frames are uniformly extracted from the training video, and the m frames are evenly divided into n global video segments of the second preset duration. According to the third preset rule, m' frames are uniformly extracted from the training video, and the m' frames are evenly divided into n local video segments of the second preset duration. The n global video segments are input into the appearance feature extraction network to obtain n global appearance features; The input appearance features are determined based on the n global appearance features; The n local video segments are input into the appearance feature extraction network to obtain n local appearance features; The initial audio features are extracted from the training video using the audio feature extraction network. The input audio features are determined based on the initial audio features; The optical flow features are extracted from the training video using the optical flow feature extraction network. Global audio positive samples and global audio negative samples are determined based on the global appearance features and the initial audio features; Local audio positive samples and local audio negative samples are determined based on the local appearance features and the initial audio features; The global audio-visual feature contrast loss is determined based on the global positive audio samples and the global negative audio samples; The local audio-visual feature contrast loss is determined based on the local positive audio samples and the local negative audio samples; Self-supervised appearance features are obtained from the training videos by training a self-supervised network; The inter-distillation loss is determined based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features. The classification loss is determined based on the input appearance features, the input audio features, and the optical flow features; The feature extraction network is optimized based on the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the cross-distillation loss, and the classification loss.

2. The method according to claim 1, characterized in that, The step of determining global positive audio samples based on the global appearance features and the initial audio features includes: The first starting time for obtaining the target global appearance feature is specified, wherein the feature extraction network extracts multiple global appearance features, and the target global appearance feature is determined from the multiple global appearance features. A first global audio segment of a first preset duration is extracted from the initial audio features based on the first start time; The global audio positive sample corresponding to the target global appearance feature is determined based on the first global audio segment.

3. The method according to claim 2, characterized in that, The step of determining global audio negative samples based on the global appearance features and the initial audio features includes: Obtain the second start time of each of the global appearance features, excluding the target global appearance feature; A second global audio segment of a first preset duration is extracted from the initial audio features based on each second start time; Each global audio negative sample is determined based on each of the second global audio segments.

4. The method according to claim 1, characterized in that, The step of determining local audio positive samples based on the local appearance features and the initial audio features includes: A third starting time for obtaining the target local appearance features, wherein the feature extraction network extracts multiple local appearance features, and the target local appearance features are determined from the multiple local appearance features; Based on the third start time, a first local audio segment satisfying a first preset rule is extracted from the initial audio features, wherein the first local audio segment includes multiple first audio slices; Obtain the first dot product similarity between the target local appearance features and the first local audio segment; A first attention weight is determined for each of the first audio slices based on the first dot product similarity. The local audio positive sample is determined based on the first attention weight and the first local audio segment.

5. The method according to claim 4, characterized in that, The step of determining local audio negative samples based on the local appearance features and the initial audio features includes: Obtain the fourth start time for each of the local appearance features, excluding the target local appearance feature; Based on each fourth start time, a second local audio segment satisfying a first preset rule is extracted from the initial audio features to obtain a plurality of second local audio segments; wherein each second local audio segment includes a plurality of second audio slices; For each of the second local audio segments, perform the following operation: Obtain the second dot product similarity between the target local appearance features and the second local audio segment; The second attention weight of the second local audio segment is determined based on the second dot product similarity. The local audio negative sample is determined based on the second attention weight and the second local audio segment.

6. The method according to claim 4 or 5, characterized in that, The expression for the first preset rule is as follows: ； In the formula, It is the duration of the initial audio features. w and b These are learnable parameters. w Indicates weight, b Indicates offset, This represents the transpose of the weights. This represents the activation function, and TAP represents average pooling along the time dimension. This represents the local appearance feature of the target, specifically the i-th local appearance feature. This represents the dynamic time window, which is the duration of each local audio negative sample or each local audio positive sample.

7. The method according to claim 5, characterized in that, The local audio-visual feature contrast loss is determined by the following formula: in, This represents the local audio / video feature contrast loss. It is a local audio positive sample. It is a local audio negative sample. B For the training batch size, It is the temperature coefficient. It is the ratio of the dynamic time window to the duration of the initial audio features. This represents the transpose of a local appearance feature.

8. The method according to claim 3, characterized in that, The global audio-visual feature contrast loss is determined by the following formula: ； in, This represents the global audio-visual feature contrast loss. It is a global positive audio sample with apparent features. It is a global audio negative sample of appearance features. B For the training batch size, It is the temperature coefficient. This represents the transpose of a global appearance feature.

9. The method according to claim 1, characterized in that, The step of determining the classification loss based on the input appearance features, the input audio features, and the optical flow features includes: The fusion features are determined based on the input appearance features, the input audio features, and the optical flow features; The predicted score of the training video is determined based on the fusion features; Obtain the true labels of the training videos; The classification loss is determined based on the true label and the predicted score.

10. The method according to claim 1, characterized in that, The step of determining the inter-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features includes: The fusion features are determined based on the input appearance features, the input audio features, and the optical flow features; The first cross-distillation fraction is determined based on the self-supervised apparent characteristics; The second interdistillation fraction is determined based on the fusion characteristics; The interdistillation loss is determined based on the first interdistillation fraction and the second interdistillation fraction.

11. The method according to claim 9 or 10, characterized in that, The step of determining the fusion features based on the input appearance features, the input audio features, and the optical flow features includes: The input appearance features, the input audio features, and the optical flow features are spatially mapped to obtain mapped input appearance features, mapped input audio features, and mapped optical flow features. Visual features are determined based on the mapped input appearance features and the mapped streamer features; The fusion feature is determined based on the visual features and the mapped input audio features.

12. A method for identification, characterized in that, The feature extraction network applicable to any one of claims 1-11 includes: The target video's input appearance features, input audio features, and optical flow features are extracted using a feature extraction network. The fusion features are determined based on the input appearance features, the input audio features, and the optical flow features; The prediction score of the target video is determined based on the fusion features; The target video is identified based on the predicted score.

13. The method according to claim 12, characterized in that, The feature extraction network includes: an appearance feature extraction network, an audio feature extraction network, and an optical flow feature extraction network; the feature extraction network extracts the input appearance features, input audio features, and optical flow features of the target video, including: According to the second preset rule, m frames are uniformly extracted from the target video, and the m frames are evenly divided and synthesized into n global video segments of the first preset duration. The n global video segments are input into the appearance feature extraction network to obtain n global appearance features; The input appearance features are determined based on the n global appearance features; Initial audio features are extracted from the target video using the audio feature extraction network. Determine the input audio features based on the initial audio features; The optical flow features are extracted from the target video using the optical flow feature extraction network.

14. The method according to claim 12, Its characteristic lies in that, the step of determining the fusion feature based on the input appearance feature, the input audio feature, and the optical flow feature includes: The input appearance features, the input audio features, and the optical flow features are spatially mapped to obtain mapped input appearance features, mapped input audio features, and mapped optical flow features. Visual features are determined based on the mapped input appearance features and the mapped streamer features; The fusion feature is determined based on the visual features and the input audio features.

15. An optimization apparatus based on the method according to any one of claims 1-11, characterized in that, include: The first execution module is used to extract initial audio features, input audio features, local appearance features, global appearance features, input appearance features, and optical flow features from the training video using a feature extraction network. The first determining module is used to determine global audio positive samples and global audio negative samples based on the global appearance features and the initial audio features; The second determining module is used to determine local audio positive samples and local audio negative samples based on the local appearance features and the initial audio features; The third determining module is used to determine the global audio-visual feature contrast loss based on the global positive audio samples and the global negative audio samples; The fourth determining module is used to determine the local audio-visual feature contrast loss based on the local audio positive samples and the local audio negative samples; The first acquisition module is used to acquire self-supervised appearance features from the training video through a self-supervised training network; The fifth determining module is used to determine the inter-distillation loss based on the self-supervised appearance features, the input appearance features, the input audio features, and the optical flow features; The sixth determining module is used to determine the classification loss based on the input appearance features, the input audio features, and the optical flow features; The second execution module is used to optimize the feature extraction network based on the global audio-visual feature contrast loss, the local audio-visual feature contrast loss, the cross-distillation loss, and the classification loss.

16. An identification device based on the method of any one of claims 12-14, characterized in that, include: The second acquisition module is used to extract the input appearance features, input audio features, and optical flow features of the target video through a feature extraction network; The seventh determining module is used to determine the fusion features based on the input appearance features, the input audio features, and the optical flow features; The eighth determining module is used to determine the prediction score of the target video based on the fusion features; The ninth determining module is used to identify the target video based on the predicted score.

17. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that, when executed by the processor, performs the method as described in any one of claims 1-11 and / or the method as described in any one of claims 12-14.

18. A storage medium, characterized in that, The computer program stored in the storage medium can be executed by one or more processors, and the computer program can be used to implement the method as described in any one of claims 1-11 and / or the method as described in any one of claims 12-14.

Images