Short video scene recognition method and system based on cyclic generation modal completion

Abstract

The present invention proposes a short video scene recognition method and system based on cyclic generation of modal completion, which relates to the technical field of short video scene recognition. It includes dividing the original data of each modality into modal complete data and modal missing data; learning the consistent representation of multimodality based on multiple modal complete data to obtain the consistent semantic representation corresponding to each modal complete data; and learning the specific representation of each modality based on the original data of each modality; fusing the consistent semantic representation corresponding to each modal complete data and the specific representation of the corresponding modality, supplementing the data of each modality after the missing data is completed into the multimodal original data, learning the consistent representation of multimodality again, and obtaining the scene classification result through the classification network. The present invention utilizes the semantic consistency between short video multimodal data to generate complementary data with rich semantics in the process of short video scene recognition, thereby improving the performance of short video scene recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of short video scene recognition, and in particular to a short video scene recognition method and system based on cyclic generation of modal completion. Background Art

[0002] With the rapid development of the Internet, short videos have emerged as a media form. As short videos appear more and more in the public eye, short video classification plays an increasingly important role in the grouping and display of videos and personalized recommendations for users. Short video scene recognition is an important task in the analysis and understanding of short video content. It plays an important auxiliary role in the understanding of short videos and is an important research direction in the understanding of short videos. Specifically, scene recognition is the observer's understanding of the environment within the field of view, including the objects contained in the environment and the relationship between objects. Important applications of scene recognition include behavior detection, target recognition, anomaly detection in specific scenarios, pedestrian recognition, video retrieval, etc. In addition, scene recognition can also assist in product recommendations, text explanations, assisted reconnaissance, autonomous driving, etc.

[0003] In conventional video scene classification, multimodal learning is mostly based on the premise that the data of each modality is complete. However, in real short videos, the data uploaded by users is not necessarily standardized, and there is a problem of missing data. At present, some multimodal learning works have taken into account the problem of missing data in some modalities, but usually the missing data is supplemented by zero-filling or nearest neighbor filling technology in the data preprocessing stage. However, the inventors found that although this data filling technology may complete the missing data, the supplemented data lacks actual semantics, and has limited effect on improving the performance of short video scene recognition. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a short video scene recognition method and system based on cyclic generation of modal completion, which utilizes the semantic consistency between short video multimodal data to generate completion data with rich semantics during the short video scene recognition process, thereby improving the performance of short video scene recognition.

[0005] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention provides a short video scene recognition method based on cyclic generation of modal completion.

[0007] The short video scene recognition method based on cyclic generation modal completion includes the following steps:

[0008] Obtain multimodal raw data in short videos, and divide the raw data of each modality into modality complete data and modality missing data;

[0009] Based on the complete data of multiple modalities, learn the consistent representation of multimodality and obtain the consistent semantic representation corresponding to the complete data of each modality; and based on the original data of each modality, learn the specific representation of each modality;

[0010] The consistent semantic representation corresponding to the complete data of each modality is combined with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed;

[0011] The data of each modality after missing data completion is added to the multimodal original data, the consistent representation of the multimodality is learned again, the consistent semantic representation corresponding to the complete modal data is updated, and the scene classification result is obtained through the classification network to complete the scene recognition.

[0012] A second aspect of the present invention provides a short video scene recognition system based on cyclic generation of modal completion.

[0013] A short video scene recognition system based on cyclic generation of modal completion, including:

[0014] The raw data acquisition and division module is configured to: acquire multimodal raw data in the short video, and divide the raw data of each modality into modality complete data and modality missing data;

[0015] The feature representation module is configured to: learn a consistent representation of multiple modalities based on the complete data of multiple modalities to obtain a consistent semantic representation corresponding to the complete data of each modality; and learn a specific representation of each modality based on the original data of each modality;

[0016] The fusion module is configured to: fuse the consistent semantic representation corresponding to the complete data of each modality with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed;

[0017] The update and recognition module is configured to: add the data of each modality after missing data completion to the multimodal original data, learn the consistent representation of the multimodality again, update the consistent semantic representation corresponding to the complete modal data, and obtain the scene classification result through the classification network to complete the scene recognition.

[0018] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the short video scene recognition method based on cyclic generation of modal completion as described in the first aspect of the present invention.

[0019] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the short video scene recognition method based on cyclic generation of modal completion as described in the first aspect of the present invention are implemented.

[0020] One or more of the above technical solutions have the following beneficial effects:

[0021] (1) The present invention provides a short video scene recognition method and system based on cyclic generative modal completion, which combines short video scene recognition with missing data completion. The cyclic generative modal completion technology is used to add the cyclic generation of missing data into the overall framework of scene recognition. The semantic consistency between short video multimodal data is utilized, and complementary data with rich semantics can be generated during the short video scene recognition process, thereby improving the performance of short video scene recognition.

[0022] (2) The present invention takes into account the situation of incomplete data and divides the original data of each modality into modal complete data and modal missing data, learns the consistent representation of multimodality based on multiple modal complete data, obtains the consistent semantic representation corresponding to each modal complete data, and makes full use of the existing modal data to learn the consistent representation of multimodality, constructs a short video scene recognition model, and thus improves the short video scene recognition performance, avoiding the problem of semantic missing caused by simply supplementing the missing data through zero-value filling or nearest neighbor filling technology in the data preprocessing stage in the prior art.

[0023] (3) The present invention learns the specific representation of each modality based on the original data of each modality, merges the consistent semantic representation corresponding to the complete data of each modality with the specific representation of the corresponding modality, obtains the data of each modality after missing data completion, adds the data of each modality after missing data completion to the multimodal original data, learns the consistent representation of multimodality again, updates the consistent semantic representation corresponding to the complete data of the modality, and adopts the above-mentioned cyclic generation method to complete the missing data of each modality in the short video, which can make full use of the existing multimodal data, thereby fully exploring the semantic consistency of each modality in the short video.

[0024] (4) During the generation phase, the present invention fully utilizes the unique information of each mode of short video and the consistency information in the common subspace, so that the generated data is more realistic.

[0025] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 This is a flow chart of the method of the first embodiment.

[0028] Figure 2 This is the data flow diagram of the first embodiment. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] Embodiment 1

[0033] This embodiment discloses a short video scene recognition method based on cyclic generation of modal completion.

[0034] In order to deeply explore the semantic representation of short video scenes, this embodiment uses the cyclic generation modality completion technology to add the cyclic generation of missing data to the overall framework of scene recognition, and utilizes the semantic consistency between short video multimodal data. It can generate semantically rich completion data in the process of short video scene recognition, thereby improving the performance of short video scene recognition. The idea of ​​this embodiment is applicable to multimodal fusion scenarios in various data missing situations, helping them to better complete the task.

[0035] In general, in this embodiment, the incomplete multimodal raw data is first divided into two parts, namely, modality complete data and modality missing data. Among them, each modality in the modality complete data is mapped to a common subspace through its own mapping network, and the consistent representation of multimodality is learned. The raw data of each modality of all samples are mapped to their own unique space through another mapping network. Then each sample can find its consistent semantic representation in the common subspace, and can also find the modality-specific representation in the unique space;

[0036] The consistent semantic representation in the common subspace and the specific representation in the unique space are merged, and the data of each modality is generated through a generative network such as a diffusion model;

[0037] The generated modal data can complete the original missing data, and then a new round of learning of common subspace and unique space can be carried out;

[0038] Finally, the samples in the common subspace are represented by the multi-layer perceptron and softmax of the classification network to obtain the recognition results. During the training process, the classification network is trained through cross entropy loss and reconstruction loss.

[0039] like Figure 1 and Figure 2 As shown, further, the short video scene recognition method based on cyclic generation modal completion includes the following steps:

[0040] Obtain multimodal raw data in short videos, and divide the raw data of each modality into modality complete data and modality missing data;

[0041] Based on the complete data of multiple modalities, learn the consistent representation of multimodality and obtain the consistent semantic representation corresponding to the complete data of each modality; and based on the original data of each modality, learn the specific representation of each modality;

[0042] The consistent semantic representation corresponding to the complete data of each modality is combined with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed;

[0043] The data of each modality after missing data completion is added to the multimodal original data, the consistent representation of the multimodality is learned again, the consistent semantic representation corresponding to the complete modal data is updated, and the scene classification result is obtained through the classification network to complete the scene recognition.

[0044] Furthermore, the complete data of multiple modalities are mapped to a common subspace respectively, and the consistent semantic representation of multimodality is learned in the common subspace;

[0045] The original data of each modality is mapped to its own unique space, and the specific representation of each modality is learned in its own unique space.

[0046] Furthermore, the scene classification results are obtained through the classification network, specifically:

[0047] Step 1: Supplement the missing data into the multimodal original data to obtain new multimodal original data;

[0048] Step 2: Update the complete modal data. Based on the updated complete modal data, learn the common subspace and the unique space again to obtain the new data of each modality after missing data completion, and add the new data of each modality after missing data completion to the new multimodal original data.

[0049] Step 3: Use the obtained common subspace feature representation as a new multimodal consistent semantic representation;

[0050] Step 4: Input the new multimodal consistent semantic representation into the multi-layer fully connected neural network, and obtain the recognition result through softmax to determine whether the recognition accuracy reaches the preset threshold or the number of training times reaches the preset number. If not, loop the above steps 1 to 3 until the recognition accuracy or the number of training times reaches the requirement.

[0051] It can be understood that in order to complete the missing data, this embodiment combines the characteristics of the unique space of each modality and the common subspace, and the above-mentioned feature combination process can be repeated multiple times, which can fully utilize the complementarity between multiple modalities. The data finally generated is more in line with the actual data distribution, so that the learned characteristics of the common subspace can be more accurate, thereby obtaining better and more accurate classification results.

[0052] The cycle is repeated many times until the recognition accuracy reaches a preset threshold or the number of trainings reaches a preset number. Finally, the features of multiple modes in the common subspace can be averaged or concatenated for feature fusion and used for the next classification.

[0053] It should be pointed out that the method in this embodiment adopts a short video scene recognition network with cyclic generative modality completion, which specifically includes a cyclic generative modality completion network and a classification network: the data information of the missing modality is supplemented by the cyclic generative modality completion network, and the multimodal data is mapped to a common subspace to obtain a multimodal semantic consistency feature representation; the classification result is obtained through the classification network.

[0054] Furthermore, the cyclic generation modality completion network specifically comprises the following operations:

[0055] S1: Divide the sample set with missing data, i.e., the multimodal original data, into two parts, one part is the data with complete data in each modality, i.e., the modality complete data; the other part is the data with missing data, i.e., the modality missing data.

[0056] The multimodal raw data includes visual modality raw data, sound modality raw data and text modality raw data. The data sets of each modality are respectively represented as follows:

[0057] X v Represents the data set of the visual modality, that is, the original data of the visual modality, X v =(X vc , X vic ). Among them, X vc The visual modality representing the complete data of each modality, X vic A visual modality indicating the presence of missing data.

[0058] X a Represents the data set of the sound mode, that is, the original data of the sound mode, Xa =(X ac , X aic ). Among them, X ac The sound mode representing the complete data of each mode, X aic The sound modality that indicates the presence of missing data.

[0059] X t Represents the data set of text mode, that is, the original data of text mode, X t =(X tc , X tic ). Among them, X tc The text modality representing the complete data of each modality, X tic A text modal indicating the presence of missing data.

[0060] S2: Each modality is mapped to a common subspace through its own feature map.

[0061] The multi-modal consistent representation described in the above embodiment refers to mapping multiple modalities to a common subspace, in which the mutual information between the multiple modal feature representations is maximized.

[0062] The mapping process of visual modality is The mapping process of sound mode is: The mapping process of text mode is

[0063] in, and All are multi-layer neural networks; P vc , P ac and P tc They are the consistent semantic representations of the visual modality, audio modality and text modality in the common subspace respectively.

[0064] The correlation between the three modes in the common subspace can be expressed by mutual information:

[0065] I(P vc ;P ac ;P tc )=H(P vc ;P ac )+H(P ac ;P tc )H(P vc ;P ac ;P tc ).

[0066] Among them, I(P vc ;P ac ;P tc) is the mutual information of the consistent semantic representation of the visual modality, the audio modality and the text modality in the common subspace; H(P vc ;P ac ) is P vc and P ac The joint entropy of ac ;P tc ) is P ac and P tc The joint entropy of vc ;P ac ;P tc ) is P vc , P ac and P tc Joint entropy of the three variables.

[0067] S3: The visual modality features in S2 are extracted using the VGG16_P1aces365 network.

[0068] Specifically:

[0069] Extract key frames from the video by taking one frame every 5 frames, and select the first 50 key frames of each video; extract key frame features.

[0070] Key frame features are extracted in the following way: all key frames are resized to 224*224 by resizing method, and then input into VGG16_Places365 network, and 4096-dimensional image features are output as visual modality features of each frame. VGG16_Places365 is a mature network model in the prior art, and will not be described in detail here.

[0071] The sound features are extracted using MFCC, and the text features are extracted using sentence2vector.

[0072] Among them, the MFCC feature is the Mel-scale Frequency Cepstral Coefficients (MFCC for short), which is the cepstral parameter extracted in the Mel scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear frequency.

[0073] S4: Each modality is mapped to a unique space through its own feature map.

[0074] The mapping process of visual modality is The mapping process of sound mode is: The mapping process of text mode is

[0075] in, and All are multi-layer neural networks; Iv ,I a and I t They are specific representations of the visual modality, the audio modality and the text modality in their respective unique spaces.

[0076] S5: Generation of missing data for each modality.

[0077] The consistent semantic representation of each modality in the common subspace and the specific representation in the corresponding unique space are integrated, and the data of each modality after missing data completion is generated through a generative network such as a diffusion model. The generated missing data after missing data completion can complete the original missing data, and then a new round of learning of the common subspace and unique space can be carried out.

[0078] The data generation process of each modality is as follows:

[0079] DM(P vc +I v )→X v′ ,DM(P ac +I a )→X a′ ,DM(P tc +I t )→X t′ .

[0080] Among them, DM() represents the Stable Diffusion diffusion model, X v′ , X a′ and X t′ Represents the data generated by each mode.

[0081] The generated data is added to the original data, and the common subspace and unique space are learned again based on the complete data.

[0082] like Figure 1 As shown, the independent feature space of modality 1, the independent feature space of modality 2, and the independent feature space of modality 3 represent the unique spaces to which the original data of the visual modality, the original data of the sound modality, and the original data of the text modality are mapped respectively:

[0083] The samples containing the above three modalities, that is, the three modal raw data in the short video, are divided, where all white columns represent complete modal data, and columns with black squares represent missing modal data;

[0084] The complete modal data of the three modalities are mapped to a common subspace respectively, and the multimodal consistent semantic representation is learned in the common subspace;

[0085] Map the original data of each modality to their own unique space, and learn the specific representation of each modality in their own unique space;

[0086] Then, the consistent semantic representation corresponding to the complete data of each modality is combined with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed, corresponding to Figure 1 Mode 1, Mode 2 and Mode 3 in ;

[0087] After that, the data of each modality after the missing data of modality 1, modality 2 and modality 3 are completed are circulated along the direction of the dotted line, and the data of each modality after the missing data is completed are added to the multimodal original data. The consistent representation of the multimodality is learned again, the consistent semantic representation corresponding to the complete modal data is updated, and the scene classification result is obtained through the classification network to complete the scene recognition.

[0088] Furthermore, the classification network includes:

[0089] S1: The common subspace feature representation obtained in the cycle generation completion network is used as a multimodal consistent feature representation.

[0090] S2: Input the multimodal consistency feature representation into a multi-layer fully connected neural network, and obtain the recognition result through softmax.

[0091] S3: Training process.

[0092] The loss function of the entire classification network consists of three parts. The first part is the mutual information of the three modes in the common subspace, the larger the better. The second part is the reconstruction loss of the generated mode, the smaller the better. The third part is the cross entropy loss calculated by softmax after the fusion features of each mode in the common subspace, the smaller the better.

[0093] The three parts of loss are calculated as follows:

[0094] L MI =-I(P vc ;P ac ;P tc ),

[0095] L Rec =||X v -X v′ || F +||X a -X a′ || F +||X t -X t′ || F ,

[0096] L cross =crossentropy(X cons , Y).

[0097] Among them, L MI is the mutual information loss of the three modalities in the common subspace, I(P vc ;P ac ;P tc ) is the mutual information of the three modes in the common subspace; L Rec is the reconstruction loss of the generated modality; L cross The cross entropy loss is calculated by softmax for the fusion features of each modality in the common subspace; X cons =softmax(MLP(concat(P vc , P ac , P tc ))), softmax() is the activation function, MLP() is the multi-layer perceptron, concat() is the concatenation operation; Y is the category label information of the data.

[0098] The final total loss function is the weighted sum of the three losses:

[0099] Loss = L MI +αL Rec +βL cross .

[0100] Among them, α and β are weight parameters.

[0101] The final scene recognition result is obtained by optimizing the loss function. The classification network is trained by minimizing the loss and using the back propagation algorithm until the classification accuracy reaches a certain set value or the algorithm converges, ending the training process.

[0102] S4: Testing process.

[0103] The untrained test set samples are input into the trained network structure, and a short video scene recognition network with modality completion is generated through a cycle to complete the missing data and output the classification results.

[0104] Embodiment 2

[0105] This embodiment discloses a short video scene recognition system based on cyclic generation of modal completion.

[0106] A short video scene recognition system based on cyclic generation of modal completion, including:

[0107] The raw data acquisition and division module is configured to: acquire multimodal raw data in the short video, and divide the raw data of each modality into modality complete data and modality missing data;

[0108] The feature representation module is configured to: learn a consistent representation of multiple modalities based on the complete data of multiple modalities to obtain a consistent semantic representation corresponding to the complete data of each modality; and learn a specific representation of each modality based on the original data of each modality;

[0109] The fusion module is configured to: fuse the consistent semantic representation corresponding to the complete data of each modality with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed;

[0110] The update and recognition module is configured to: add the data of each modality after missing data completion to the multimodal original data, learn the consistent representation of the multimodality again, update the consistent semantic representation corresponding to the complete modal data, and obtain the scene classification result through the classification network to complete the scene recognition.

[0111] Embodiment 3

[0112] The purpose of this embodiment is to provide a computer-readable storage medium.

[0113] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the short video scene recognition method based on cyclic generation of modal completion as described in Example 1 of the present disclosure.

[0114] Embodiment 4

[0115] The purpose of this embodiment is to provide an electronic device.

[0116] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the short video scene recognition method based on cyclic generation of modal completion as described in Example 1 of the present disclosure are implemented.

[0117] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0118] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0119] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A short video scene recognition method based on cyclic generation modal completion, characterized in that: The following steps are involved: Obtain multimodal raw data in short videos, and divide the raw data of each modality into modality complete data and modality missing data; Based on the complete data of multiple modalities, learn the consistent representation of multimodality and obtain the consistent semantic representation corresponding to the complete data of each modality; and based on the original data of each modality, learn the specific representation of each modality, specifically mapping the complete data of multiple modalities to a common subspace respectively, and learn the consistent semantic representation of multimodality in the common subspace; Map the original data of each modality to their own unique space, and learn the specific representation of each modality in their own unique space; The consistent semantic representation corresponding to the complete data of each modality is combined with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed; Update the complete modal data. Based on the updated complete modal data, learn the common subspace and the unique space again to obtain the new data of each modality after the missing data is completed. Add the new data of each modality after the missing data is completed to the new multimodal original data. Specifically, the consistent semantic representation corresponding to the complete data of each modality is integrated with the specific representation of the corresponding modality, which can be expressed as: ， ， Among them, DM() represents the Stable Diffusion diffusion model; , and They are respectively the consistent semantic representations of visual modality, audio modality and textual modality in the common subspace; , and They are specific representations of the visual modality, the audio modality, and the textual modality in their own unique spaces; , and Represents the data after missing data completion generated by visual modality, audio modality and text modality respectively; The data of each modality after missing data completion is added to the multimodal original data, the consistent representation of the multimodality is learned again, the consistent semantic representation corresponding to the complete modal data is updated, and the scene classification result is obtained through the classification network to complete the scene recognition.

2. The short video scene recognition method based on cyclic generation modal completion as claimed in claim 1, characterized in that: The multimodal original data includes visual modality original data, sound modality original data and text modality original data.

3. The short video scene recognition method based on cyclic generation modal completion as claimed in claim 2, characterized in that: Multiple modal complete data are mapped to a common subspace, specifically: The mapping process of visual modality is , the mapping process of sound mode is ,The mapping process of text mode is ;in , and All are multi-layer neural networks; The raw data of each modality is mapped to its own unique space, specifically: The mapping process of visual modality is , the mapping process of sound mode is ,The mapping process of text mode is ;in , and All are multi-layer neural networks; in, A data set representing a visual modality, Visual modalities that represent complete data for each modality; A data set representing a sound modality, Sound modes representing complete data for each mode; A data set representing a text modality, A text modality representing the complete data of each modality; , and They are respectively the consistent semantic representations of visual modality, audio modality and textual modality in the common subspace; , and They are specific representations of the visual modality, the audio modality and the text modality in their respective unique spaces.

4. The short video scene recognition method based on cyclic generation modal completion as claimed in claim 1, characterized in that: The classification results are obtained through the classification network, specifically: Step 1: Supplement the missing data into the multimodal original data to obtain new multimodal original data; Step 2: Update the complete modal data. Based on the updated complete modal data, learn the common subspace and the unique space again to obtain the new data of each modality after missing data completion, and add the new data of each modality after missing data completion to the new multimodal original data. Step 3: Use the obtained common subspace feature representation as a new multimodal consistent semantic representation; Step 4: Input the new multimodal consistent semantic representation into the multi-layer fully connected neural network, and obtain the recognition result through softmax to determine whether the recognition accuracy reaches the preset threshold or the number of training times reaches the preset number. If not, loop the above steps 1 to 3 until the recognition accuracy or the number of training times reaches the requirement.

5. The short video scene recognition method based on cyclic generation modal completion as claimed in claim 1, characterized in that: The loss function of the classification network is: ， in, and is the weight parameter; is the mutual information loss of the three modalities in the common subspace, is the mutual information of the three modes in the common subspace; is the reconstruction loss for the generated modality; A data set representing a visual modality, A data set representing a sound modality, A data set representing a text modality; Calculate the cross entropy loss through softmax for the fusion features of each modality in the common subspace; , softmax() is the activation function, MLP() is the multi-layer perceptron, concat() is the connection operation, It is the category label information of the data.

6. A short video scene recognition system based on cyclic generation modal completion, characterized in that: include: The raw data acquisition and division module is configured to: acquire multimodal raw data in the short video, and divide the raw data of each modality into modality complete data and modality missing data; The feature representation module is configured to: learn the consistent representation of multimodality based on the complete data of multiple modalities, and obtain the consistent semantic representation corresponding to the complete data of each modality; and learn the specific representation of each modality based on the original data of each modality, specifically mapping the complete data of multiple modalities to a common subspace respectively, and learning the consistent semantic representation of multimodality in the common subspace; Map the original data of each modality to their own unique space, and learn the specific representation of each modality in their own unique space; The fusion module is configured to: fuse the consistent semantic representation corresponding to the complete data of each modality with the specific representation of the corresponding modality to obtain the data of each modality after the missing data is completed; Update the complete modal data. Based on the updated complete modal data, learn the common subspace and the unique space again to obtain the new data of each modality after the missing data is completed. Add the new data of each modality after the missing data is completed to the new multimodal original data. Specifically, the consistent semantic representation corresponding to the complete data of each modality is integrated with the specific representation of the corresponding modality, which can be expressed as: ， ， Among them, DM() represents the Stable Diffusion diffusion model; , and They are respectively the consistent semantic representations of visual modality, audio modality and textual modality in the common subspace; , and They are specific representations of the visual modality, the audio modality, and the textual modality in their own unique spaces; , and Represents the data after missing data completion generated by visual modality, audio modality and text modality respectively; The update and recognition module is configured to: add the data of each modality after missing data completion to the multimodal original data, learn the consistent representation of the multimodality again, update the consistent semantic representation corresponding to the complete modal data, and obtain the scene classification result through the classification network to complete the scene recognition.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the short video scene recognition method based on cyclic generation of modal completion as described in any one of claims 1 to 5 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the short video scene recognition method based on cyclic generation of modal completion as described in any one of claims 1 to 5 are implemented.

Images