Method, device, medium and equipment for training and classification of dynamic data classification network

Abstract

Embodiments of the present application provide a training and classification method, device, medium and equipment of a dynamic data classification network. The training method sets a single-modal feature extraction network to share the same self-attention layer, bridges the gap between visual and language two modes, and combines a representation learning method and a semi-supervised training method to train the dynamic data classification network, so that the classification network can accurately obtain the feature information bridging multiple modes, accurately fuse the multi-modal information included in the dynamic data, and accurately classify the dynamic data to be classified.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of information processing technology, and more specifically, the embodiments of the present invention relate to a training method for dynamic data classification networks, as well as dynamic data classification methods, apparatus, media, and devices. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Dynamic data refers to content that uses images or videos combined with detailed text descriptions to convey users' immediate emotions, thoughts, or opinions, such as user updates on social media platforms. During the dynamic data publishing process, the platform needs to review the content to improve its quality. To enhance review efficiency, the dynamic data must be categorized before review (also known as "multimodal multi-label classification").

[0004] In related technologies, multimodal multi-label classification is achieved by introducing deep neural networks to learn feature representations of dynamic data, using similarity constraints between modalities to simulate the intrinsic relationships between different modalities, and introducing supervised training of labeled multimodal data to capture label dependencies.

[0005] However, since visual and text modalities have different feature tendencies, the above scheme has poor information fusion effect between visual and text modalities, making it difficult to accurately integrate the information of the two modalities for data classification. This leads to inaccurate model classification. Furthermore, the scheme uses supervised training, and the model classification effect is poor when there are few supervised samples. Summary of the Invention

[0006] In view of this, in order to solve the above problems, the embodiments of the present invention aim to provide a training method for a dynamic data classification network, a dynamic data classification method, a corresponding apparatus, a storage medium, and a computing device.

[0007] In a first aspect of the present invention, a method for training and classifying a dynamic data classification network is provided, wherein the dynamic data includes visual data and text data, and the dynamic data classification network includes a feature extraction network and a classifier; the feature extraction network includes a visual feature extraction network and a text feature extraction network; the method includes:

[0008] Obtain the first dynamic data sample with category labels;

[0009] The visual feature extraction network and the text feature extraction network are used to extract features from the first dynamic data sample to obtain corresponding visual features and text features, wherein the visual feature extraction network and the text feature extraction network share the network parameters of the attention layer.

[0010] Obtain the image-text matching results of the visual features and text features, and adjust the network parameters of the feature extraction network based on the image-text matching results and the classification labels;

[0011] Based on the feature extraction network, the classifier is semi-supervised trained using the first dynamic data sample and the second dynamic data sample without classification labels.

[0012] Optionally, the step of extracting features from the first dynamic data sample using the visual feature extraction network and the text feature extraction network respectively to obtain corresponding visual features and text features includes:

[0013] The visual data is converted into target data with a set encoding format, and a contrastive language-image pre-trained model is used to obtain pre-extracted visual features of the target data; the set encoding format is used to convert the visual data into data in the form of text characters;

[0014] The pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

[0015] Optionally, each layer of the visual feature extraction network includes at least a first self-attention layer, a cross-attention layer, and a first feedforward neural network; the parameters of the first self-attention layer include a query index; the step of inputting the pre-extracted visual features into the visual feature extraction network to obtain the corresponding visual features includes:

[0016] For each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the output is obtained by the first self-attention layer processing the received query index.

[0017] The output of the cross-attention layer is input into the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network. The output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

[0018] Optionally, the output of the first self-attention layer includes an output tensor determined based on the query index; the output tensor is determined using the attention weight calculation formula based on the query vector, key vector, and value vector corresponding to the query index.

[0019] Optionally, when the output tensor is determined according to the attention weight calculation formula, the first self-attention layer is provided with multiple attention heads; the output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimension of the attention heads, and processing the concatenation result based on a linear transformation matrix.

[0020] Optionally, the step of obtaining pre-extracted visual features of the target data using a contrastive language-image pre-trained model includes:

[0021] In response to the fact that the visual data is in the form of an image, the first image is input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features;

[0022] Alternatively, in response to the visual data being in video format, keyframe images of the visual data are acquired; the keyframe images are then input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features.

[0023] Optionally, obtaining the image-text matching result of the visual features and text features includes:

[0024] Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

[0025] Optionally, for using image-text feature pairs as input for image-text matching, the method further includes:

[0026] Divide the image and text feature pairs corresponding to the first dynamic data sample into at least three groups;

[0027] The first group is designated as the positive sample;

[0028] The text features of each image-text feature pair in the second group are replaced with the corresponding pseudo-text features, and the replaced second group is determined as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature;

[0029] The visual features of each image-text feature pair in the third group are replaced with the corresponding pseudo-visual features, and the replaced third group is determined as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature.

[0030] The positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

[0031] Optionally, the step of semi-supervising training of the classifier using the first dynamic data sample and a second dynamic data sample without classification labels, based on the feature extraction network, includes:

[0032] The visual and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample.

[0033] The classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function. The classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample. The second loss function is correspondingly set with control weights. The control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

[0034] Optionally, the control weights are adjusted based on the classification accuracy of the classifier;

[0035] When the classification accuracy is less than a first threshold, the control weight is set to 0; when the classification accuracy is greater than or equal to the first threshold and less than a second threshold, the control weight increases linearly with the classification accuracy and its maximum value is a preset hyperparameter; when the classification accuracy is greater than or equal to the second threshold, the control weight is set to the hyperparameter.

[0036] Optionally, during the semi-supervised training of the classifier, after the semi-supervised training is completed, the method further includes:

[0037] The evaluation set is used to determine the target classification categories that have a classification accuracy lower than a set threshold and where no performance evaluation metric meets the set threshold.

[0038] Based on the data sources of the evaluation set, determine the classification accuracy of each data source on the classifier;

[0039] Obtain a third dynamic data sample from the data source with the highest classification accuracy corresponding to the data source, and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label;

[0040] The classifier loss function is adjusted according to the target classification category, and the classifier is trained alternately based on the first dynamic data sample and the second dynamic data sample.

[0041] Optionally, obtaining the first dynamic data sample includes:

[0042] Retrieve first dynamic data with a first category label from a designated data storage system; the first category label indicates a preset category.

[0043] Second dynamic data with a second category label is obtained from the outside, the second category label is mapped to the preset category, and the second category label is replaced with a third category label corresponding to the mapped category.

[0044] The first dynamic data sample is generated based on the data distribution of each category in the first dynamic data and the second dynamic data.

[0045] In a second aspect of the present invention, a dynamic data classification method is provided, the method comprising:

[0046] Acquire dynamic data to be classified; the dynamic data includes visual data and text data;

[0047] The dynamic data to be classified is input into a dynamic data classification network to obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network described above.

[0048] Based on the predicted classification results, the classification result of the dynamic data to be classified is determined.

[0049] In a third aspect of the present invention, a training apparatus for a dynamic data classification network is provided, wherein the dynamic data includes visual data and text data, and the dynamic data classification network includes a feature extraction network and a classifier; the feature extraction network includes a visual feature extraction network and a text feature extraction network; the apparatus includes:

[0050] The sample acquisition module is used to acquire the first dynamic data sample with classification labels.

[0051] The feature extraction module is used to extract features from the first dynamic data sample using the visual feature extraction network and the text feature extraction network respectively, to obtain corresponding visual features and text features, wherein the visual feature extraction network and the text feature extraction network share the network parameters of the attention layer;

[0052] The feature extraction network training module is used to obtain the image-text matching results of the visual features and text features, and to adjust the network parameters of the feature extraction network according to the image-text matching results and the classification labels;

[0053] The classifier training module is used to perform semi-supervised training on the classifier based on the feature extraction network, using the first dynamic data sample and the second dynamic data sample without classification labels.

[0054] Optionally, the feature extraction module is specifically used for:

[0055] The visual data is converted into target data with a set encoding format, and a contrastive language-image pre-trained model is used to obtain pre-extracted visual features of the target data; the set encoding format is used to convert the visual data into data in the form of text characters;

[0056] The pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

[0057] Optionally, each layer of the visual feature extraction network includes at least a first self-attention layer, a cross-attention layer, and a first feedforward neural network; the parameters of the first self-attention layer include a query index; the feature extraction module, when inputting the pre-extracted visual features into the visual feature extraction network to obtain the corresponding visual features, may include:

[0058] For each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the output is obtained by the first self-attention layer processing the received query index.

[0059] The output of the cross-attention layer is input into the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network. The output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

[0060] Optionally, the output of the first self-attention layer includes an output tensor determined based on the query index; the output tensor is determined using the attention weight calculation formula based on the query vector, key vector, and value vector corresponding to the query index.

[0061] Optionally, the first self-attention layer is provided with multiple attention heads; the output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimension of the attention heads, and processing the concatenation result based on a linear transformation matrix.

[0062] Optionally, the feature extraction module, when used to obtain pre-extracted visual features of the target data using a contrastive language-image pre-trained model, may include:

[0063] In response to the fact that the visual data is in the form of an image, the first image is input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features;

[0064] Alternatively, in response to the visual data being in video format, keyframe images of the visual data are acquired; the keyframe images are then input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features.

[0065] Optionally, the feature extraction network training module is specifically used for:

[0066] Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

[0067] Optionally, the feature extraction network training module is specifically used for:

[0068] Divide the image and text feature pairs corresponding to the first dynamic data sample into at least three groups;

[0069] The first group is designated as the positive sample;

[0070] The text features of each image-text feature pair in the second group are replaced with the corresponding pseudo-text features, and the replaced second group is determined as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature;

[0071] The visual features of each image-text feature pair in the third group are replaced with the corresponding pseudo-visual features, and the replaced third group is determined as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature.

[0072] The positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

[0073] Optionally, the classifier training module is specifically used for:

[0074] The visual and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample.

[0075] The classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function. The classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample. The second loss function is correspondingly set with control weights. The control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

[0076] Optionally, the control weight is adjusted according to the classification accuracy of the classifier; when the classification accuracy is less than a first threshold, the control weight is 0; when the classification accuracy is greater than or equal to the first threshold and less than a second threshold, the control weight increases linearly with the classification accuracy and its maximum value is a preset hyperparameter; when the classification accuracy is greater than or equal to the second threshold, the control weight is the hyperparameter.

[0077] Optionally, the classifier training module further includes:

[0078] The evaluation set is used to determine the target classification categories that have a classification accuracy lower than a set threshold and where no performance evaluation metric meets the set threshold.

[0079] Based on the data sources of the evaluation set, determine the classification accuracy of each data source on the classifier;

[0080] Obtain a third dynamic data sample from the data source with the highest classification accuracy corresponding to the data source, and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label;

[0081] The classifier loss function is adjusted according to the target classification category, and the classifier is trained alternately based on the first dynamic data sample and the second dynamic data sample.

[0082] Optionally, the sample acquisition module is specifically used for:

[0083] Retrieve first dynamic data with a first category label from a designated data storage system; the first category label indicates a preset category.

[0084] Second dynamic data with a second category label is obtained from the outside, the second category label is mapped to the preset category, and the second category label is replaced with a third category label corresponding to the mapped category.

[0085] The first dynamic data sample is generated based on the data distribution of each category in the first dynamic data and the second dynamic data.

[0086] In a fourth aspect of the present invention, a dynamic data classification apparatus is provided, the apparatus comprising:

[0087] A dynamic data acquisition module is used to acquire dynamic data to be classified; the dynamic data includes visual data and text data.

[0088] The classification module is used to input the dynamic data to be classified into the dynamic data classification network and obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network described above.

[0089] The classification result acquisition module is used to determine the classification result of the dynamic data to be classified based on the predicted classification result.

[0090] In a fifth aspect of the present invention, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method or dynamic data classification method of the above-described dynamic data classification network.

[0091] In a sixth aspect of the present invention, a computing device is provided, comprising: a processor and a memory; the memory being used to store a computer program; and the processor being used to execute the training method or dynamic data classification method of the dynamic data classification network by invoking the computer program.

[0092] The embodiments of this invention utilize the separate extraction of single-modal features and set up a single-modal feature extraction network that shares the same self-attention layer to bridge the gap between visual and linguistic modalities. Combined with representation learning methods, this dynamic data classification network can accurately acquire feature information that bridges multiple modalities, achieving accurate fusion of multimodal information. At the same time, a semi-supervised algorithm is used to obtain an accurate classifier by training with data that has and does not have classification labels. This solves the problem of poor training effect when there is a small amount of sample data with classification labels, improves training efficiency, and meets the needs of information reviewers and users.

[0093] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0094] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0095] Figure 1A schematic diagram of the network framework of a dynamic data classification network according to an embodiment of the present invention is shown.

[0096] Figure 2 A flowchart illustrating a training method for a dynamic data classification network according to an embodiment of the present invention is shown schematically.

[0097] Figure 3A A flowchart illustrating an embodiment of the present invention for obtaining training samples with classification labels for a dynamic data classification network is shown.

[0098] Figure 3B The diagram illustrates a process for obtaining training samples with classification labels, using a dynamic data publishing platform as an example, according to an embodiment of the present invention.

[0099] Figure 4 A flowchart illustrating an embodiment of the present invention for obtaining visual features using a visual feature extraction network is shown schematically.

[0100] Figure 5A A schematic diagram of the network structure of a feature extraction network for a dynamic data classification network according to an embodiment of the present invention is shown.

[0101] Figure 5B A flowchart illustrating an embodiment of the present invention for obtaining visual features using a visual feature extraction network is shown schematically.

[0102] Figure 6 A flowchart illustrating an embodiment of the present invention for obtaining image-text matching results is shown schematically.

[0103] Figure 7 A flowchart illustrating a method for semi-supervised training of a classifier according to an embodiment of the present invention is shown schematically.

[0104] Figure 8 A flowchart illustrating a dynamic data classification method according to an embodiment of the present invention is shown schematically.

[0105] Figure 9 A block diagram of a training apparatus for a dynamic data classification network according to an embodiment of the present invention is shown schematically.

[0106] Figure 10 A block diagram of a dynamic data classification device according to an embodiment of the present invention is shown schematically;

[0107] Figure 11 A schematic diagram of a storage medium according to an embodiment of the present invention is shown.

[0108] Figure 12A schematic diagram of a computing device according to an embodiment of the present invention is shown.

[0109] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0110] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0111] Those skilled in the art will understand that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0112] According to embodiments of the present invention, a training method, apparatus, medium, and computing device for dynamic data classification networks and dynamic data classification are proposed.

[0113] It is important to understand in this article that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and that any naming is for distinction only and has no limiting meaning.

[0114] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0115] Before introducing the training method of the dynamic data classification network provided by this invention, a brief explanation of the key terms and background technology involved in this invention will be given first.

[0116] Multimodal: expressing or perceiving things from multiple modalities. Multimodal can include homogeneous modalities, such as images taken from two separate cameras, or heterogeneous modalities, such as pictures and text. Typically, the main modalities studied include "3V," namely Verbal, Vocal, and Visual. In this invention, multimodal refers to the visual and textual aspects included in dynamic data.

[0117] Multiple-instance learning (MIL): A special type of supervised learning method suitable for a specific data format called "multi-instance" data. In MILL, a sample is represented as a bag of instances, rather than a single sample. Each bag contains multiple instances, which can be either positive or negative. In traditional supervised learning tasks, each sample has an explicit label, while in MILL, the bag of instances has labels, but the labels within each instance may be unknown or partially known. The goal of MILL is to learn a model from the bags of instances that can classify new bags of instances.

[0118] Multilayer perceptron (MLP): A feedforward artificial neural network that maps a set of input vectors to a set of output vectors. It consists of multiple node layers, each fully connected to the next layer. Except for the input nodes, each node is a neuron with a non-linear activation function. The MLP is trained using a supervised learning method with backpropagation. As a generalization of the perceptron, it overcomes the weakness of the perceptron in recognizing linearly inseparable data and can achieve non-linear discriminative expressions.

[0119] Semi-supervised learning is a learning paradigm that lies between supervised and unsupervised learning. In semi-supervised learning, both labeled and unlabeled data are used simultaneously for model training. Labeled data consists of samples with explicit labels, while unlabeled data consists of samples without explicit labels. The goal of semi-supervised learning is to leverage the information from unlabeled data to augment labeled data, thereby improving learning performance and enhancing the model's generalization ability and overall performance.

[0120] CLIP (Contrastive Language-Image Pre-training) is an advanced multimodal pre-training model developed by OpenAI that uses contrastive learning to simultaneously understand the relationships between text and images. Chinese-CLIP is a multimodal pre-trained model based on 200 million native Chinese text-image pairs. Its training dataset includes Chinese subsets from LAION-5B, Chinese data from Wukong, and translated text-image data from COCO and VisualGenome. It can quickly perform text-image feature and similarity calculation, cross-modal retrieval, and zero-shot image classification in the Chinese domain.

[0121] Self-attention mechanism: It is used to focus on different positions in the input sequence and calculate the correlation between each position and all other positions. By weighted summation of the correlations between different positions, the contextual representation of each position can be obtained.

[0122] Cross-attention mechanism: used to handle the relationship between two different input sequences, it can associate information from one input sequence with another input sequence to extract the correlation between the two input sequences.

[0123] Self-ensemble methods: a type of semi-supervised learning method that improves model performance by using unlabeled data through multiple training and ensembles. The idea behind self-ensemble methods is to train multiple models using partially labeled and unlabeled data during the training process, and then integrate the prediction results of these models.

[0124] Dynamic data refers to a form of data that expresses a user's immediate emotions and thoughts through videos or images and corresponding text descriptions. The videos or images belong to visual modality information, while the text descriptions belong to text modality information. For example, on social media platforms, users can post dynamic content containing multiple images and related descriptions to showcase their lives or opinions. During the posting of dynamic data, data classification is necessary before reviewing the content to improve review efficiency. Since dynamic data includes information from different modalities, this data classification is also known as multimodal multi-label classification.

[0125] The methods for implementing multimodal multi-label classification in related technologies can include the following schemes:

[0126] Option 1: By introducing a deep neural network to learn the feature representation of dynamic data, using similarity constraints between modalities to simulate the intrinsic relationship between different modalities, and introducing supervised training of labeled multimodal data to capture label dependencies, multimodal multilabel classification is achieved.

[0127] However, the fusion effect of visual modality and text modality information of this scheme is poor. Moreover, this scheme performs well on the VOC2007 (Pascal Visual Object Class) dataset and the LabelMe dataset. It is suitable for target recognition tasks in images, but it is difficult to accurately integrate the information of the two modalities for data classification. Its model classification results have low accuracy.

[0128] Option 2: Utilizing the concept of multi-instance learning, data from different modalities is organized into multiple packets, each containing multiple instances. An end-to-end framework, M3DN (Multi-modal Multi-instance Multi-label Deep Network), is employed. Through a multi-instance packet processing layer, consistent representations and predictions at the packet level across different modalities are obtained. Simultaneously, based on optimal transport theory, the correlation between multiple labels is considered. Combining the packet-level prediction results from the model output, the classification label of each packet is inferred. Based on the classification label of each packet, different strategies, such as voting or averaging, are used to obtain the final classification label for the entire multimodal data.

[0129] However, the text modality of dynamic data includes data from a single instance, i.e., descriptive text. This approach uses multi-instance learning to eliminate inconsistencies in performance and unclear relationships between modalities. It is not suitable for data classification tasks of dynamic data. Furthermore, this approach cannot accurately integrate text modality and visual modality information, resulting in low accuracy of its classification results for dynamic data.

[0130] Option 3: Utilizing a multimodal convolutional neural network with a maximum marginal cross-correlation learning method, this approach leverages the correlation between class labels in later convolutional and fully connected layers to group labels and represent each image as a bag of visual instances. It obtains the instance correlation within a single image, combines the group descriptions to generate multimodal instances, and treats the group descriptions as contextual environments. This achieves the integration of visual and textual modalities. By utilizing these multimodal instances and contextual environments, and based on the learned features and label correlations, the corresponding multi-label output is obtained.

[0131] However, this scheme cannot accurately extract the high-level features required for each individual label, and its core lies in the correlation learning of multiple instance objects within a single image and the differentiation of visually similar objects. It has low correlation with text modal information, resulting in the inability to accurately combine visual and text information.

[0132] In summary, solutions based on related technologies are insufficient for accurately classifying data by comprehensively utilizing information from different modalities. This invention proposes a training method for a dynamic data classification network. By extracting features from each modality separately and setting a shared self-attention layer for these features, the gap between visual and linguistic modalities is bridged. Combined with representation learning methods, this dynamic data classification network can accurately acquire feature information that bridges multiple modalities, achieving accurate fusion of multimodal information. Simultaneously, a semi-supervised algorithm is used to train an accurate classifier using data with and without classification labels. This solves the problem of poor training performance when there is a shortage of labeled samples, improving training efficiency and meeting the needs of information reviewers and users.

[0133] Based on the dynamic data classification network trained by the above training method, the present invention also provides a multimodal data classification method. This classification method inputs the dynamic data to be classified into the trained dynamic data classification network, and obtains the classification result of the dynamic data according to the predicted classification result output by the dynamic data classification network, thereby realizing accurate classification of dynamic data using the feature fusion result of text modality and visual modality.

[0134] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0135] Exemplary methods

[0136] This invention provides a training method for dynamic data classification networks, see [link to relevant documentation]. Figure 1 The illustrated network structure of this dynamic data classification network includes at least a feature extraction network and a classifier. The feature extraction network extracts different modal features from the dynamic data, and the classifier takes the output of the feature extraction network as input to obtain the classification label of the dynamic data. Since the dynamic data includes both visual and textual data, the feature extraction network is divided into a visual feature extraction network and a textual feature extraction network, which are used to extract corresponding visual and textual features from the dynamic data, respectively. Both the visual and textual feature extraction networks include multi-layer neural network structures, and the self-attention layers within the same layer of the two extraction networks share their network parameters.

[0137] In this invention, the feature extraction network is first trained and its parameters are adjusted using a representation learning method. After the feature extraction network training is completed, the classifier is semi-supervised using first dynamic data samples with classification labels and second dynamic data samples without separate labels. Figure 2 As shown, the training method for the dynamic data classification network provided by this invention may include the following steps:

[0138] S201, Obtain the first dynamic data sample with classification labels;

[0139] The classification label is a preset mark or label used to identify the category to which the dynamic data sample belongs. It can be a number, text or other form of identification. Based on this method, a classification network for dynamic data is trained. The first dynamic data sample is sample data including visual data and text data with known classification results.

[0140] The first dynamic data sample can be obtained from a designated data storage system with classification labels or from external sources such as web crawlers, constructing a dataset for this first dynamic data sample as training samples. During the acquisition of data with classification labels, data acquisition can be based on the distribution of the classification categories to ensure a balanced distribution of sample data across different categories.

[0141] S202, the visual feature extraction network and the text feature extraction network are used to extract features from the first dynamic data sample to obtain corresponding visual features and text features, wherein the visual feature extraction network and the text feature extraction network share the network parameters of the attention layer.

[0142] The visual feature extraction network is used to extract corresponding visual features from dynamic data. After constructing the network, the network parameters of each layer are initialized. The network parameters of the self-attention layer include a query index, which guides the self-attention layer to learn how to focus on different parts of the input data. This query index, as a learnable weight matrix, guides the network's attention distribution in the self-attention layer, helping the network dynamically adjust the degree of attention to different parts of the input data according to the current task requirements. Specifically, during network parameter initialization, the query index in the first layer of the visual feature extraction network can be generated using random number generation methods such as uniformly distributed random numbers or normally distributed random numbers to create a random vector with specified dimensions and range requirements. The query indices in other layers besides the first layer are the output results of the previous layer of the visual feature extraction network.

[0143] The first dynamic data sample, or the visual data included in the first dynamic data sample, is input into the visual feature extraction network to obtain the visual features output from the last layer of the network. Alternatively, feature extraction can be performed on the visual data of the first dynamic data sample first, converting the visual data, such as images or videos in the dynamic data, into tensors, and then inputting the converted tensors into the visual feature extraction network. The conversion of visual data into tensors can be achieved using various libraries and tools, such as NumPy or TensorFlow in Python, or through feature extraction using the CLIP model. When using the CLIP model, the visual data from the first dynamic data sample can be directly input into the CLIP model, or the visual data can be first converted to a set encoding format, such as base64 encoding, and then the encoded data can be input into the CLIP model.

[0144] The text feature extraction network is used to learn text semantic information. After the construction of the text feature extraction network and the initialization of the network parameters are completed, the text data of the first dynamic data sample can be input into the text feature extraction network, or the text data can be concatenated into a long text and segmented into words, and the segmentation result can be input into the text feature extraction network, so that the text feature extraction network can use self-attention layers and other structural layers such as feedforward neural network layers to map the input of this layer into new text features, which are then sent to the next layer or used as the output of the text feature extraction network.

[0145] In this embodiment, the self-attention layer of the text feature extraction network shares network parameters with the corresponding self-attention layer of the visual feature extraction network. During training, the network parameters of the self-attention layer can learn the association information and common feature representations between visual data and text data. These network parameters can be applied to both visual and text data, enabling the feature extraction network of the dynamic data classification network to learn how to dynamically associate visual data and text data in the self-attention layer, thereby achieving effective interaction and fusion between the two.

[0146] S203, obtain the image-text matching result of the visual features and text features, and adjust the network parameters of the feature extraction network according to the image-text matching result and the classification label;

[0147] The image-text matching refers to using representation learning to map the visual features and text features to a shared low-dimensional representation space. Within this representation space, it determines whether the visual features and text features belong to the same topic or category, and outputs the predicted classification labels corresponding to the matching visual and text features. In this embodiment, the image-text matching can include any of the following representation learning tasks: image-text category matching, image-text comparison learning, or image-to-text generation.

[0148] The image-text matching result includes at least the predicted classification labels corresponding to the matched visual features and text features. Based on the difference between the predicted classification labels and the actual matching of the matched visual features and text features and their respective classification labels, the network parameters of the visual feature extraction network and the text feature extraction network are adjusted by minimizing the preset feature extraction loss function and backpropagation algorithm until the training convergence condition is met, such as the number of iterations reaching a set threshold or the loss function converging.

[0149] The feature extraction loss function can include the loss for correct matching of image features and text features, and the loss for incorrect matching. The weighted sum of the two is used as the loss function. The loss for correct matching refers to the loss when image features and text features match correctly and the predicted classification label is also correct. The loss for incorrect matching refers to the loss when image features and text features do not match correctly or the predicted classification label is incorrect. The loss for correct matching can use the cross-entropy loss function to measure the difference between the predicted classification label and the true classification label. The loss for incorrect matching can be set according to the matching score and the threshold.

[0150] S204, Based on the feature extraction network, the classifier is semi-supervised trained using the first dynamic data sample and the second dynamic data sample without classification labels.

[0151] The second dynamic data sample without classification labels may include data without classification labels obtained from a specified data storage system, or data with classification labels obtained from external sources and then processed to remove the classification labels, thereby constructing the second dynamic data sample.

[0152] After the feature extraction network is trained, the network parameters of the feature extraction network are frozen, and the classifier is trained using a semi-supervised learning method. The semi-supervised training steps are repeated until the performance of the classifier on the validation set converges or the loss function converges to a small value.

[0153] The classifier can be a multilayer perceptron, whose input consists of visual and textual features output from the feature extraction network. The two-dimensional features are concatenated and then input into the multilayer perceptron. The output dimension is adjusted to a preset number of categories, and the output is normalized using an activation function to obtain the output category. Alternatively, by increasing the number of classifier layers and adding skip connections between the encoder and decoder layers, the pressure on higher layers of the classifier to represent details can be reduced, allowing unsupervised and supervised learning to be combined.

[0154] In semi-supervised learning, the classifier can be initially trained using labeled first dynamic data samples through supervised learning to initialize the classifier's parameters and provide a good parameter foundation for subsequent semi-supervised learning.

[0155] Semi-supervised learning methods can include pseudo-labeling methods, which involve training a classifier on a second set of unlabeled dynamic data samples and using the classifier's predictions for each data sample as pseudo-labels. The classifier is then retrained after merging the pseudo-labeled second set of dynamic data samples with the first set of dynamic data samples. Self-training, as a specific pseudo-labeling method, uses a classifier trained on labeled data to predict the second set of unlabeled dynamic data samples. High-confidence predictions are used as pseudo-labels, which are then used to augment the labeled first set of dynamic data samples. This self-training process iterates, continuously updating the classifier and pseudo-labels.

[0156] In the embodiments of the present invention, visual feature extraction networks and text feature extraction networks are set up separately to extract visual features and text features respectively. The network parameters of the self-attention layer are set up for the two extraction networks to share the same network parameters. This allows the network parameters of the self-attention layer to learn the correlation information between visual modalities and text modalities during training, automatically adjust the degree of attention to different parts to better integrate different modal information, and maintain attention to the features of each modal information extracted by the extraction network itself. After the feature extraction network is trained by combining representation learning with a loss function, a semi-supervised algorithm is used to obtain an accurate classifier with a small amount of data. Thus, a dynamic data classification network that can bridge the feature information of multiple modalities and realize the effective interaction between visual features and text features is obtained.

[0157] Furthermore, training the feature extraction network and the classifier separately makes the entire dynamic data classification network more reusable and flexible. The same feature extraction network can be reused on different tasks, and only the classifier part needs to be adjusted to adapt to different tasks.

[0158] In some alternative implementations, such as Figure 3A As shown, the first dynamic data sample can be obtained in the following way:

[0159] S301, Obtain first dynamic data with a first classification label from a designated data storage system; the first classification label indicates a preset classification category;

[0160] The designated data storage system may include the data storage system of the business platform that requires dynamic data classification. This data storage system stores dynamic data with classification tags set in the historical period as well as dynamic data without classification tags set. The preset classification category refers to the category set by the business platform.

[0161] For the dynamic data stored in the specified data storage system, the dynamic data samples can be filtered and cleaned by constructing rules, such as deleting meaningless dynamic data or dynamic data containing prohibited words, and dividing the dynamic data according to whether it has a classification label. The dynamic data with classification labels is used as part of the first dynamic data sample, and the dynamic data without classification labels is used as part of the second dynamic data sample required for the semi-supervised training of the classifier.

[0162] S302, Obtain second dynamic data with a second classification label from the outside, map the second classification label to the preset classification category, and replace the second classification label with a third classification label corresponding to the mapped classification category;

[0163] The amount of dynamic data with category labels in the specified data storage system is small, and some categories have insufficient data or missing dynamic data with the category labels. At the same time, crawlers or other technologies can be used to obtain similar dynamic data from external platforms or devices outside the specified data storage system to supplement the data.

[0164] For externally acquired second dynamic data with a second category label, the second dynamic data can be filtered and cleaned first, and then the second category label can be mapped to the corresponding preset category in the specified data storage system. The category label of the mapped category can then be redefined as the category label of the second dynamic data, so that the second dynamic data can fall into the preset category.

[0165] S303, Generate the first dynamic data sample based on the data distribution of each category in the first dynamic data and the second dynamic data.

[0166] After obtaining the first dynamic data and the second dynamic data, the second dynamic data can be used to supplement each category according to the category distribution of the first classification label of the first dynamic data, so as to achieve a balanced distribution of dynamic data volume of each category. The dynamic data with classification labels after supplementing the categories is determined as the first dynamic data sample, which is used to train the feature extraction network.

[0167] In the embodiments of the present invention, by acquiring similar dynamic data from external sources and mapping it onto existing classification labels, the workload and cost of manual annotation are reduced, the diversity and richness of the original dataset are increased, the data imbalance problem is alleviated, the model's ability to identify each category is improved, and the model's generalization ability and performance are enhanced.

[0168] Based on the foregoing description of obtaining the first dynamic data sample and the second dynamic data sample, this embodiment uses a dynamic data publishing platform as an example to further illustrate the data sample acquisition.

[0169] See Figure 3B The training dataset acquisition process shown is divided into two stages: the acquisition and cleaning of on-site labeled data (i.e., dynamic data samples with classification labels in the dynamic data publishing platform), the construction of evaluation data, and the crawling and import of external data.

[0170] Phase 1: Acquisition, cleaning, and classification of on-site labeled data:

[0171] (1) Construct anti-spam and anti-blank rules through the backend database of the dynamic data publishing platform, collect all dynamic data in the past period and dynamic data samples with manually labeled classification tags; (2) Filter and clean the matched data by constructing rules to obtain enough dynamic data; (3) Separate the labeled data from the unlabeled data and analyze the category distribution of the labeled data to facilitate subsequent data supplementation.

[0172] For example, referring to the preset classification categories shown in Table 1, assuming the classification categories are divided into four major categories and twenty-two subcategories, with each subcategory belonging to one of the major categories:

[0173]

[0174]

[0175] Table 1

[0176] The category distribution in the dynamic data publishing platform refers to dividing the data with category labels in the publishing platform according to the twenty-two detailed categories and obtaining the number of labeled data samples in each detailed category.

[0177] Phase Two: Evaluation of Data Construction and External Data Crawling and Importation

[0178] (1) Extract 5-10 records from each detailed category in the labeled on-site dataset to construct an evaluation dataset; (2) Use web crawling technology to crawl labeled off-site data, i.e., labeled data from other social web pages or platforms outside the dynamic data publishing platform, and filter and clean the crawled data so that the labels of the off-site data can be mapped to the detailed categories shown in Table 1 above; (3) Combine the category distribution of the labeled on-site dataset with the off-site data to supplement it, so that the amount of data between each detailed category is as balanced as possible, and use the labeled data after the category is supplemented and balanced as the first dynamic data sample.

[0179] For the second dynamic data sample without category labels, the main data sample can be the unlabeled data from within the site, supplemented with some labeled data from outside the site. After removing the labels, the data sample can be cleaned and deduplicated to construct the second dynamic data sample.

[0180] In some alternative implementations, the dynamic data classification based on the present invention is a multimodal multi-label classification task. The dynamic data includes at least one piece of text information and multiple images, with relatively complex visual information that is difficult to integrate. To facilitate the analysis of the correlation between the visual and text modalities of the dynamic data and to better fuse data from different modalities, the visual feature extraction network and text feature extraction network are used respectively in step S202 to extract features from the first dynamic data sample, obtaining corresponding visual and text features. See [link to relevant documentation]. Figure 4 As shown, this can be achieved in the following way:

[0181] S401, the visual data is converted into target data with a set encoding format, and the pre-extracted visual features of the target data are obtained using the CLIP model; the set encoding format is used to convert the visual data into data in the form of text characters;

[0182] Converting visual data into text characters involves transforming pixel information in an image into text data, which can be achieved through encoding format conversion. The encoding format can include, but is not limited to, Base64, Base32, Base85, Base91, ASCII85, etc. Each encoding format has different applications in different scenarios, and the appropriate encoding format can be selected based on the characteristics of the data and transmission requirements.

[0183] Base64 encoding uses 64 characters to represent arbitrary binary data for transmitting text data over a network. It can convert image data into text character format for easier transmission and processing across various systems. In this embodiment, Base64 encoding is chosen. When converting the visual data (i.e., the image or video in the dynamic data) into Base64 encoded target data, the visual data is first converted into binary data, then Base64 encoded, ultimately yielding text character data.

[0184] When using the CLIP model to obtain pre-extracted visual features of target data with the set encoding format, if the visual data in the first dynamic data sample is in the form of images (i.e., multiple images), all images can be input into the CLIP model to obtain the pre-extracted visual features. Alternatively, since the first image in the dynamic data is representative, the first image can also be input into the CLIP model to obtain the pre-extracted visual features. If the visual data in the first dynamic data sample is in the form of video (i.e., a video clip), the keyframe images of the visual data are first obtained, and then the keyframe images are input into the CLIP model to obtain the pre-extracted visual features.

[0185] In this embodiment, since the high-order CLIP model and the basic parameter version CLIP model have the same feature extraction effect in this task and the processing time of the basic parameter version model is less, the basic parameter version CLIP model, that is, the CLIP model with fewer parameters and shallower network structure, is used to extract features from visual data.

[0186] S402, the pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

[0187] Before inputting the pre-extracted visual features into the visual feature extraction network, the pre-extracted visual features, the corresponding segmented text data, and the classification labels of the first dynamic data sample can be aligned. The aligned pre-extracted visual features and text data are then fed into their respective feature extraction networks. This alignment operation preprocesses the pre-extracted visual features and text data to ensure they have a consistent feature representation or input format, thus meeting the input requirements of the feature extraction network. For example, image features can be scaled, cropped, transformed, or normalized to ensure all image data have similar dimensions, color channels, and other features, enabling effective comparison and processing of images of different sizes or styles within the visual feature extraction network. Text data can be cropped to limit its length, either by truncating or padding it to a fixed length, ensuring a uniform input size and avoiding computational mismatches or difficulties caused by inconsistent text lengths.

[0188] In some embodiments, after obtaining the pre-extracted visual features, an image mask layer of all 1s can be generated based on the pre-extracted visual features, that is, a matrix of the same size as the pre-extracted visual features with all element values ​​of 1 is generated. The image mask layer is then passed as input to the self-attention layer and the cross-attention layer to instruct the visual feature extraction network to pay attention to all feature locations when processing image features, thereby initializing the network parameters of each layer of the feature extraction network based on the image mask layer.

[0189] In the embodiments of the present invention, by first converting the visual data into the encoding format of the target data, the computational complexity can be reduced and the risk of overfitting can be reduced. Based on the pre-extracted visual features obtained from the encoded data, the rich feature representations learned by the pre-trained model on large-scale data can be used to better capture the semantic information of the visual data, extract rich feature vectors, and achieve cross-modal learning between text and vision, providing a data foundation for the subsequent feature extraction network to understand the relationship between text description and visual content.

[0190] In some embodiments, see Figure 5A The network structure of the visual feature extraction network shown includes at least a first self-attention layer, a cross-attention layer, and a first feedforward neural network in each layer. The learnable parameters corresponding to the first self-attention layer include a query index. The query index in the first layer of the network is randomly initialized, while the query indices of other layers are the outputs of the corresponding preceding layer's first feedforward neural network. This query index indicates the degree of attention the self-attention layer of the visual feature extraction network learns to pay attention to different parts of the input data. Using this query index and the weight matrix of the self-attention layer, self-interaction of the self-attention layer can be achieved, enabling each element (token) in the input of the self-attention layer to learn the correlation with other tokens and its own areas of interest.

[0191] After obtaining the first dynamic data sample, the following can be used: Figure 4 The CLIP model shown can be used to extract pre-extracted visual features, or other methods can be used to extract corresponding pre-extracted visual features from images or videos in dynamic data. For example, large language models such as GPT-4 can be used to obtain the pre-extracted visual features. First, the pre-extracted visual features of the visual data in the first dynamic data sample are obtained. Based on the extracted pre-extracted visual features, the aforementioned step S202, which involves using a visual feature extraction network to obtain the corresponding visual features, means inputting the pre-extracted visual features into the visual feature extraction network to obtain the corresponding visual features. Based on the structure of the visual feature extraction network, such as... Figure 5B As shown, pre-extracted visual features can be input into the network and visual features can be obtained in the following way:

[0192] S501, for each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the output is obtained by the first self-attention layer processing the received query index;

[0193] In this embodiment, the input to the self-attention layer is the aforementioned query index, and the input to the cross-attention layer is the output of the self-attention layer and the pre-extracted visual features corresponding to the visual data in the dynamic data sample. The cross-attention layer can extract the correlation between the output and the pre-extracted visual features and input both into the cross-attention layer so that the cross-attention layer can interact with the two inputs to further learn the visual features.

[0194] The output of the self-attention layer is obtained based on the query index of this layer, and may include an output tensor determined based on the query index. The output tensor can be obtained by using the attention weight calculation formula based on the query vector, key vector and value vector corresponding to the query index. The corresponding query vector, key vector and value vector are obtained by linearly transforming the query index through the weight matrix of the self-attention layer.

[0195] The learnable parameters of this self-attention layer include at least the query index X and the weight matrix W. Q W K W V This learnable parameter is continuously optimized during the training process. Based on this learnable parameter, the output tensor is the output of the following attention weight formula (1):

[0196]

[0197] Where Q = XW Q K = XW K V = XW V d k The dimensions of Q and K are used to scale the attention size. Q represents the query vector, which measures the correlation between the current position and other positions; K represents the key vector, which provides information about other positions to calculate attention weights; and V represents the numerical vector, which contains information multiplied by the attention weights. The self-attention mechanism calculates the attention weights based on the relationship between the query vector Q, the key vector K, and the numerical vector V, which determines which parts of the input sequence the network should focus on when focusing.

[0198] Alternatively, in order to handle multiple concerns of query index and past feature information to comprehensively understand the input information, a self-attention mechanism with multiple attention heads can be adopted. That is, multiple attention heads are set in the first self-attention layer. The input of the self-attention layer is split into multiple sub-tensors. Each sub-tensor represents the attention calculation of a "head". Each attention head has an independently learned Q, K, V weight matrix. This independently learned weight matrix allows each attention head to focus on different parts of the input and learn different representations.

[0199] When multiple attention heads are set in the first self-attention layer, the above output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimension of the attention heads, and processing the concatenation result based on a linear transformation matrix.

[0200] That is, when calculating the output of the self-attention layer, attention weights are calculated for each attention head, and the input Value is weighted and summed using these weights to obtain the output of each head. Finally, the outputs of each head are concatenated to generate the final output tensor.

[0201] See the i-th attention head shown in formula (2) below. i The attention weight calculation formula is as follows: i takes the value [1, h]. The vector corresponding to the weight matrix Q, K, V under this attention head after linear transformation of the query index is obtained, and the attention weight of this attention head is calculated based on this vector.

[0202] head i =Attention(XW i Q XW i K XW i V (2)

[0203] After obtaining the self-attention weights of each attention head, the output tensor is obtained as shown in formula (3). The outputs of each attention head are concatenated according to the dimension of the attention head, and the concatenation result is processed based on the linear transformation matrix to generate the output tensor under the multi-attention head mechanism.

[0204] Multihead(X)=Concat(head1,...,head h W o (3)

[0205] Among them, the W o The weighting parameters are linearly varying and can be set according to actual needs.

[0206] S502, the output of the cross-attention layer is input to the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network, and the output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

[0207] After receiving the output of the first self-attention layer, the cross-attention layer interacts with the frozen pre-extracted visual features. The calculation of the attention weight of the cross-attention layer can be referred to the calculation process shown in the following formula (4):

[0208]

[0209] Where X′ is the output of the first self-attention layer, Y is the pre-extracted visual feature, and Q′ = XW′ Q K′=V′=XW′ K d r The dimension of the key-value vector K′ is used as a parameter in the denominator to scale the attention distribution.

[0210] The cross-attention layer first calculates the attention score matrix Q′K′. T Divide the attention score matrix by The attention score matrix is ​​scaled, and a softmax function is applied to the scaled matrix to obtain the attention weight distribution. This weight distribution is then used to perform a weighted summation of the value matrix (V') to obtain the final cross-attention output. Through these calculations, the cross-attention layer can cross-focus on two inputs.

[0211] The first feedforward neural network is used to perform non-linear transformation on the context representation of each position. By stacking multiple encoder layers, it can extract high-level abstract features of the input layer by layer. It can include at least fully connected layers and activation functions. The output of the cross-attention layer is used as the input of the first feedforward neural network. After being processed by the feedforward neural network, it is mapped to a new query index and sent to the next layer or used as the visual features output by the visual feature extraction network for subsequent training tasks.

[0212] In the embodiments of the present invention, based on the parameter sharing of the self-attention layer in the same layer of the visual feature extraction network and the text feature extraction network, the self-attention layer can learn visual and text information simultaneously during the training process. The input of the self-attention layer of the visual feature extraction network is set to the above-mentioned query index, and the input of the cross-attention layer is the output result of the self-attention layer and the pre-extracted visual features. This allows the visual features to be guided and adjusted by the text information, enhancing the correlation between vision and text. The acquired visual features have better representational ability, thereby enhancing the understanding ability of the feature extraction network in multimodal tasks.

[0213] In some alternative implementations, Figure 5AThe diagram also includes the network structure of a text feature extraction network. Each layer of this text feature extraction network may include at least a second self-attention layer and a second feedforward neural network layer. The second self-attention layer shares the network parameters of the first self-attention layer in the corresponding layer of the visual feature extraction network. Based on the structure of this text feature extraction network, the process of obtaining the corresponding text features using the text feature extraction network in step S202 can be achieved in the following way:

[0214] The text information is concatenated into a long text and then segmented into words to obtain the word segmentation results.

[0215] The word segmentation result is input to the second self-attention layer, so that the second self-attention layer determines the output result based on the query vector, key vector and value vector corresponding to the word segmentation result using the attention weight calculation formula and transmits it to the second feedforward neural network.

[0216] The output of the second feedforward neural network is transmitted to the second self-attention layer of the next layer of the text feature extraction network, and the output of the second feedforward neural network in the last layer of the text feature extraction network is used as the text feature.

[0217] In the embodiments of the present invention, by introducing network parameters of a second self-attention layer and a shared self-attention layer, joint training of the visual modality and the text modality is achieved, making the text feature extraction network more adaptable to complex semantic tasks and improving the performance and generalization ability of the model.

[0218] In some alternative implementations, the image-text matching result of obtaining the visual features and text features described in step S203 above can be achieved in the following ways:

[0219] Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

[0220] That is, for each dynamic data in the first dynamic data sample, its visual features and corresponding text features constitute an image-text feature pair. This image-text feature pair is used as the input for image-text matching. This image-text matching can predict whether the visual features and text features in the input image-text feature pair match, and output the corresponding matching prediction result. It can also use the visual features and text features that predict the match as a whole to predict the classification label, and output the corresponding classification label prediction result. Based on the actual matching situation and the true classification label result, the loss function is calculated.

[0221] Alternatively, in some alternative implementations, such as Figure 6As shown, the image-text matching result of obtaining the visual features and text features described in step S203 above can also be achieved in the following ways:

[0222] S601, the above-mentioned image and text feature pairs corresponding to the first dynamic data sample are divided into at least three groups;

[0223] The first dynamic data sample is divided into at least three batches, each batch containing a certain number of dynamic data samples. The above-mentioned image and text feature pairs corresponding to the dynamic data samples in the batch constitute a group.

[0224] S602, the first group is determined as a positive sample;

[0225] Since the image-text feature pair includes the visual features and the text features belonging to the same dynamic data, each image-text feature pair in any group is a corresponding match between the visual features and the text features, and any group with such a corresponding match is determined as a positive sample.

[0226] S603, replace the text features of each image-text feature pair in the second group with the corresponding pseudo-text features, and determine the replaced second group as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature;

[0227] S604, replace the visual features of each image-text feature pair in the third group with the corresponding pseudo-visual features, and determine the replaced third group as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature;

[0228] For the remaining groups other than the first group, for each image-text feature pair included in one half of the remaining groups, the matching text feature is replaced with another text feature that has the highest similarity to that text feature. Similarly, for each image-text feature pair included in the other half of the remaining groups, the matching image feature is replaced with another image feature that has the highest similarity to that image feature, thus obtaining the negative sample group.

[0229] S605, the positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

[0230] After obtaining the image-text matching result through any of the above methods, the cross-entropy loss function can be calculated by comparing the predicted classification label output by the image-text matching with the true classification label, and the network parameters of the feature extraction network can be adjusted through the backpropagation algorithm to minimize the loss function.

[0231] It is understandable that, for the two implementation methods of obtaining image-text matching results mentioned above, at least one of the methods can be used to obtain the image-text matching results to complete the training of the feature extraction network, that is, only one method can be used or the two methods can be combined.

[0232] In an embodiment of the present invention, by dividing the image and text features corresponding to the first dynamic data sample into different groups and introducing pseudo-text features and pseudo-visual features as negative samples, and using pseudo-text features and pseudo-visual features with high similarity as negative samples, the network parameters of the feature extraction network can be adjusted based on the image and text matching results of the sample. This can guide the feature extraction network to learn the association between text features and visual features more accurately, enabling the network to better capture the semantic relationship between text and image, improve the expressive power of features, and effectively improve the generalization ability of the model and reduce the risk of overfitting by introducing different types of negative samples.

[0233] In some alternative implementations, for the feature extraction network described in step S204 above, the classifier is semi-supervised trained using the first dynamic data sample and the second dynamic data sample without classification labels. Before the semi-supervised training, the classifier is pre-initialized, which can be done using random initialization or dynamic data samples with classification labels for initial training. After initialization, see [link to relevant documentation]. Figure 7 As shown, semi-supervised training can be achieved through the following steps:

[0234] S701, the visual features and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample;

[0235] Since the second dynamic data sample does not have a classification label, pseudo-classification labels are generated using the prediction results of the classifier on the second dynamic data sample.

[0236] Among them, the pseudo-classification labeling method shown in the following formula (5) can determine the classification category with the highest probability output by the classifier as the pseudo-classification label of the second dynamic data sample.

[0237]

[0238] Among them, f i (x) represents the probability that the classifier's output predicts the classifier's class label.

[0239] Alternatively, for generating pseudo-labels, a self-integration method can be used, which integrates the same model at different iterations and under different data augmentation and regularization conditions to construct better pseudo-labels.

[0240] S702, the classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function; the classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample; the second loss function is correspondingly set with control weights; the control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

[0241] After obtaining the second dynamic data sample with pseudo-classification labels, the first dynamic data sample and the second dynamic data sample are input into the aforementioned feature extraction network to obtain the corresponding visual features and text features. The visual features and text features can be concatenated and then fed into the classifier. The first loss function and the second loss function are calculated according to the output of the classifier, and the network parameters of the classifier are adjusted by minimizing the loss function.

[0242] In this embodiment, the idea of ​​cross-entropy regularization can be used to transform the second dynamic data sample without classification labels into the regularization term of the classifier loss function. That is, data with pseudo-classification labels are directly regarded as data with classification labels. Cross-entropy is used to evaluate the error magnitude, and the errors of the two parts of data are combined by controlling the weight parameter. For example, the classifier loss function can be as shown in formula (6):

[0243]

[0244] Where n is the batch size of the first dynamic data sample with classification labels, n′ is the batch size of the second dynamic data sample with pseudo-classification labels, C is the output class, and α(t) is the control weight corresponding to the unlabeled data sample. The above formula can be divided into two parts: the left side of the plus sign is the first loss function for the data with classification labels, and the right side of the plus sign is the second loss function for the pseudo-labeled data. The specific loss function used, the pseudo-cross-entropy loss function, is shown in formula (7):

[0245] L(y i ,f i )=-y i logf i -(1-y i log(1-f) i (7)

[0246] In this module, the control weights determine the role of unlabeled data costs in network updates. Initially, when the classifier is introduced, its classification accuracy is low, and the generated pseudo-classification labels are also quite noisy; therefore, the control weights need to be set to 0. During training, the classification accuracy continuously improves, and the control weights can be adjusted based on the classification accuracy. For example, see the following formula (8), where the control weights exhibit a linear relationship with the classification accuracy:

[0247]

[0248] Wherein, variable t represents the classification accuracy of the classifier, T1 and T2 are the first and second thresholds of the classification accuracy, respectively, with the first threshold being less than the second threshold, and α f The hyperparameter is a pre-set parameter, and its value can be 0.5.

[0249] That is, when the classification accuracy is less than the first threshold, the control weight is 0; when the classification accuracy is greater than or equal to the first threshold and less than the second threshold, the control weight increases linearly with the classification accuracy and its maximum value is a preset hyperparameter; when the classification accuracy is greater than or equal to the second threshold, the control weight is the hyperparameter.

[0250] In the embodiments of this invention, the concept of cross-entropy regularization is used to transform second dynamic data samples without classification labels into data with pseudo-classification labels. This reduces the amount of data that needs to be manually labeled, saving manpower and time costs, and expanding the scale of the training dataset. Data with pseudo-classification labels is directly regarded as data with classification labels. Cross-entropy is used to evaluate the magnitude of the error. By controlling the weight parameters, the influence of pseudo-classification label data is adjusted, continuously increasing the loss ratio of pseudo-label data, guiding the classifier to improve training accuracy, and reducing the impact of noise. This allows a relatively accurate classifier to be trained with less labeled data. At the same time, combining the training methods of labeled and unlabeled data can not only reduce the risk of overfitting but also improve the accuracy and robustness of classification.

[0251] In some alternative implementations, after the training in step S702 is completed and the classifier's performance is evaluated using the evaluation set, if any performance evaluation metric fails to meet a set threshold, the method may further include the following supplementary training step, which includes:

[0252] (1) Use the evaluation set to determine the target classification category where the classification accuracy is lower than a set threshold and any performance evaluation index does not meet the set index threshold; (2) Determine the classification accuracy of each data source on the classifier according to the data source of the evaluation set; (3) Obtain the third dynamic data sample of the data source with the highest classification accuracy corresponding to the data source and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label; (4) Adjust the classifier loss function according to the target classification category and train the classifier alternately according to the first dynamic data sample and the second dynamic data sample.

[0253] That is, for each category whose classification accuracy is lower than the set threshold, the performance evaluation index value of each category is calculated first. The performance evaluation index may include at least one of precision, recall and F1 score. At the same time, the data source of the evaluation set is used as a different dimension to calculate the overall accuracy of each data source on the classifier. The data source may include supervised data (evaluation data of the labels to be classified) in the site (specified data storage system), supervised data from outside the site (externally obtained), and supervised data combining in-site and out-of-site data.

[0254] Since the multi-dimensional performance evaluation metrics of the classifier did not meet the standards, the training set was resampled. The high classification accuracy of the data source indicates that the data from that data source has good class discrimination. Therefore, the first dynamic data sample with classification labels was supplemented by introducing data from the data source type with the highest overall accuracy. At the same time, the classification accuracy of the categories below the set threshold was included as part of the loss function in the classifier training. The classifier was further trained based on the supplemented first dynamic data and second dynamic data samples to further improve the classification accuracy.

[0255] In the embodiments of the present invention, the classification effect of the classifier is evaluated by adjusting the distribution of classification categories and combining multi-source and multi-dimensional indicators. The classifier is then resampled and further trained based on the evaluation results. During the training process, the classification categories with relatively poor performance are included as part of the loss function, which makes the classification accuracy higher and more consistent with the real situation.

[0256] Based on the training method of the dynamic data classification network described in the foregoing embodiments, this invention also provides a dynamic data classification method. This method is applicable to various classification scenarios involving visual data and text data. See [link to relevant documentation]. Figure 8 As shown, this classification method may include the following steps:

[0257] S801, Obtain the dynamic data to be classified; the dynamic data includes visual data and text data;

[0258] S802, the dynamic data to be classified is input into the dynamic data classification network to obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network described above;

[0259] S803, Based on the predicted classification result, determine the classification result of the dynamic data to be classified.

[0260] In the embodiments of the present invention, the classification network trained based on the aforementioned training method for dynamic data classification networks can accurately extract and characterize the multimodal features of dynamic data. By using this classification network to classify the dynamic data to be classified, the dynamic data can be classified quickly and accurately, thereby better meeting the needs of information reviewers and users, improving user experience and satisfaction, and providing more effective support for applications such as information review.

[0261] The execution order of the steps in the above-described process is not limited to the order shown in the flowchart. Furthermore, the description of each step can be implemented in software, hardware, or a combination thereof. For example, those skilled in the art can implement it as software code, which can be computer-executable instructions capable of implementing the logical functions corresponding to the steps. When implemented in software, the executable instructions can be stored in memory and executed by the processor in the system.

[0262] Exemplary device

[0263] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 9 An apparatus according to an exemplary embodiment of the present invention will be described.

[0264] The specific implementation process of the functions and roles of each module in the following apparatus can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here. For the apparatus embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments.

[0265] Figure 9 A training apparatus for a dynamic data classification network according to an embodiment of the present invention is schematically illustrated. The dynamic data includes visual data and text data, and the dynamic data classification network includes a feature extraction network and a classifier. The feature extraction network includes a visual feature extraction network and a text feature extraction network. The apparatus includes:

[0266] The sample acquisition module 910 is used to acquire the first dynamic data sample with classification labels;

[0267] The feature extraction module 920 is used to extract features from the first dynamic data sample using the visual feature extraction network and the text feature extraction network respectively, to obtain corresponding visual features and text features, wherein the visual feature extraction network and the text feature extraction network share the network parameters of the attention layer.

[0268] The feature extraction network training module 930 is used to obtain the image-text matching results of the visual features and text features, and to adjust the network parameters of the feature extraction network according to the image-text matching results and the classification labels;

[0269] The classifier training module 940 is used to perform semi-supervised training on the classifier based on the feature extraction network, using the first dynamic data sample and the second dynamic data sample without classification labels.

[0270] In some alternative implementations, the feature extraction module is specifically used for:

[0271] The visual data is converted into target data with a set encoding format, and a contrastive language-image pre-trained model is used to obtain pre-extracted visual features of the target data; the set encoding format is used to convert the visual data into data in the form of text characters;

[0272] The pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

[0273] In some alternative implementations, each layer of the visual feature extraction network includes at least a first self-attention layer, a cross-attention layer, and a first feedforward neural network; the parameters of the first self-attention layer include a query index; the feature extraction module, when inputting the pre-extracted visual features into the visual feature extraction network to obtain the corresponding visual features, may include:

[0274] For each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the output is obtained by the first self-attention layer processing the received query index.

[0275] The output of the cross-attention layer is input into the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network. The output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

[0276] In some alternative implementations, the output of the first self-attention layer includes an output tensor determined based on the query index; the output tensor is determined using an attention weight calculation formula based on the query vector, key vector, and value vector corresponding to the query index.

[0277] In some alternative implementations, the first self-attention layer is provided with multiple attention heads; the output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimensions of the attention heads, and processing the concatenation result based on a linear transformation matrix.

[0278] In some alternative implementations, the feature extraction module, when used to obtain pre-extracted visual features of the target data using a contrastive language-image pre-trained model, may include:

[0279] In response to the fact that the visual data is in the form of an image, the first image is input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features;

[0280] Alternatively, in response to the visual data being in video format, keyframe images of the visual data are acquired; the keyframe images are then input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features.

[0281] In some alternative implementations, the feature extraction network training module is specifically used for:

[0282] Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

[0283] In some alternative implementations, the feature extraction network training module is specifically used for:

[0284] Divide the image and text feature pairs corresponding to the first dynamic data sample into at least three groups;

[0285] The first group is designated as the positive sample;

[0286] The text features of each image-text feature pair in the second group are replaced with the corresponding pseudo-text features, and the replaced second group is determined as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature;

[0287] The visual features of each image-text feature pair in the third group are replaced with the corresponding pseudo-visual features, and the replaced third group is determined as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature.

[0288] The positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

[0289] In some alternative implementations, the classifier training module is specifically used for:

[0290] The visual and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample.

[0291] The classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function. The classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample. The second loss function is correspondingly set with control weights. The control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

[0292] In some alternative implementations, the control weights are adjusted based on the classification accuracy of the classifier; when the classification accuracy is less than a first threshold, the control weights are set to 0; when the classification accuracy is greater than or equal to the first threshold and less than a second threshold, the control weights increase linearly with the classification accuracy and the maximum value is a preset hyperparameter; when the classification accuracy is greater than or equal to the second threshold, the control weights are set to the hyperparameter.

[0293] In some alternative implementations, the classifier training module further includes:

[0294] The evaluation set is used to determine the target classification categories that have a classification accuracy lower than a set threshold and where no performance evaluation metric meets the set threshold.

[0295] Based on the data sources of the evaluation set, determine the classification accuracy of each data source on the classifier;

[0296] Obtain a third dynamic data sample from the data source with the highest classification accuracy corresponding to the data source, and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label;

[0297] The classifier loss function is adjusted according to the target classification category, and the classifier is trained alternately based on the first dynamic data sample and the second dynamic data sample.

[0298] In some alternative implementations, the sample acquisition module is specifically used for:

[0299] Retrieve first dynamic data with a first category label from a designated data storage system; the first category label indicates a preset category.

[0300] Second dynamic data with a second category label is obtained from the outside, the second category label is mapped to the preset category, and the second category label is replaced with a third category label corresponding to the mapped category.

[0301] The first dynamic data sample is generated based on the data distribution of each category in the first dynamic data and the second dynamic data.

[0302] See Figure 10 As shown, the present invention also provides a dynamic data classification device, the device comprising:

[0303] The dynamic data acquisition module 1010 is used to acquire dynamic data to be classified; the dynamic data includes visual data and text data.

[0304] The classification module 1020 is used to input the dynamic data to be classified into the dynamic data classification network and obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network described above.

[0305] The classification result acquisition module 1030 is used to determine the classification result of the dynamic data to be classified based on the predicted classification result.

[0306] Exemplary media

[0307] After introducing the methods and apparatus of exemplary embodiments of the present invention, the following references are made. Figure 11 A readable storage medium according to an exemplary embodiment of the present invention will be described.

[0308] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disc read-only memory (CD-ROM) containing program code, which can run on a device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0309] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0310] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0311] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RE, etc., or any suitable combination thereof.

[0312] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0313] Exemplary computing device

[0314] After introducing the methods, apparatus, and media of exemplary embodiments of the present invention, the following references are made. Figure 12 A computing device according to an exemplary embodiment of the present invention will be described.

[0315] Figure 12The computing device 1200 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0316] like Figure 12 As shown, the computing device 1200 is presented in the form of a general-purpose computing device. The components of the computing device 1200 may include, but are not limited to: at least one processing unit 1201, at least one storage unit 1202, and a bus 1203 connecting different system components (including the processing unit 1201 and the storage unit 1202).

[0317] Bus 1203 includes a data bus, a control bus, and an address bus.

[0318] Storage unit 1202 may include readable media in the form of volatile memory, such as random access memory (RAM) 12021 and / or cache memory 12022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 12023.

[0319] Storage unit 1202 may also include a program / utility 12025 having a set (at least one) of program modules 12024, such program modules 12024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0320] The computing device 1200 can also communicate with one or more external devices 1204 (such as a keyboard, pointing device, etc.).

[0321] This communication can be performed via input / output (I / O) interface 1205. Furthermore, the computing device 1200 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1206. Figure 12 As shown, network adapter 1206 communicates with other modules of computing device 1200 via bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0322] It should be noted that although several units / modules or sub-units / modules of the training of the dynamic data classification network and the dynamic data classification device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0323] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0324] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A training method for a dynamic data classification network, characterized in that, The dynamic data includes visual data and text data, and the dynamic data classification network includes a feature extraction network and a classifier. The feature extraction network includes a visual feature extraction network and a text feature extraction network; the method includes: Obtain the first dynamic data sample with category labels; The visual feature extraction network and the text feature extraction network are used to extract features from the first dynamic data sample to obtain corresponding visual features and text features. The visual feature extraction network and the text feature extraction network share the network parameters of the self-attention layer. Each layer of the visual feature extraction network includes at least a first self-attention layer, a cross-attention layer and a first feedforward neural network. Obtain the image-text matching results of the visual features and text features, and adjust the network parameters of the feature extraction network based on the image-text matching results and the classification labels; Based on the feature extraction network, the classifier is semi-supervised trained using the first dynamic data sample and the second dynamic data sample without classification labels; The visual features were extracted in the following way: For each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the pre-extracted visual features are obtained by feature extraction of the visual data of the first dynamic data sample, and the output is obtained by the first self-attention layer processing the received query index; The output of the cross-attention layer is input into the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network. The output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

2. The method according to claim 1, characterized in that, The step of extracting features from the first dynamic data sample using the visual feature extraction network and the text feature extraction network respectively to obtain corresponding visual features and text features includes: The visual data is converted into target data with a set encoding format, and a contrastive language-image pre-trained model is used to obtain pre-extracted visual features of the target data; the set encoding format is used to convert the visual data into data in the form of text characters; The pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

3. The method according to claim 1, characterized in that, The output of the first self-attention layer includes an output tensor determined based on the query index; the output tensor is determined using the attention weight calculation formula based on the query vector, key vector, and value vector corresponding to the query index.

4. The method according to claim 3, characterized in that, The first self-attention layer is configured with multiple attention heads; the output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimensions of the attention heads, and processing the concatenation result based on a linear transformation matrix.

5. The method according to claim 2, characterized in that, The process of obtaining pre-extracted visual features of the target data using a contrastive language-image pre-trained model includes: In response to the fact that the visual data is in the form of an image, the first image is input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features; Alternatively, in response to the visual data being in video format, keyframe images of the visual data are acquired; the keyframe images are then input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features.

6. The method according to claim 1, characterized in that, The step of obtaining the image-text matching result of the visual features and text features includes: Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

7. The method according to claim 6, characterized in that, The method further includes: Divide the image and text feature pairs corresponding to the first dynamic data sample into at least three groups; The first group is designated as the positive sample; The text features of each image-text feature pair in the second group are replaced with the corresponding pseudo-text features, and the replaced second group is determined as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature; The visual features of each image-text feature pair in the third group are replaced with the corresponding pseudo-visual features, and the replaced third group is determined as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature. The positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

8. The method according to claim 1, characterized in that, The step of semi-supervised training of the classifier using the first dynamic data sample and the second dynamic data sample without classification labels, based on the feature extraction network, includes: The visual and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample. The classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function. The classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample. The second loss function is correspondingly set with control weights. The control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

9. The method according to claim 8, characterized in that, The control weights are adjusted based on the classification accuracy of the classifier; If the classification accuracy is less than the first threshold, the control weight is set to 0. When the classification accuracy is greater than or equal to the first threshold and less than the second threshold, the control weight value increases linearly with the classification accuracy and the maximum value is a preset hyperparameter. When the classification accuracy is greater than or equal to the second threshold, the control weight is taken as the hyperparameter.

10. The method according to claim 8, characterized in that, After the training is completed, the method further includes: The evaluation set is used to determine the target classification categories that have a classification accuracy lower than a set threshold and where no performance evaluation metric meets the set threshold. Based on the data sources of the evaluation set, determine the classification accuracy of each data source on the classifier; Obtain a third dynamic data sample from the data source with the highest classification accuracy corresponding to the data source, and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label; The classifier loss function is adjusted according to the target classification category, and the classifier is trained alternately based on the first dynamic data sample and the second dynamic data sample.

11. The method according to claim 1, characterized in that, The process of obtaining the first dynamic data sample with classification labels includes: Retrieve first dynamic data with a first category label from a designated data storage system; the first category label indicates a preset category. Second dynamic data with a second category label is obtained from the outside, the second category label is mapped to the preset category, and the second category label is replaced with a third category label corresponding to the mapped category. The first dynamic data sample is generated based on the data distribution of each category in the first dynamic data and the second dynamic data.

12. A dynamic data classification method, characterized in that, The method includes: Acquire dynamic data to be classified; the dynamic data includes visual data and text data; The dynamic data to be classified is input into a dynamic data classification network to obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network according to any one of claims 1-11; Based on the predicted classification results, the classification result of the dynamic data to be classified is determined.

13. A training device for a dynamic data classification network, characterized in that, The dynamic data includes visual data and text data, and the dynamic data classification network includes a feature extraction network and a classifier. The feature extraction network includes a visual feature extraction network and a text feature extraction network; the device includes: The sample acquisition module is used to acquire the first dynamic data sample with classification labels. The feature extraction module is used to extract features from the first dynamic data sample using the visual feature extraction network and the text feature extraction network respectively, to obtain corresponding visual features and text features. The visual feature extraction network and the text feature extraction network share the network parameters of the self-attention layer. Each layer of the visual feature extraction network includes at least a first self-attention layer, a cross-attention layer, and a first feedforward neural network. The feature extraction network training module is used to obtain the image-text matching results of the visual features and text features, and to adjust the network parameters of the feature extraction network according to the image-text matching results and the classification labels; The classifier training module is used to perform semi-supervised training on the classifier based on the feature extraction network, using the first dynamic data sample and the second dynamic data sample without classification labels. The visual features were extracted in the following way: For each layer of the visual feature extraction network, the output of the first self-attention layer and the pre-extracted visual features are input to the cross-attention layer; the pre-extracted visual features are obtained by feature extraction of the visual data of the first dynamic data sample, and the output is obtained by the first self-attention layer processing the received query index; The output of the cross-attention layer is input into the first feedforward neural network, and the output of the first feedforward neural network is transmitted to the next layer of the visual feature extraction network. The output of the first feedforward neural network in the last layer is determined as the corresponding visual feature.

14. The apparatus according to claim 13, characterized in that, The feature extraction module is specifically used for: The visual data is converted into target data with a set encoding format, and the pre-extracted visual features of the target data are obtained using a contrastive language-image pre-trained model. The specified encoding format is used to convert the visual data into text character data; The pre-extracted visual features are input into the visual feature extraction network to obtain the corresponding visual features.

15. The apparatus according to claim 13, characterized in that, The output of the first self-attention layer includes an output tensor determined based on the query index; the output tensor is determined using the attention weight calculation formula based on the query vector, key vector, and value vector corresponding to the query index.

16. The apparatus according to claim 15, characterized in that, The first self-attention layer is configured with multiple attention heads; the output tensor is obtained by concatenating the outputs of the multiple attention heads corresponding to the query index according to the dimensions of the attention heads, and processing the concatenation result based on a linear transformation matrix.

17. The apparatus according to claim 14, characterized in that, The feature extraction module, when used to obtain pre-extracted visual features of the target data using a contrastive language-image pre-trained model, includes: In response to the fact that the visual data is in the form of an image, the first image is input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features; Alternatively, in response to the visual data being in video format, keyframe images of the visual data are acquired; the keyframe images are then input into the contrastive language-image pre-trained model to obtain the pre-extracted visual features.

18. The apparatus according to claim 13, characterized in that, The feature extraction network training module is specifically used for: Using image-text feature pairs as input for image-text matching, the image-text matching prediction result of the image-text feature pairs and the prediction result of the image-text category label are obtained; the image-text feature pairs include the visual features and the text features that belong to the same dynamic data.

19. The apparatus according to claim 18, characterized in that, The feature extraction network training module is specifically used for: Divide the image and text feature pairs corresponding to the first dynamic data sample into at least three groups; The first group is designated as the positive sample; The text features of each image-text feature pair in the second group are replaced with the corresponding pseudo-text features, and the replaced second group is determined as the first negative sample; the pseudo-text feature refers to the other text feature in the second group that has the highest similarity to the text feature; The visual features of each image-text feature pair in the third group are replaced with the corresponding pseudo-visual features, and the replaced third group is determined as the second negative sample; the pseudo-visual feature refers to the other visual feature in the third group that has the highest similarity to the visual feature. The positive sample, the first negative sample, and the second negative sample are concatenated and used as the input for image-text matching.

20. The apparatus according to claim 13, characterized in that, The classifier training module is specifically used for: The visual and text features of the second dynamic data sample extracted by the feature extraction network are input into the classifier, and the classification output by the classifier is determined as the pseudo-classification label of the second dynamic data sample. The classifier is trained using the first dynamic data sample and the second dynamic data sample with pseudo-classification labels, and the network parameters of the classifier are adjusted by minimizing a preset classifier loss function. The classifier loss function includes a first loss function for the first dynamic data sample and a second loss function for the second dynamic data sample. The second loss function is correspondingly set with control weights. The control weights are used to indicate the weight of the second dynamic data sample in adjusting the network parameters of the classifier.

21. The apparatus according to claim 20, characterized in that, The control weights are adjusted based on the classification accuracy of the classifier; If the classification accuracy is less than the first threshold, the control weight is set to 0. When the classification accuracy is greater than or equal to the first threshold and less than the second threshold, the control weight value increases linearly with the classification accuracy and the maximum value is a preset hyperparameter. When the classification accuracy is greater than or equal to the second threshold, the control weight is taken as the hyperparameter.

22. The apparatus according to claim 20, characterized in that, The classifier training module also includes: The evaluation set is used to determine the target classification categories that have a classification accuracy lower than a set threshold and where no performance evaluation metric meets the set threshold. Based on the data sources of the evaluation set, determine the classification accuracy of each data source on the classifier; Obtain a third dynamic data sample from the data source with the highest classification accuracy corresponding to the data source, and incorporate it into the first dynamic data sample; the third dynamic data sample has a classification label; The classifier loss function is adjusted according to the target classification category, and the classifier is trained alternately based on the first dynamic data sample and the second dynamic data sample.

23. The apparatus according to claim 13, characterized in that, The sample acquisition module is specifically used for: Retrieve first dynamic data with a first category label from a designated data storage system; the first category label indicates a preset category. Second dynamic data with a second category label is obtained from the outside, the second category label is mapped to the preset category, and the second category label is replaced with a third category label corresponding to the mapped category. The first dynamic data sample is generated based on the data distribution of each category in the first dynamic data and the second dynamic data.

24. A dynamic data classification device, characterized in that, The device includes: A dynamic data acquisition module is used to acquire dynamic data to be classified; the dynamic data includes visual data and text data. A classification module is used to input the dynamic data to be classified into a dynamic data classification network and obtain the predicted classification result output by the dynamic data classification network; wherein, the dynamic data classification network is trained using the training method of the dynamic data classification network according to any one of claims 1-11; The classification result acquisition module is used to determine the classification result of the dynamic data to be classified based on the predicted classification result.

25. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method as described in any one of claims 1 to 11 or the method as described in claim 12.

26. A computing device, characterized in that, include: Processor, memory; The memory is used to store computer programs; The processor is configured to execute the method as described in any one of claims 1 to 11 or the method as described in claim 12 by invoking the computer program.

Images