A visual language object classification method based on cross-modal dense attention

Abstract

The application discloses a visual language object classification method based on cross-modal dense attention, and the method comprises the following steps: acquiring an image to be processed containing at least one object to be classified and text information to be processed, wherein the image to be processed matches the text information to be processed; inputting the image to be processed and the text information to be processed into a target object classification model which is pre-trained to obtain the category of each object to be classified; wherein the target object classification model comprises two encoders, a sequence attention module, a cross-modal attention module and a classification module. The technical scheme of the embodiment of the application realizes the effect of accurately classifying the objects to be classified contained in the image to be processed, alleviates the semantic gap problem between different modal data by adopting the sequence attention module, further improves the precision of the multi-modal classification task, and achieves the effect of improving the robustness of the model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visual language object classification method based on cross-modal dense attention. Background Technology

[0002] Currently, research has been conducted on using artificial intelligence technology to classify objects in images, thereby meeting users' needs for classifying target objects.

[0003] When using relevant models to classify target objects in practice, existing target object classification methods are based on only a single modality of object data, such as images, text, or speech containing the object. The disadvantage of this classification method is that it cannot make full use of the target object's data in various modalities and cannot establish the relationship between different modal data, resulting in low classification accuracy and slow classification speed. Summary of the Invention

[0004] This invention provides a visual language object classification method based on cross-modal dense attention, so as to improve the accuracy of multimodal fine-grained classification.

[0005] According to one aspect of the present invention, a visual language object classification method based on cross-modal dense attention is provided, the method comprising:

[0006] Obtain an image to be processed and text information to be processed that contain at least one object to be classified, wherein the image to be processed matches the text information to be processed;

[0007] The image to be processed and the text information to be processed are input into a pre-trained target object classification model to obtain the category to which each object to be classified belongs;

[0008] The target object classification model includes two encoders, a sequence attention module, a cross-modal attention module, and a classification module.

[0009] The technical solution of this invention obtains a processing image and processing text information containing at least one object to be classified, inputs the processing image and processing text information into a pre-trained target object classification model, and obtains the category to which each object belongs. This solves the problem in the prior art that classification is based on a single modality of the object, which cannot fully utilize the data of the object in various modalities and cannot establish the relationship between different modal data, resulting in low classification accuracy and slow classification speed. It achieves the effect of accurately classifying the objects to be classified contained in the processing image. By adopting a sequence attention module, it alleviates the semantic gap problem between different modal data, further improves the accuracy of multimodal classification tasks, and achieves the effect of improving the robustness of the model.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a visual language object classification method based on cross-modal dense attention according to Embodiment 1 of the present invention;

[0013] Figure 2 This is a schematic diagram of the image to be processed according to Embodiment 1 of the present invention;

[0014] Figure 3 This is a flowchart of a visual language object classification method based on cross-modal dense attention according to Embodiment 1 of the present invention;

[0015] Figure 4 This is a flowchart of a visual language object classification method based on cross-modal dense attention according to Embodiment 2 of the present invention;

[0016] Figure 5 This is a flowchart of a visual language object classification method based on cross-modal dense attention according to Embodiment 2 of the present invention;

[0017] Figure 6 This is a flowchart of a visual language object classification method based on cross-modal dense attention provided in Embodiment 3 of the present invention;

[0018] Figure 7 This is a flowchart of a training method for a classification model of an object to be trained according to Embodiment 3 of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Example 1

[0022] Figure 1 This is a flowchart of a visual language object classification method based on cross-modal dense attention, provided in Embodiment 1 of the present invention. This embodiment is applicable to classifying objects contained in images and text information to be processed. The method can be executed by a visual language object classification device based on cross-modal dense attention, which can be implemented in hardware and / or software and can be configured in a terminal and / or server. Figure 1 As shown, the method includes:

[0023] S110. Obtain the image to be processed and the text information to be processed, which contain at least one object to be classified.

[0024] In this embodiment, the image to be processed can be an image received by the server or client and captured in real time by the user through a camera device, or it can be a stored image retrieved by the server or client from a relevant database. The image may include one or more objects; these objects are the objects to be classified. Of course, in practical applications, the objects to be classified can be any object, such as animals or people, and the number of objects to be classified in the image can be one or more; this embodiment does not specifically limit this. The text information to be processed can be a textual description of the objects to be classified. The images to be processed and the text information to be processed are matched; that is, images and text information containing the same objects to be classified can be obtained in pairs. For example, as shown... Figure 2 As shown, when the image to be processed is Figure 2 When the image shown is displayed, the corresponding text information to be processed can be "Banana BreadPancakes with Cinnamon Cream Cheese Syrup Cooking Classy".

[0025] It should be noted that in specific application scenarios, the image and text information to be processed can be acquired in real time or periodically. Alternatively, when a user uploads an image or text information, the image and text information can be acquired, and the image can be used as the image to be processed and the text information can be used as the text information to be processed. This embodiment does not impose any specific limitations on this.

[0026] It should also be noted that the number of images to be processed and text information to be processed can be one or more sets. That is, the technical solution provided by the embodiments of the present invention can process one set of images to be processed and text information to be processed, or it can process multiple sets of images to be processed and text information to be processed at the same time. This embodiment does not make specific limitations on this.

[0027] In practical applications, when there are multiple objects to be classified, all objects in the image to be processed can be considered as objects to be classified, and the corresponding text information to be processed can be determined based on the image. When there is only one object to be classified, the object to be classified in the image can be pre-labeled, and the labeled image can be uploaded to the server so that the server can store the feature attributes of the object to be classified. After acquiring multiple images to be processed, when at least one of these images is detected to contain the feature attributes of the object to be classified, the object in the current image is identified as the object to be classified, and the corresponding text information to be processed is determined based on the image. This allows the neural network model of this embodiment to process the image to be processed and the text information to be processed, and determine the category to which the object to be classified belongs.

[0028] S120. Input the image to be processed and the text information to be processed into the pre-trained target object classification model to obtain the category of each object to be classified.

[0029] In this embodiment, after acquiring the image and text information to be processed, they can be input into a pre-trained target object classification model. The model then processes the image and text information. The target object classification model can be a pre-trained neural network model for target object recognition and classification. For example, it can be a deep learning network model trained based on an attention mechanism, including encoder and Transformer modules. The target object classification model includes two encoders, a sequence attention module, a cross-modal attention module, and a classification module.

[0030] The encoder can be a neural network model that processes image or text information into corresponding feature dimensions. Optionally, the encoder can include an image encoder and a text encoder. In this embodiment, the image encoder can be a neural network model for implementing image encoding. For example, the image encoder can be a deep residual convolutional neural network model, such as a ResNet50 network. The text encoder can be a neural network model for extracting features from text information. For example, the text encoder can be a Transformer model based on BPE encoding, such as an 8-head attention, 12-layer, 512-dimensional Transformer.

[0031] The sequence attention module can be a neural network module that determines the fine-grained relationship between image features and text features. For example, the sequence attention module can be a BERT (Bidirectional Encoder Representation from Transformers) model, a fine-grained attention feature fusion module based on a multi-layer Transformer structure, such as a 12-head attention, 12-layer, 768-dimensional BERT model. Those skilled in the art will understand that the BERT model is a pre-trained language representation model that, by employing a novel masked language model (MLM), can generate deep bidirectional language representations.

[0032] Among them, the cross-modal attention module can be a module that fuses feature information. It can achieve the effect of suppressing noisy modes and highlighting dominant modes through feature fusion of different modes.

[0033] The classification module can be a neural network model that determines the category of an object based on feature information. For example, the classification module can be a Multi-Layer Perceptron (MLP) neural network. It should be noted that when the image contains multiple objects to be classified (i.e., a multi-label classification task with multiple correct labels), a sigmoid activation function can be applied to the fully connected layers in the classification module, and the binary cross-entropy loss function can be used for training. For multi-class classification tasks, where each image contains only one object to be classified and multiple images are classified simultaneously, a softmax activation function can be applied to the fully connected layers in the classification module, and the cross-entropy loss function can be used for training.

[0034] In practical applications, after acquiring the image and text information containing the object to be classified, these can be input into the target object classification model. First, feature extraction is performed on the image and text information using two encoders, yielding corresponding image and text features. Then, a sequence attention module enhances the image and text features, resulting in enhanced image and text features. Next, a cross-modal attention module fuses the enhanced image and text features. Finally, a classification module processes the fused features to obtain the final classification result for the object to be classified. For example, when the image to be processed is... Figure 2 When the image shown is the largest object in the image, its corresponding category can be dessert; when the image includes both people and animals, and there are multiple objects to be classified, its corresponding category can be human and animal.

[0035] For example, such as Figure 3 As shown, after acquiring the image and text information to be processed, features can be extracted. Then, the extracted image and text features are aligned based on contrastive learning. Next, cross-modal feature enhancement is performed on the processed image and text features based on the sequence attention module. Furthermore, the enhanced image and text features are fused based on the cross-modal attention module. Finally, the fused features are processed based on the classification module to obtain the category of the object to be classified.

[0036] The technical solution of this invention obtains a processing image and processing text information containing at least one object to be classified, inputs the processing image and processing text information into a pre-trained target object classification model, and obtains the category to which each object belongs. This solves the problem in the prior art that classification is based on a single modality of the object, which cannot fully utilize the data of the object in various modalities and cannot establish the relationship between different modal data, resulting in low classification accuracy and slow classification speed. It achieves the effect of accurately classifying the objects to be classified contained in the processing image. By adopting a sequence attention module, it alleviates the semantic gap problem between different modal data, further improves the accuracy of multimodal classification tasks, and achieves the effect of improving the robustness of the model.

[0037] Example 2

[0038] Figure 4 This is a flowchart of a visual language object classification method based on cross-modal dense attention provided in Embodiment 2 of the present invention. Based on the foregoing embodiments, S120 is further refined, and its specific implementation can be found in the technical solution of this embodiment. Technical terms that are the same as or similar to those in the above embodiments will not be repeated here.

[0039] like Figure 4 As shown, the method includes:

[0040] S210. Obtain the image to be processed and the text information to be processed, which contain at least one object to be classified.

[0041] S220. Based on two encoders, the image to be processed and the text information to be processed are processed respectively to obtain the features of the image to be processed and the features of the text to be processed.

[0042] In this embodiment, since the information input to the target object classification model consists of different modalities, namely image information and text information, an image encoder and a text encoder can be used to process the image and text information respectively when encoding them based on the encoder. The image features can be vectors representing the feature information of the image after feature extraction. Similarly, the text features can be vectors representing the feature information of the text information after feature extraction.

[0043] In practical applications, after the image and text information to be processed are input into the target object classification model, due to the different extracted feature information, they can be processed by different encoders to obtain the image features and text features.

[0044] Optionally, the image to be processed and the text information to be processed are processed based on two encoders respectively to obtain the image features to be processed and the text features to be processed, including: extracting the image features of the image to be processed based on the image encoder to obtain the image features to be processed; and extracting the text features of the text information to be processed based on the text encoder to obtain the text features to be processed.

[0045] In practical implementation, an image encoder extracts features from the image to be processed, thereby obtaining image features containing the feature information of the image to be processed. Simultaneously, a text encoder extracts features from the text to be processed, thereby obtaining text features containing the feature information of the text to be processed. For example, when the image encoder is a ResNet50 network, the feature vector of the last layer of the network can be used as the image features during feature extraction; when the text encoder is a Transformer model based on BPE encoding, the feature vector of the highest layer in the Transformer can be used as the text features during feature extraction.

[0046] It should be noted that the image features to be processed and the text features to be processed can be feature vectors of the same dimension. In this embodiment, the image features to be processed and the text features to be processed can be represented by a matrix.

[0047] It should also be noted that after obtaining the image features and text features to be processed, these two features can be normalized to achieve stable forward input distribution and faster model convergence.

[0048] S230. Perform cross-modal alignment processing on the image features and text features to be processed respectively to obtain the image features and text features to be applied.

[0049] In this embodiment, the image encoder and text encoder can be deployed in the same module. After obtaining the image features and text features to be processed, since these two types of features are feature vectors from different modalities, they can be aligned to make them more similar. This alignment process can be called cross-modal alignment, or cross-modal alignment pre-training based on contrastive learning. For example, the module containing the image encoder and text encoder can be a CLIP model. The image features to be applied can be the feature vectors obtained after cross-modal alignment of the image features to be processed. The text features to be applied can be the feature vectors obtained after cross-modal alignment of the text features to be processed.

[0050] In practical applications, after obtaining the image features and text features to be processed based on the image encoder and text encoder, these two features can be pre-trained to map them to a unified feature space, so that the image features and text features to be processed can be better matched and the similarity between the two features can be enhanced.

[0051] In practical implementation, the image features and text features to be processed can be based on the open-source dataset in the CLIP model. This open-source dataset contains several pre-matched image-text pairs. First, feature extraction is performed on each image-text pair in this open-source dataset to obtain the corresponding image feature vector matrix E. Image and text feature vector matrix E Text Then, the image feature vector matrix E can be calculated using matrix multiplication. Image With text feature vector matrix E Text The cosine similarity is calculated using the cross-entropy loss function to calculate the loss values ​​of the image feature vector matrix and the text feature vector matrix respectively. The two loss values ​​are added together and then averaged to obtain the target loss value, which can be used for pre-training. Further, the cosine similarity between the image features and the text features to be processed is calculated, and the probability distributions predicted after mapping the image to the text space and the probability distributions after mapping the text to the image space are obtained through Softmax. These two probability distributions can be optimized to make them closer to the real distributions corresponding to the open source dataset. Based on the optimized probability distributions, the image features and text features to be applied are obtained.

[0052] S240. The sequence attention module processes the image features and text features to be applied to obtain the image features and text features to be used.

[0053] In this embodiment, after obtaining the image features and text features to be applied, they can be input into the sequence attention module. Based on the multi-layer Transformer structure in the sequence attention module, the image features and text features to be applied are processed to obtain the image features and text features to be used.

[0054] Optionally, the sequence attention module processes the image features and text features to be applied to obtain the image features and text features to be used, including: concatenating the image features and text features to be applied to obtain concatenated features to be processed; inputting the concatenated features to be processed into the sequence attention module to obtain concatenated features to be used; and segmenting the concatenated features to be used to obtain the image features and text features to be used.

[0055] In practical applications, after obtaining the image features and text features to be applied, these two types of features can be concatenated to obtain concatenated features to be processed. Then, the concatenated features to be processed are input into the sequence attention module. Since the Transformer is the core structure of the sequence attention module, and the attention mechanism is the most critical part of the Transformer, the concatenated features to be processed can be processed based on the attention mechanism to obtain the concatenated features to be used. Those skilled in the art will understand that the attention mechanism mainly involves three feature spaces: query space, key space, and value space. The concatenated features to be processed can be projected into these three feature spaces respectively, and the concatenated features to be processed can be updated based on the projection matrices of these three feature spaces to obtain the concatenated features to be used. Finally, since the image features and text features to be applied are feature vectors of the same dimension, after obtaining the concatenated features to be used, they can be segmented based on the dimension of the feature to finally obtain image features and text features to be used of the same dimension. The advantages of this setup are: it allows for the acquisition of fine-grained relationships between image features and text features, alleviates the semantic gap between feature data from different modalities, improves the multimodal feature space, and enhances the accuracy of multimodal classification tasks.

[0056] For example, suppose X∈R s×d Let X be the concatenated features to be processed, where s is the number of image features and text features to be applied, and d is the dimension of the concatenated features to be processed; projecting X onto the query space, key space, and value space respectively, we can obtain Q = XW. Q K = XW K and V=XW V Where Q represents the concatenation feature to be processed in the query space, and W... Q This represents the projection matrix applied to project X into the query space, where K is the concatenation feature to be processed in the key space, and W... K Let V represent the projection matrix applied to project X into the bond space, V be the splicing feature to be processed in the value space, and W be the projection matrix applied to project X into the bond space. V This represents the projection matrix applied to project X into the value space. Furthermore, X is updated based on Q, K, and V, and the updated X can be... Among them, QK T Let d represent the similarity matrix. k The update coefficients are used to ensure the stability of the softmax gradient. Finally, the updated spliced ​​features to be processed, i.e. the spliced ​​features to be used, are segmented to obtain the image features and text features to be used.

[0057] S250. Based on the cross-modal attention module, the image features and text features to be used are fused to obtain the fused features to be classified.

[0058] In this embodiment, after obtaining the image features and text features to be used, they can be input into the cross-modal attention module to perform fusion processing on the image features and text features to be used, thereby obtaining the fused features to be classified.

[0059] Optionally, the cross-modal attention module is used to fuse the image features and text features to be used to obtain the fused features to be classified, including: determining a first weight value and a second weight value corresponding to the image features and text features to be used respectively; multiplying the image features to be used with the first weight value to obtain a first fused feature; multiplying the text features to be used with the second weight value to obtain a second fused feature; and adding the first fused feature and the second fused feature to obtain the fused features to be classified.

[0060] In practical applications, after inputting the image features and text features to be used into the cross-modal attention module, in order to suppress noisy modalities and highlight the dominant modalities, a weighted processing of the image features and text features to be used can be performed using keyless attention. Specifically, a first weight value and a second weight value corresponding to the image features and text features to be used are determined respectively. Then, the image features to be used and the first weight value are multiplied to obtain the first feature to be fused. The text features to be used are multiplied with the second weight value to obtain the second feature to be fused. Finally, the first feature to be fused and the second feature to be fused are added together to obtain the fused feature to be classified.

[0061] For example, I′ [CLS] As the image feature to be used, T′ [CLS] The first weight value, as the text feature to be used, can be determined based on the following formula:

[0062] e I =w T I' [CLS]

[0063] e T =w T T′ [CLS]

[0064]

[0065] Among them, w T It can represent the transpose of the learnable parameter vector, λ can represent the first weight value, I′ [CLS] For the image features to be used, T′[CLS] The text feature to be used can be exp, which can be used for exponential calculation.

[0066] Furthermore, the features to be classified and fused can be determined based on the following formula:

[0067] c=λI′ [CLS] +(1-λ)T′ [CLS]

[0068] Where c can represent the feature to be classified and fused.

[0069] S260. Based on the classification module, the features to be classified and fused are processed to obtain the category to which each object to be classified belongs.

[0070] In practical applications, after obtaining the fusion features to be classified, they can be input into the classification module. Based on the processing of the fusion features by the classification module, the category of each object to be classified contained in the image and text information to be processed can be obtained.

[0071] It should be noted that the activation function can be determined based on the number of objects to be classified contained in the image or text information to be processed.

[0072] For example, such as Figure 5 As shown, it can be Figure 2 The image shown is used as the image to be processed to illustrate the above data processing process: First, the image to be processed and the text information to be processed are input into the target object classification model. Based on the image encoder, feature extraction is performed on the image to be processed to obtain the image features. At the same time, based on the text encoder, feature extraction is performed on the text information to be processed to obtain the text features. Then, the image features and text features to be processed are aligned based on contrastive learning to obtain the image features I to be used. [CLS] and the text features to be used E [CLS] Furthermore, based on the sequence attention module, I [CLS] and E [CLS] After processing, the image features I′ to be used can be obtained. [CLS] and the text features T′ to be used [CLS] Then, based on the cross-modal attention module, I′ [CLS] and T′ [CLS] Feature fusion is performed, and finally, the fused features are processed based on the classification module to obtain the category to which each object to be classified belongs.

[0073] The technical solution of this invention involves acquiring a processing image and processing text information containing at least one object to be classified, then processing the processing image and processing text information separately using two encoders to obtain processing image features and processing text features. Next, cross-modal alignment processing is performed on the processing image features and processing text features to obtain application image features and application text features. Then, the application image features and application text features are processed using a sequence attention module to obtain usage image features and usage text features. Further, the usage image features and usage text features are fused using a cross-modal attention module to obtain classification fused features. Finally, the classification fused features are processed using a classification module to obtain the category to which each object belongs. This achieves the effect of accurately classifying the objects to be classified contained in the processing image. By employing a sequence attention module, the semantic gap problem between different modal data is alleviated, further improving the accuracy of multimodal classification tasks and enhancing the robustness of the model.

[0074] Example 3

[0075] Figure 6 This is a flowchart of a visual language object classification method based on cross-modal dense attention provided in Embodiment 3 of the present invention. Based on the foregoing embodiments, a target object classification model can be pre-trained to classify the objects to be identified from the image and text information to be processed. Specific implementation methods can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0076] like Figure 6 As shown, the method includes:

[0077] S310. Determine at least one training sample.

[0078] The training samples include: training sample images and training sample text information. The training sample images and training sample text information include the object to be classified and the theoretical output category corresponding to the training sample images and training sample text information.

[0079] The training sample images can be images captured by a camera, images reconstructed by an image reconstruction model, or images pre-stored in storage. Each image includes one or more objects, which can be used as the objects to be classified. The training sample text information can be textual descriptions of the corresponding training sample images. The theoretical output category can be the theoretical category to which each object in the image to be classified belongs.

[0080] Specifically, before training the classification model for the objects to be classified, multiple training samples can be obtained to train the model. It should be noted that to improve the model's accuracy, as many and varied training samples as possible should be obtained. Multiple training sample images containing the objects to be classified, along with corresponding training sample text information, are acquired. Furthermore, these training sample images and text information are processed to obtain the theoretical output categories corresponding to the training sample images and text information, thus constructing a rich set of training samples based on the above method.

[0081] S320. Input the training samples into the classification model of the object to be trained to obtain the first actual output category.

[0082] It should be noted that for each training sample, the S320 method can be used to train it, thereby obtaining the target object classification model.

[0083] The model parameters in the object classification model to be trained can be set to default values. The model parameters are then corrected using training samples to obtain the target object classification model. The first actual output category can be the classification category of the object to be classified after inputting the training sample images and training sample text information into the object classification model to be trained.

[0084] In this embodiment, the object classification model to be trained includes two encoders, a sequence attention module, a cross-modal attention module, and a classification module.

[0085] In practical applications, after inputting training sample images and training sample text information into the classification model to be trained, features can be extracted from the training sample images and training sample text information through different encoders to obtain the first actual image features and the first actual text features. Then, based on the sequence attention module, feature enhancement processing is performed on the first actual image features and the first actual text features to obtain the second actual image features and the second actual text features. Further, based on the cross-modal attention module, feature fusion processing is performed on the second actual image features and the second actual text features to obtain the actual fused features. Finally, based on the classification module, the actual fused features are processed to obtain the first actual output category.

[0086] S330. Perform loss processing on the first actual output category and the theoretical output category according to the pre-set joint loss function.

[0087] Optionally, the joint loss function is determined based on the cross-entropy loss function and the class center loss function.

[0088] For example, the joint loss function can be determined based on the following formula:

[0089] L uni =αL ce +(1-α)L cct

[0090] Among them, L uni It can represent the joint loss function, L ce It can represent the cross-entropy loss function, L cct It can represent the class center loss function, and α can represent the balance factor, the value of which can be set to 0.5.

[0091] In practical applications, to improve the prediction performance of fine-grained classification tasks, a loss function can be applied to the first actual output class and the theoretical output class based on the joint loss function obtained by weighted fusion of the two loss functions.

[0092] It should be noted that the joint loss function can be determined based on any two loss functions, and this embodiment does not impose any specific limitations on it.

[0093] S340. Based on the first loss value, adjust the parameters of the sequence attention module, cross-modal attention module, and classification module to obtain the classification model of the object to be processed.

[0094] The model structure of the object classification model to be processed is the same as that of the object classification model to be trained.

[0095] It should be noted that when initializing the model parameters of the object classification model to be trained, the CLIP parameters can be used as the initial parameters for both encoders, while the parameters in the sequence attention module, cross-modal attention module, and classification module are randomly initialized. Since the object classification model to be trained contains both CLIP parameters and parameters initialized at any time, some modules may overfit or underfit during the model parameter adjustment process. Based on this, a multi-stage training method can be used to train the entire model.

[0096] The first loss value can be the difference between the first actual output category and the theoretical output category.

[0097] In practical applications, when using the first loss value to correct the parameters of the sequence attention module, cross-modal attention module, and classification module in the training classification model, the initial parameters of the two encoders can be fixed, and only the parameters of the above three modules can be adjusted. Specifically, the convergence of the joint loss function can be used as the training objective, such as whether the training error is less than a preset error, whether the error change tends to stabilize, or whether the current number of iterations is equal to the preset number. If the convergence condition is met, such as the training error of the joint loss function being less than the preset error, or the error change trend tending to stabilize, it indicates that the training of the object classification model is complete, and iterative training can be stopped. If the convergence condition is not met, other training samples can be obtained to continue training the object classification model until the training error of the joint loss function is within the preset range. When the training error of the joint loss function converges, the current object classification model can be used as the object classification model to be processed, thus enabling the next training stage to train the object classification model to be processed.

[0098] S350. Input the training samples into the object classification model to obtain the second actual output category.

[0099] It should be noted that for each training sample, the S350 method can be used to train it, thereby obtaining the target object classification model.

[0100] The second actual output category can be the classification category of the object to be classified after the training sample images and training sample text information are input into the object classification model.

[0101] It should be noted that since the model structure of the object classification model to be processed and the object classification model to be trained are the same, the process of processing the training samples based on the classification model to be processed to obtain the second actual output category is the same as the process in S320, and will not be described in detail here.

[0102] S360. Based on the joint loss function, perform loss processing on the second actual output category and the theoretical output category.

[0103] In practical applications, the loss can be applied to the first actual output category and the theoretical output category based on the joint loss function obtained by weighted fusion of the two loss functions.

[0104] S370. Based on the second loss value, adjust the model parameters of the object classification model to obtain the target object classification model.

[0105] The second loss value can be the difference between the second actual output category and the theoretical output category.

[0106] In practical applications, when adjusting the model parameters of the classification model using the second loss value, the initial parameters of the two encoders can be unfrozen. This allows for individual adjustment of the encoder and module parameters during the model parameter tuning process. Specifically, the convergence of the joint loss function can be used as a training objective, such as whether the training error is less than a preset error, whether the error change tends to stabilize, or whether the current number of iterations equals a preset number. If convergence conditions are met, such as the training error of the joint loss function being less than the preset error, or the error change trend stabilizing, it indicates that the classification model for the object being processed has been trained successfully, and iterative training can be stopped. If convergence conditions are not met, other training samples can be obtained to continue training the classification model until the training error of the joint loss function is within a preset range. When the training error of the joint loss function converges, the trained classification model for the object being processed can be used as the target object classification model. That is, by inputting the image and text information containing the object to be classified into the target object classification model, the category of the object to be classified can be accurately obtained.

[0107] For example, such as Figure 7 The training process of the object classification model is shown below: 1. Initialize the model parameters of the object classification model; 2. Freeze the parameters of the two encoders (such as the image encoder and the text encoder); 3. Adjust the parameters of the sequence attention module, the cross-modal attention module and the classification module; 4. Unfreeze the parameters of the two encoders; 5. Adjust the parameters of the entire model.

[0108] S380. Obtain the image to be processed and the text information to be processed, which contain at least one object to be classified.

[0109] S390. Input the image to be processed and the text information to be processed into the pre-trained target object classification model to obtain the category of each object to be classified.

[0110] The technical solution of this invention involves training a target object classification model to obtain a trained target object classification model. Then, based on this target object classification model, the image to be processed and the text information to be processed, which contain at least one object to be identified, are processed to obtain the category of each object to be identified. This achieves the effect of accurately classifying the objects to be classified contained in the image to be processed. By employing a sequence attention module, the semantic gap between different modal data is alleviated, further improving the accuracy of multimodal classification tasks and enhancing the robustness of the model.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A visual language object classification method based on cross-modal dense attention, characterized in that, include: Obtain an image to be processed and text information to be processed that contain at least one object to be classified, wherein the image to be processed matches the text information to be processed; The image to be processed and the text information to be processed are input into a pre-trained target object classification model to obtain the category to which each object to be classified belongs; The target object classification model includes two encoders, a sequence attention module, a cross-modal attention module, and a classification module. The step of inputting the image to be processed and the text information to be processed into a pre-trained target object classification model to obtain the category of each object to be classified includes: The image to be processed and the text information to be processed are processed by two encoders respectively to obtain the features of the image to be processed and the features of the text to be processed. Cross-modal alignment processing is performed on the image features to be processed and the text features to be processed respectively to obtain the image features to be applied and the text features to be applied. The sequence attention module processes the image features to be applied and the text features to be applied to obtain the image features to be used and the text features to be used. The cross-modal attention module is used to fuse the image features and text features to be used to obtain the fused features to be classified. The classification module processes the fusion features to be classified to obtain the category of each object to be classified.

2. The method according to claim 1, characterized in that, The two encoders include an image encoder and a text encoder. The process of processing the image to be processed and the text information to be processed based on the two encoders respectively to obtain image features and text features to be processed includes: Based on the image encoder, image features of the image to be processed are extracted to obtain the image features to be processed; and, The text features of the text to be processed are obtained by extracting the text features of the text information to be processed based on the text encoder.

3. The method according to claim 1, characterized in that, The process of processing the image features and text features to be applied based on the sequence attention module to obtain the image features to be used and the text features to be used includes: The image features to be applied and the text features to be applied are concatenated to obtain the concatenated features to be processed. The splicing features to be processed are input into the sequence attention module to obtain the splicing features to be used; The splicing features to be used are segmented to obtain the image features to be used and the text features to be used.

4. The method according to claim 1, characterized in that, The process of fusing the image features and text features to be used based on the cross-modal attention module to obtain the fused features to be classified includes: Determine a first weight value and a second weight value corresponding to the image feature to be used and the text feature to be used, respectively; The image feature to be used is multiplied by the first weight value to obtain the first feature to be fused; The text feature to be used is multiplied by the second weight value to obtain the second feature to be fused; The first feature to be fused and the second feature to be fused are added together to obtain the feature to be classified and fused.

5. The method according to claim 1, characterized in that, Also includes: Determine at least one training sample, wherein the training sample includes: a training sample image and training sample text information, wherein the training sample image and the training sample text information include an object to be classified and a theoretical output category corresponding to the training sample image and the training sample text information; The training samples are input into the object classification model to be trained to obtain the first actual output category. The object classification model to be trained includes two encoders, a sequence attention module, a cross-modal attention module, and a classification module. According to the pre-set joint loss function, loss processing is performed on the first actual output category and the theoretical output category; Based on the first loss value, the parameters of the sequence attention module, the cross-modal attention module, and the classification module are adjusted to obtain the object classification model to be processed, wherein the object classification model to be processed has the same model structure as the object classification model to be trained; The training samples are input into the object classification model to obtain the second actual output category; According to the joint loss function, loss processing is performed on the second actual output category and the theoretical output category; The model parameters of the object classification model to be processed are adjusted based on the second loss value to obtain the target classification model.

6. The method according to claim 5, characterized in that, The joint loss function is determined based on the cross-entropy loss function and the class center loss function.

Images