Cross-modal image-text matching training method and device, storage medium, and electronic device
Through the cross-modal graphic matching training method of significance object detection and feature fusion, the scalability and generalization problems of the pre-training method of graphic matching in the prior art when the application scenario changes are solved, and more accurate and Personalized joint search results of pictures and texts.
Patent Information
- Application Number
- CN202111464736.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-03
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-03
AI Technical Summary
The existing pre-training method for graphic matching based on object detection is high in the face of changes in application scenarios, the training cost is high, the model is scalable and generalized, and it is assumed that the detected object category and text description are strong semantic correlations in practice. Invalid, unable to effectively process multiple attribute descriptions in the image.
Using a cross-modal graphic matching training method based on significance object detection, the visual-language matching pre-training is performed by fusing the significance object mask map with the input image, and the visual-language matching pre-training is performed using lightweight image and text encoder to generate images and Vector features of the text and model updates through cosine similarity and cross entropy loss.
It improves the scalability and generalization ability of the graphic matching model, can conduct joint graphic searches more accurately and personalized, reduces the dependence on strong correlation assumptions, and supports various forms of joint graphic search tasks.
Smart Images

Figure CN114283430B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present application relate to image and text matching training technology, and more particularly to a cross-modal image and text matching training method and device, storage medium, and electronic device. Background Art
[0002] In recent years, with the development of computer vision and natural language processing technology, cross-modal deep learning has attracted more and more attention. Cross-modal deep learning technology can be widely used in many fields such as subtitle generation, visual question answering, visual dialogue, image-text retrieval, and text-based image generation. Vision-language pretraining (VLP) is an important research field of cross-modal deep learning. It can not only generate image descriptions, but also be used for cross-modal problems such as image-text retrieval. The more influential ones include large-scale image-text matching pre-training models (Contrastive Language–Image Pre-training, CLIP), etc. CLIP can map images to the categories of text descriptions and describe the learned visual concepts in natural language. Its superior performance supports various types of image-text joint retrieval (image-text retrieval, image-based search, multi-text search, image-text joint search).
[0003] The key to cross-modal learning is how to understand the content of cross-modal data and measure the semantic similarity between different modal data. Many technical solutions now use object detection or labeling of image region feature regression as part of the pre-training target. For example, the multimodal pre-training method (Object-Semantics Aligned Pre-training, OSCAR) uses the object labels detected in the image as the positioning anchor points at the visual and language semantic levels to simplify the learning task of semantic alignment between images and texts. However, these methods require a powerful object detection model, such as Faster-RCNN, and the training set required for object detection must be annotated in advance, with a small number of categories and coarse category semantic granularity. The image-text matching pre-training method based on object detection often leads to a sharp increase in the cost of the training pipeline when facing changes in application scenarios, and the scalability and generalization of the model are reduced. The method based on common object detection is a method of hard aligning image semantics with text semantics, with the implicit assumption that the detected object category and text description have a strong semantic correlation. However, this strong correlation assumption is often invalid in practice. For example, an image of several athletes playing football may be described as "an intense football game". The target concept of "athlete" has been weakened by this description. Moreover, it can be seen from this example that a target may have multiple attribute descriptions. For example, the athletes in the picture can also be described as "people, players, teenagers" and other text attributes. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a cross-modal image and text matching training method and device, storage medium, and electronic device, which can realize multi-modal image and text matching training expression, and are convenient for application in fields such as image search.
[0005] According to a first aspect of the present application, a cross-modal image-text matching training method is provided, comprising:
[0006] The image to be trained is input into an image encoder, and the image encoder scales the image to be trained so that the image to be trained reaches the set length and width of the image to be input; it is determined whether the image to be input has an edited salient target mask map, and if so, the salient target mask map and the input image are input into a mask fusion module, and the fusion features of the input image are obtained through the mask fusion module; if the image to be input does not have a salient target mask map, salient target detection is performed on the image to be input to generate a salient target mask map of the input image, and the salient target mask map and the input image are input into a mask fusion module, and the fusion features of the input image are obtained through the mask fusion module; the fusion features are input into an image attention encoding module, and the input fusion features are encoded into image vector features through a neural network model;
[0007] and inputting the description of the image content of the image to be trained as text into a text encoder, the text encoder segmenting the text, converting the text into a token vector, and performing transformer-based text encoding processing on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature;
[0008] After the training reaches the selected number of samples, the image encoder and the text encoder perform modal interaction on the image vector features and the text vector features respectively, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
[0009] Preferably, the performing salient object detection on the image to be input to generate a salient object mask map of the input image includes:
[0010] After the input image of the set resolution is subjected to multi-layer 3×3 convolution processing, it is subjected to multiple downsampling and stacking processing respectively. When the feature resolution is reduced to the set threshold, the downsampled low-resolution features are subjected to upsampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then upsampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion;
[0011] The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the upsampled features to 1 channel, and then the spatial attention features are generated through the sigmoid activation function;
[0012] The spatial attention feature and the input feature are dot-multiplied to obtain a salient target probability map of the input image, and a salient target mask map of the input image is generated based on the salient target probability map.
[0013] Preferably, the step of obtaining the fusion features of the input image through the mask fusion module includes:
[0014] Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function;
[0015] After cascading the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 convolution and a Sigmoid activation function are performed to obtain a mask-based gated attention feature;
[0016] The initial features of the input image are processed by dot multiplication with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is performed to perform feature fusion to obtain the fused features of the input image.
[0017] Preferably, encoding the input fusion features into image vector features through a neural network model includes:
[0018] The image attention encoding module aligns objects with significant semantic information based on prior information of significant object areas, and encodes the fused features into image vectors of set dimensions.
[0019] According to a second aspect of the present application, a cross-modal image-text matching training device is provided, comprising:
[0020] The image encoder receives the image to be trained, scales the image to be trained so that the image to be trained reaches the set length and width of the image to be input; determines whether the image to be input has an edited salient target mask map, and if so, inputs the salient target mask map and the input image to a mask fusion module; if the image to be input does not have a salient target mask map, inputs the image to be input to a salient target detection module;
[0021] A salient object detection module, used to perform salient object detection on the image to be input, and generate a salient object mask map of the input image;
[0022] A mask fusion module, configured to receive the salient object mask map and the input image, and generate a fusion feature of the input image based on the salient object mask map and the input image;
[0023] An image attention encoding module, used for receiving fused features and encoding the fused features into image vector features through a neural network model; and
[0024] A text encoder, used for receiving a description of the image content of the image to be trained as an input text, performing word segmentation on the text, converting the text into a token vector, and performing transformer-based text encoding processing on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature;
[0025] A processing unit is used to perform modal interaction on the image vector features and the text vector features respectively by the image encoder and the text encoder after the training reaches the selected number of samples, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
[0026] Preferably, the salient object detection module is further used for:
[0027] After the input image of the set resolution is processed by 3×3 convolution, it is downsampled and stacked multiple times. When the feature resolution is reduced to the set threshold, the downsampled low-resolution features are subjected to upsampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then upsampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion;
[0028] The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the upsampled features to 1 channel, and then the spatial attention features are generated through the sigmoid activation function;
[0029] The spatial attention feature and the input feature are dot-multiplied to obtain a salient target probability map of the input image, and a salient target mask map of the input image is generated based on the salient target probability map.
[0030] Preferably, the salient object detection module is further used for:
[0031] Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function;
[0032] After cascading the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 convolution and a Sigmoid activation function are performed to obtain a mask-based gated attention feature;
[0033] The initial features of the input image are processed by dot multiplication with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is performed to perform feature fusion to obtain the fused features of the input image.
[0034] Preferably, the image attention encoding module is further used to: align targets with significant semantic information based on prior information of significant target areas, and encode the fused features into an image vector of a set dimension.
[0035] According to a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the cross-modal image-text matching training method based on salient target detection.
[0036] According to a fourth aspect of the present application, a readable non-transitory storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the cross-modal image-text matching training method based on salient target detection described above are implemented.
[0037] The cross-modal image-text matching training method and device based on salient object detection, storage medium, and electronic device provided in the embodiment of the present application first use the pre-trained salient object detection to obtain the salient object area mask map, then fuse the features of the input image and the salient object area mask map, and then use the dual-tower model for visual-language matching pre-training; wherein, the dual-tower model of visual-language matching pre-training draws on the model architecture of CLIP and changes the image and text encoders to a lighter network model. The image encoder of the original CLIP is regarded as an equal-weighted image encoder, while the image encoder fused with salient object detection can be regarded as a non-equal-weighted image feature encoder with spatial attention. After the pre-training is completed, the salient area mask input by the model can be modified accordingly, and the output of the image encoder will also change according to the different mask areas. The embodiment of the present application can not only be applied to various forms of conventional image-text joint retrieval, but also can be used to perform image-text joint retrieval in a more accurate and personalized manner by directly modifying the semantic features of the image through manual interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a process flow of a cross-modal image-text matching training method based on salient object detection provided in an embodiment of the present application;
[0039] Figure 2 A schematic diagram of the encoding process of the image encoder provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of the encoding process of the text encoder provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of cross-modal vector feature modal interaction provided in an embodiment of the present application;
[0042] Figure 5 A schematic diagram of a spatial attention salient target detection network structure provided in an embodiment of the present application;
[0043] Figure 6 A schematic diagram of the structure of a residual neural network module provided in an embodiment of the present application;
[0044] Figure 7 A schematic diagram of the structure of a spatial attention module provided in an embodiment of the present application;
[0045] Figure 8 A schematic diagram of the structure of a downsampling module provided in an embodiment of the present application;
[0046] Fig. 9 A schematic diagram of the structure of an upsampling feature fusion module provided in an embodiment of the present application;
[0047] Fig.10A schematic diagram of the structure of the mask fusion module provided in the embodiment of the present application;
[0048] Fig.11 A schematic diagram of the composition structure of a cross-modal image-text matching training device based on salient object detection provided in an embodiment of the present application;
[0049] Fig.12 A structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0050] The essence of the technical solution of the embodiment of the present application is explained in detail below with reference to examples.
[0051] Figure 1 A flow chart of a cross-modal image-text matching training method based on salient object detection provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the cross-modal image-text matching training method based on salient object detection in the embodiment of the present application includes the following processing steps:
[0052] Step 101: input the image to be trained into an image encoder, and the image encoder encodes the image to be trained to obtain image vector features of the image to be trained.
[0053] In an embodiment of the present application, an image to be trained is input into an image encoder, and the image encoder scales the image to be trained so that the image to be trained reaches the set length and width of the image to be input; it is determined whether there is an edited salient target mask map for the image to be input, and if so, the salient target mask map and the input image are input into a mask fusion module, and a fusion feature of the input image is obtained through the mask fusion module; if the image to be input does not have a salient target mask map, salient target detection is performed on the image to be input to generate a salient target mask map of the input image, and the salient target mask map and the input image are input into a mask fusion module, and a fusion feature of the input image is obtained through the mask fusion module; the fusion feature is input into an image attention encoding module, and the input fusion feature is encoded into an image vector feature through a neural network model.
[0054] In an embodiment of the present application, generating a salient object mask map of the input image includes:
[0055] After the input image of the set resolution is processed by multi-layer 3×3 convolution, it is down-sampled and stacked multiple times. When the feature resolution is reduced to the set threshold, the down-sampled low-resolution features are subjected to up-sampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then up-sampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion; the set threshold of the feature resolution here includes 1024, 512, or 2048, etc.
[0056] The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the upsampled features to 1 channel, and then the spatial attention features are generated through the sigmoid activation function;
[0057] The spatial attention feature and the input feature are dot-multiplied to obtain a salient target probability map of the input image, and a salient target mask map of the input image is generated based on the salient target probability map.
[0058] Wherein, obtaining the fusion features of the input image through the mask fusion module includes:
[0059] Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function;
[0060] After cascading the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 convolution and a Sigmoid activation function are performed to obtain a mask-based gated attention feature;
[0061] The initial features of the input image are processed by dot multiplication with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is performed to perform feature fusion to obtain the fused features of the input image.
[0062] Among them, the input fusion features are encoded into image vector features through the neural network model, including:
[0063] The image attention encoding module aligns objects with significant semantic information based on prior information of significant object areas, and encodes the fused features into image vectors of set dimensions.
[0064] Step 102: Input the description of the image content of the image to be trained as text into a text encoder, the text encoder performs word segmentation on the text, converts the text into a token vector, and performs transformer-based text encoding on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature.
[0065] In the embodiment of the present application, the architecture of the text encoder adopts the same transformer architecture as the original CLIP, except that in order to reduce parameters and computation, the number of transformer layers is reduced from 12 to 8, and the number of heads is changed to 12. The output of the text encoder is a vector of the same dimension as the image vector feature.
[0066] Step 103, after the training reaches the selected number of samples, the image encoder and the text encoder perform modal interaction on the image vector features and the text vector features respectively, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
[0067] In an embodiment of the present application, in the image-text matching training stage, after using batch_size=N image-text pair data, the image encoder and the text encoder will output the image and text vector features respectively, and then the two cross-modal vector features will be modally interacted to calculate the cosine similarity and cross entropy loss of the vectors, so as to update the model parameters using the reverse gradient. At the same time, due to the lack of corresponding image saliency mask label data in the training stage, the embodiment of the present application uses SD-Net to detect the input image and automatically generate a mask map. When the pre-training is completed, the model can map the image to the category described by the text, use natural language to describe the learned visual concepts, and can use the image and text encoders as feature vector extractors for other downstream tasks such as image-text retrieval. Here, the number of training samples can be set according to the parameter accuracy requirements of the training model.
[0068] The embodiments of the present application are further described in detail below with reference to specific examples.
[0069] In the embodiments of the present application, it is assumed that the text is only a sparse abstract representation of the image, and its main description content is strongly correlated with the salient areas in the image. From a common sense point of view, this is also in line with the behavioral habit of humans to describe visually significant image areas that can be quickly detected, especially in scenarios where only a simple description of the image is required. Salient Object Detection is to find and mark the most obvious and prominent objects in an image or video. Different from target detection that frames the position of objects and identifies object category labels, salient target detection is to segment salient subject targets at the pixel level. Typical methods include HED, U2-Net, etc. The output of salient object detection is a feature map of the same size as the input. The pixel value corresponding to the feature map is in the range of 0 to 1, representing the probability that the pixel belongs to the salient area (foreground).
[0070] The embodiment of the present application first uses the pre-trained salient target detection to obtain the salient object area mask map, then performs feature fusion of the input image and the salient object area mask map, and then uses the twin-tower model for visual-language matching pre-training. The twin-tower model for visual-language matching pre-training can refer to the model architecture of CLIP and change the image and text encoder to a lighter network model. If the image encoder of the original CLIP is regarded as an equal-weighted image encoder, then the image encoder fused with the salient object detection can be regarded as a non-equal-weighted image feature encoder with spatial attention.
[0071] Another advantage of the embodiment of the present application is that after the pre-training of image-text matching is completed, the salient area mask of the model input can be manually modified. As the mask area changes, the output of the image encoder will also change accordingly, while the traditional image feature encoder can only encode and output one semantic feature for the same image and cannot change the representation of the image semantics. Therefore, the embodiment of the present application can not only be applied to various forms of conventional image-text joint retrieval, but also can perform image-text joint retrieval in a more accurate and personalized manner by directly modifying the image semantic features through manual interaction.
[0072] The embodiment of the present application adopts a visual-language matching pre-training model to perform matching training of images and texts. The visual-language matching pre-training model of the embodiment of the present application is mainly composed of two functional modules: an image encoder and a text encoder. The image-text matching pre-training model is a parallel dual-tower mode. The image encoder and the text encoder are independent of each other, and there is no interaction in the intermediate process. The input image can be an RGB image, and the text is a brief description of the content of this RGB image. Figure 2 The encoding process of the image encoder of the embodiment of the present application is shown as follows: Figure 2As shown in the figure, it includes: first receiving the input RGB image Image, scaling the input 3-channel RGB image Image to an image Img of a specified H×W resolution, such as H=W=224; then judging whether manual editing is performed and whether there is a salient target mask map that has been edited, if so, using the manually edited salient target mask map and the image Img as inputs of the mask fusion module, otherwise, using the salient target detection module and the corresponding post-processing to automatically generate the corresponding salient target mask; finally, the image attention encoding module encodes the input fusion features into a 512-dimensional image vector feature through a neural network model.
[0073] Figure 3 A schematic diagram of the encoding process of the text encoder provided in the embodiment of the present application, such as Figure 3 As shown, the encoding process of the text encoder of the embodiment of the present application includes: first, the input text is converted into a token vector through a word segmenter, and then the token is input into a transformer-based text encoding module, and finally it is also encoded into a 512-dimensional text vector feature.
[0074] Figure 4 A schematic diagram of cross-modal vector feature modal interaction provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in an embodiment of the present application, in the training phase, after using batch_size=N image-text pair data, the image encoder and the text encoder respectively output the image and text vector features, and then perform modal interaction on these two cross-modal vector features, calculate the cosine similarity and cross entropy loss of the vectors, and then use the reverse gradient to update the model parameters. At the same time, due to the lack of corresponding image saliency mask label data in the training phase, a method is adopted in which SD-Net detects the input image and automatically generates a mask. When the pre-training is completed, the model can map images to categories described in text, describe the learned visual concepts in natural language, and can use image and text encoders as feature vector extractors for other downstream tasks such as image and text retrieval.
[0075] The image encoder of the embodiment of the present application mainly includes a salient target detection module, a mask fusion module and an image attention encoding module. The mask map can be provided by the salient target detection module or drawn manually. This not only allows the image encoder to pay more attention to the feature expression of the mask area image, but also supports the operation mode of manually selecting the key encoding of the local area of the image, which is convenient for manually adjusting the focus and accuracy of the image semantic encoding.
[0076] The salient object detection module of the embodiment of the present application can be replaced by any salient object detection neural network. As an implementation method, a lightweight spatial attention salient object detection network SD-Net can be designed. Figure 5 The schematic diagram of the spatial attention salient target detection network structure provided for the embodiment of the present application shows the SD-Net model architecture. The embodiment of the present application can obtain a probability map of each pixel in the input image belonging to a salient target through the SD-Net model. Assuming a 3-channel RGB image with a resolution of 224×224 is input, SD-Net will output a 224×224 salient target probability map, and after threshold processing, it will finally output a 1-channel salient target mask map with a resolution of 224×224.
[0077] Among them, in order to enhance the model's ability to perceive spatial features, the embodiment of the present application can improve a residual neural network module, which uses deep separation convolution to reduce computational complexity and memory requirements, and combines a spatial attention mechanism. Figure 6 A schematic diagram of the residual neural network module structure provided in the embodiment of the present application is shown in FIG. Figure 6 As shown in the figure, in the residual neural network module, C represents the number of output channels of each module, and “SRB×2” represents the continuous stacking of 2 SRB modules. Figure 7 A schematic diagram of the spatial attention module structure provided in an embodiment of the present application, Figure 7 The Spartial Attention Module (SAM) only uses the spatial attention mechanism in the Convolutional Block Attention Module (CBAM) and removes the channel attention mechanism. This method first performs a global mean pooling and a global maximum pooling based on the channel, and then cascades the two results based on the channel. Then, after a 7×7 convolution operation with a large kernel, the dimension is reduced to 1 channel. Then, the spatial attention feature is generated through the sigmoid activation function. Finally, the spatial attention feature is multiplied by the input feature of the module to obtain the final generated feature, such as Figure 7 At the same time, the embodiment of the present application uses a downsampling module DSM to perform downsampling and adjust the number of channels. The module first uses a 1×1 convolution operation to change the number of channels from C to C′, and then uses a maximum pooling operation to perform 2 times downsampling, as shown in FIG. Figure 8 shown.
[0078] In the sampling stage, the UFFM module is used for upsampling and feature fusion. The module first uses 1×1 convolution to reduce the dimension of low-resolution features to reduce the amount of calculation, then upsamples to the same scale as the high-resolution features, and then uses 1×1 convolution and 3×3 convolution to perform channel dimension reduction and feature fusion, such as Fig. 9As shown in the figure, the channel dimension reduction parameter is C_3=C_1. After the last layer of feature fusion, interpolation upsampling is performed to restore it to the same scale as the input image, and then two layers of 3×3 and 1×1 convolution and other related operations are performed in sequence. The number of channels in the final output probability feature layer is 1.
[0079] Finally, after obtaining the significant target probability map, the present application takes a certain threshold to generate a mask map. In the embodiment of the present application, the threshold is selected as 0.7. When the probability is greater than 0.7, the pixel is considered to be a significant target pixel and is reset to a mask value of 1; otherwise, it is a non-significant target pixel and is reset to a mask value of 0. After completing the above operations, the mask map of the significant target area is obtained.
[0080] Without significantly increasing the number of model parameters and the amount of computation, the SD-Net of the embodiment of the present application uses the spatial attention mechanism of the SRB module to enhance the model's ability to express the spatial position information of significant objects. At the same time, SD-Net integrates more multi-scale features in the upsampling stage, and the resulting mask image has a more accurate texture of the significant target contour.
[0081] In order to better guide the fusion of the input image and the salient target mask map of the image, in the embodiment of the present application, the two images are not directly concatenated into an input array with 4 channels. Instead, the input image and the mask map are input separately and then fused through the gated attention mechanism, such as Fig.10 As shown in the figure. First, the RGB input image and the salient target mask map are scaled to the same appropriate resolution, such as H=W=224. After the RGB image and mask map are subjected to two layers of 3×3 convolution, BN regularization, ReLU activation function and other operations, the initial features F_rgb and F_mask of H×W×32 and H×W×16 are obtained respectively. After the F_rgb and F_mask features are cascaded and 1×1 convolution and other operations are used for feature fusion and channel dimension reduction, a 5×5 large convolution and Sigmoid activation function are used to obtain the mask-based gated attention feature F_gate. Then, the F_rgb feature is multiplied with F_gate. Next, the product feature is cascaded again with the initial feature F_rgb of the input image, and finally a 1×1 convolution and other operations are used for feature fusion to obtain an output feature of size H×W×64. It should be noted that the cascaded fusion features are used as the final output features instead of the features multiplied by the gated attention features because the fusion features can obtain richer contrast semantics, which is equivalent to fusing the contrast features of two partially identical images.
[0082] The mask fusion module of the embodiment of the present application is characterized in that it does not directly concatenate the mask map channel with the RGB 3 channels of the original image, but instead uses the features extracted after convolution of the two parameters to adaptively guide the model to express and correct objects in the salient target area using the spatial attention mechanism. This is more conducive to avoiding interference with the subsequent image-text matching process due to local recognition accuracy problems of the mask map during training, thereby reducing the requirements for mask accuracy.
[0083] The image attention encoding module of the embodiment of the present application adopts the image encoding module based on resnet50 and multi-head attention mechanism in CLIP. The difference is that the image attention encoding module of the embodiment of the present application eliminates the related operations such as the first three layers of convolution of resnet50, and uses the output of the mask fusion module as the input of the image attention encoding module. The image attention encoding module finally encodes the input fusion feature of size H×W×64 into a 512-dimensional vector.
[0084] Different from the input of the original CLIP image encoding, the input of the image attention encoding module incorporates the prior information of the salient object area, making it easier to align objects with salient semantic information, making the encoding method of the image encoder more in line with people's habit of describing salient objects when describing images. At the same time, the encoder based on the convolutional neural network has a stronger inductive bias ability, which is more conducive to the application in the training of non-large data sets.
[0085] The architecture of the text encoder in the embodiment of the present application adopts the same transformer architecture as the original CLIP, except that in order to reduce parameters and computation, we reduce the number of transformer layers from 12 to 8 and change the number of heads to 12. The output of the text encoder is also a 512-dimensional vector.
[0086] The embodiment of the present application integrates the prior knowledge of salient target detection into the image-text pre-training method, thereby improving the encoding capability of the image encoder in the pre-training model for salient targets. In addition, the embodiment of the present application helps people to modify image encoding based on their personal subjective needs, which is conducive to extracting features that are more in line with people's subjective objects, and has application potential in retrieval tasks such as image search.
[0087] Fig.11 A schematic diagram of the composition structure of a cross-modal image-text matching training device based on salient object detection provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the cross-modal image-text matching training device based on salient object detection in the embodiment of the present application includes:
[0088] The image encoder 110 receives the image to be trained, scales the image to be trained to make the image to be trained reach the set length and width of the image to be input; determines whether the image to be input has an edited salient target mask map, and if so, inputs the salient target mask map and the input image to a mask fusion module; if the image to be input does not have a salient target mask map, inputs the image to be input to a salient target detection module 111;
[0089] A salient object detection module 111 is used to perform salient object detection on the image to be input, and generate a salient object mask map of the input image;
[0090] A mask fusion module 112, configured to receive the salient object mask map and the input image, and generate a fusion feature of the input image based on the salient object mask map and the input image;
[0091] An image attention encoding module 113, configured to receive fused features and encode the fused features into image vector features through a neural network model; and
[0092] A text encoder 114, configured to receive a description of the image content of the image to be trained as an input text, perform word segmentation on the text, convert the text into a token vector, and perform transformer-based text encoding processing on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature;
[0093] The processing unit 115 is used to perform modal interaction on the image vector features and the text vector features respectively after the training reaches the selected number of samples, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
[0094] As an implementation method, the salient object detection module 111 is further used for:
[0095] After the input image of the set resolution is processed by multi-layer 3×3 convolution, it is down-sampled and stacked multiple times respectively. When the feature resolution is reduced to the set threshold, the down-sampled low-resolution features are subjected to up-sampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then up-sampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion;
[0096] The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the pixel data to one channel, and then the spatial attention features are generated through the sigmoid activation function;
[0097] The spatial attention feature and the input feature are dot-multiplied to obtain a salient target probability map of the input image, and a salient target mask map of the input image is generated based on the salient target probability map.
[0098] As an implementation method, the salient object detection module 111 is further used for:
[0099] Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function;
[0100] After cascading the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 convolution and a Sigmoid activation function are performed to obtain a mask-based gated attention feature;
[0101] The initial features of the input image are processed by dot multiplication with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is performed to perform feature fusion to obtain the fused features of the input image.
[0102] As an implementation method, the image attention encoding module 113 is also used to: align targets with significant semantic information based on prior information of significant target areas, and encode the fused features into an image vector of a set dimension.
[0103] In an exemplary embodiment, the above-mentioned processing units of the cross-modal image-text matching training device based on salient target detection in the embodiment of the present application can be implemented by one or more central processing units (CPU, Central Processing Unit), image processing units (GPU, Graphics Processing Unit), baseband processors (BP, Base Processor), application-specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic devices (PLD, Programmable Logic Device), complex programmable logic devices (CPLD, Complex Programmable Logic Device), field programmable gate arrays (FPGA, Field-Programmable Gate Array), general processors, controllers, microcontrollers (MCU, Micro Controller Unit), microprocessors (Microprocessors), or other electronic components.
[0104] In the disclosed embodiment, Fig.11 The specific manner in which each processing unit in the cross-modal image-text matching training device based on salient object detection performs operations has been described in detail in the embodiments of the method, and will not be elaborated on here.
[0105] Below, reference Fig.12 The electronic device 12 according to the embodiment of the present application is described.
[0106] like Fig.12 As shown, the electronic device 12 includes one or more processors 121 and a memory 122 .
[0107] The processor 121 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 12 to perform desired functions.
[0108] The memory 122 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 121 may run the program instructions to implement the cross-system network device calling method of each embodiment of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0109] In one example, the electronic device 12 may further include: an input device 123 and an output device 124, which are connected via a bus system and / or other forms of connection mechanisms ( Fig.12 ) interconnected.
[0110] The input device 123 may include, for example, a keyboard, a mouse, etc.
[0111] The output device 124 can output various information to the outside, including the determined distance information, direction information, etc. The output device 124 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0112] Of course, to simplify, Fig.12 Only some of the components in the electronic device 12 that are relevant to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 12 may also include any other appropriate components according to specific application conditions.
[0113] The embodiment of the present application also records a readable non-transitory storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the cross-modal image-text matching training method based on salient object detection are implemented.
[0114] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.
[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0116] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0117] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0118] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0119] The above is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A cross-modal image-text matching training method, characterized in that: The method comprises: Input the image to be trained into an image encoder, and the image encoder scales the image to be trained so that the image to be trained reaches the set length and width of the image to be input; determine whether the image to be input has an edited salient target mask map, and if so, input the salient target mask map and the input image into a mask fusion module, and obtain the fusion features of the input image through the mask fusion module; if the image to be input does not have a salient target mask map, perform salient target detection on the image to be input, generate a salient target mask map of the input image, and input the salient target mask map and the input image into a mask fusion module, and obtain the fusion features of the input image through the mask fusion module, and perform salient target detection on the image to be input to generate a salient target mask map of the input image, further comprising: After the input image of the set resolution is processed by multi-layer 3×3 convolution, it is down-sampled and stacked multiple times respectively. When the feature resolution is reduced to the set threshold, the down-sampled low-resolution features are subjected to up-sampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then up-sampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion; The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the upsampled features to 1 channel, and then the spatial attention features are generated through the sigmoid activation function; Performing a dot product of the spatial attention feature and the input feature to obtain a salient target probability map of the input image, and generating a salient target mask map of the input image based on the salient target probability map; Input the fused features into the image attention encoding module, and encode the input fused features into image vector features through a neural network model; and inputting the description of the image content of the image to be trained as text into a text encoder, the text encoder segmenting the text, converting the text into a token vector, and performing transformer-based text encoding processing on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature; After the training reaches the selected number of samples, the image encoder and the text encoder perform modal interaction on the image vector features and the text vector features respectively, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
2. The method according to claim 1, characterized in that The step of obtaining the fusion feature of the input image through the mask fusion module includes: Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function; After cascading the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 large convolution and Sigmoid activation function are performed to obtain a mask-based gated attention feature; The initial features of the input image are dot-multiplied with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is used to perform feature fusion to obtain the fused features of the input image.
3. The method according to claim 2, characterized in that The step of encoding the input fusion features into image vector features through a neural network model includes: The image attention encoding module aligns objects with significant semantic information based on prior information of significant object areas, and encodes the fused features into image vectors of set dimensions.
4. A cross-modal image-text matching training device, characterized in that: The device comprises: The image encoder receives the image to be trained, scales the image to be trained so that the image to be trained reaches the set length and width of the image to be input; determines whether the image to be input has an edited salient target mask map, and if so, inputs the salient target mask map and the input image to a mask fusion module; if the image to be input does not have a salient target mask map, inputs the image to be input to a salient target detection module; A salient object detection module is used to perform salient object detection on the image to be input and generate a salient object mask map of the input image. The salient object detection module is also used to: After the input image of the set resolution is processed by multi-layer 3×3 convolution, it is down-sampled and stacked multiple times respectively. When the feature resolution is reduced to the set threshold, the down-sampled low-resolution features are subjected to up-sampled feature fusion UFFM processing; that is, firstly, 1×1 convolution is used to reduce the dimension of the low-resolution features, and then up-sampled to the same scale as the high-resolution features, and 1×1 convolution and 3×3 convolution are used in sequence to perform channel dimension reduction and feature fusion; The features processed by UFFM are interpolated and upsampled to restore to the same scale as the input image, and then 3×3 convolution and 1×1 convolution are performed to reduce the dimension of the upsampled features to 1 channel, and then the spatial attention features are generated through the sigmoid activation function; Performing a dot product of the spatial attention feature and the input feature to obtain a salient target probability map of the input image, and generating a salient target mask map of the input image based on the salient target probability map; A mask fusion module, configured to receive the salient object mask map and the input image, and generate a fusion feature of the input image based on the salient object mask map and the input image; An image attention encoding module, used for receiving fused features and encoding the fused features into image vector features through a neural network model; and A text encoder, used for receiving a description of the image content of the image to be trained as an input text, performing word segmentation on the text, converting the text into a token vector, and performing transformer-based text encoding processing on the token vector to obtain a text vector feature encoded with the same dimension as the image vector feature; A processing unit is used to perform modal interaction on the image vector features and the text vector features respectively by the image encoder and the text encoder after the training reaches the selected number of samples, calculate the cosine similarity and cross entropy loss of the vectors, and use reverse gradient to update the model parameters of the image encoder and the text encoder.
5. The device according to claim 4, characterized in that The salient object detection module is further used for: Scaling the input image and the salient target mask map to the same set resolution, and respectively obtaining the underlying features of the input image and the salient target mask map after processing with two layers of 3×3 convolution, BN regularization, and ReLU activation function; After concatenating the underlying features of the input image and the salient target mask map, a 1×1 convolution is performed to perform feature fusion and channel dimension reduction, and then a 5×5 convolution and a Sigmoid activation function are used to obtain a mask-based gated attention feature; The initial features of the input image are processed by dot multiplication with the gated attention features, and the features after dot multiplication are cascaded with the initial features of the input image again, and then 1×1 convolution is performed to perform feature fusion to obtain the fused features of the input image.
6. The device according to claim 5, characterized in that The image attention encoding module is also used to: align targets with significant semantic information based on prior information of significant target areas, and encode the fused features into an image vector of a set dimension.
7. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the cross-modal image-text matching training method described in any one of claims 1 to 3.
8. A readable non-transitory storage medium, characterized in that: The storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the cross-modal image-text matching training method described in any one of claims 1 to 3 are implemented.
