Image-text matching and grayscale sensitivity detection method, device and equipment

Abstract

The present application relates to an image-text matching and grayscale sensitivity detection method, apparatus, and device. The method comprises: calculating the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection; performing feature extraction on the text input and the image input that passes the grayscale sensitivity detection, deploying a cross-attention mechanism in the feature alignment stage, and designing a feature alignment loss function by enhancing the diversity of multimodal representations; designing a distribution balance loss function by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment; designing a pixel protection regularization loss function by calculating all pixel values ​​of the image input that passes the grayscale sensitivity detection, and then optimizing and distilling the pre-trained CLIP model, and using the optimized CLIP model to output the adjusted image and the degree of matching between the image and each text prompt. The use of this method can improve the image retrieval accuracy and image-text matching accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image-text matching and grayscale sensitivity detection method, device and equipment. Background Art

[0002] With the rapid development of artificial intelligence (AI), multimodal learning, as a bridge connecting different types of data, such as images and text, has become a key research direction in the field. The CLIP model, a classic language-image comparison model, has attracted considerable attention for its exceptional performance in multimodal tasks. The CLIP model achieves cross-modal functionality by learning the associations between images and text, providing powerful technical support for applications such as image retrieval and image-text matching. These two areas have widespread and important applications in fields such as intelligent security, e-commerce recommendations, and digital libraries, requiring extremely high accuracy and security.

[0003] However, the widespread deployment of the CLIP model in real-world applications has gradually raised security concerns. Research has shown that the CLIP model is vulnerable to malicious attacks. Attackers can use carefully crafted tampering methods to disrupt the correct matching relationship between image and text, thereby misleading the model. Some studies have shown that this vulnerability of the CLIP model likely stems from the gap between its modalities: the difference between image embeddings and text embeddings. This vulnerability not only limits the application of the CLIP model in security-critical scenarios but also poses challenges to data authenticity and integrity, significantly compromising image retrieval accuracy. Furthermore, insufficient semantic understanding within the image and text modalities creates a significant semantic gap between the two, making accurate image-text matching difficult to achieve. Summary of the Invention

[0004] Based on this, it is necessary to provide an image-text matching and grayscale sensitivity detection method, device and equipment that can improve image retrieval accuracy and image-text matching accuracy to address the above technical problems.

[0005] An image-text matching and grayscale sensitivity detection method, the method comprising:

[0006] Obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection;

[0007] Perform feature extraction on text input and image input through grayscale sensitive detection to obtain image features and text features;

[0008] In the feature alignment stage, we deploy a cross-attention mechanism and design a feature alignment loss function by enhancing the diversity of multimodal representations. We design a distribution balancing loss function to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment. We design a pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection.

[0009] The pre-trained CLIP model is optimized and distilled according to the feature alignment loss function, distribution balance loss function, pixel protection regularization loss function, and the pre-set contrast optimization feature loss function to obtain the optimized CLIP model;

[0010] The optimized CLIP model is used to output the adjusted image and the degree of match between the image and each text prompt.

[0011] An image-text matching and grayscale sensitivity detection device, comprising:

[0012] Grayscale sensitivity detection and feature extraction module, used to obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection; perform feature extraction on the text input and the image input that passes the grayscale sensitivity detection to obtain image features and text features;

[0013] The loss function design module is used to deploy the cross-attention mechanism in the feature alignment stage and design the feature alignment loss function by enhancing the diversity of multimodal representations; the distribution balance loss function is designed to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment; and the pixel protection regularization loss function is designed by calculating all pixel values ​​of the image input through grayscale sensitive detection.

[0014] The model optimization and image matching module is used to optimize and distill the pre-trained CLIP model according to the feature alignment loss function, distribution balance loss function, pixel protection regularization loss function, and the pre-set contrast optimization feature loss function to obtain the optimized CLIP model; the optimized CLIP model is used to output the adjusted image and the degree of match between the image and each text prompt.

[0015] The above-mentioned image-text matching and grayscale sensitivity detection method, device and equipment first obtain image and text inputs, perform grayscale sensitivity detection by calculating the grayscale sensitivity difference value of the image, identify images that may be tampered with, and ensure data reliability. Feature extraction is then performed on the text input and the image input that has undergone grayscale sensitivity detection, which can accurately capture the key information of the image and text, laying the foundation for subsequent matching. Grayscale sensitivity detection can effectively enhance the robustness of image features and reduce interference caused by small perturbations of the image. In the feature alignment stage, the deployment of the cross-attention mechanism enables the model to focus on the key parts of the image and text that are related to each other, and enhance the semantic connection between the two; by enhancing the diversity of multimodal representations, the feature alignment loss function is designed, and the cosine similarity between image features and text features is used as the calculation basis to guide the image features to align with the target text features, prompting the model to learn more accurate cross-modal mapping relationships and reduce matching errors caused by semantic gaps. The distribution balance loss function balances the feature distribution of the optimized image by maximizing the mean cosine similarity between image and text features and minimizing the second-order moment. This effectively avoids matching bias caused by large differences in feature distribution, ensuring that images and text with similar semantics are more closely clustered in the embedding space, improving matching accuracy. The pixel protection regularization loss function constrains and protects image pixels by calculating all pixel values ​​of the image input after grayscale-sensitive detection, preventing malicious attacks from tampering with image pixels and ensuring the authenticity and integrity of image features, thereby improving the model's robustness against adversarial examples. Finally, the pre-trained CLIP model is optimized and distilled by combining the feature alignment loss function, the distribution balance loss function, and the pixel protection regularization loss function with a pre-set contrast optimization feature loss function. This multi-pronged approach adjusts model parameters, enabling the optimized CLIP model to more accurately output the adjusted image and the degree of match between the image and each text prompt. By addressing modality gaps, ensuring image quality, and detecting image tampering, this application effectively improves image retrieval accuracy and image-text matching accuracy, providing strong support for the development of multimodal applications in a secure and reliable environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flow chart of an image-text matching and grayscale-sensitive detection method in one embodiment;

[0017] Figure 2 Schematic diagram of the technical framework of an image-text matching and grayscale-sensitive detection method in one embodiment;

[0018] Figure 3 is a structural block diagram of an image-text matching and grayscale sensitivity detection device in one embodiment;

[0019] Figure 4FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0021] In one embodiment, Figure 1 As shown, a method for image-text matching and grayscale sensitivity detection is provided, comprising the following steps:

[0022] Step 102: Obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection.

[0023] Malicious image tampering disrupts its proper matching relationship with text, and the CLIP score of tampered images changes abnormally after grayscale conversion. This detection can identify tampered images and exclude or further process them. In image retrieval and image-text matching, avoiding the use of tampered images in calculations ensures the authenticity and integrity of input data, improves the reliability of model output results, and indirectly improves retrieval and matching accuracy. If an attacker tampered with an image to artificially inflate the CLIP score, grayscale-sensitive detection can identify the image as abnormal and not output it as a correct match result.

[0024] Step 104 : performing feature extraction on the text input and the image input detected by grayscale sensitivity to obtain image features and text features.

[0025] like Figure 2 As shown in FIG, an image encoder is used to process the input image to extract image features, and a text encoder is used to process the input text to obtain text features.

[0026] Step 106, deploying a cross-attention mechanism in the feature alignment stage and designing a feature alignment loss function by enhancing the diversity of multimodal representations; designing a distribution balance loss function to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment; and designing a pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection.

[0027] The CLIP model suffers from a modality gap between image and text embeddings, leading to inaccurate matching. During the feature alignment phase, the deployment of a cross-attention mechanism enables the model to focus on key, interconnected parts of the image and text, strengthening the semantic connection between them. A feature alignment loss function designed to enhance the diversity of multimodal representations, based on the cosine similarity between image and text features, guides image features to align with target text features, enabling the model to learn more accurate cross-modal mappings and reduce matching errors caused by the semantic gap.

[0028] Furthermore, because the degree of association between different textual cues and images can vary significantly, traditional methods can easily lead models to be overly biased towards certain specific cues. The distribution balance loss function calculates the variance of the cosine similarity between image and textual features to balance the feature distribution of the optimized image, avoiding over-bias towards certain specific cues and bringing it closer to the distribution of the target semantic concept. In image retrieval, this ensures that the model responds appropriately to all relevant textual cues, and does not ignore other reasonable matches due to the dominance of one type of cues in the feature space. In image-text matching, this ensures a more balanced and reasonable match between images and different textual cues, avoiding mismatches caused by oversensitivity to some textual cues and thus improving matching accuracy.

[0029] Furthermore, when optimizing images to match text semantics, pixel values ​​may fluctuate unnaturally, impacting image quality and feature stability. The pixel-protection regularization loss function calculates all pixel values ​​in the image and constrains them to a reasonable range to prevent unnatural image variations during the optimization process. In image retrieval and image-text matching, stable image features enable the model to make decisions based on reliable visual information, avoiding feature misinterpretation due to image quality degradation, and thus improving accuracy.

[0030] Step 108 , optimize and distill the pre-trained CLIP model according to the feature alignment loss function, the distribution balance loss function, the pixel protection regularization loss function, and the preset contrast optimization feature loss function to obtain an optimized CLIP model; use the optimized CLIP model to output the adjusted image and the degree of match between the image and each text prompt.

[0031] By combining the feature alignment loss function, distribution balance loss function and pixel protection regularization loss function with the pre-set contrast optimization feature loss function, the pre-trained CLIP model is optimized and distilled to adjust the model parameters, so that the optimized CLIP model can more accurately output the adjusted image and the degree of matching between the image and each text prompt, ultimately achieving a significant improvement in image retrieval accuracy and image-text matching accuracy.

[0032] The above-mentioned image-text matching and grayscale sensitivity detection method, device and equipment first obtain image and text inputs, perform grayscale sensitivity detection by calculating the grayscale sensitivity difference value of the image, identify images that may be tampered with, and ensure data reliability. Feature extraction is then performed on the text input and the image input that has undergone grayscale sensitivity detection, which can accurately capture the key information of the image and text, laying the foundation for subsequent matching. Grayscale sensitivity detection can effectively enhance the robustness of image features and reduce interference caused by small perturbations of the image. In the feature alignment stage, the deployment of the cross-attention mechanism enables the model to focus on the key parts of the image and text that are related to each other, and enhance the semantic connection between the two; by enhancing the diversity of multimodal representations, the feature alignment loss function is designed, and the cosine similarity between image features and text features is used as the calculation basis to guide the image features to align with the target text features, prompting the model to learn more accurate cross-modal mapping relationships and reduce matching errors caused by semantic gaps. The distribution balance loss function balances the feature distribution of the optimized image by maximizing the mean cosine similarity between image and text features and minimizing the second-order moment. This effectively avoids matching bias caused by large differences in feature distribution, ensuring that images and text with similar semantics are more closely clustered in the embedding space, improving matching accuracy. The pixel protection regularization loss function constrains and protects image pixels by calculating all pixel values ​​of the image input after grayscale-sensitive detection, preventing malicious attacks from tampering with image pixels and ensuring the authenticity and integrity of image features, thereby improving the model's robustness against adversarial examples. Finally, the pre-trained CLIP model is optimized and distilled by combining the feature alignment loss function, the distribution balance loss function, and the pixel protection regularization loss function with a pre-set contrast optimization feature loss function. This multi-pronged approach adjusts model parameters, enabling the optimized CLIP model to more accurately output the adjusted image and the degree of match between the image and each text prompt. By addressing modality gaps, ensuring image quality, and detecting image tampering, this application effectively improves image retrieval accuracy and image-text matching accuracy, providing strong support for the development of multimodal applications in a secure and reliable environment.

[0033] In one embodiment, calculating the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection includes:

[0034] Calculate the grayscale sensitivity difference of the image input

[0035] ;

[0036] in, Indicates the number of text prompts, Representing an image With the i Text prompts The cosine similarity between represents a grayscale image, i Indicates the sequence number of the text prompt.

[0037] In one embodiment, a dual-threshold detection mechanism is set up, and when the grayscale sensitivity difference value of the image input is higher than a first threshold and its grayscale image is lower than a second threshold, it is determined to be a tampered image;

[0038] When the image input is higher than the first threshold and its grayscale image is higher than the second threshold, grayscale sensitive detection is performed. The specific process is as follows:

[0039] ;

[0040] ;

[0041] in, Represents the grayscale sensitivity difference value, represents the first threshold, It represents the CLIP similarity score of the original image. Indicates the second threshold.

[0042] A dynamic threshold function based on the complexity of image content is introduced, and the underlying features of the image such as texture and gradient are extracted through ResNet-50 to build a threshold adaptive model:

[0043] ;

[0044] in, is the base threshold (default 0.2), is the image feature vector, is the characteristic mean, is the coefficient.

[0045] In one embodiment, a cross-attention mechanism is deployed in the feature alignment stage and a feature alignment loss function is designed by enhancing the diversity of multimodal representations, including:

[0046] In the feature alignment stage, a cross-attention mechanism is deployed to achieve dynamic feature focusing by calculating the semantic importance weights of textual cues; the semantic importance is:

[0047] ;

[0048] in, To query text features, is the feature vector of the i-th text prompt, is the feature vector of the j-th text prompt, n is the total number of text prompts;

[0049] In the feature alignment stage, the cross-attention mechanism is deployed and the feature alignment loss function is designed by enhancing the diversity of multimodal representation:

[0050] ;

[0051] in, Indicates the number of text prompts, Represents image features, i Indicates the sequence number of the text prompt. Represents text features, and Represents different model intermediate layer features, is the Euclidean distance between features, is the temperature parameter, Represents all feature pairs in the batch Take expectations.

[0052] In one embodiment, the distribution balancing loss function is designed by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment:

[0053] ;

[0054] in, n is the total number of text prompts, i Indicates the sequence number of the text prompt.

[0055] In one embodiment, a pixel protection regularization loss function is designed by calculating all pixel values ​​of the image input through grayscale sensitive detection:

[0056] ;

[0057] in, represents the rectified linear unit function, represents the image input, Indicates the upper limit of image pixels, Indicates the lower limit of the image pixel value, It represents the expectation of all pixel values ​​of the image input.

[0058] In one embodiment, an unsupervised pre-training framework based on InfoNCE loss is constructed, and the feature space is optimized by comparing the original image with the grayscale image, the original image with the enhanced image, etc. The preset comparison optimization feature loss function is:

[0059] ;

[0060] in, is a positive sample, is a negative sample, is the temperature parameter, represents the feature vector of the original image I, Represents the feature vector of the positive sample image I', represents the feature vector of the kth negative sample image, Indicates the total number of negative samples.

[0061] In one embodiment, optimizing a pre-trained CLIP model based on a feature alignment loss function, a distribution balance loss function, and a pixel protection regularization loss function includes:

[0062] The total loss function for optimizing the pre-trained CLIP model is:

[0063] ;

[0064] in, represents the feature alignment loss function, represents the distribution balance loss function, represents the regularized loss function, Represents the contrast optimization feature loss function, 、 and Indicates different weights.

[0065] In another embodiment, knowledge distillation technology can also be used to build a teacher-student network architecture, compressing the optimized CLIP model into a lightweight version. The student model uses the MobileViT architecture and designs a distillation loss function training model: , represents the feature vector extracted by the student model for the input image, represents the feature vector extracted by the optimized CLIP model for the same input, such as the original CLIP-L / 14.

[0066] It should be understood that although Figure 1 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0067] In one embodiment, Figure 3As shown, an image-text matching and grayscale sensitivity detection device is provided, including: a grayscale sensitivity detection and feature extraction module 302, a loss function design module 304 and a model optimization and image matching module 306, wherein:

[0068] Grayscale sensitivity detection and feature extraction module 302 is used to obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection; and perform feature extraction on the text input and the image input that passes the grayscale sensitivity detection to obtain image features and text features;

[0069] The loss function design module 304 is used to deploy a cross-attention mechanism in the feature alignment stage and design a feature alignment loss function by enhancing the diversity of multimodal representations; design a distribution balancing loss function to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment; and design a pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection;

[0070] The model optimization and image matching module 306 is configured to optimize and distill the pre-trained CLIP model according to a feature alignment loss function, a distribution balance loss function, a pixel protection regularization loss function, and a preset contrast optimization feature loss function to obtain an optimized CLIP model; and output the adjusted image and the degree of match between the image and each text prompt using the optimized CLIP model.

[0071] For the specific definition of an image-text matching and grayscale sensitivity detection device, please refer to the definition of an image-text matching and grayscale sensitivity detection method above, which will not be repeated here. The various modules in the above-mentioned image-text matching and grayscale sensitivity detection device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0072] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image-text matching and grayscale sensitivity detection method is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0073] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0074] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0075] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image-text matching and grayscale sensitivity detection method, characterized in that: The method comprises: Obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection; Perform feature extraction on text input and image input through grayscale sensitive detection to obtain image features and text features; In the feature alignment stage, we deploy a cross-attention mechanism and design a feature alignment loss function by enhancing the diversity of multimodal representations. We design a distribution balancing loss function to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment. We design a pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection. Optimizing and distilling the pre-trained CLIP model according to the feature alignment loss function, the distribution balance loss function, the pixel protection regularization loss function, and the preset contrast optimization feature loss function to obtain an optimized CLIP model; Outputting the adjusted image and the degree of matching between the image and each text prompt using the optimized CLIP model; Calculate the grayscale sensitivity difference value of the image input for grayscale sensitivity detection, including: Calculate the grayscale sensitivity difference of the image input: ； in, Indicates the number of text prompts, Representing an image With the i Text prompts The cosine similarity between represents a grayscale image, i Indicates the sequence number of the text prompt; The pixel protection regularization loss function is designed by calculating all pixel values ​​of the image input through grayscale sensitive detection: ； in, represents the rectified linear unit function, represents the image input, Indicates the upper limit of image pixels, Indicates the lower limit of the image pixel value, It represents the expectation of all pixel values ​​of the image input.

2. The method according to claim 1, characterized in that The method further comprises: A dual-threshold detection mechanism is set up. When the grayscale sensitivity difference value of the image input is higher than the first threshold and its grayscale image is lower than the second threshold, it is judged as a tampered image; When the image input is higher than the first threshold and its grayscale image is higher than the second threshold, grayscale sensitive detection is performed. The specific process is as follows: ； ； in, Represents the grayscale sensitivity difference value, represents the first threshold, is the CLIP similarity score of the original image, represents the second threshold; A dynamic threshold function based on image content complexity is introduced, and the underlying features of the image, including texture and gradient, are extracted through ResNet-50 to build a threshold adaptive model: ； in, is the basic threshold, is the image feature vector, is the characteristic mean, is the coefficient.

3. The method according to claim 1, characterized in that In the feature alignment stage, we deploy a cross-attention mechanism and design a feature alignment loss function by enhancing the diversity of multimodal representations, including: In the feature alignment stage, a cross-attention mechanism is deployed to achieve dynamic feature focusing by calculating the semantic importance weights of textual cues; the semantic importance is: ； in, To query text features, is the feature vector of the i-th text prompt, is the feature vector of the j-th text prompt, n is the total number of text prompts; In the feature alignment stage, the cross-attention mechanism is deployed and the feature alignment loss function is designed by enhancing the diversity of multimodal representation: ； in, Indicates the number of text prompts, Represents image features, i Indicates the sequence number of the text prompt. Represents text features, and Represents different model intermediate layer features, is the Euclidean distance between features, is the temperature parameter, Represents all feature pairs in the batch Take expectations.

4. The method according to claim 1, wherein The distribution balance loss function is designed by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment: ； in, n is the total number of text prompts, i Indicates the sequence number of the text prompt.

5. The method according to claim 1, wherein The preset contrast optimization feature loss function is: ； in, is a positive sample, is a negative sample, is the temperature parameter, represents the feature vector of the original image I, Represents the feature vector of the positive sample image I', represents the feature vector of the kth negative sample image, Indicates the total number of negative samples.

6. The method according to claim 1, characterized in that The pre-trained CLIP model is optimized according to the feature alignment loss function, distribution balance loss function and pixel protection regularization loss function, including: The total loss function for optimizing the pre-trained CLIP model is: ； in, represents the feature alignment loss function, represents the distribution balance loss function, represents the regularized loss function, Represents the contrast optimization feature loss function, 、 and Indicates different weights.

7. An image-text matching and grayscale sensitivity detection device, characterized in that: The device comprises: Grayscale sensitivity detection and feature extraction module, used to obtain image input and text input; calculate the grayscale sensitivity difference value of the image input to perform grayscale sensitivity detection, including: Calculate the grayscale sensitivity difference of the image input: ； in, Indicates the number of text prompts, Representing an image With the i Text prompts The cosine similarity between represents a grayscale image, i Indicates the sequence number of the text prompt; extract features from the text input and the image input through grayscale sensitive detection to obtain image features and text features; The loss function design module is used to deploy the cross-attention mechanism in the feature alignment stage and design the feature alignment loss function by enhancing the diversity of multimodal representation; design the distribution balance loss function to balance the feature distribution of the optimized image by maximizing the mean of the cosine similarity between image features and text features and minimizing the second-order moment; design the pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection, including: designing the pixel protection regularization loss function by calculating all pixel values ​​of the image input through grayscale sensitive detection is: ； in, represents the rectified linear unit function, represents the image input, Indicates the upper limit of image pixels, Indicates the lower limit of the image pixel value, Indicates the expectation of all pixel values ​​of the image input; The model optimization and image matching module is used to optimize and distill the pre-trained CLIP model according to the feature alignment loss function, distribution balance loss function, pixel protection regularization loss function, and the preset contrast optimization feature loss function to obtain the optimized CLIP model; the optimized CLIP model is used to output the adjusted image and the degree of match between the image and each text prompt.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images