Image processing method, device and electronic equipment

Abstract

The present disclosure provides an image processing method, device and electronic equipment. The method comprises: obtaining image-text data, the image-text data comprising homologous first and second images and description text corresponding to the first and second images; obtaining first and second image features corresponding to the first and second images through a target image encoder, and obtaining first and second text features of the description text corresponding to the first and second images through a target text encoder; constructing a hard negative sample feature group according to the first and second image features and the first and second text features; determining a comprehensive loss value corresponding to the hard negative sample feature group of a plurality of image-text data through a target loss function, and adjusting parameters of the target image encoder and the target text encoder according to the comprehensive loss value. The present disclosure can improve the resolution capability of the image encoder for fine-grained differences in images.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and more specifically, to an image processing method, apparatus, and electronic device. Background Technology

[0002] Visual language models (VLMs) such as CLIP have achieved significant success in connecting vision and language, but they still fall short in understanding fine-grained details, particularly in the perception of color, quantity, and spatial relationships. Fine-grained enhancement methods, such as region-level contrastive learning, self-distillation methods, and hard negative sample mining strategies, each have their limitations, such as reliance on additional manual region annotation, limitations on the capabilities of teacher models, and a lack of visual similarity or fine-grained control in the generated samples (e.g., random replacement of text or local image regions).

[0003] Therefore, there is a need for better methods to enhance the model's ability to understand images at a finer granular level.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image processing method, apparatus, and electronic device for at least improving the ability of an image encoder to understand the fine-grained detail similarity of images.

[0006] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring image-text comparison data, the image-text comparison data including a first image and a second image, and descriptive text corresponding to the first image and descriptive text corresponding to the second image, wherein the first image and the second image are source images and have fine-grained difference regions, the pixel area ratio of the fine-grained difference regions in the first image and the second image is less than a preset value; acquiring first image features and second image features corresponding to the first image and the second image through a target image encoder, and acquiring first text features of the descriptive text corresponding to the first image and second image and second text features of the descriptive text corresponding to the second image through a target text encoder; constructing a hard-to-bear sample feature group based on the first image features, the second image features, the first text features, and the second text features, the hard-to-bear sample feature group including a first element, a second element, and a third element, wherein the first element and the second element correspond to the same image but different feature types, and the second element and the third element have the same feature types but different corresponding images; determining a comprehensive loss value corresponding to the hard-to-bear sample feature group of multiple image-text comparison data through a target loss function, and adjusting the parameters of the target image encoder and the target text encoder based on the comprehensive loss value.

[0007] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising: an image encoder; wherein the image encoder is a target image encoder as described in any of the preceding image processing methods, and the image encoder is trained by any of the preceding image processing methods.

[0008] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any of the preceding methods based on instructions stored in the memory.

[0009] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the image processing method as described in any of the preceding claims.

[0010] This embodiment of the present disclosure uses a target image encoder and a target text encoder to extract features from two images with fine-grained differences in the same source region and the corresponding text. Then, multiple hard-negative sample feature groups are constructed based on the extracted features. The loss value corresponding to the multiple hard-negative sample feature groups is determined by a target loss function. Then, the target image encoder and the target text encoder are optimized based on the loss value. This allows the target image encoder to improve its fine-grained visual semantic understanding ability by learning from multiple hard-negative samples.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0013] Figure 1 This is a flowchart of an image processing method in an exemplary embodiment of this disclosure.

[0014] Figure 2 This is a schematic diagram of fine-grained differences in an exemplary embodiment of this disclosure.

[0015] Figure 3 This is a schematic diagram illustrating the construction of a hard-to-bear sample feature set in an exemplary embodiment of this disclosure.

[0016] Figure 4 This is a sub-flowchart of step S4 in an exemplary embodiment of this disclosure.

[0017] Figure 5 This is a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0018] Figure 6 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0020] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0021] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of an image processing method in an exemplary embodiment of this disclosure.

[0023] refer to Figure 1 The image processing method 100 may include:

[0024] Step S1: Obtain image-text comparison data, which includes a first image and a second image, as well as descriptive text corresponding to the first image and descriptive text corresponding to the second image. The first image and the second image are from the same source, and there are fine-grained difference regions between the first image and the second image. The pixel area ratio of the fine-grained difference regions in the first image and the second image is less than a preset value.

[0025] Step S2: Obtain the first image features and the second image features corresponding to the first image and the second image through the target image encoder; obtain the first text features of the descriptive text corresponding to the first image and the second text features of the descriptive text corresponding to the second image through the target text encoder.

[0026] Step S3: Construct a hard-to-bear sample feature group based on the first image feature and the second image feature, the first text feature and the second text feature. The hard-to-bear sample feature group includes a first element, a second element and a third element. The first element and the second element correspond to the same image but different feature types. The second element and the third element have the same feature types but different corresponding images.

[0027] Step S4: Determine the comprehensive loss value corresponding to the difficult negative sample feature group of multiple image-text comparison data through the target loss function, and adjust the parameters of the target image encoder and the target text encoder according to the comprehensive loss value.

[0028] This embodiment of the present disclosure uses a target image encoder and a target text encoder to extract features from two images with fine-grained differences in the same source region and the corresponding text. Then, multiple hard-negative sample feature groups are constructed based on the extracted features. The loss value corresponding to the multiple hard-negative sample feature groups is determined by a target loss function. Then, the target image encoder and the target text encoder are optimized based on the loss value. This allows the target image encoder to improve its fine-grained visual semantic understanding ability by learning from multiple hard-negative samples.

[0029] The following is a detailed description of each step of the image processing method 100.

[0030] In step S1, image-text comparison data is obtained. The image-text comparison data includes a first image and a second image, as well as descriptive text corresponding to the first image and descriptive text corresponding to the second image. The first image and the second image are from the same source, and there are fine-grained difference regions between the first image and the second image. The pixel area ratio of the fine-grained difference regions in the first image and the second image is less than a preset value.

[0031] To improve the fine-grained visual semantic understanding capability of the model (image encoder), embodiments of this disclosure use source images and their corresponding descriptive text, along with a text encoder, to train the image encoder.

[0032] Among them, homologous images refer to images that have been modified from the same source. Specifically, they can be images derived from the same original image through local modifications or other operations.

[0033] For example, the second image is an image formed by editing and modifying the first image (since the two are relative, it can also be that the first image is an image formed by editing and modifying the second image), or the first image and the second image are images formed by the same original image after different editing and modification.

[0034] In some embodiments, the fine-grained difference region is a tiny difference region. For example, the first image and the second image are similar in shape but different in species (the first image is a kiwi, and the second image is a kiwifruit), and the rest are the same; or, the first image and the second image are different in color only in the color of a certain element, and the rest are the same.

[0035] In these embodiments, the proportion of pixel area in the fine-grained difference region between the first image and the second image is less than a preset value, and the proportion of pixel area in the second image is also less than a preset value. This preset value is, for example, 5%.

[0036] In other embodiments, the fine-grained difference region is a small, local and non-essential difference region. For example, in the first image and the second image, the pixel area ratio of the fine-grained difference region is less than 5% (preset value), and the average color difference of the corresponding pixels in the fine-grained difference region in the first image and the second image is less than 8 (0-255 color gamut) (preset color threshold).

[0037] In some other embodiments, the fine-grained difference region is a region of subtle feature differences (such as the texture of a bird's feathers or the shape of a car's headlights) found in subcategories within the same major category, i.e., a region of local feature differences between objects of the same type.

[0038] Those skilled in the art can select two images containing any of the aforementioned fine-grained difference regions to form images in the image-text comparison data, based on the weaknesses of the target image encoder, in order to specifically train the target image encoder's ability to recognize such fine-grained difference regions.

[0039] Figure 2 This is a schematic diagram of fine-grained differences in an exemplary embodiment of this disclosure.

[0040] refer to Figure 2 The left side is the first image 21, and the right side is the second image 22. There is a fine-grained difference region 201 between the first image 21 and the second image 22. In the first image 21, the grapes in the fine-grained difference region 201 are green, while in the second image 22, the grapes in the fine-grained difference region 201 are purple. All other parts of the first image 21 and the second image 22 are completely identical.

[0041] This embodiment of the disclosure uses homologous images with fine-grained differences as training images to ensure that the difference between the first image and the second image is only a minor change in the local area of ​​the image from the same source, rather than a natural difference between the images. This allows the image decoder to be trained to recognize meaningful minor differences, rather than abnormal changes between images.

[0042] Unlike related technologies, in this embodiment of the disclosure, in addition to using two images to train the target image encoder, a target text encoder is additionally set to assist in training the target image encoder.

[0043] In an exemplary embodiment, the target text encoder can be any existing text encoder that can be improved during training, preferably a small text encoder with a few parameters. After training of the target image encoder is complete, the target image encoder operates independently, and the target text encoder does not work together with the target image encoder but can participate in the training of the next target image encoder. Each time it participates in training, the parameters of the target text encoder are modified during backpropagation, so there is no need to fix the parameters of the target text encoder.

[0044] In order to enable the target text encoder to assist in training the target image encoder, this embodiment of the present disclosure also adds descriptive text corresponding to the first image (hereinafter referred to as the first descriptive text) and descriptive text corresponding to the second image (hereinafter referred to as the second descriptive text) to the training dataset and the image-text comparison data.

[0045] In some embodiments, the second image is an image formed by editing the first image using local editing instructions; alternatively, the first image and the second image are images formed from the same original image after different editing modifications. The first descriptive text is automatically generated by a third-party model based on the first image, and the second descriptive text is automatically generated by the same third-party model based on the second image.

[0046] In other embodiments, the first image is generated based on the first descriptive text, the second image is generated based on partial editing instructions for the first image and the first image, and the second descriptive text is generated based on partial editing instructions for the first image and the first descriptive text.

[0047] In the above embodiments, both the generation of the second image and the generation of the second descriptive text can be automatically generated by a third-party model. The third-party model for generating the second image and the third-party model for generating the second descriptive text can be the same or different third-party models.

[0048] In some other embodiments, the first descriptive text is automatically generated by a third-party model based on the first image, or the first image is generated based on the first descriptive text. Meanwhile, the second descriptive text is generated based on partial modifications to the first descriptive text, and the second image is generated based on the descriptive text corresponding to the second image.

[0049] exist Figure 2 In the example shown, the first description text corresponding to the first image is: "A realistic photographic style wooden dining table, with sunlight creating shadows through the window. On the table are pale yellow bananas with small brown spots, a bunch of light green grapes, and a full-colored, smooth-surfaced red apple."; the second description text corresponding to the second image is: "A realistic photographic style wooden dining table, with sunlight creating shadows through the window. On the table are pale yellow bananas with small brown spots, a bunch of purplish-red grapes, and a full-colored, smooth-surfaced red apple."

[0050] Regardless of how the first image, second image, first descriptive text, and second descriptive text are generated, multiple image local editing instructions and text local editing instructions can be pre-stored, and then these image local editing instructions and text local editing instructions can be used to assist in generating at least one of the first image, second image, first descriptive text, and second descriptive text.

[0051] In addition, a source image can be stored, and then multiple second images can be generated based on the multiple image local editing instructions and the source image. The source image can then be added as the first image to the image-text comparison data to form multiple sets of image-text comparison data. Alternatively, multiple images with fine-grained differences between them can be generated based on the multiple image local editing instructions and the source image. These images can then be randomly combined in pairs to form the first and second images in the multiple image-text comparison data, thereby forming multiple sets of image-text comparison data.

[0052] To enrich the training data, multiple source images can be stored, and multiple sets of image-text comparison data corresponding to each source image can be generated in the manner described above, thereby forming more image-text comparison data.

[0053] In an exemplary embodiment, the natural fine-grained similarity of the instruction editing dataset (such as UltraEdit[7]) can be directly reused to construct the image-text comparison data, avoiding manual annotation or complex proposals. In addition, one or more third-party models can be used in the process of generating the image-text comparison data, and the third-party model does not need to be used after the image-text comparison data is generated. Subsequently, the generated image-text comparison data can be stored according to the correspondence relationship so that it can be called in step S2.

[0054] In step S2, the first image features and the second image features corresponding to the first image and the second image are obtained by the target image encoder, and the first text features of the descriptive text corresponding to the first image and the second text features of the descriptive text corresponding to the second image are obtained by the target text encoder.

[0055] The embodiments of this disclosure train the target image encoder's semantic understanding ability of fine-grained differences. Therefore, the understanding results of the target image encoder on the image must first be obtained so that it can be evaluated and modified in the future.

[0056] Therefore, in step S2, the first image, the second image, the first descriptive text, and the second descriptive text in a set of image-text comparison data are mapped to the same high-dimensional shared embedding space through the target image encoder and the target text encoder, thereby converting them into image features and text features that can be computed.

[0057] Let the target image encoder be denoted as f(·). The target image encoder is responsible for mapping the input image (first image or second image) to a high-dimensional shared embedding space, generating an image embedding representation, i.e., image features. Let the first image be denoted as Is, and the second image as It, then the features of the first image are denoted as f(Is), and the features of the second image are denoted as f(It). The target image encoder can be a commonly used Transformer-based architecture (such as ViT) image encoder.

[0058] Let the target text encoder be denoted as g(·). The target text encoder is responsible for receiving text input (such as the first descriptive text Ts or the second descriptive text Tt) and mapping it to the same shared embedding space as the image embedding, generating text embeddings, i.e., the first text feature and the second text feature are denoted as g(Ts) and g(Tt), respectively. The target text encoder can be a commonly used Transformer-based architecture (such as a variant of BERT or GPT) text encoder.

[0059] In this context, the shared embedding space refers to the goal of the target image encoder and the target text encoder to map semantically related images and texts to neighboring locations in this space.

[0060] In order to accurately measure the target image encoder’s ability to understand images, N sets of image-text comparison data are used for feature embedding. That is, in step S2, N sets of corresponding embedding feature groups are extracted from the N sets of image-text comparison data. Each embedding feature group includes a set of first image features, second image features, first text features, and second text features corresponding to the image-text comparison data.

[0061] In step S3, a hard-to-bear sample feature group is constructed based on the first image feature and the second image feature, the first text feature and the second text feature. The hard-to-bear sample feature group includes a first element, a second element and a third element. The first element and the second element correspond to the same image but different feature types. The second element and the third element have the same feature types but different corresponding images.

[0062] In this embodiment of the disclosure, the image encoder is trained using positive sample pairs and hard negative sample pairs. Positive sample pairs are images and their corresponding descriptive texts, such as (first image, first descriptive text) and (second image, second descriptive text). Hard negative sample pairs are images and their corresponding descriptive texts for another image in the image-text comparison data, such as (first image, second descriptive text) and (second image, first descriptive text).

[0063] This enables the image encoder to identify the correspondence between positive sample pairs and the non-correspondence between hard negative sample pairs.

[0064] For the same image or text, positive sample pairs and hard negative sample pairs can form a hard negative sample feature group consisting of three elements. The first and second elements in the hard negative sample feature group constitute a positive sample pair, and the first and third elements constitute a hard negative sample pair.

[0065] In an exemplary embodiment, the difficult-to-bear sample feature group includes a first group and a second group for evaluating the distance between image features and text features. The first element of the first group is a first image feature, the second element is a first text feature, and the third element is a second text feature. The first element of the second group is a second image feature, the second element is a second text feature, and the third element is a first text feature.

[0066] In an exemplary embodiment, the difficult-to-bear sample feature group includes a third group and a fourth group for evaluating the distance from text features to image features. The first element of the third group is a first text feature, the second element is a first image feature, and the third element is a second image feature. The first element of the fourth group is a second text feature, the second element is a second image feature, and the third element is a first image feature.

[0067] That is, a set of image-text comparison data can be used to construct four types of difficult-to-bear sample feature groups. Some or all of the above types of difficult-to-bear sample feature groups can be selected for construction.

[0068] When training a target image encoder using N sets of image-text comparison data, M classes of difficult-to-bear sample feature groups can be generated for each set of image-text comparison data. The first, second, and third elements of the m-th class of difficult-to-bear sample feature groups for each set of image-text comparison data are identical, where N ≥ 1 and 1 ≤ m ≤ M. For example, let M = 2, the first class of difficult-to-bear sample feature groups is (first image feature, first text feature, second text feature), and the second class of difficult-to-bear sample feature groups is (second image feature, second text feature, first text feature). Then, the first class of difficult-to-bear sample feature groups for all N sets of image-text comparison data will be (first image feature, first text feature, second text feature), and the second class of difficult-to-bear sample feature groups will be (second image feature, second text feature, first text feature). And so on.

[0069] In addition to first forming features based on image-text comparison data and then forming hard-to-bear sample feature groups based on the features, it is also possible to directly form hard-to-bear sample groups based on image-text comparison data, and then input the hard-to-bear sample groups into the target image encoder and the target text encoder for embedding mapping to form the hard-to-bear sample feature groups corresponding to each hard-to-bear sample group (the hard-to-bear sample groups are formed directly based on the image and text, while the hard-to-bear sample feature groups are formed based on the image features and text features).

[0070] Figure 3 This is a schematic diagram illustrating the construction of a hard-to-bear sample feature set in an exemplary embodiment of this disclosure.

[0071] refer to Figure 3Obtain an instruction editing dataset (such as UltraEdit), which consists of multiple samples. Each sample includes a tuple (source image Is, source image description text Ts, editing instruction Te, target image It, target image description text Tt). The target image It is obtained by editing the source image Is according to the editing instruction Te, and the target image description text Tt is the descriptive text for the target image It. The source image is the first image, the target image is the second image, the source image description text Ts is the first description text, and the target image description text Tt is the second description text.

[0072] Then, positive sample pairs and hard negative sample pairs are constructed based on the data edited according to the instructions. Positive sample pairs include (source image Is, source image description text Ts) and (target image It, target image description text Tt); hard negative sample pairs include (source image Is, target image description text Tt) and (target image It, source image description text Ts), and so on. The hard negative samples in these hard negative sample pairs are visually and semantically very close to the positive samples, but due to subtle changes introduced by the editing instructions, they exhibit key similarities, posing a challenge to the model.

[0073] Next, based on the positive sample pairs and the hard negative sample pairs, construct hard negative sample groups, such as (source image Is, source image description text Ts, target image description text Tt) and (target image It, target image description text Tt, source image description text Ts), etc.

[0074] Finally, the difficult-to-bear sample group is input into the target image encoder and the target text encoder to obtain the difficult-to-bear sample feature group corresponding to the difficult-to-bear sample group, such as (first image feature f(Is), first text feature g(Ts), second text feature g(Tt)) and (second image feature f(It), second text feature g(Tt), first text feature g(Ts)).

[0075] Regardless of the order in which the hard-negative sample feature groups are formed, the hard-negative sample feature groups are evaluated using the target loss function to quantify the target image encoder's ability to understand the differences between positive and hard-negative samples.

[0076] In step S4, the comprehensive loss value corresponding to the difficult negative sample feature group of multiple image-text comparison data is determined by the target loss function, and the parameters of the target image encoder and the target text encoder are adjusted according to the comprehensive loss value.

[0077] In some embodiments, a loss value can be given based on each class of hard-to-bear sample feature groups, and then the loss values ​​corresponding to each class of hard-to-bear sample feature groups can be summed to obtain the comprehensive loss value for this training (i.e., the target image encoder and the target text encoder for the images and text input in this batch).

[0078] Figure 4 This is a sub-flowchart of step S4 in an exemplary embodiment of this disclosure.

[0079] refer to Figure 4 In an exemplary embodiment, step S4 may include:

[0080] Step S41: Obtain the m-th difficult-to-bear sample feature group corresponding to N sets of image-text comparison data. The first element, second element, and third element in the m-th difficult-to-bear sample feature group are the same. N≥1, 1≤m≤M, and M is the number of difficult-to-bear sample feature groups corresponding to one set of image-text comparison data.

[0081] Step S42: Based on the m-th class of difficult-to-bear sample feature group corresponding to the N sets of image-text comparison data, determine the m-th loss value corresponding to the m-th class of difficult-to-bear sample feature group through the m-th loss term corresponding to the m-th class of difficult-to-bear sample feature group. The target loss function consists of M loss terms.

[0082] Step S43: Obtain the comprehensive loss value based on the sum of the M loss values.

[0083] exist Figure 4 In the embodiment shown, for each group of hard-to-bear sample features, the loss value is calculated using a loss term, and the target loss function consists of M loss terms.

[0084] In some embodiments, each loss term has the same form.

[0085] In an exemplary embodiment, for a given anchor point (e.g., the first image xi in the i-th set of image-text comparison data), its corresponding positive sample (e.g., the first descriptive text yi in the i-th set of image-text comparison data), and a set of hard negative samples for that anchor point (e.g., the second descriptive text yj′ in the i-th set of image-text comparison data), the loss term L NegCL (X,Y,Y') has the following form:

[0086]

[0087] Among them, L NegCL Let f(x) represent the m-th loss term corresponding to the m-th class of difficult-to-bear sample feature group, and let X, Y, and Y' be the sets of the first, second, and third elements corresponding to the m-th class of difficult-to-bear sample feature group of N sets of image-text comparison data, respectively; i Let g(y) be the first element in the m-th class of difficult-to-bear sample feature group of the i-th batch of image-text comparison data. i ) represents the second element in the m-th class of difficult-to-bear sample feature group of the i-th batch of image-text comparison data, g(y′) j) represents the third element in the m-th class of difficult-to-bear sample feature group of the j-th batch of image-text comparison data, g(y) k ) represents the second element in the m-th class of difficult negative sample feature group of the k-th batch of image-text comparison data, and τ is the temperature hyperparameter.

[0088] The core of this formula lies in the denominator. This includes negative samples y from the in-batch (N sets of image-text comparison data). k (That is, all descriptive texts in the N sets of image-text comparison data that do not match the first image xi in the i-th set of image-text comparison data), and additionally added with x i Highly similar but with key fine-grained similarity, difficult-to-bear samples y′ j By adding these difficult-to-bear samples to the denominator, the model is motivated to work harder to distinguish the anchor point x. i Its positive sample y i And these extremely challenging, difficult-to-bear samples y′ m .

[0089] Next, for the image-to-text hard negative sample feature set, calculate the image-to-text hard negative sample loss.

[0090] For the first type of hard negative sample feature group (m=1), the first image Is is the anchor point, the first descriptive text Ts is the positive sample, and the second descriptive text Tt is the hard negative sample, i.e. (Xs,Ys,Yt), where Xs={Is}, Ys={Ts}, Yt={Tt}.

[0091] The corresponding loss function term, or loss term, is expressed as:

[0092] L NegCL (Xs,Ys,Yt)(2)

[0093] The hard-to-bear sample feature set and the corresponding loss term are used to train the target image encoder and the target text encoder to align the first image Is with the correct first descriptive text Ts and distinguish it from the second descriptive text Tt that has been changed due to editing instructions, given a first image Is.

[0094] For the second type of hard negative sample feature group (m=2), the first image It is taken as the anchor point, the first image description text Tt is the positive sample, and the second image description text Ts is the hard negative sample. That is, (Xt,Yt,Ys), where Xt={It}, Yt={Tt}, Ys={Ts}.

[0095] The corresponding loss function term, or loss term, is expressed as:

[0096] L NegCL (Xt,Yt,Ys)(3)

[0097] The hard-to-bear sample feature set and the corresponding loss term are used to train the target image encoder and the target text encoder so that, given a second image It, they can align it with the correct second descriptive text Tt and distinguish it from the first descriptive text Ts that has been changed due to editing instructions.

[0098] Image-to-text hard-negative sample loss It is the sum of these two loss terms:

[0099]

[0100] For the hard negative sample feature set of text-to-image, calculate the text-to-image hard negative sample loss.

[0101] For the third type of hard negative sample feature group (m=3), the first descriptive text Ts is the anchor point, the first image Is is the positive sample, and the second image It is the hard negative sample, i.e. (Ys, Xs, Xt).

[0102] The corresponding loss function term, or loss term, is expressed as:

[0103] L NegCL (Ys,Xs,Xt)(5)

[0104] The difficult-to-bear sample feature set and the corresponding loss term are used to train the target image encoder and the target text encoder to align the target image encoder with the correct first image Is and distinguish it from the second image It that has been changed due to editing instructions, given a first descriptive text Ts.

[0105] For the fourth type of hard negative sample feature group (m=4), the second descriptive text Tt is used as the anchor point, the second image It is the positive sample, and the first image Is is the hard negative sample, i.e. (Yt, Xt, Xs).

[0106] The corresponding loss function term, or loss term, is expressed as:

[0107] L NegCL (Yt,Xt,Xs)(6)

[0108] The hard-to-bear sample feature set and the corresponding loss term are used to train the target image encoder and the target text encoder so that, given a second image It, they can align it with the correct second descriptive text Tt and distinguish it from the first descriptive text Ts that has been changed due to editing instructions.

[0109] Image-to-text hard-negative sample loss It is the sum of these two loss terms:

[0110]

[0111] Finally, the contrastive learning loss function of the resulting negative samples is the target loss function L. HN Image-to-text poor sample loss Image-to-text poor sample loss sum:

[0112]

[0113] In other embodiments of this disclosure, the form of the loss term corresponding to different types of difficult sample groups may also be different, and at least one loss term is in the form of the above formula (1).

[0114] Backpropagation is performed on the target image encoder and the target text encoder based on the value of the target loss function to train both simultaneously. Finally, training stops and the parameters of the target image encoder are fixed when the value of the target loss function reaches a preset threshold, or when the loss function value no longer decreases significantly in multiple consecutive iterations, or when the number of training iterations reaches a preset maximum.

[0115] The trained target image encoder can be applied to scenarios such as image retrieval, image classification, cross-modal matching (e.g., image-text semantic alignment), fine-grained image feature extraction, image content understanding, visual question answering (VQA), and feature encoding in image generation tasks. It achieves efficient image representation and subsequent processing by embedding the output image features.

[0116] In the training process of the target image encoder, this disclosure utilizes positive sample pairs and difficult-negative sample pairs to construct difficult-negative sample pairs, thereby obtaining a difficult-negative sample feature group. A specially designed symmetric difficult-negative sample contrastive loss function is then used to train the image encoder (i.e., the Visual Language Model (VLM)). This enables the image encoder to learn and distinguish visual semantic similarity introduced by subtle editing instructions, thereby enhancing its fine-grained understanding ability. By effectively generating high-quality difficult-negative samples using an instruction editing dataset and training the model using the symmetric difficult-negative sample contrastive loss function, the shortcomings of existing visual language models in fine-grained visual semantic understanding can be addressed.

[0117] During the verification process, the OpenAI open-source CLIPViT-L / 14 was used as the baseline model for the experiment and as the target image encoder. The image encoder trained using the training method proposed in this embodiment improved the average accuracy from 18.5% to 30.4% on the MMVP [1] benchmark for fine-grained difference recognition ability.

[0118] method MMVP accuracy OpenAICLIP 18.5％ Image encoder of this disclosure embodiment 30.4％

[0119] Therefore, the present invention generates difficult-to-bear samples by cross-combining a first image with a second descriptive text, etc., while preserving visual context similarity (e.g., only modifying color / attributes), which is closer to the real challenge than random replacement and generates higher-quality difficult-to-bear samples.

[0120] By using a symmetric contrastive loss function, the trained model is forced to learn to distinguish subtle similarities (such as color and spatial relationships) in both image-to-text and text-to-image directions, resulting in explicit fine-grained training. This approach achieves better training results compared to the unidirectional or implicit learning methods of existing approaches.

[0121] Finally, it can be trained directly using the semantic similarity of editing instructions without relying on teacher models or external tools (such as object detectors), resulting in better end-to-end scalability.

[0122] Corresponding to the above method embodiments, this disclosure also provides an image processing apparatus that can be used to execute the above method embodiments.

[0123] Figure 5 This is a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0124] refer to Figure 5 The image processing device 500 may include:

[0125] Image encoder 51;

[0126] The image encoder is the target image encoder described in any of the above embodiments of the image processing method, and the image encoder is trained by the image processing method described in any of the above embodiments.

[0127] Since the functions of the device 500 have been described in detail in their respective method embodiments, they will not be repeated here.

[0128] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0129] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0130] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0131] The following reference Figure 6 To describe an electronic device 600 according to this embodiment of the present invention. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0132] like Figure 6 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0133] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform the method shown in the embodiments of this disclosure.

[0134] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 6201 and / or cache 6202, and may further include read-only memory (ROM) 6203.

[0135] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0137] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0139] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0140] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0144] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0145] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0146] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. An image processing method, characterized in that, include: Obtain image-text comparison data, which includes a first image and a second image, as well as descriptive text corresponding to the first image and descriptive text corresponding to the second image. The first image and the second image are from the same source, and there are fine-grained difference regions between the first image and the second image. The pixel area ratio of the fine-grained difference regions in the first image and the second image is less than a preset value. The first image features and the second image features corresponding to the first image and the second image are obtained by the target image encoder, and the first text features of the descriptive text corresponding to the first image and the second text features of the descriptive text corresponding to the second image are obtained by the target text encoder. A difficult-to-bear sample feature group is constructed based on the first image feature and the second image feature, the first text feature and the second text feature. The difficult-to-bear sample feature group includes a first element, a second element and a third element. The first element and the second element correspond to the same image but different feature types. The second element and the third element have the same feature types but different corresponding images. The comprehensive loss value corresponding to the difficult negative sample feature group of multiple image-text comparison data is determined by the target loss function, and the parameters of the target image encoder and the target text encoder are adjusted according to the comprehensive loss value.

2. The image processing method as described in claim 1, characterized in that, The second image is generated based on the partial editing instructions for the first image and the first image, and the description text corresponding to the second image is generated based on the partial editing instructions for the first image and the description text corresponding to the first image.

3. The image processing method as described in claim 1, characterized in that, The descriptive text corresponding to the second image is generated based on a partial modification of the descriptive text corresponding to the first image, and the second image is generated based on the descriptive text corresponding to the second image.

4. The image processing method as described in claim 1, characterized in that, The hard-to-bear sample feature set includes a first group and a second group for evaluating the distance between image features and text features. The first element of the first group is the first image feature, the second element is the first text feature, and the third element is the second text feature. The first element of the second group is the second image feature, the second element is the second text feature, and the third element is the first text feature.

5. The image processing method as described in claim 1 or 4, characterized in that, The hard-to-bear sample feature set includes a third group and a fourth group for evaluating the distance between text features and image features. The first element of the third group is a first text feature, the second element is a first image feature, and the third element is a second image feature. The first element of the fourth group is a second text feature, the second element is a second image feature, and the third element is a first image feature.

6. The image processing method as described in claim 1, characterized in that, The comprehensive loss value corresponding to the difficult-to-bear sample feature group of multiple image-text comparison data is determined by the objective loss function, including: Obtain the m-th difficult-to-bear sample feature group corresponding to N sets of image-text comparison data. The first element, second element, and third element in the m-th difficult-to-bear sample feature group are the same. N≥1, 1≤m≤M, and M is the number of difficult-to-bear sample feature groups corresponding to one set of image-text comparison data. Based on the m-th class of hard-to-bear sample feature group corresponding to the N sets of image-text comparison data, the m-th loss value corresponding to the m-th class of hard-to-bear sample feature group is determined by the m-th loss term corresponding to the m-th class of hard-to-bear sample feature group. The target loss function consists of M loss terms. The comprehensive loss value is obtained by summing the M loss values.

7. The image processing method as described in claim 6, characterized in that, The loss items include: Among them, L NegCL Let f(x) represent the m-th loss term corresponding to the m-th class of difficult-to-bear sample feature group, and let X, Y, and Y' be the sets of the first, second, and third elements corresponding to the m-th class of difficult-to-bear sample feature group of N sets of image-text comparison data, respectively; i Let g(y) be the first element in the m-th class of difficult-to-bear sample feature group of the i-th batch of image-text comparison data. i ) represents the second element in the m-th class of difficult-to-bear sample feature group of the i-th batch of image-text comparison data, g(y′) j ) represents the third element in the m-th class of difficult-to-bear sample feature group of the j-th batch of image-text comparison data, g(y) k ) represents the second element in the m-th class of difficult-to-bear sample feature group of the k-th batch of image-text comparison data, and τ is the temperature hyperparameter.

8. An image processing apparatus, characterized in that, include: Image encoder; Wherein, the image encoder is the target image encoder as described in any one of the image processing methods as described in claims 1-7, and the image encoder is trained by the image processing method as described in any one of claims 1-7.

9. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1-7 based on instructions stored in the memory.

10. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the method as claimed in any one of claims 1-7.

Images