Methods, apparatuses, devices, and media for processing instances of objects in images

Abstract

Methods, apparatuses, devices, and media for processing instances of objects in images are provided. In one method, a plurality of samples are received, a sample in the plurality of samples including an image and a hint associated with an object in the image. A first contrastive training objective of an instance representation model is determined using a plurality of images in the plurality of samples and objects in the plurality of images, the instance representation model describing an association between the image and an instance representation of the image, the instance representation being features extracted for an instance of the object in the image. A second contrastive training objective of the instance representation model is determined using the plurality of images in the plurality of samples and a plurality of hints. The instance representation model is obtained based on the first contrastive training objective and the second contrastive training objective. The instance representation can be made to pay more attention to local features of a region where the object in the image is located, thereby improving the accuracy of the instance representation model.

Description

Technical Field

[0001] Exemplary implementations of this disclosure generally relate to image processing, and more particularly to methods, apparatus, devices, and computer-readable storage media for processing instances of objects in an image. Background Technology

[0002] Machine learning techniques have been widely used in image processing. Currently, technical solutions based on machine learning have been proposed for processing image-text pairs. However, in some applications, images may include multiple different objects, and it is desirable to determine the relationships between each object and the text. While techniques using manually labeled training data have been proposed, manual labeling is time-consuming and labor-intensive. Furthermore, the accuracy of models trained using unlabeled data is unsatisfactory. Therefore, how to handle object instances in images more effectively has become a challenging and hotly debated area in image processing. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for processing instances of objects in an image is provided. In this method, a plurality of samples are received, the samples including images and cues associated with objects in the images. Using the plurality of images from the plurality of samples and the objects in the plurality of images, a first contrastive training objective for an instance representation model is determined, the instance representation model describing the association between images and instance representations of images, the instance representation being features extracted for instances of objects in the images. Using the plurality of images from the plurality of samples and the plurality of cues, a second contrastive training objective for the instance representation model is determined. Based on the first and second contrastive training objectives, an instance representation model is obtained.

[0004] In a second aspect of this disclosure, an apparatus for processing instances of objects in an image is provided. The apparatus includes: a receiving module configured to receive a plurality of samples, the samples including images and cues associated with objects in the images; a first determining module configured to determine a first contrastive training objective for an instance representation model using the plurality of images and objects in the plurality of samples, the instance representation model describing the association between images and instance representations of images, the instance representation being features extracted for instances of objects in the images; a second determining module configured to determine a second contrastive training objective for the instance representation model using the plurality of images and cues in the plurality of samples; and an acquiring module configured to acquire the instance representation model based on the first and second contrastive training objectives.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] In the following detailed description, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent, taken in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram of a sample to be processed according to an implementation of this disclosure is shown;

[0010] Figure 2 A block diagram is shown illustrating an instance of processing objects in an image according to some implementations of this disclosure;

[0011] Figure 3 A block diagram illustrating the structure of an image processing model according to some implementations of this disclosure is shown;

[0012] Figure 4 A block diagram illustrating the structure of an instance decoder according to some implementations of this disclosure is shown;

[0013] Figure 5 A block diagram illustrating the determination of a first contrastive training target according to some implementations of this disclosure is shown;

[0014] Figure 6 A block diagram illustrating comparative learning training according to some implementations of this disclosure is shown;

[0015] Figure 7 A block diagram illustrating object detection according to some implementations of this disclosure is shown;

[0016] Figure 8 A block diagram is shown representing the distribution in the feature space of some implementations according to this disclosure;

[0017] Figure 9 A flowchart is shown illustrating a method for processing instances of objects in an image according to some implementations of this disclosure;

[0018] Figure 10 A block diagram of an apparatus for processing objects in an image according to some implementations of this disclosure is shown; and

[0019] Figure 11 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0020] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0028] Example Environment

[0029] In recent years, Visual Language Pre-training (VLP) has attracted considerable attention from researchers and has been widely applied. Its aim is to learn from a large number of image-text pairs and acquire knowledge that can be generalized to downstream tasks. In the context of this disclosure, images in an e-commerce environment will be used as a specific example of images to describe the specific details of the technical solution of this disclosure. Here, goods or items in the e-commerce environment can be referred to as "objects," and goods or items appearing in advertising images can be referred to as "instances of objects" (or simply instances). See first. Figure 1 An example of a sample including image-text pairs is described according to an exemplary implementation of this disclosure.

[0030] Figure 1 A block diagram 100 of a sample to be processed according to an implementation of this disclosure is shown. For example... Figure 1As shown, in an e-commerce environment, an image can be an advertising image used to promote a product. For example, for the product "face cream," image 112 and prompt 114 can be provided. Here, image 112 can include "face cream," that is, image 112 includes an instance 116 of the object "face cream." Further, prompt 114 can include a description (e.g., a text description) associated with the instance of the object in image 112. At this time, sample 110 can include image 112 and prompt 114. Similarly, sample 120 can include image 122 and prompt 124, and image 122 can include an instance 126 of the object. Sample 130 can include image 132 and prompt 134, and image 132 can include instances 136 and 138 of the object. Sample 140 can include image 142 and prompt 144, and image 142 can include an instance 146 of the object.

[0031] For ease of description, samples 110, 120, 130, and 140 can be referred to as the first sample, the second sample, the third sample, and the fourth sample, respectively. Images 112, 122, 132, and 142 in each sample can be referred to as the first image, the second image, the third image, and the fourth image, respectively. Hints 114, 124, 134, and 144 in each sample can be referred to as the first hint, the second hint, the third hint, and the fourth hint, respectively.

[0032] The booming development of e-commerce has brought great convenience to people's daily lives, leading to a wide range of item / commodity-based application tasks, such as item classification, item retrieval, and item recommendation. Therefore, building a general-purpose basic model that is applicable to large-scale e-commerce applications can improve applicability and reduce training costs.

[0033] Figure 1 This illustrates the differences between natural images and item images. In natural images, typically most pixels are semantically related to the text. However, in the e-commerce domain, this relevance is less pronounced (e.g., "frying pan" or "coffee machine" might only occupy a small portion of the entire image). Furthermore, item images often consist of multiple sources, such as advertising videos, product pages, customer reviews, and so on.

[0034] To more accurately determine the relationships between items and text, a technique has been developed that utilizes manually labeled training data, and then uses the labeled training data to perform training. However, manual labeling involves a significant investment of manpower and time.

[0035] Several technical solutions have been proposed for visual language pre-training, and significant progress has been made in various downstream tasks of visual language. These methods are based on large-scale image-text pairs to learn reusable general multimodal representations across various tasks. Business-specific multimodal representation learning techniques have been proposed, based on VLP methods to pre-train on rich business image-text pairs, thereby learning image-level representations of items. These techniques only consider the global alignment between images and text, without discovering the specific features contained in e-commerce data for learning instance-centric representations. In other words, existing techniques cannot effectively reflect the local features of instances in images. Directly applying these VLP methods to the e-commerce domain still has shortcomings. If the entire image is simply treated as a single entity for training, foreground and background will inevitably be confused. Therefore, to build a foundational model that can be well generalized to various e-commerce applications, it is necessary to learn item-related instance representations, that is, instance-centric representations of objects in the image.

[0036] At this point, enabling the model to focus on object instances in the presence of background interference has become a research hotspot and challenge. The aim is to process object instances in images more effectively, thereby improving the accuracy of downstream processing tasks.

[0037] An overview of image processing

[0038] Due to the significant differences between natural images and object images, directly applying these frameworks to image-level representations in e-commerce does not yield ideal modeling performance. To at least partially address the shortcomings of existing technologies, an instance-centric multi-modal pre-training technique, called ECLIP (E-commerce Contrastive Language-Image Pre-training), is proposed.

[0039] See Figure 2 Describe the overview of this technical solution. Figure 2 A block diagram 200 is shown illustrating an instance of processing objects in an image according to some implementations of this disclosure. For example... Figure 2 As shown, in the pre-training process 212, an instance representation model 230 can be obtained based on a first contrastive training objective 210 (inter-object multimodal learning) and a second contrastive training objective 220 (intra-object multimodal learning). Here, the instance representation model 230 can describe the association between instances of images. An instance representation is a feature extracted from instances of objects in an image. In other words, this instance representation is closely related to instances of objects in the image and can effectively reflect the specific features of instances in the image.

[0040] To perform the training process, multiple samples can be received, each of which may include an image and a cue associated with an instance of an object in the image. For example... Figure 2 As shown, sample 240 may include image 242 and prompt 244. Specifically, image 242 may include an instance of object 246, and in an e-commerce context, image 242 may be an image including a face cream, and prompt 244 may be an advertising slogan for the face cream, etc.

[0041] In the pre-training process 212, a first contrastive training objective 210 for the instance representation model can be determined based on contrastive learning techniques, utilizing multiple images 242 from multiple samples and objects 246 within those images. A second contrastive training objective 220 for the instance representation model 230 can be determined based on contrastive learning techniques, utilizing multiple images 242 from multiple samples and multiple cues 244. Furthermore, the instance representation model 230 can be obtained based on the first contrastive training objective 210 and the second contrastive training objective 220.

[0042] It will be understood that the instance representation model 230 described herein can be a general multimodal base model that is scalable to a wide range of downstream applications in e-commerce. Here, the first contrastive training objective 210 and the second contrastive training objective 220 can serve as pre-training tasks to improve the accuracy and generality of the instance representation model 230.

[0043] The first contrastive training objective 210 brings the representations of identical objects closer together and pushes the representations of different objects further apart. Besides the presented items, the appearance of positive image samples varies greatly. Making their representations closer in the feature space than negative sample pairs encourages instance queries to focus on the visual region corresponding to the desired item. The second contrastive training objective 220 ensures that only positive queries aggregate the semantics of foreground instances, while negative queries do not. Combining these two preprocessing tasks improves the overall instance representation model 230's ability to learn instance-specific representations of objects from images.

[0044] Detailed information on image processing

[0045] According to an exemplary implementation of this disclosure, instance representation model 230 can be implemented based on an encoder and decoder architecture. Figure 3A block diagram 300 illustrating the structure of an image processing model according to some implementations of this disclosure is shown. Specifically, an image encoder 310 can be used to obtain an image representation of an image, a cue encoder 320 can be used to obtain a cue representation of a cue, and an instance decoder 330 can be used to determine an instance representation 350 of the image based on the image representation and the cue representation. According to an exemplary implementation of this disclosure, two independent encoders can be used to process object-related images and text. Decoders corresponding to the encoders described above can be implemented to aggregate instance-centric object representations without additional manual annotation.

[0046] like Figure 3 As shown, the image processing model may include an image encoder 210, a cue encoder 320, and an instance decoder 330. The symbol x = (x I x T Let x represent the input sample. I and x T Images and text describing the information of the corresponding objects, respectively. For example, x I Image 112 can be represented, and x T This can be represented as cue 114. At this point, image encoder 310 and cue encoder 320 can encode the image-text pair into a sequence of feature embeddings. For example, embeddings 352 and 354 can be obtained. Then, a modality-related projection layer can be used to map the linear sequence of feature embeddings to a joint multimodal feature space. These projected embeddings are further decoded to produce instance representation 350.

[0047] See Figure 3 Describe the two single-mode encoders in more detail. (e.g.) Figure 3 As shown, the image encoder 310 can receive image x I ∈R H×W×C Here, H, W, and C represent the height, width, and number of channels of the image, respectively. According to an exemplary implementation of this disclosure, in determining the image representation, the image can be divided into multiple blocks, and multiple block features and image features of the image can be determined for each block. Subsequently, the image representation can be determined based on the determined features.

[0048] like Figure 3 As shown on the left, an image can be divided into N (e.g., using currently known and / or future-developed visual converters) Figure 3 The diagram shows 3×3 (or other numbers) non-overlapping patches 312. These patches can be flattened into one-dimensional input labels for linear projection to obtain the patch features v1, ..., v of each patch. NFurthermore, positional embeddings can be added. Through hierarchical feature encoding, embedding 352 can be obtained, that is, the visual embedding sequence {v}. cls v1, ..., v N}, where v cls The special markers [CLS] (i.e., image features) used to encode the entire image information are used to obtain an image representation. In other words, image features and block features can be concatenated to obtain an image representation.

[0049] In determining the cue representation, the cue encoder 320 can be used to divide the cue 114 into multiple words (e.g., sub-words), and determine multiple word features of each word and the cue features of the entire cue. Then, the cue representation can be determined based on the determined features.

[0050] like Figure 3 As shown on the right, the cue encoder 320 can be used to process the cue 114. This encoder employs a similar transducer structure, such as a BERT (Bidirectional Encoder Representations from Transformer) based structure. The object cue x can be input into the cue encoder 320. T For example, the prompt may include multiple subwords, and here the text representation of the prompt can be symbolized as M subwords. Similar to the image encoder 310, multiple word features w1, ..., w can be determined. M Furthermore, a special [CLS] marker can be added at the beginning of the sequence to summarize the cue semantics. After encoding, the generated embedding 354, i.e., the language embedding sequence, is represented as {w cls w1, ..., w M}

[0051] In traditional VLP technology solutions, after obtaining the various embeddings described above, one can utilize... and Alignment of orthogonal image pairs is achieved through contrastive learning of 350. Here, g I (·) and g T (·) represents the projection described above. While effective in general domains, this design only considers the alignment between global image and text semantics. However, in images within the e-commerce domain, only the regions containing the desired object instances are foreground information corresponding to the text description. Modeling such image-level alignments does not accurately learn the semantics of the regions where objects reside in the image. Therefore, positive samples 334 and negative samples 332 can be constructed, and then an instance-centric representation 350 can be learned based on the first contrastive training objective 210 and the second contrastive training objective 220.

[0052] According to an exemplary implementation of this disclosure, a set of learnable tags, referred to as instance queries, can be used to represent potential instances in an image. It will be understood that although the details of the prompt have been described above using text as an example, alternatively and / or additionally, the prompt may include an image format. That is, the prompt can be multimodal and can include either text or image formats. According to an exemplary implementation of this disclosure, multiple query representations of multiple queries performed in an image can be determined separately. Here, each query may be associated with a specific text or image of the multimodal prompt. That is, it is desirable to specify the instance probed by the prompt content. Specifically, an instance query can be represented as... And it can be achieved using the following formula 1:

[0053]

[0054] In Formula 1, The modality indicating the prompt, i.e., g I (v cls ) or g T (w cls ), and These are learnable location embeddings and type embeddings, indicating the probe region of the query and the modality type of the prompt. These queries can be aggregated from encoded visual features into instance representations 350 (i.e., ...) by instance decoder 330. During pre-training, for a given sample, there is only one positive cue (about the same object), and the remaining T-1 cues are for other objects.

[0055] The instance decoder 330 can use a set of learnable tags, which can be referred to as instance queries. In each decoder block, these instance queries are updated by interacting with encoded visual features. Through a stack of multiple blocks, they progressively probe for instances of potential objects throughout the image. Furthermore, each instance query is conditioned on specific text or images of a multimodal cue. This design makes it specialized for a particular instance type indicated by the content of its associated cue. Therefore, by specifying the content of the multimodal cue, the instance decoder can adaptively discover corresponding instances.

[0056] The instance decoder 330 can employ a multi-layer structure; for example, it can use a 6-layer structure found in a conventional converter, or it can include more or fewer layers. Here, the multi-layer structure can be sequentially connected, and the instance representation output from layer 0 can be used as the input to layer 1, the instance representation output from layer 1 can be used as the input to layer 2, and so on. Within each layer structure, all encoded... Projecting onto the same feature space as the cue, generating an embedding sequence. This serves as the block representation corresponding to each block.

[0057] Initially, the instance is represented by H. 0 It can be zero-initialized and then fed into the instance decoder 330. The instance decoder 330 can then read all the aforementioned embeddings: Z, Q, and H. 0 As its input, the instance representation can be determined based on multiple block representations, multiple query representations, and the previous instance representation preceding the instance representation. Each layer structure can contain L stacked blocks, each block consisting of a slot attention layer and a self-attention layer.

[0058] According to an exemplary implementation of this disclosure, at each layer in the instance decoder 330, slot attention operations can be performed on multiple query representations, multiple block representations, and previous instance representations to determine the instance representation. That is, at each layer, the instance representation output by the previous layer can be updated to determine the instance representation of the current layer. See also Figure 4 Describe the structure of each layer in the instance decoder 330, Figure 4 A block diagram 400 illustrating the structure of an instance decoder according to some implementations of this disclosure is shown. For example... Figure 4 As shown, the embeddings 410, 412, and 414 identified above can be input into the slot attention layer 420. Here, the slot attention layer 420 can adaptively update the instance representation 350 through interaction with the encoded visual embeddings.

[0059] According to an exemplary implementation of this disclosure, corresponding weights are determined for multiple query representations, multiple block representations, and previous instance representations, respectively. Further, a similarity matrix is ​​determined using the multiple query representations, multiple block representations, previous instance representations, and their corresponding weights, and then the instance representation is determined using the similarity matrix. Specifically, for the l-th slot attention layer, the similarity matrix can be calculated first. This is achieved through a dot product attention mechanism. Specifically, the similarity matrix can be determined in the following way:

[0060]

[0061] In Formula 2, W z and W q It is a learnable projection parameter matrix, H l-1 It is an instance representation generated from layer (l-1). Alternatively and / or additionally, the similarity matrix M can be further normalized using a softmax function over T queries:

[0062]

[0063] At this point, the generated similarity matrix M actually performs soft assignment by calculating the semantic similarity between N visual tags and T instance queries. In this way, each visual tag can be distributed to a specific query based on its similarity score. To aggregate the information of the visual tags into their assigned input queries, the updated instance representation can be determined based on a weighted average calculated from the similarity matrix M.

[0064]

[0065] Finally, the instance representation of layer l is H. l Updates can be performed using the remaining connections:

[0066]

[0067] In Formula 5, W v and W o These are linear transformation parameters. Furthermore, multiple other layers can exist above the slot attention layer 420, for example, an addition & normalization layer 430, a self-attention layer 440, an addition & normalization layer 450, a forward feed layer 460, and an addition & normalization layer 470. Here, the self-attention layer 440 can be used to perform information propagation between each query. Specifically, it provides the previously updated H... l It employs a standard multi-head self-attention mechanism, followed by a fully connected forward feed network. After L consecutive decoder blocks, the final instance representation H is obtained. L It is worth noting that, since multimodal cues only participate in the similarity calculation in Equation 2, the generated H... L It contains only visual modal information.

[0068] Will understand, Figure 4 Taking only one layer of the instance decoder as an example, the instance decoder 330 can be implemented with more, fewer, or different layers, alternatively and / or additionally. In this case, the instance decoder 330 operates similarly to clustering image tags, where each instance query serves as the center of a cluster. Within each decoder block, it determines the location of each tag by measuring the distance between each tag and a center in the semantic space. The cluster centers are then updated softly (Equation 4) based on the calculated distance. By stacking multiple decoder blocks, it can implicitly force each query to focus on a specific region and aggregate instance-level representations.

[0069] The structure of the instance representation model has been described, and the pre-training objectives involved in the training process will be described in detail below. It will be understood that ECLIP here is trained using a large number of unfiltered samples and may have multiple pre-training tasks. According to an exemplary implementation of this disclosure, the instance representation model 230 can be updated using a contrastive loss function. Here, the contrastive loss function can include two directions, thus there is an image-to-cue contrastive loss function and a cue-to-image contrastive loss function.

[0070] According to one exemplary implementation of this disclosure, image-to-text contrastive learning can obtain better unimodal representations. Multiple samples can be given. At this time, image x I and text x T Similarity estimation between them can be: The image-to-text contrast loss function can be determined based on the following formula 6. Furthermore, the pre-training process can be performed based on the following contrastive loss function. Here, the goal of pre-training is to bring image-text pairs related to the same object closer together and to push image-text pairs related to different objects further apart.

[0071]

[0072] Furthermore, the text-to-image contrast loss function can be determined based on the following formula 7. And a pre-training process is performed. At this point, the goal of pre-training is to bring text-image pairs about the same object closer together and push text-image pairs about different objects further apart.

[0073]

[0074] In the above formula, τ is the learnable temperature parameter, and B is the number of samples. Furthermore, the overall objective of contrastive learning can be defined based on loss functions in two directions:

[0075]

[0076] According to an exemplary implementation of this disclosure, the first contrastive training target 210 can be determined based on multimodal contrastive learning between objects. See also Figure 5 To describe the outline of determining the first contrastive training objective 210, Figure 5 A block diagram 500 is shown illustrating the determination of the first contrastive training target 210 according to some implementations of this disclosure. For example... Figure 5 As shown, it can be used Figure 1 Images from multiple samples are used to determine the first contrastive training target 210.

[0077] Here, image 112 in sample 110 and image 122 in sample 120 involve the same object, "face cream," while image 112 in sample 110 and image 132 in sample 130 involve different objects ("face cream" and "frying pan," respectively). These images can then be used to construct image pairs 510 and 520, respectively, to determine the training objectives between objects based on contrastive learning.

[0078] According to an exemplary implementation of this disclosure, the first contrastive training target 210 can be made to bring the instance representations of images 112 and 122 closer together and push the instance representations of images 112 and 132 further apart. In other words, it is expected that the first contrastive training target 210 will bring the instance representations of the positive image pairs 510 closer together and push the instance representations of the negative image pairs 520 further apart.

[0079] See Figure 6 This describes more details on determining the first contrastive training objective. Figure 6 A block diagram 600 illustrating comparative learning training according to some implementations of this disclosure is shown. For example... Figure 6 As shown, a momentum model (e.g., momentum models 610 and 614) is maintained during pre-training, which can indicate the exponential moving average of the original model. Figure 6 As shown, image 112 (i.e., the current sample x) can be processed using the basic model 612, and image 122 (i.e., the positive sample x) can be processed using momentum models 610 and 614 respectively. + ) and image 122 (that is, negative sample x) - At this point, each model can output instance representations 620, 622, and 624 respectively. The parameters of each model can be adjusted by moving the model in a direction that brings the instance representations 620 and 622 closer together and in a direction that moves the instance representations 622 and 624 further apart.

[0080] Specifically, for object sample x i The positive hints generated by the basic model and the momentum model are respectively represented as and Inter-object comparison loss This can be expressed by the following formula:

[0081]

[0082] In Equation 9, samples i and j are positive sample pairs. This is a set of negative samples belonging to other objects. The objective function maximizes the similarity between different samples of the same object while minimizing the similarity between mismatched objects. Since the images of the objects are collected from different sources, their background appearances are often diverse. Therefore, This encourages the generated instance representations to be highly relevant to the desired object, thus helping to align positive cues with corresponding image tags in a fine-grained manner.

[0083] According to an exemplary implementation of this disclosure, an additional instance-text (i.e., text in the prompt) matching loss related to multimodal learning between objects can be provided. This loss can predict whether the instance and text description match. Specifically, given an instance-text pair, the instance-text matching loss can be determined based on the following formula 10:

[0084]

[0085] In Equation 10, ⊙ represents the Hadamard product, and f(·) represents the mapping layer: That is, the matching loss between instances and text can be defined using a typical binary cross-entropy loss function. Furthermore, Equation 10 can be used to perform training during the pre-training process.

[0086] According to an exemplary implementation of this disclosure, a second contrastive training target 220 can be determined based on in-object multimodal learning. In this case, the second contrastive training target 220 can bring closer the distance between instance representations and cues involving the same object, and widen the distance between instance representations and cues involving different objects. In other words, among the multiple samples described above, the distance between the instance representation of image 112 and cue 114 in sample 110 can be brought closer, and the distance between the instance representation of image 132 and cue 114 in sample 130 can be widened.

[0087] For a given sample, during pre-training, only one positive cue describes the object displayed in the image of that sample, while the remaining T-1 cues come from other objects. In this case, the second contrastive training objective 220 aims to ensure that only positive queries can detect the correct foreground instance, while negative queries cannot. Text supervision can be used to apply the intra-object contrastive loss. Assuming index r represents a positive query, the second contrastive loss training objective 220 can be determined based on the following formula 11:

[0088]

[0089] In Formula 11, r represents a positive query, relative to all T-1 negative queries, which brings the positive query closer to the object description.

[0090] Alternatively and / or additionally, entropy regularization can be introduced for the similarity matrix M, that is, the similarity loss function between the image and the cue can be determined based on the following formula 12:

[0091]

[0092] This approach prevents overly smooth similarity scores across all queries.

[0093] According to an exemplary implementation of this disclosure, the overall training objective of the pre-training process can be determined based on some or all of the multiple loss functions described above. Specifically, the overall training objective can be determined based on one or more of the formulas 6 to 12. According to an exemplary implementation of this disclosure, the overall training objective can be determined based on the following formula 13:

[0094]

[0095] It will be understood that although the process of determining instance representation was described above using an e-commerce environment as a specific application context, the technical solutions described above can be implemented in other environments. Here, the object can be an item, commodity, person, or animal, and the prompt can include a description of that item, person, or animal. For example, the technical solutions described above can be implemented in an animal recognition application environment. In this way, instance representation can capture more specific features of a particular animal in an image, thereby improving the accuracy of recognition.

[0096] It will be understood that, although text was used as a specific example of a cue above, cuees can alternatively and / or additionally be represented using image formats. This allows for a more flexible definition of the cue's content, increasing the diversity of samples to be processed. It will also be understood that the training process described above is merely illustrative, and its specific details can be modified based on the needs of the specific application environment; for example, the training process may include more, fewer, or different procedures.

[0097] According to an exemplary implementation of this disclosure, in the instance decoder architecture, instance-level semantics are explicitly aggregated by introducing a set of learnable instance queries. Furthermore, two specially configured preprocessing tasks are proposed, enabling the model to focus on desired object instances without relying on costly manual annotations.

[0098] After pre-training, the generated base model can be used to extract object instance representations with minimal operations. Specifically, given an object sample... We first encode the image-text pairs into an embedding sequence using a single-modal encoder. Then, the text description g... T (w clsThe global representation of a query is treated as a positive query and fed into a decoder connected to T-1 negative queries. For convenience, negative queries can be represented from a standard Gaussian distribution. Sampling is performed. Alternative and / or additional locations can be used to set up negative queries based on other methods. The generated... The generated representation of positive queries was subsequently applied to a wide range of downstream e-commerce tasks.

[0099] The process of obtaining instance representation model 230 has been described. Multiple open datasets can be used to perform the pre-training process described above, and the trained model can be used to perform downstream tasks. According to an exemplary implementation of this disclosure, when determining the pre-training dataset, pre-training data can be collected from popular e-commerce websites. Specifically, the pre-training dataset may include 15 million different objects and over 100 million images, covering approximately 9,000 different categories, such as clothing, daily necessities, instruments, etc. For each object, it has a corresponding prompt (i.e., text description) and multiple images from object detail pages, customer reviews, and corresponding advertising videos. During pre-training, positive sample pairs are constructed by sampling images belonging to the same object from different sources.

[0100] For example, the image encoder 310 can be configured, for example, using the standard ViT network and initialized with weights pre-trained from ImageNet. The cue encoder 320 can be implemented based on the BERTbase architecture. The instance decoder 330 has 6 identical layers and 20 instance queries. Alternatively and / or additionally, two variants of ViT can be used: ViTB / 16 (with 250 million parameters) and ViT-L / 16 (with 480 million parameters), thus implementing instance representation models 230 with different accuracies. During preprocessing, the input image is resized to 224×224 using random cropping and horizontal flip prediction, and the text is tagged with word pieces of a maximum length of 55.

[0101] In the context of this disclosure, instance representation models can be obtained using unfiltered training data. This is built upon the inherent characteristics of the e-commerce data itself, such as... Figure 1 As shown, objects typically have multiple image samples from different sources (e.g., merchants, customer reviews, accompanying advertising videos, etc.). Although these instances may look different due to variations in camera view or scene, they all contain the same object entity. Utilizing the instance-centric multimodal learning model described above can improve image processing performance in the e-commerce domain.

[0102] According to an exemplary implementation of this disclosure, an instance representation model can be used to determine the target instance representation of instances in a target image within a target sample, and downstream tasks associated with the target sample can be performed based on the target instance representation. Specifically, the ECLIP technique can be directly applied to classification evaluation without fine-tuning (i.e., one-shot). Table 1 shows a comparison of the CLIP and ECLIP techniques.

[0103] Table 1 Performance comparison of performing item classification tasks

[0104]

[0105] As shown in Table 1, the ECLIP solution significantly surpasses the accuracy of all existing solutions. For example, compared to CLIP, the accuracy is improved by 10.16%, demonstrating the superiority of the instance representation according to an exemplary implementation of this disclosure.

[0106] Furthermore, Table 2 presents a performance comparison of the retrieval tasks, using accuracy at different recall rates R@K to represent performance. As shown in Table 2, the ECLIP technology solution demonstrates significant advantages.

[0107] Table 2 Performance Comparison of Retrieval Tasks (Image-to-Text / Text-to-Image)

[0108]

[0109] further, Figure 7 A block diagram 700 illustrating object detection according to some implementations of this disclosure is shown. For example... Figure 7 As shown, images 710 and 720 illustrate objects identified by performing a conventional CLIP technique on images 130 and 140, and images 712 and 722 illustrate objects identified by performing an ECLIP technique on images 130 and 140, respectively. Figure 7 In the images, lighter-colored areas (i.e., areas with higher heat) represent the regions where the identified objects are located. It can be seen that the object locations shown in images 712 and 722 are more accurate.

[0110] Figure 8 A block diagram 800 is shown illustrating an example representation of the distribution in the feature space according to some implementations of this disclosure. Figure 8Distribution 810 shows the distribution of the representations of each image determined using the conventional CLIP technique in the feature space, and distribution 820 shows the distribution of the instance representations of each image determined using the ECLIP technique in the feature space. It can be seen that in distribution 820, instance representations of the same object are closer together and farther away from instance representations of different objects. In this way, the feature representations determined according to the ECLIP technique can more accurately represent the relevant features of instances of objects in the image.

[0111] In summary, the ECLIP approach can be directly pre-trained using a large amount of e-commerce related data, and can successfully extract more general, semantically richer, and robust instance representations. Extensive experimental results show that, without further fine-tuning, the ECLIP approach significantly outperforms existing methods on a wide range of downstream tasks, demonstrating its strong portability to real-world e-commerce applications.

[0112] In the context of this disclosure, the proposed ECLIP is particularly suitable for e-commerce scenarios. Unlike conventional global representations of images, ECLIP successfully obtains instance-centric representations through its decoder architecture. By fully leveraging the natural characteristics of e-commerce data and the proposed preprocessing task, ECLIP achieves fine-grained alignment capabilities to realize the desired object instances without relying on any manual annotations. Furthermore, the base model generated after pre-training on large-scale object data can be seamlessly generalized to downstream e-commerce applications. Comprehensive experimental results further demonstrate the superiority of ECLIP: it achieves significant improvements over existing state-of-the-art methods on various real-world e-commerce tasks without any fine-tuning.

[0113] Example process

[0114] Figure 9 A flowchart is shown of a method 900 for processing objects in an image according to some implementations of this disclosure. For example... Figure 9 As shown, at box 910, multiple samples are received, including images and cues associated with objects in the images. At box 920, using the multiple images and objects in the multiple samples, a first contrastive training objective for the instance representation model is determined. The instance representation model describes the association between instances of images, where the instances are features extracted for instances of objects in the images. At box 930, using the multiple images and cues in the multiple samples, a second contrastive training objective for the instance representation model is determined. And at box 940, the instance representation model is obtained based on the first and second contrastive training objectives.

[0115] According to an exemplary implementation of this disclosure, a plurality of samples include a first sample, a second sample, and a third sample, wherein a first image in the first sample and a second image in the second sample include instances of the same object, and the first image and the third image in the third sample include instances of different objects, wherein determining a first contrast training target includes: bringing the first contrast training target closer to the distance between the first instance representation of the first image and the second instance representation of the second image, and pushing the first contrast training target further away from the distance between the first instance representation and the third instance representation of the third image.

[0116] According to an exemplary implementation of this disclosure, the first prompt in the first sample and the second prompt in the second sample point to the same object, and the first prompt and the third prompt in the third sample point to different objects, wherein determining the second contrast training target includes: bringing the second contrast training target closer to the distance between the first instance representation and the first prompt, and pushing the second contrast training target further away from the distance between the third instance representation and the first prompt.

[0117] According to an exemplary implementation of this disclosure, the method 900 further includes: using an image encoder to obtain an image representation of an image; using a cue encoder to obtain a cue representation of a cue; and using an instance decoder to determine an instance representation of an image based on the image representation and the cue representation.

[0118] According to an exemplary implementation of this disclosure, obtaining an image representation includes: dividing an image into multiple blocks using an image encoder; determining multiple block features of the multiple blocks and image features of the image; and determining an image representation based on the multiple block features and image features.

[0119] According to an exemplary implementation of this disclosure, obtaining a prompt representation includes: using a prompt encoder to divide the prompt into multiple words; determining multiple word features of the multiple words and prompt features of the prompt; and determining a prompt representation based on the multiple word features and prompt features.

[0120] According to an exemplary implementation of this disclosure, determining an instance representation includes: determining multiple query representations of multiple queries executed in an image; and determining an instance representation based on a previous instance representation preceding the multiple query representations, multiple block representations, and the instance representation.

[0121] According to an exemplary implementation of this disclosure, determining an instance representation includes performing a slot attention operation on multiple query representations, multiple block representations, and a previous instance representation to determine the instance representation.

[0122] According to an exemplary implementation of this disclosure, performing slot attention operations to determine an instance representation includes: determining corresponding weights for multiple query representations, multiple block representations, and a previous instance representation, respectively; determining a similarity matrix using the multiple query representations, multiple block representations, and the previous instance representation and the corresponding weights; and using the similarity matrix to determine the instance representation.

[0123] According to an exemplary implementation of this disclosure, the method 900 further includes updating the instance representation model based on at least one of the following: an image-to-cue contrastive loss function; a cue-to-image contrastive loss function; a cross-entropy loss function between instance representation and cue representation; and a similarity loss function between image and cue.

[0124] According to one exemplary implementation of this disclosure, the object includes an item, and the prompt includes a description of the item.

[0125] According to one exemplary implementation of this disclosure, the description includes either text data or image data.

[0126] According to an exemplary implementation of this disclosure, the method 900 further includes: using an instance representation model to determine a target instance representation of an instance in a target image of a target sample; and performing a downstream task associated with the target sample based on the target instance representation.

[0127] Example devices and equipment

[0128] Figure 10 A block diagram of an apparatus 1000 for processing instances of objects in an image, according to some implementations of the present disclosure, is shown. The apparatus 1000 includes: a receiving module 1010 configured to receive a plurality of samples, the samples including images and cues associated with objects in the images; a first determining module 1020 configured to determine a first contrastive training objective of an instance representation model using the plurality of images and objects in the plurality of samples, the instance representation model describing the association between images and instance representations of images, the instance representation being features extracted for instances of objects in the images; a second determining module 1030 configured to determine a second contrastive training objective of the instance representation model using the plurality of images and cues in the plurality of samples; and an acquiring module 1040 configured to acquire the instance representation model based on the first and second contrastive training objectives.

[0129] According to an exemplary implementation of this disclosure, a plurality of samples include a first sample, a second sample, and a third sample, wherein a first image in the first sample and a second image in the second sample include instances of the same object, and the first image and the third image in the third sample include instances of different objects, wherein the first determining module includes: a zoom-in module configured to cause a first contrast training target to zoom in the distance between the first instance representation of the first image and the second instance representation of the second image, and a push-out module configured to cause the first contrast training target to push out the distance between the first instance representation and the third instance representation of the third image.

[0130] According to an exemplary implementation of this disclosure, the first prompt in the first sample and the second prompt in the second sample point to the same object, and the first prompt and the third prompt in the third sample point to different objects, wherein the second determining module includes: a zoom-in module configured to make the second contrast training target zoom in the distance between the first instance representation and the first prompt, and a push-out module configured to make the second contrast training target push out the distance between the third instance representation and the first prompt.

[0131] According to an exemplary implementation of this disclosure, it further includes: an image processing module configured to acquire an image representation of an image using an image encoder; a cue processing module configured to acquire a cue representation of a cue using a cue encoder; and a representation determination module configured to determine an instance representation of an image based on the image representation and the cue representation using an instance decoder.

[0132] According to an exemplary implementation of this disclosure, an image processing module is configured to include: a segmentation module configured to segment an image into multiple blocks using an image encoder; a feature determination module configured to determine multiple block features of the multiple blocks and image features of the image; and an image representation determination module configured to determine an image representation based on the multiple block features and the image features.

[0133] According to an exemplary implementation of this disclosure, the prompt processing module includes: a segmentation module configured to segment the prompt into multiple words using a prompt encoder; a feature determination module configured to determine multiple word features of the multiple words and prompt features of the prompt; and a prompt representation determination module configured to determine a prompt representation based on the multiple word features and the prompt features.

[0134] According to an exemplary implementation of this disclosure, the representation determination module includes: a query module configured to determine multiple query representations of multiple queries performed in an image, respectively; and a query-based determination module configured to determine an instance representation based on a previous instance representation preceding the multiple query representations, multiple block representations, and instance representations.

[0135] According to an exemplary implementation of this disclosure, a query-based determination module is configured to include: a slot attention module configured to perform slot attention operations on multiple query representations, multiple block representations, and previous instance representations to determine an instance representation.

[0136] According to an exemplary implementation of this disclosure, the slot attention module includes: a weight determination module configured to determine corresponding weights for multiple query representations, multiple block representations, and previous instance representations, respectively; a similarity determination module configured to determine a similarity matrix using the multiple query representations, multiple block representations, and previous instance representations and the corresponding weights; and a similarity-based determination module configured to determine instance representations using the similarity matrix.

[0137] According to an exemplary implementation of this disclosure, the model further includes an update module configured to update the instance representation model based on at least one of the following: an image-to-cue contrastive loss function; a cue-to-image contrastive loss function; a cross-entropy loss function between the instance representation and the cue representation; and a similarity loss function between the image and the cue.

[0138] According to one exemplary implementation of this disclosure, the object includes an item, and the prompt includes a description of the item.

[0139] According to one exemplary implementation of this disclosure, the description includes either text data or image data.

[0140] According to an exemplary implementation of this disclosure, the apparatus further includes: an instance representation determination module configured to determine a target instance representation of an instance in a target image of a target sample using an instance representation model; and a task execution module configured to perform a downstream task associated with the target sample based on the target instance representation.

[0141] Figure 11 A block diagram of a device 1100 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 11 The computing device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein.

[0142] like Figure 11As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage devices 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0143] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1100.

[0144] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 11 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1120 may include computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0145] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0146] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device that enables computing device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0147] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0148] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0152] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing instances of objects in an image, comprising: Receive multiple samples, wherein the samples include images and cues associated with objects in the images; Using multiple images from the multiple samples and objects in the multiple images, a first contrastive training target for the instance representation model is determined. The instance representation model describes the association between an image and its instance representation, which is a feature extracted for an instance of an object in the image. Using multiple images and multiple cues from the multiple samples, a second contrastive training objective for the instance representation model is determined; as well as Based on the first and second contrastive training objectives, the instance representation model is obtained. The plurality of samples includes a first sample, a second sample, and a third sample. The first image in the first sample and the second image in the second sample include instances of the same object, and the first image and the third image in the third sample include instances of different objects. Determining the first contrast training target includes: bringing the first contrast training target closer to the first instance representation of the first image and the second instance representation of the second image, and pushing the first contrast training target further apart from the first instance representation and the third instance representation of the third image. Wherein the first prompt in the first sample and the second prompt in the second sample point to the same object, and the first prompt and the third prompt in the third sample point to different objects, wherein determining the second contrast training target includes: making the second contrast training target bring the distance between the first instance representation and the first prompt closer, and making the second contrast training target push the distance between the third instance representation and the first prompt further apart.

2. The method according to claim 1, further comprising: An image representation of the image is obtained using an image encoder; The prompt encoder is used to obtain the prompt representation of the prompt; as well as The instance representation of the image is determined using an instance decoder based on the image representation and the cue representation.

3. The method according to claim 2, wherein obtaining the image representation comprises: The image is divided into multiple blocks using the image encoder. Determine the block features of the plurality of blocks and the image features of the image, respectively; as well as The image representation is determined based on the multiple block features and the image features.

4. The method according to claim 2, wherein obtaining the prompt representation includes: The prompt encoder is used to divide the prompt into multiple words; Determine the word features of the multiple words and the prompt features of the prompt; as well as The prompt representation is determined based on the multiple word features and the prompt features.

5. The method of claim 3, wherein determining the instance represents: Determine multiple query representations for the multiple queries executed in the image, respectively; as well as The instance representation is determined based on the plurality of query representations, the plurality of block representations, and the previous instance representation preceding the instance representation.

6. The method of claim 5, wherein determining the instance representation comprises: Slot attention operations are performed on the multiple query representations, the multiple block representations, and the previous instance representation to determine the instance representation.

7. The method of claim 6, wherein performing the slot attention operation to determine the instance representation comprises: Determine the corresponding weights for the multiple query representations, the multiple block representations, and the previous instance representation, respectively; A similarity matrix is ​​determined using the multiple query representations, the multiple block representations, the previous instance representations, and the corresponding weights; as well as The similarity matrix is ​​used to determine the instance representation.

8. The method of claim 1, further comprising updating the instance representation model based on at least one of the following: The contrast loss function from the image to the prompt; The contrast loss function provided to the image; The instance representation and the prompt representation are represented by the cross-entropy loss function; The similarity loss function between the image and the prompt.

9. The method of claim 1, wherein the object comprises an item, and the prompt comprises a description of the item.

10. The method of claim 9, wherein the description includes any one of text data and image data.

11. The method of claim 1, further comprising: The instance representation model is used to determine the target instance representation of the instance in the target image of the target sample; as well as Downstream tasks associated with the target sample are executed based on the target instance representation.

12. An apparatus for processing instances of objects in an image, comprising: A receiving module is configured to receive multiple samples, wherein the samples include images and cues associated with objects in the images; The first determining module is configured to determine a first contrastive training target for an instance representation model using multiple images from the plurality of samples and objects in the plurality of images, wherein the instance representation model describes the association between an image and an instance representation of the image, and the instance representation is a feature extracted for an instance of an object in the image. The second determining module is configured to determine a second contrastive training target for the instance representation model using multiple images and multiple cues from the plurality of samples. as well as The acquisition module is configured to acquire the instance representation model based on the first contrastive training objective and the second contrastive training objective. The plurality of samples includes a first sample, a second sample, and a third sample, wherein the first image in the first sample and the second image in the second sample include instances of the same object, and the first image and the third image in the third sample include instances of different objects. The first determining module is further configured to: cause the first contrast training target to bring the distance between the first instance representation of the first image and the second instance representation of the second image closer, and cause the first contrast training target to push the distance between the first instance representation and the third instance representation of the third image further apart; Wherein the first prompt in the first sample and the second prompt in the second sample point to the same object, the first prompt and the third prompt in the third sample point to different objects, and the second determining module is further configured to: cause the second contrast training target to bring the distance between the first instance representation and the first prompt closer, and cause the second contrast training target to push the distance between the third instance representation and the first prompt further away.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images