Methods, apparatus, devices, and media for processing images using machine learning models

Abstract

Methods, apparatus, devices, and media for processing images using machine learning models are provided. The machine learning model identifies at least one candidate object from an image and includes: a feature extraction model describing the association between features of the image and at least one candidate object; and a classification scoring model describing the association between features and classification scores of at least one candidate object, the classification score representing the probability that at least one candidate object is classified as foreground in the image. In one method, update parameters associated with the classification scoring model are determined based on the classification scores of at least one candidate object and ground truth classification scores of at least one ground truth object in the image. The classification scoring model is updated based on the update parameters associated with the classification scoring model. The use of update parameters associated with the classification scoring model to update the feature extraction model is prevented. This can improve the training accuracy of the machine learning model, thereby improving the accuracy of object recognition.

Description

Technical Field

[0001] Exemplary implementations of this disclosure generally relate to image processing, and more particularly to methods, apparatus, devices, and computer-readable storage media for processing images using machine learning models. Background Technology

[0002] Machine learning technology has been widely used in image processing. Currently, technical solutions based on machine learning for object recognition in images have been proposed. For example, objects can be pre-labeled in images, and these labeled images can be used to train machine learning models. However, the images used as training data cannot possibly include all objects in the real world. This means that when using a trained machine learning model to process an image, it can only identify the labeled objects from the training data, and cannot identify unlabeled objects. Therefore, how to train machine learning models more effectively to improve recognition accuracy has become a challenge and a hot topic in the field of image processing. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for processing images using a machine learning model is provided. Here, the machine learning model is used to identify at least one candidate object from an image. The machine learning model includes: a feature extraction model for describing the association between features of the image and at least one candidate object; and a classification scoring model for describing the association between features of at least one candidate object and a classification score of at least one candidate object, the classification score representing the probability that at least one candidate object is classified as foreground in the image. In this method, update parameters associated with the classification scoring model are determined based on the classification scores of at least one candidate object and ground truth classification scores of at least one ground truth object in the image. The classification scoring model is updated based on the update parameters associated with the classification scoring model. Updating the feature extraction model using the update parameters associated with the classification scoring model is prevented.

[0004] In a second aspect of this disclosure, an apparatus is provided for processing images using a machine learning model. Here, the machine learning model is used to identify at least one candidate object from the image. The machine learning model includes: a feature extraction model for describing the association between features of the image and at least one candidate object; and a classification scoring model for describing the association between features of the at least one candidate object and a classification score of the at least one candidate object, the classification score representing the probability that the at least one candidate object is classified as foreground in the image. The apparatus includes: a determination module configured to determine update parameters associated with the classification scoring model based on the classification scores of the at least one candidate object and ground truth classification scores of at least one ground truth object in the image; an update module configured to update the classification scoring model based on the update parameters associated with the classification scoring model; and a blocking module configured to prevent the feature extraction model from being updated using the update parameters associated with the classification scoring model.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] In the following detailed description, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent, taken in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing an image used as training data for a machine learning model in an implementation of this disclosure;

[0010] Figure 2 A block diagram illustrating the structure of a machine learning model for recognizing objects from images according to some implementations of this disclosure is shown;

[0011] Figure 3A block diagram is shown illustrating a process for determining update parameters of a mask model according to some implementations of this disclosure;

[0012] Figure 4 A block diagram is shown illustrating a process for determining update parameters of a bounding box model according to some implementations of this disclosure;

[0013] Figure 5 A block diagram is shown illustrating a process for determining update parameters of a positioning scoring module according to some implementations of this disclosure;

[0014] Figure 6 A block diagram of multiple candidate objects in an image is shown according to some implementations of this disclosure;

[0015] Figure 7 A block diagram is shown illustrating a process for selecting positive and negative samples from multiple candidate objects according to some implementations of this disclosure;

[0016] Figure 8 A block diagram is shown illustrating the use of machine learning models to identify objects from an image to be processed, according to some implementations of this disclosure;

[0017] Figure 9 A flowchart is shown illustrating a method for processing images using a machine learning model according to some implementations of this disclosure;

[0018] Figure 10 A block diagram of an apparatus for processing images using a machine learning model, according to some implementations of this disclosure, is shown; and

[0019] Figure 11 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0020] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0028] Example Environment

[0029] In the context of this disclosure, an object can represent an entity with a tangible shape in an image, such as, but not limited to, people, animals, objects, etc. In the context of this disclosure, fruit will be used as an example of an object to describe how to identify objects in an image. In other application environments, video can include other types of objects. For example, in a logistics management system, packages during logistics transportation can be identified and tracked; in a traffic monitoring environment, individual vehicles in a road environment can be identified and tracked, and so on.

[0030] See Figure 1 Describe the outline used to train the machine learning model. Figure 1 A block diagram 100 illustrates an image used as training data for a machine learning model, representing an implementation of this disclosure. Figure 1 As shown, image 110 can be used as training data to train a machine learning model. Here, it is expected that the machine learning model can learn the relationships between images and one or more objects within the images from the training data. Subsequently, the image to be processed can be input into the trained machine learning model so that the machine learning model can identify the objects within the image.

[0031] Image 110 can be used to train a machine learning model, where image 110 may include ground truth objects 112, ..., and 114. Here, a ground truth object refers to an object in image 110 that is pre-labeled (e.g., manually and / or otherwise). During training, the machine learning model can continuously learn the relationships between the image and one or more ground truth objects 112, ..., and 114 in the image. It will be understood that although image 110 also includes other objects (e.g., lemons, apples, strawberries, etc.), these other objects are not labeled. For example, image 110 may further include unlabeled object 116 (lemon). Alternatively and / or additionally, image 110 may include only two objects, pineapple and banana, and both of these objects are labeled as ground truth objects.

[0032] Will understand, although Figure 1 The single image 110 used as training data is shown only schematically. A large number of training images can be obtained, and each training image can include the same or different types of ground truth objects. The training process can be performed iteratively using a large number of training images to obtain an optimized machine learning model.

[0033] However, because the real world contains a vast array of fruits, it is difficult to provide training images that include all types of fruits, and it is also difficult to label each type of fruit in the training images. This causes machine learning models to learn only about a limited number of fruits (i.e., the labeled fruits), and thus, when identifying objects from an image to be processed, they can only recognize the fruits that have already been labeled in the training images. In other words, assuming that the labeled fruits in the training images only include pineapples and bananas, when an image containing pineapples, bananas, and lemons is input into the machine learning model, the model can only recognize pineapples and bananas, but cannot recognize lemons, and the machine learning model may incorrectly identify lemons as background.

[0034] At this point, how to train machine learning models in a more effective way, and enable them to identify unlabeled objects in training images to improve recognition accuracy, has become a challenge and a hot topic in the field of image processing.

[0035] Architecture of object recognition model

[0036] To at least partially address the shortcomings of the prior art, a method for processing images using a machine learning model is proposed according to an exemplary implementation of this disclosure. See also Figure 2 This describes a summary of an exemplary implementation according to the present disclosure. Figure 2 A block diagram 200 illustrates the structure of a machine learning model for recognizing objects from images, according to some implementations of this disclosure. For example... Figure 2 As shown, the machine learning model 210 may include multiple internal models. For example, a backbone model 220 may be implemented based on multiple network models currently known and / or to be developed in the future. This backbone model 220 can perform processing on the received image, such as identifying candidate objects in the image based on a query, so that the downstream feature extraction model 222 can extract features of each candidate object in the image. The feature extraction model 222 can describe the association between the image and the features of at least one candidate object. In other words, the feature extraction model 222 can extract features 224 of each candidate object.

[0037] Furthermore, the machine learning model 210 may include: a mask model 232 for describing the relationship between the features of at least one candidate object and the region of at least one candidate object; a bounding box model 234 for describing the relationship between the features of at least one candidate object and the bounding box of at least one candidate object in the image; a localization scoring model 236 for describing the relationship between the features of at least one candidate object and the localization score of at least one candidate object, wherein the localization score represents the difference between the localization of at least one candidate object and the ground truth localization of at least one ground truth object; and a classification scoring model 238 for describing the relationship between the features of at least one candidate object and the classification score of at least one candidate object, wherein the classification score represents the probability that at least one candidate object is classified as foreground in the image.

[0038] According to an exemplary implementation of this disclosure, the machine learning model 210 may further include a contrastive learning model 230. Here, the contrastive learning model 230 can adjust the parameters of the upstream modules based on contrastive learning, so that the feature extraction model 220 can narrow the distance between the features of the labeled object and the features of candidate objects of the same type as the labeled object, and widen the distance between the features of the labeled object and the features of candidate objects of a different type from the labeled object. According to an exemplary implementation of this disclosure, the aforementioned modules in the machine learning model 210 can be trained individually and / or in combination, so that the machine learning model 210 can obtain information about the association between images and objects.

[0039] Model training process

[0040] According to an exemplary implementation of this disclosure, update parameters for each model (e.g., update gradients determined based on a loss function) can be determined based on various ground-value data of an image, so that each model can be trained accordingly. For example, update parameters associated with a classification scoring model can be determined based on the classification scores of at least one candidate object in the image and the ground-value classification scores of at least one ground-value object in the image. It will be understood that, generally speaking, an image may include foreground and background. For example, as Figure 1 As shown in image 110, various fruits belong to the foreground of image 110, and the white blank area belongs to the background of image 110. The probability of a candidate object belonging to the foreground can be represented by a value between [0,1].

[0041] The probability of a candidate object being labeled as foreground can be used as a classification score. In this case, the ground truth label of image 110 can include foreground and background regions, with a classification score of 1 for the foreground region and a classification score of 0 for the background region. During forward propagation, update parameters (referred to as classification update parameters) associated with classification scoring model 238 can be determined based on the classification scores of at least one candidate object and the ground truth classification scores of at least one ground truth object in the image. Specifically, features 224 can be input into classification scoring model 238 to predict the classification scores of candidate objects. Then, the predicted classification scores can be compared with the ground truth scores of ground truth objects (i.e., classification scores of 0 or 1 in the ground truth label) to determine the loss function of classification scoring model 238. Subsequently, the update gradient of the model can be determined in the direction that minimizes the loss function, and this gradient is used as the update parameters for updating classification scoring model 238.

[0042] According to an exemplary implementation of this disclosure, the classification scoring model 238 can be updated based on determined update parameters; however, as Figure 2 As shown by the dashed line 260, the backpropagation of this update parameter can be truncated, and the update gradient associated with the classification scoring model 238 is not used when updating the upstream models (e.g., backbone model 220, feature extraction model 222). In other words, the use of classification update parameters is prevented when updating upstream models.

[0043] Using the exemplary implementation of this disclosure, the classification update parameters are used only to update the update parameters of the classification scoring model 238, and not to update the upstream model. In this way, on the one hand, it can be ensured that the classification scoring model 238 can learn the semantic knowledge related to the ground truth foreground and ground truth background in image 110; on the other hand, by not considering foreground and background related knowledge when updating the feature extraction model 222, the generated features 224 will not overemphasize the boundary between the foreground and background of the image. In this way, when the machine learning model 210 processes new types of fruit (e.g., lemons), lemons will not be identified as background regions, but rather as new types of fruit not labeled in the training data.

[0044] The process of training the classification and scoring model 238 has been described; further details regarding the training of other models will be provided below. According to an exemplary implementation of this disclosure, the mask model 232 can be updated based on information in the ground truth annotations of image 110. Here, the mask model 232 can describe the association between features of at least one candidate object and the region in which the candidate object is located in image 110. Specifically, based on a comparison of the ground truth mask and the predicted mask output by the mask model 232, a corresponding loss function can be determined, thereby determining the update parameters (i.e., update gradients) used to update the mask model 232. For ease of description, the update parameters associated with the mask model 232 can be referred to as mask update parameters.

[0045] Figure 3 A block diagram 300 illustrates a process for determining update parameters of a mask model according to some implementations of this disclosure. For example... Figure 3 As shown, the truth labeling of image 110 may include a truth object 112 and a corresponding truth mask 310, which can represent the pixel range of the truth object 112 in image 110. It will be understood that... Figure 3 The ground truth mask 310 in the diagram is merely illustrative; in a real-world environment, the ground truth mask 310 can have higher accuracy. During training, features 224 can be input into the mask model 232, and the mask model 232 can output a prediction mask 320.

[0046] The predicted mask 320 can be compared with the ground truth mask 310 to determine the corresponding loss function and thus the mask update parameters used to update the mask model 232. The mask update parameters can then be used to update the mask model 232. Furthermore, since the quality of features 224 will affect the performance of the mask model 232, the mask update parameters can be considered during the update of the feature extraction model 222. In other words, the mask update parameters will be used to update both the upstream model and the mask model 232. In this way, the parameters of the feature extraction model 232 can be updated in a way that makes the generated features more helpful in narrowing the difference between the predicted mask 320 and the ground truth mask 310.

[0047] According to an exemplary implementation of this disclosure, the bounding box model 234 can be updated based on information in the ground truth annotation of image 110. Here, the bounding box model 234 can describe the association between features of at least one candidate object and the bounding box of at least one candidate object in the image. Specifically, the corresponding loss function can be determined based on the comparison between the ground truth bounding box and the predicted bounding box output by the bounding box model 234, thereby determining the update parameters (i.e., update gradients) used to update the bounding box model 234. For ease of description, the update parameters associated with the bounding box model 234 can be referred to as bounding box update parameters.

[0048] Figure 4 A block diagram 400 illustrates a process for determining update parameters of bounding box model 234 according to some implementations of this disclosure. For example... Figure 4 As shown, the ground truth label of image 110 may include a ground truth object 112 and a corresponding ground truth bounding box 410, which can represent the rectangular range of the ground truth object 112 in image 110. During training, features 224 can be input into the bounding box model 234, and the bounding box model 234 can output a predicted bounding box 420.

[0049] The predicted bounding box 420 can be compared with the ground truth bounding box 410 to determine the corresponding loss function and thus the bounding box update parameters used to update the bounding box model 234. The bounding box update parameters can then be used to update the bounding box model 234. Furthermore, since the quality of features 224 will affect the performance of the bounding box model 234, the bounding box update parameters can be considered during the update of the feature extraction model 222. In other words, the bounding box update parameters will be used to update both the upstream model and the bounding box model 234. In this way, the parameters of the feature extraction model 232 can be updated in a way that makes the generated features more helpful in narrowing the difference between the predicted bounding box 420 and the ground truth bounding box 410.

[0050] According to an exemplary implementation of this disclosure, the localization scoring model 236 can be updated based on the localization information of each object. Here, the localization scoring model 236 can describe the correlation between the features of at least one candidate object and the localization score of at least one candidate object. Specifically, the localization of at least one candidate object can be compared with the ground truth localization of at least one ground truth object to determine the corresponding loss function and thus determine the update parameters (i.e., update gradients) used to update the localization scoring model 236. For ease of description, the update parameters associated with the localization scoring model 236 can be referred to as localization update parameters.

[0051] Figure 5 A block diagram 500 illustrates a process for determining updated parameters of a location scoring model according to some implementations of this disclosure. For example... Figure 5 As shown, the localization score of the candidate object can be determined based on the intersection-union ratio of the predicted bounding box 420 and the ground truth bounding box 410, according to Formula 1 below.

[0052]

[0053] Formula 1

[0054] In Formula 1, This indicates the location score of the candidate object. This represents the truth bounding box 410, and... This indicates a predicted bounding box of 420. In this way, the localization score of each candidate object can be determined through simple mathematical operations. This will be understood... The score ranges from 0 to 1, with higher values ​​indicating more accurate localization. When the predicted bounding box 420 coincides with the ground truth bounding box 410, A score of 1 indicates that the location scoring model 236 has the highest accuracy.

[0055] According to an exemplary implementation of this disclosure, each candidate object identified from image 110 can be processed in a similar manner and the corresponding [object] can be determined. Scoring. The corresponding loss function can be determined, thereby determining the localization update parameters used to update the localization scoring model 236. The localization update parameters can be used to update the localization scoring model 236. Furthermore, since the quality of feature 224 will affect the performance of the localization scoring model 236, the localization update parameters can be considered during the update of the feature extraction model 222. In other words, the localization update parameters will be used to update the upstream model and the localization scoring model 236. In this way, the parameters of the feature extraction model 232 can be updated to make the generated features more conducive to improving the localization score (e.g., closer to 1).

[0056] According to an exemplary implementation of this disclosure, the machine learning model 210 further includes a contrastive learning model 230. The contrastive learning model 230 uses contrastive learning principles and ensures that the distribution of features 224 in the feature space 240 is consistent with the distribution of features of the labeled objects. Specifically, it is expected that the update parameters generated by the contrastive learning model 230 can update the feature extraction model 222 in the following directions: bringing closer the distance between the features of the labeled objects and the features of objects corresponding to the labeled objects, and distancing the distance between the features of the labeled objects and the features of objects not corresponding to the labeled objects.

[0057] Based on the principle of contrastive learning, positive and negative samples can be selected from at least one candidate object for contrastive learning. See also Figure 6 To describe more details about the sample selection, the Figure 6 A block diagram 600 showing multiple candidate objects in an image according to some implementations of this disclosure is illustrated. For example... Figure 6 As shown, suppose multiple candidate objects 610, 612, 614, ..., and 616 are identified from image 110. Each candidate object can be compared with the ground truth object in the ground truth annotation of image 110. Specifically, a global queue can be constructed to store the relevant features of the candidate objects corresponding to the ground truth objects; see [link to relevant documentation]. Figure 7This describes more details about selecting positive and negative samples based on a global queue. Figure 7 A block diagram 700 illustrates a process for selecting positive and negative samples from multiple candidate objects according to some implementations of this disclosure.

[0058] Each ground truth object 112, ..., 114 can be input into the feature extraction model 222 to determine the corresponding ground truth features. The features of each candidate object can be compared with the features of each ground truth object to select similar candidate objects from among the candidate objects. At this point, both the features of the candidate objects and the ground truth features of the ground truth objects are output by the feature extraction model 222. Therefore, by comparing the two types of features, it is helpful to select positive and negative samples for training the contrastive learning model. Figure 7 As shown, features 712, ..., 714 corresponding to similar candidate objects 112, ..., 114 can be added to the global queue 710.

[0059] According to an exemplary implementation of this disclosure, pooling operations can be performed on the global queue 710 to determine feature centers 716. In this way, feature centers 716 can reflect the distribution trend of features of ground truth objects in feature space 240, thus further aiding in the selection of positive and negative samples. For example, candidate objects with a small distance from feature center 716 can be selected as positive samples, and candidate objects with a large distance from feature center 716 can be selected as negative samples. In this way, by performing the training process using the selected positive and negative samples, the feature extraction model 222 can be updated in a way that improves the degree to which the extraction model 222 can distinguish objects by features.

[0060] like Figure 7 As shown, multiple candidate objects 610, 612, 614, ..., 616... can be sorted in ascending order of distance to generate sequence 720. In this sequence, the features of the candidate objects at the beginning of sequence 720 are closer to the feature center, and therefore these candidate objects can be used as positive samples. The features of the candidate objects at the end of sequence 720 are more distant from the feature center, and therefore these candidate objects can be used as negative samples.

[0061] According to an exemplary implementation of this disclosure, positive and negative samples can be selected from both ends of sequence 720, respectively. For example, a first number (e.g., a positive integer) with a small distance can be selected from the beginning position of sequence 720. Candidates can be selected as positive samples. Alternatively, negative samples can be selected from positions following each positive sample in sequence 720. Specifically, a second number (e.g., a positive integer) with a large distance from the tail position can be selected. Candidate objects are selected as negative samples. In this case, positive samples are easily distinguishable simple positive samples, and negative samples are easily distinguishable simple negative samples. In this way, models can be trained by oriented towards directions that make features more helpful in distinguishing objects.

[0062] It will be understood that although the above sample selection method makes it easier for the model to distinguish different objects, the positive and negative samples are too different and therefore too simplistic, which is not conducive to the model distinguishing the details between the various objects. Therefore, a second number of candidate objects can be selected from those adjacent to the first number of candidate objects in sequence 720 as negative samples.

[0063] like Figure 7 As shown, in the already selected If candidate objects 610, ..., 612 are used as positive samples 730, then the sample immediately adjacent to positive sample 730 can be selected. Candidate objects 614, ..., 616 are used as negative samples 740. Since negative sample 740 is adjacent to positive sample 730, it has a smaller distance from feature center 716 compared to the candidate objects located at the end of sequence 720. In this case, negative sample 740 will be used as a "hard" negative sample in training. In this way, the model can learn more details to distinguish each object, thus making the output feature 224 more helpful in distinguishing objects in the image.

[0064] The specific process for selecting samples is described below. A sample of length can be maintained. The global queue 710 is used to store the features of labeled ground truth objects in the training images. Specifically, the features included in the images... After an object is processed by feature extraction model 222, it can generate Features (dimension: ,in (Representing the dimension of the feature). One or more candidate objects most similar to the truth object can be selected from the above features, and the features of the selected candidate objects are added to the global queue 710. Furthermore, the global queue 710 can be pooled to obtain the feature centers 716 (using vectors). (This indicates that...) Furthermore, it can be... Positive and negative sample selection is performed among the candidate objects to complete contrastive learning in the feature space 240.

[0065] like Figure 7As shown on the right, in feature space 240, a pentagram represents the feature center 716, a circle represents positive samples 730, and a square represents negative samples 740. In this configuration, positive samples 730 are closer to the feature center 716, while negative samples 740 are farther from it. This approach makes the features output by the feature extraction module more effective in distinguishing between different objects.

[0066] According to an exemplary implementation of this disclosure, positive and negative samples can be selected based on an optimal transmission algorithm. Specifically, it can be calculated... One predicted candidate and Loss matrix between labeled true objects The loss can be determined by comprehensively considering both classification and localization scores. Specifically, the loss between the predicted candidate object and the ground truth object can be determined based on the following formula 2:

[0067]

[0068] Formula 2

[0069] In Formula 2, Indicates loss, and These represent the weights and loss associated with the classification and scoring model, respectively. and These represent the weights and loss associated with the bounding box model, respectively. Alternatively and / or additionally, the weights can be determined based on both the loss inherent to the bounding box and the loss associated with IoU. Candidate samples can be ranked based on their loss to form sequence 720. Furthermore, the sample with the minimum loss can be selected from sequence 720. Each candidate object is used as a positive sample, and after selecting the positive sample... Several candidate objects are used as negative samples. At this time, Each candidate object can be used as a hard negative sample.

[0070] According to an exemplary implementation of this disclosure, after determining positive samples and After obtaining a number of negative samples, these samples can be used to perform training so that the feature extraction model 222 is updated in a direction that is more helpful in distinguishing objects. Specifically, assuming that using... Representing the sample space, using Representing the features of positive samples (objects) and utilizing The features representing negative samples (background) can be used to determine the contrast loss based on the following formula 2.

[0071]

[0072] Formula 3

[0073] In Formula 3, Indicates comparative loss, Representing the sample space, using Features representing positive samples Represents the features of negative samples, and The feature center is represented. The update parameters (e.g., update gradients) associated with the contrastive learning model 230 can be determined based on the contrastive loss defined in Equation 3. For ease of description, the update parameters associated with the contrastive learning model 230 can be referred to as the contrastive update gradients. Furthermore, the contrastive update gradients can be used to update the feature extraction model 222 during training.

[0074] In this way, the features in the queue correspond to the labeled objects in the training data, making it easy to distinguish labeled objects from unlabeled objects, thus improving the discriminative power of the features. Furthermore, the position of the feature center in the feature space is stable, thereby improving the consistency of the features. In this way, the discriminative power and consistency of the features output by the feature extraction model can be improved, which in turn helps to improve the accuracy of machine learning models in extracting objects from images.

[0075] According to an exemplary implementation of this disclosure, the image 110 used as training data may include at least one labeled ground truth object, and the image 110 further includes at least one unlabeled object. In other words, the training data may not label all objects in the real world. In this case, by utilizing the exemplary implementation of this disclosure, the boundary between foreground and background in the training image can be weakened by truncating the update gradient generated by the loss function of the classification scoring model 238, thereby preventing the machine learning model 210 from incorrectly identifying an object of unknown classification (e.g., a lemon) as background.

[0076] Furthermore, through the contrastive learning process described above, the features output by feature extraction model 222 can narrow the distance between candidate objects and labeled objects, and widen the distance between candidate objects and unlabeled objects. In this way, it is easier to update feature extraction model 222 in a direction that improves object recognition accuracy.

[0077] It will be understood that although the process of training the feature extraction model 222 based on the loss functions associated with each model has been described above, all and / or parts of the above process can be combined to train the feature extraction model 222. The process of determining the overall loss function based on loss functions from multiple aspects will be described below with specific formulas.

[0078] According to an exemplary implementation of this disclosure, the overall loss function during training can be determined by combining the loss functions associated with the multiple aspects described above (mask model 232, bounding box model 234, localization scoring model 236, classification scoring model 238, and contrastive learning model 230). The overall loss function can be determined based on the following formula 4:

[0079]

[0080] Formula 4

[0081] In Formula 4, Represents the overall loss function. and These represent the weights and loss of the classification and scoring model 238, respectively. and These represent the weights and loss associated with the bounding box model 234, respectively. and These represent the weights and loss associated with mask model 232, respectively. and Let represent the weights and loss related to the location scoring model 236, respectively. and These represent the weights and loss associated with the contrastive learning model 230, respectively. In this way, multiple factors can be comprehensively considered when training the feature extraction model 222, making the features 224 output by the feature extraction model 222 more suitable for object recognition. This approach can further improve the accuracy of object recognition.

[0082] It will be understood that the training process described above is merely illustrative, and the specific details of the training process can be modified based on the needs of the specific application environment. For example, the training process may include more, fewer, or different processes.

[0083] Model application process

[0084] The training process of each model in machine learning model 210 has been described above. The following section describes how to use machine learning model 210 to identify objects in an image to be processed. According to an exemplary implementation of this disclosure, an image to be processed can be input into machine learning model 210, at which point machine learning model 210 can identify and output relevant information about each object.

[0085] Figure 8 A block diagram 800 illustrates the use of a machine learning model to identify objects from an image to be processed, according to some implementations of this disclosure. For example... Figure 8As shown, a trained machine learning model 210 can be used to identify objects in the image 810 to be processed. Specifically, the bounding box model 234 in the machine learning model 210 can be used to determine the bounding boxes of each candidate object in the image 810 to be processed. In this way, the location of each identified object can be clearly indicated, thereby improving the user's visual experience.

[0086] like Figure 8 As shown, objects 820 (pineapple), ..., and object 822 (lemon) can be identified. At this point, object 820 is a ground truth object identified in image 110 used as training data (i.e., the fruit "pineapple" is identified in the training data), and object 822 is an unidentified object in image 110 used as training data (i.e., the fruit "lemon" is not identified in the training data). Using the exemplary implementation of this disclosure, machine learning model 210 can identify not only the identified objects in the training data but also the unidentified objects in the training data.

[0087] According to an exemplary implementation of this disclosure, the scores of identified candidate objects can be determined based on the various models in machine learning model 210. Specifically, the image 810 to be processed can be input into machine learning model 210, and the localization score of each candidate object in the image 810 to be processed can be determined using localization scoring model 236. For example, the localization scores of objects 820 and 830 can be 0.9 and 0.8, respectively. Further, the classification score of each candidate object in the image 810 to be processed can be determined using classification scoring model 238. For example, the localization scores of objects 820 and 830 can be 0.7 and 0.6, respectively. Further, the final score of each candidate object in the image 810 to be processed can be determined based on the geometric mean of the localization score and the classification score of each candidate object.

[0088] According to an exemplary implementation of this disclosure, the final score of each candidate object can be determined based on the average of the location score and the classification score. In this case, the final score of object 820 can be expressed as... And the final score of object 822 can be expressed as Here, the final score can represent the probability that the identified object is an accurate object. Therefore, the aforementioned final score can indicate that object 820 has an 80% probability of being an actual object in the image 810 to be processed, and object 822 has a 70% probability of being an actual object in the image 810 to be processed. Using the exemplary implementation of this disclosure, factors of both location scoring and classification scoring can be considered comprehensively, thereby improving the accuracy of the final score for determining the object.

[0089] It will be understood that in the example above, the image 810 to be processed includes multiple objects. Although object 822 is an unlabeled object in the training data (that is, at least one unlabeled object is not included in the labeled ground truth objects in the image used to train the machine learning model), the machine learning model 210 can still recognize object 822. In this way, the machine learning model trained based on the above method can provide more flexible object recognition capabilities in an open-world environment. At this point, only training data including a portion of the labeled objects is required to obtain a machine learning model capable of recognizing unlabeled objects.

[0090] According to an exemplary implementation of this disclosure, the mask model 232 in the machine recognition model 210 can determine the regions of each candidate object in the image 810 to be processed. For example, mask regions can be overlaid on the image 810 to indicate the pixel regions covered by each object. In this way, the extent of each object can be represented in a more precise manner; for example, green can be used to represent the extent of object 820, and yellow can be used to represent the extent of object 822.

[0091] The training process described above can be implemented on various public datasets, and the trained machine learning model can be used to perform image recognition operations. The model's performance can be evaluated using class-agnostic methods, specifically Average Recall (AR) and Mean Average Precision (mAP). Experiments show that on public datasets such as COCO, LVSI, UVO, and Objects365, the machine learning model obtained using the process described above outperforms currently known machine learning models in both AR and mAP performance.

[0092] It will be understood that although the above description of the specific process of object recognition uses only fruit as an example of an object in an image, in the context of this disclosure, objects can include a variety of tangible entities in the real world, including but not limited to people, animals, objects, etc. Furthermore, although the object recognition process was described above using the example of recognizing an object in a single image, objects in multiple images can be recognized. For example, objects in individual video frames of a video sequence can be recognized. Having identified objects in individual video frames, these objects can be further tracked during video training.

[0093] Example process

[0094] Figure 9A flowchart of a method 900 for processing an image using a machine learning model, according to some implementations of this disclosure, is shown. The machine learning model is used to identify at least one candidate object from the image. The machine learning model includes: a feature extraction model for describing the association between features of the image and at least one candidate object; and a classification scoring model for describing the association between features of at least one candidate object and a classification score of at least one candidate object, the classification score representing the probability that at least one candidate object is classified as foreground in the image. Specifically, at box 910, update parameters associated with the classification scoring model are determined based on the classification scores of at least one candidate object and the ground truth classification scores of at least one ground truth object in the image. At box 920, the classification scoring model is updated based on the update parameters associated with the classification scoring model. At box 930, updating the feature extraction model using the update parameters associated with the classification scoring model is prevented.

[0095] According to an exemplary implementation of this disclosure, the machine learning model further includes a localization scoring model, which describes the association between features of at least one candidate object and localization scores of at least one candidate object, whereby the localization score represents the difference between the localization of at least one candidate object and the ground truth localization of at least one ground truth object. The method 900 further includes: determining update parameters associated with the localization scoring model based on the localization of at least one candidate object and the ground truth localization of at least one ground truth object; and updating a feature extraction model based on the update parameters associated with the localization scoring model.

[0096] According to an exemplary implementation of this disclosure, the machine learning model further includes a mask model that describes the association between features of at least one candidate object and regions of at least one candidate object, and the method 900 further includes: determining update parameters associated with the mask model based on regions of at least one candidate object and ground truth regions of at least one ground truth object; and updating the feature extraction model using the update parameters associated with the mask model.

[0097] According to an exemplary implementation of this disclosure, the machine learning model further includes a bounding box model, which describes the association between features of at least one candidate object and the bounding box of at least one candidate object in an image, and the method 900 further includes: determining update parameters associated with the bounding box model based on the bounding boxes of at least one candidate object and the ground bounding boxes of at least one ground object; and updating the feature extraction model based on the update parameters associated with the bounding box model.

[0098] According to an exemplary implementation of this disclosure, the machine learning model further includes a contrastive learning model, and the method 900 further includes: selecting positive and negative samples for contrastive learning from at least one candidate object; determining update parameters associated with the contrastive learning model using the positive and negative samples; and updating the feature extraction model using the update parameters associated with the contrastive learning model.

[0099] According to an exemplary implementation of this disclosure, selecting positive and negative samples includes: determining a sequence of at least one candidate object based on a comparison of at least one candidate object and at least one truth object; and selecting positive and negative samples from the sequence.

[0100] According to an exemplary implementation of this disclosure, determining a sequence of at least one candidate object includes: selecting similar candidate objects from at least one candidate object that are similar to at least one true object based on a comparison of the features of at least one candidate object and the true features of at least one true object; determining feature centers using the features of the similar candidate objects; and determining a sequence of at least one candidate feature objects based on the distance between the features of at least one candidate object and the feature centers.

[0101] According to an exemplary implementation of this disclosure, determining the distance between a feature and a feature center of at least one candidate object includes: determining the distance based on an optimal transmission strategy.

[0102] According to an exemplary implementation of this disclosure, selecting positive samples from a sequence includes: selecting a first number of candidate objects from the ends of the sequence as positive samples.

[0103] According to an exemplary implementation of this disclosure, selecting a negative sample from a sequence includes: selecting a second number of candidate objects as negative samples from among other candidate objects following a first number of candidate objects in the sequence.

[0104] According to an exemplary implementation of this disclosure, selecting a second number of candidate objects as negative samples includes: selecting a second number of candidate objects as negative samples from among other candidate objects that are adjacent to the first number of candidate objects.

[0105] According to one exemplary implementation of this disclosure, the image includes at least one labeled truth object, and the image further includes at least one unlabeled object.

[0106] According to an exemplary implementation of this disclosure, the method 900 further includes: inputting an image to be processed into a machine learning model; determining a location score for at least one candidate object in the image to be processed using a location scoring model; determining a classification score for at least one candidate object in the image to be processed using a classification scoring model; and determining a final score for at least one candidate object in the image to be processed based on the geometric mean of the location score and the classification score.

[0107] According to one exemplary implementation of this disclosure, the image to be processed includes at least one unlabeled object, while the labeled ground truth objects in the image used to train the machine learning model do not include at least one unlabeled object.

[0108] According to an exemplary implementation of this disclosure, the method 900 further includes at least one of the following: determining the region of at least one candidate object in the image to be processed using a mask model; and determining the bounding box of at least one candidate object in the image to be processed using a bounding box model.

[0109] Example devices and equipment

[0110] Figure 10 A block diagram of an apparatus 1000 for processing images using a machine learning model, according to some implementations of the present disclosure, is shown. The machine learning model is used to identify at least one candidate object from an image. The machine learning model includes: a feature extraction model for describing the association between features of the image and at least one candidate object; and a classification scoring model for describing the association between features of at least one candidate object and a classification score of at least one candidate object, the classification score representing the probability that at least one candidate object is classified as foreground in the image.

[0111] The apparatus 1000 includes: a determining module 1010 configured to determine update parameters associated with a classification scoring model based on classification scores of at least one candidate object and ground truth classification scores of at least one ground truth object in an image; an updating module 1020 configured to update the classification scoring model based on the update parameters associated with the classification scoring model; and a blocking module 1030 configured to block the use of update parameters associated with the classification scoring model to update the feature extraction model.

[0112] According to an exemplary implementation of this disclosure, the machine learning model further includes a localization scoring model, which describes the association between features of at least one candidate object and localization scores of at least one candidate object, wherein the localization score represents the difference between the localization of at least one candidate object and the ground truth localization of at least one ground truth object, and the apparatus further includes: a localization score-based determination module configured to determine update parameters associated with the localization scoring model based on the localization of at least one candidate object and the ground truth localization of at least one ground truth object; and an update module 1020 further configured to update the feature extraction model based on the update parameters associated with the localization scoring model.

[0113] According to an exemplary implementation of this disclosure, the machine learning model further includes a mask model, which describes the association between features of at least one candidate object and regions of at least one candidate object, and the apparatus further includes: a mask-based determination module configured to determine update parameters associated with the mask model based on regions of at least one candidate object and ground truth regions of at least one ground truth object; and an update module 1020 further configured to update the feature extraction model using the update parameters associated with the mask model.

[0114] According to an exemplary implementation of this disclosure, the machine learning model further includes a bounding box model, which describes the association between features of at least one candidate object and the bounding boxes of at least one candidate object in an image, and the apparatus further includes: a bounding box-based determination module configured to determine update parameters associated with the bounding box model based on the bounding boxes of at least one candidate object and the ground truth bounding boxes of at least one ground truth object; and an update module 1020 further configured to update the feature extraction model based on the update parameters associated with the bounding box model.

[0115] According to an exemplary implementation of this disclosure, the machine learning model further includes a contrastive learning model, and the apparatus further includes: a selection module configured to select positive and negative samples for contrastive learning from at least one candidate object; a determination module based on contrastive learning configured to determine update parameters associated with the contrastive learning model using the positive and negative samples; and an update module 1020 further configured to update the feature extraction model using the update parameters associated with the contrastive learning model.

[0116] According to an exemplary implementation of this disclosure, the selection module includes: a sequence determination module configured to determine a sequence of at least one candidate object based on a comparison of at least one candidate object and at least one truth object; and a sample selection module configured to select positive samples and negative samples from the sequence.

[0117] According to an exemplary implementation of this disclosure, the sequence determination module includes: a similar object selection module configured to select similar candidate objects from at least one candidate object that are similar to at least one true object based on a comparison of features of at least one candidate object and true features of at least one true object; a center determination module configured to determine feature centers using features of the similar candidate objects; and a sorting module configured to determine a sequence of at least one candidate feature objects based on the distance between features of at least one candidate object and feature centers.

[0118] According to one exemplary implementation of this disclosure, the sorting module is further configured to: determine the distance based on the optimal transmission strategy.

[0119] According to an exemplary implementation of this disclosure, the sample selection module includes: selecting a first number of candidate objects from the ends of the sequence as positive samples.

[0120] According to an exemplary implementation of this disclosure, the sample selection module includes: selecting a second number of candidate objects as negative samples from other candidate objects following a first number of candidate objects in the sequence.

[0121] According to an exemplary implementation of this disclosure, the sample selection module includes: selecting a second number of candidate objects from among other candidate objects that are adjacent to a first number of candidate objects as negative samples.

[0122] According to one exemplary implementation of this disclosure, the image includes at least one labeled truth object, and the image further includes at least one unlabeled object.

[0123] According to an exemplary implementation of this disclosure, the apparatus further includes: an input module configured to input an image to be processed into a machine learning model; a localization scoring determination module configured to determine a localization score of at least one candidate object in the image to be processed using a localization scoring model; a classification scoring determination module configured to determine a classification score of at least one candidate object in the image to be processed using a classification scoring model; and a final scoring determination module configured to determine a final score of at least one candidate object in the image to be processed based on the geometric mean of the localization score and the classification score.

[0124] According to one exemplary implementation of this disclosure, the image to be processed includes at least one unlabeled object, while the labeled ground truth objects in the image used to train the machine learning model do not include at least one unlabeled object.

[0125] According to an exemplary implementation of this disclosure, the apparatus further includes at least one of the following: a mask determination module configured to determine a region of at least one candidate object in an image to be processed using a mask model; and a bounding box determination module configured to determine the bounding box of at least one candidate object in the image to be processed using a bounding box model.

[0126] Figure 11 A block diagram of a device 1100 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 11 The computing device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 11 The computing device 1100 shown can be used to achieve, for example Figure 9 Method 900 is shown.

[0127] like Figure 11 As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage devices 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0128] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1100.

[0129] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 11As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1120 may include computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0130] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0131] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device that enables computing device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0132] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0133] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0134] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0135] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0137] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing images using a machine learning model, said machine learning model for identifying at least one candidate object from the image, said machine learning model comprising: A feature extraction model is used to describe the association between the features of the image and the at least one candidate object; And a classification scoring model for describing the correlation between the features of the at least one candidate object and the classification score of the at least one candidate object, wherein the classification score represents the probability that the at least one candidate object is classified as foreground in the image, the method comprising: Based on the classification scores of the at least one candidate object and the ground truth classification scores of the at least one ground truth object in the image, update parameters associated with the classification scoring model are determined; The classification scoring model is updated based on the update parameters associated with it; and Prevent the feature extraction model from being updated using update parameters associated with the classification scoring model. The machine learning model further includes a location scoring model, which describes the correlation between the features of the at least one candidate object and the location score of the at least one candidate object, wherein the location score represents the difference between the location of the at least one candidate object and the ground truth location of the at least one ground truth object, and the method further includes: Based on the localization of the at least one candidate object and the true localization of the at least one true object, update parameters associated with the localization scoring model are determined; and The feature extraction model is updated based on the update parameters associated with the localization scoring model.

2. The method of claim 1, wherein the machine learning model further comprises a mask model, the mask model describing the association between features of the at least one candidate object and regions of the at least one candidate object, and the method further comprises: Based on the regions of the at least one candidate object and the ground value regions of the at least one ground value object, update parameters associated with the mask model are determined; as well as The feature extraction model is updated using the update parameters associated with the mask model.

3. The method of claim 2, wherein the machine learning model further comprises a bounding box model, the bounding box model describing the association between features of the at least one candidate object and the bounding boxes of the at least one candidate object in the image, and the method further comprises: Based on the bounding boxes of the at least one candidate object and the truth bounding boxes of the at least one truth object, the update parameters associated with the bounding box model are determined; as well as The feature extraction model is updated based on the update parameters associated with the bounding box model.

4. The method of claim 3, wherein the machine learning model further comprises a contrastive learning model, and the method further comprises: Select positive and negative samples for contrastive learning from the at least one candidate object; Using the positive and negative samples, the update parameters associated with the contrastive learning model are determined; as well as The feature extraction model is updated using the update parameters associated with the contrastive learning model.

5. The method of claim 4, wherein selecting the positive sample and the negative sample comprises: Based on the comparison between the at least one candidate object and the at least one truth object, a sequence of the at least one candidate object is determined; as well as Select the positive sample and the negative sample from the sequence.

6. The method of claim 5, wherein determining the sequence of the at least one candidate object comprises: Based on the comparison between the features of the at least one candidate object and the truth features of the at least one truth object, similar candidate objects that are similar to the at least one truth object are selected from the at least one candidate object; Feature centers are determined using the features of the similar candidate objects; as well as A sequence of at least one candidate feature object is determined based on the distance between the feature of the at least one candidate object and the feature center.

7. The method of claim 6, wherein determining the distance between a feature of the at least one candidate object and the feature center comprises: The distance is determined based on the optimal transmission strategy.

8. The method of claim 5, wherein selecting the positive sample from the sequence comprises: A first number of candidate objects are selected from the ends of the sequence as the positive samples.

9. The method of claim 8, wherein selecting the negative sample from the sequence comprises: A second number of candidate objects are selected from the remaining candidate objects following the first number of candidate objects in the sequence to serve as the negative samples.

10. The method of claim 9, wherein selecting the second number of candidate objects as the negative samples comprises: From the other candidate objects, a second number of candidate objects are selected from those adjacent to the first number of candidate objects to serve as the negative samples.

11. The method of claim 3, wherein the image includes at least one labeled truth object, and the image further includes at least one unlabeled object.

12. The method of claim 11, further comprising: The image to be processed is input into the machine learning model; The location scoring model is used to determine the location score of at least one candidate object in the image to be processed; The classification scoring model is used to determine the classification score of at least one candidate object in the image to be processed; as well as The final score of the at least one candidate object in the image to be processed is determined based on the geometric mean of the localization score and the classification score.

13. The method of claim 12, wherein the image to be processed includes at least one unlabeled object, and the at least one unlabeled object is not included in the labeled ground truth objects in the image used to train the machine learning model.

14. The method of claim 12, further comprising at least one of the following: The mask model is used to determine the region of at least one candidate object in the image to be processed; The bounding box model is used to determine the bounding box of at least one candidate object in the image to be processed.

15. An apparatus for processing an image using a machine learning model, the machine learning model being used to identify at least one candidate object from the image, the machine learning model comprising: A feature extraction model is used to describe the association between the features of the image and the at least one candidate object; And a classification scoring model for describing the correlation between the features of the at least one candidate object and the classification score of the at least one candidate object, wherein the classification score represents the probability that the at least one candidate object is classified as foreground in the image, the apparatus comprising: The determination module is configured to determine update parameters associated with the classification scoring model based on the classification scores of the at least one candidate object and the ground truth classification scores of at least one ground truth object in the image; An update module is configured to update the classification scoring model based on update parameters associated with the classification scoring model; and A blocking module is configured to prevent the feature extraction model from being updated using update parameters associated with the classification scoring model. The machine learning model further includes a location scoring model, which describes the correlation between the features of the at least one candidate object and the location score of the at least one candidate object, wherein the location score represents the difference between the location of the at least one candidate object and the ground truth location of the at least one ground truth object, and the apparatus further includes: The determining module is further configured to determine update parameters associated with the location scoring model based on the location of the at least one candidate object and the truth location of the at least one truth object; and The update module is further configured to update the feature extraction model based on update parameters associated with the localization scoring model.

16. An electronic device comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement the method according to any one of claims 1 to 14 when executed by a processor.

Images