Training methods for object recognition models, object recognition methods, electronic devices

Abstract

This application provides a training method for an object recognition model, an object recognition method, an image recognition method, an electronic device, and a storage medium. The object recognition model training method uses a first set of image samples labeled with pseudo-labels to perform a first-stage training on an initial model, obtaining a pre-trained model. The pseudo-labels are determined using an unlabeled first set of image samples. A second-stage training is then performed using a second set of image samples labeled with soft labels, obtaining the object recognition model. The soft labels are determined using both the unlabeled first and second set of image samples. The image recognition method provided in this application divides the training process of the object recognition model into two stages. In the second stage, soft labels can be directly obtained using the pseudo-labels from the first stage. The second stage eliminates the need for clustering calculations, thus saving computational overhead associated with clustering.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a training method for an object recognition model, an object recognition method, an image recognition method, an electronic device, and a storage medium. Background Technology

[0002] In model training scenarios, the cost of manual annotation is constantly increasing. Therefore, how to train models using unsupervised methods has become an increasingly popular research area.

[0003] Unsupervised training works by learning from unlabeled samples to discover structural knowledge within the training set, thereby solving problems such as classification and prediction.

[0004] In scenarios where large-scale unlabeled samples are used for training, clustering is typically employed to obtain pseudo-label information, thereby completing model training. However, the clustering process places enormous demands on the GPU's memory and computational resources. Taking image samples collected by roadside image acquisition devices as an example, the clustering process leads to high resource consumption due to the large amount of content in the image samples. Summary of the Invention

[0005] This application provides a method for training an object recognition model, a method for object recognition, a method for image recognition, an electronic device, and a storage medium.

[0006] In a first aspect, embodiments of this application provide a method for object identification, which may include the following steps:

[0007] Obtain candidate images;

[0008] The reference image containing the target object and the candidate image are input into a pre-trained object recognition model to identify the candidate image containing the target object.

[0009] Secondly, embodiments of this application provide a method for training an object recognition model, which may include the following steps:

[0010] The initial model is trained in the first stage using the first set of image samples labeled with pseudo-labels, resulting in a pre-trained model; the pseudo-labels are determined using the first set of unlabeled image samples.

[0011] The model after initial training is trained in the second stage using a second set of image samples labeled with soft tags to obtain an object recognition model; the soft tags are determined using the first set of unlabeled image samples and the second set of unlabeled image samples.

[0012] Thirdly, embodiments of this application provide an image recognition method, applying augmented reality devices and / or virtual reality devices, including:

[0013] Obtain candidate images;

[0014] The reference image containing the target object and the candidate image are input into a pre-trained object recognition model to identify the candidate image containing the target object.

[0015] Render candidate images containing the target object onto the display of an augmented reality device and / or a virtual reality device.

[0016] Fourthly, embodiments of this application provide an object recognition apparatus, which may include:

[0017] The acquisition module is used to acquire candidate images;

[0018] The recognition module is used to input reference images containing the target object and candidate images into a pre-trained object recognition model to identify candidate images containing the target object.

[0019] Fifthly, embodiments of this application provide a training apparatus for an object recognition model, which may include:

[0020] The first-stage training module is used to train the initial model using the first set of image samples labeled with pseudo-labels, resulting in a pre-trained model; the pseudo-labels are determined using the first set of unlabeled image samples.

[0021] The second-stage training module is used to perform a second-stage training on the initially trained model using a second set of image samples labeled with soft tags, in order to obtain an object recognition model; the soft tags are determined using the unlabeled first set of image samples and the unlabeled second set of image samples.

[0022] Sixthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0023] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0024] Compared with the prior art, this application has the following advantages:

[0025] According to the embodiments of this application, compared to directly using clustering techniques to cluster and label unlabeled sample features to obtain pseudo-labels for model training, this embodiment can divide the training process of the object recognition model into two stages. The first stage involves clustering and labeling unlabeled sample features to obtain pseudo-labels. In the second stage, the clustering and labeling results or pseudo-labels from the first stage can be directly used to obtain soft labels. That is, no further clustering calculations are needed in the second stage, especially for image samples acquired by roadside image acquisition devices. For complex image sample processing, this can save the computational overhead caused by clustering calculations. Simultaneously, since the second stage uses samples labeled with soft labels for model training, the generalization ability of the model can be improved, ensuring the recognition accuracy of the object recognition model.

[0026] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0027] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0028] Figure 1 A schematic diagram illustrating the scenario of the object recognition method provided in this application;

[0029] Figure 2 This is a flowchart of an object recognition method according to an embodiment of this application;

[0030] Figure 3 This is a flowchart of a training method for an object recognition model according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram illustrating the determination of a fake label according to an embodiment of this application;

[0032] Figure 5 This is a schematic diagram illustrating the determination of a soft tag according to an embodiment of this application;

[0033] Figure 6 This is a schematic diagram illustrating the determination of a class center according to an embodiment of this application;

[0034] Figure 7 This is a schematic diagram illustrating the updating of the class center according to an embodiment of this application;

[0035] Figure 8This is a structural block diagram of an object recognition device according to an embodiment of this application;

[0036] Figure 9 This is a structural block diagram of a training device for an object recognition model according to an embodiment of this application; and

[0037] Figure 10 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0038] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0039] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0040] First, the terms used in this application will be explained.

[0041] Deep Learning: Deep learning refers to a collection of algorithms that use various machine learning algorithms on multi-layered neural networks to solve various problems involving images, text, audio, and video. Examples include object recognition in images, text summarization, and the generation of relevant information from audio and video. The core of deep learning is feature learning, which aims to acquire hierarchical feature information through layered networks, thereby solving the significant challenge of manually designing features in the past.

[0042] Artificial Neural Networks (ANNs): Also known simply as Neural Networks or Connectionist Models. An ANN is a computational model that abstracts and builds upon the neural network of the human brain from an information processing perspective, forming different networks with different connection methods. ANNs possess self-learning capabilities. For example, in image recognition scenarios, when image samples and their corresponding labels are input into an ANN, the network will learn to recognize similar images through its self-learning function.

[0043] Meta-learning enables models to tune hyperparameters, allowing them to quickly learn new tasks based on existing knowledge. Meta-learning solves the problem of "learning to learn." The difference between meta-learning and traditional machine learning is that machine learning involves manual hyperparameter tuning followed by direct training of a deep model for a specific task, while meta-learning first trains a good set of hyperparameters on other tasks and then trains the model on the specific task to optimize the hyperparameters.

[0044] Clustering: In unsupervised learning, the labeling information of training samples is unknown. Its goal is to reveal the inherent patterns in the data by learning from unlabeled training samples, providing a foundation for further data analysis. Clustering divides a sample set into several typically disjoint subsets, each called a cluster. Data samples within the same cluster are considered to belong to the same category, and training for tasks such as classification is based on this.

[0045] Figure 1 This is a schematic diagram illustrating an application scenario for implementing the method of the embodiments of this application. Figure 1 The example illustrates a process of querying a candidate image set for images containing a target object. The query can be performed by an electronic device such as a smartphone or tablet. The target object can be a person, animal, clothing, electronic product, vehicle, etc. The candidate image set can be images captured from a video or images from the internet. For example, the video could be road traffic video. Alternatively, the video could be a user-shot video or a film / TV show. Figure 1 In the scenario shown, the target object is a vehicle.

[0046] The specific query principle is briefly described below. On one hand, an image containing the target object is input into a pre-trained object recognition model. This model can extract the features of the target object and obtain its feature representation. On the other hand, a set of candidate images can be input into the pre-trained object recognition model so that it can extract the features of the candidate objects contained in each candidate image and obtain the feature representation of each candidate object. Finally, by comparing the similarity between the feature representations of the target object and the feature representations of the candidate objects, the object recognition model can determine the candidate images with a higher probability of containing the target object.

[0047] Furthermore, the features of candidate images can be obtained through pre-storage. For example, when candidate images are updated, features can be extracted from the updated subsequent images. This can improve efficiency in query scenarios.

[0048] This proposed solution can be applied to searching for missing persons, animals, etc., as well as hit-and-run vehicles. Alternatively, it can be used to search for actors' works or similar products. For example, an actor's image can be input into the object recognition model, which can then be used to search for the actor's film and television works in a film and television database. Another example is that images of clothing or electronic products that a user likes can be input into the object recognition model, which can then be used to search for similar or identical clothing items on shopping websites.

[0049] This application provides a method for object recognition, such as... Figure 2 The flowchart shown is a method for object recognition according to an embodiment of this application, which may include:

[0050] Step S201: Obtain candidate images.

[0051] Candidate images can be images captured from road traffic videos. Alternatively, candidate images can be images captured from videos on the internet. Or, candidate images can be images from the internet itself, such as images appearing in an application. These candidate images can be real-time images, historical images, or images already present on a webpage. Furthermore, candidate images can be single images or sets of multiple images.

[0052] Step S202: Input the reference image containing the target object and the candidate image into the pre-trained object recognition model to identify the candidate image containing the target object.

[0053] The target object can be a person, animal, vehicle, clothing, electronic product, etc. A pre-trained object recognition model compares each candidate image with a reference image to determine if the candidate image contains the target object. Alternatively, the pre-trained object recognition model can determine the probability of the target object appearing in the candidate image. Finally, the candidate image containing the target object is selected as the target image, or the candidate image with a probability of the target object appearing higher than a corresponding probability threshold is selected as the target image.

[0054] Taking an animal as an example, a photo of a lost pet can be used as a reference image. Road video images captured by devices within a certain distance of the pet owner's community can be used as candidate images to facilitate the search for the lost pet.

[0055] Taking a person as an example, a photograph of the missing person can be used as a reference image. Road video images captured by cameras within a certain distance of the missing person's community can be used as candidate images to facilitate the search for the missing person.

[0056] Taking a vehicle as the target object as an example, a photograph of the vehicle involved in the accident can be used as a reference image. Road video images collected by data acquisition devices within the district, county, or other area where the accident occurred can be used as candidate images to facilitate the tracking of the vehicle.

[0057] Taking clothing and electronic products as examples, photos of these items can be used as reference images. Product images from the corresponding vertical categories on shopping websites can be used as candidate images to enable searching for the same or similar clothing and electronic products.

[0058] One possible implementation may also include the following steps:

[0059] Using candidate images containing the target object, relevant information about the target object is determined; the relevant information includes at least one of the target object's identification information and the target object's location information.

[0060] The relevant information for the target object includes its identification information and / or location information. For identification information, taking a vehicle as an example, the identification information could be the vehicle's license plate number determined through image recognition. For an animal as an example, the identification could be the animal's breed and coat color determined through image recognition, such as an orange cat or a Dalmatian. Regarding location, the installation location or area of ​​the acquisition device that detected the target image can be determined based on its identification number. This determined installation location or area can then be used as the target object's location information.

[0061] This application provides a method for training an object recognition model, such as... Figure 3 The flowchart shown is a training method for an object recognition model according to an embodiment of this application, which may include:

[0062] Step S301: Use the first set of image samples labeled with pseudo-labels to perform the first stage of training on the initial model to obtain the pre-trained model; the pseudo-labels are determined using the first set of unlabeled image samples.

[0063] Pseudo-labels are a concept in unsupervised training, referring to labels that are automatically assigned to unlabeled samples rather than manually. In one scenario, pseudo-labels can be determined by initially training the model on a small amount of labeled data. This initial trained model can then be used to predict unlabeled data, yielding predicted labels. These predicted labels can then be used as pseudo-labels for the unlabeled data.

[0064] Alternatively, in another scenario, pseudo-labels can be determined by using an untrained model to predict features from unlabeled data. These features are then clustered using a clustering algorithm to obtain the clustering results. The pseudo-labels can then be derived from these clustering results. The untrained model can be a general feature extraction model.

[0065] Pseudo-labels are used to annotate image samples in the first image sample set. Taking human image samples as an example, pseudo-labels can be feature representations of different people in the human image samples. For example, pseudo-labels can be used to represent multiple dimensions of information such as gender, height, skin color, hair color, and clothing.

[0066] By utilizing deep learning and meta-clustering techniques, combined with an artificial neural network, the initial model is input with a set of image samples labeled with pseudo-labels. The initial model can then obtain label prediction results. There will be differences between the predicted labels and the pseudo-labels, which can be represented by a loss function. The role of the loss function can be understood as follows: when the predicted labels obtained by the initial model during forward propagation are close to the pseudo-labels, the loss function takes a smaller value; conversely, the loss function increases in value. Furthermore, the loss function is a function with the parameters in the initial model as independent variables.

[0067] Using the aforementioned differences, the parameters in the initial model are adjusted. These differences are backpropagated through each layer of the initial model, and the parameters of each layer are adjusted accordingly until the output of the initial model converges or achieves the expected result.

[0068] This allows for the first stage of training the initial model. After this first stage of training, the initial model can be trained into a pre-trained model.

[0069] Step S302: Use the second set of image samples labeled with soft labels to perform a second stage of training on the initially trained model to obtain an object recognition model; the soft labels are determined using the unlabeled first set of image samples and the unlabeled second set of image samples.

[0070] Soft labels, also a concept in unsupervised training, refer to the labels that appear after the labeled data has been discretized. For example, an image sample might have the original label "cat," but the sample also contains dogs and people. If only the label "cat" is used to represent the image sample's features, the model training results might be affected by noise. By setting soft labels like "dog" and "person," the model's predicted probabilities can be used for training instead of the original image labels.

[0071] Soft labels can be determined using an unlabeled first image sample set and an unlabeled second image sample set. The unlabeled second image sample set can be input into the initially trained model to obtain feature extraction results for the unlabeled second image sample set. For example, the second image sample set may have n feature extraction results. Pseudo-labels can be obtained using the unlabeled first image sample set. Based on the similarity between each feature extraction result and the pseudo-label, the corresponding soft label can be obtained. Alternatively, as mentioned earlier, feature clustering results can be obtained using the unlabeled first image sample set. The cluster centers can then be obtained. Based on the similarity between each feature extraction result and the cluster centers, the corresponding soft label can also be obtained.

[0072] The second set of image samples, after being soft-labeled, can be used to train the initially trained model in the second stage. The second stage of training is considered complete when certain conditions are met. These conditions can be determined based on factors such as the number of training epochs, the number of samples involved in training, the training duration, and whether the loss function has converged. For example, using the soft-labeled second set of image samples to optimize the parameters of the initially trained model, achieving 'a' epochs of parameter optimization confirms that the specified condition is met. The value 'a' can be determined empirically and is a positive integer. Other examples include the number of samples participating in the second stage of training reaching 500,000 or 1,000,000, which confirms that the specified condition is met. Similarly, the second stage of training duration reaching 200 hours or 1000 hours also confirms that the specified condition is met. Furthermore, the convergence of a specified loss function can also be used as a condition.

[0073] Compared to directly using clustering techniques to cluster and label unlabeled sample features to obtain pseudo-labels for training, the current implementation breaks down the object recognition model training process into two stages. The first stage involves clustering and labeling unlabeled sample features to obtain pseudo-labels. The second stage directly utilizes the clustering and labeling results or pseudo-labels from the first stage to obtain soft labels. That is, no further clustering calculations are needed in the second stage, thus saving computational overhead. Simultaneously, since the second stage uses samples with soft labels for model training, the model's generalization ability is improved, ensuring the accuracy of the object recognition model.

[0074] Combination Figure 4 As shown, in one possible implementation, the method for determining pseudo-tags may include:

[0075] Step S401: Use the initial model to extract features from the first set of unlabeled image samples to obtain the first feature extraction result.

[0076] By inputting the unlabeled first set of image samples into the initial model, the first feature extraction result can be obtained. The first feature extraction result is... Figure 4 In, it is represented by X1. Figure 4 In the example shown, the image sample contains multiple vehicles. Based on this, the first feature extraction result can characterize the vehicle type, color, number of passengers in the vehicle, and the vehicle's location in the image sample.

[0077] Step S402: Perform clustering processing on the first feature extraction results to obtain at least one feature cluster; the feature cluster is used to characterize the features of the candidate object, which is the object that appears in each image sample of the unlabeled first image sample set.

[0078] The candidate objects contained in each image sample in the sample set may differ. For example, the candidate object in the m-th image sample may be a vehicle, the candidate object in the (m+1)-th image sample may be a vehicle and a pedestrian, and the candidate object in the (m+2)-th image sample may be a pedestrian, a pet, and lane lines. The purpose of clustering is to group features belonging to the same candidate object together from image samples. Here, m is a positive integer.

[0079] For example, the clustering process of the first feature extraction result can be performed using the K-means algorithm, the density-based spatial clustering of applications with noise (DBScan), etc.

[0080] Step S403: Determine pseudo-labels based on feature clusters.

[0081] There are several ways to determine pseudo-labels based on feature clusters. For example, any single feature can be used as a pseudo-label. Another example is to randomly select multiple features and perform AND or OR operations between them, obtaining the pseudo-label based on the result. Yet another example is to calculate the average of multiple features, using the result as the pseudo-label. The pseudo-label can be represented by encoding.

[0082] Combination Figure 5 As shown, in one possible implementation, the method for determining the soft label includes:

[0083] Step S501: Use the pre-trained model to extract features from the unlabeled second image sample set to obtain the second feature extraction result.

[0084] In the unlabeled second image sample set, the image samples can be labeled from 1 to k. k is a positive integer, meaning the second image sample set can include k image samples. It's easy to understand that multiple second image sample sets can be constructed, with each set containing unique image samples. For example, in the first second image sample set, the image samples can be labeled from 1 to k. In the second second image sample set, the image samples can be labeled from k+1 to 2k.

[0085] By inputting the unlabeled second set of image samples into the initially trained model, the corresponding features can be obtained. The corresponding features are... Figure 5 This is represented as X2. The initially trained model can correspond to... Figure 5 In f θ For example, inputting the first image from the second image sample set into the pre-trained model yields i features. Inputting the second image from the second image sample set into the pre-trained model yields j features. Both i and j are positive integers, and their values ​​may be the same or different.

[0086] After feature extraction is performed on each image sample in the second image sample set, all the features of the second image sample can be obtained. That is, all the features of the second image sample can correspond to the second feature extraction results, and there can be multiple second feature extraction results.

[0087] Step S502: Determine soft labels by using the similarity between the second feature extraction results and the class centers; the class centers are determined using the unlabeled first image sample set and are used to represent features.

[0088] The essence of a class center is a feature representation. Class centers can be determined using an unlabeled first set of image samples, and there can be multiple class centers. After obtaining the second feature extraction result, for the i-th feature, the similarity with each class center can be calculated, yielding a similarity calculation result. For example, the score of the similarity calculation result is proportional to the degree of similarity. Based on the similarity calculation result, the class center with the highest score is selected as the soft label for the i-th feature. Similarity calculation can employ methods such as Euclidean distance and cosine similarity. Once the soft labels are determined, they can be used to label each image sample in the second set of image samples. The soft labels can be represented by encoding. Figure 5 In this context, the process of determining the class center is illustrated using robot icons. Robot icons can represent soft tags used to identify robots. That is, robots are identified through soft tags, and a similarity comparison process is then performed.

[0089] Combination Figure 6As shown, in one possible implementation, the method for determining the class center includes:

[0090] Step S601: Use the initial model to extract features from the first set of unlabeled image samples to obtain the first feature extraction result.

[0091] By inputting the unlabeled first set of image samples into the initial model, the first feature extraction result can be obtained. The first feature extraction result is... Figure 6 In, it is represented by X1. Figure 6 In the example shown, the image sample contains multiple vehicles. Based on this, the first feature extraction result can characterize the vehicle type, color, number of passengers in the vehicle, and the vehicle's location in the image sample.

[0092] Step S602: Perform clustering processing on the first feature extraction results to obtain at least one feature cluster; the feature cluster is used to characterize the features of candidate objects appearing in the unlabeled first image sample set.

[0093] The candidate objects contained in each image sample in the sample set may differ. For example, the candidate object in the m-th image sample may be a vehicle, the candidate object in the (m+1)-th image sample may be a vehicle and a pedestrian, and the candidate object in the (m+2)-th image sample may be a pedestrian, a pet, and lane lines. The purpose of clustering is to group features belonging to the same candidate object together from image samples. Here, m is a positive integer.

[0094] For example, clustering the results of the first feature extraction can be performed using algorithms such as K-means clustering or density-based noise-based spatial clustering.

[0095] Step S603: Calculate the class center for each feature cluster to obtain the corresponding class center calculation result.

[0096] Class center calculation can be implemented using class center algorithms. For example, a mean-based algorithm can be directly used to obtain the class center of each feature cluster. Alternatively, a weighted mean-based algorithm can be used. For instance, the mean-based algorithm can be used to calculate the average value, and then the distance between each feature in the feature cluster and the average value can be calculated. Weights for each feature cluster are set based on these distances, and a weighted mean-based algorithm is then used to obtain the class center of each feature cluster. Alternatively, features whose distance from the average value is greater than a corresponding distance threshold can be filtered out, and the remaining features can be used to calculate the average value again to obtain the class centers.

[0097] Combination Figure 7 As shown, one possible implementation also includes the step of updating the class center:

[0098] The steps for updating the class center include:

[0099] Step S701: Use the pre-trained model to extract features from the first set of image samples after pseudo-labeling to obtain the third feature extraction result.

[0100] Figure 7 middle f θ’ This can represent the initial model. After the first stage of training, the initial model yields a pre-trained model. The third feature extraction result is obtained by extracting features from the first set of image samples after pseudo-labeling using the pre-trained model.

[0101] Step S702: Update the class centers using the results of the third feature extraction.

[0102] Since the first stage of training is based on the first set of image samples after pseudo-labeling, and the class centers are obtained using feature clusters, there may be discrepancies between the third feature extraction result and the class centers. Therefore, in cases of discrepancies, the class centers can be updated using the third feature extraction result. For example, the average of the third feature extraction result and the class centers can be calculated, and the class centers can be updated using this average. Alternatively, the class centers can be directly replaced with the third feature extraction result.

[0103] In one possible implementation, the specified conditions are determined based on the convergence of a loss function of a specified type;

[0104] The loss function is calculated using the model trained in the second stage;

[0105] The specified type of loss function includes at least one of the weighted ternary loss function and the consistency loss function.

[0106] Image samples from the second set of soft-labeled images are input into the model during the second stage of training to obtain feature prediction results. The convergence of a specified type of loss function can be used to determine whether the second stage of training can be terminated based on the feature prediction results. For example, if the specified type of loss function has not converged, the optimization effect of the model parameters can be examined based on the convergence of the loss function, and the parameters can be iteratively optimized. If the specified type of loss function converges, the second stage of training can be terminated, resulting in a trained object recognition model.

[0107] The specified type of loss function can be the ternary loss function. The ternary loss function is used to test the feature prediction results, and the test results are used to optimize the parameters in the model. The principle of the ternary loss function is to use three image samples to form a triplet. In the triplet, the first and second image samples (positive samples) are similar, while the first and third image samples (negative samples) are dissimilar. For example, the first image sample is an image containing the target object, the positive samples are candidate images containing the target object, and the negative samples are candidate images not containing the target object. The goal of the triplet-based loss function is to detect whether the distance between similar samples is less than the distance between dissimilar samples, and whether the difference meets expectations. Based on this, the model trained in the second stage can retrieve the target image containing the target object from the candidate images.

[0108] Furthermore, a weighted ternary loss function can be used to verify the feature prediction results. The weighted ternary loss function involves weighting some or all samples. For example, a first image sample is selected, and positive and negative samples are chosen based on feature similarity. Specifically, a similarity threshold can be set; samples with similarity scores above the threshold are considered positive, and those below the threshold are considered negative. Furthermore, the weights for each positive and negative sample can be assigned based on the similarity results.

[0109] In addition, the consistency loss function can be used to test the feature prediction results, and the test results can be used to optimize the parameters in the model. The principle of the consistency loss function is to process the image containing the target object by performing random mosaic, random smearing, affine transformation, random rotation or mirroring, etc., and then test whether the model can use the processed image as the target image containing the target object in the query.

[0110] In one possible implementation, the number of samples in the second image sample set is n times the number of samples in the first image sample set; n > 1.

[0111] In the current implementation, the number of samples in the first image sample set can be much smaller than the number of samples in the second image sample set. For example, the value of n can be 10, 50, or even 100. Based on this, the amount of data that needs to be aggregated in the first stage of training can be reduced.

[0112] This application provides an image recognition method that can be applied to augmented reality (AR) devices and / or virtual reality (VR) devices. The method may include the following steps:

[0113] S801: Obtain candidate images.

[0114] Candidate images can be images captured from road traffic videos. Alternatively, candidate images can be images captured from videos on the internet. Or, candidate images can be images from the internet itself, such as images appearing in an application. These candidate images can be real-time images, historical images, or images already present on a webpage. Furthermore, candidate images can be single images or sets of multiple images.

[0115] S802: Input the reference image containing the target object and the candidate image into the pre-trained object recognition model to identify the candidate image containing the target object.

[0116] The target object can be a person, animal, vehicle, clothing, electronic product, etc. A pre-trained object recognition model compares each candidate image with a reference image to determine if the candidate image contains the target object. Alternatively, the pre-trained object recognition model can determine the probability of the target object appearing in the candidate image. Finally, the candidate image containing the target object is selected as the target image, or the candidate image with a probability of the target object appearing higher than a corresponding probability threshold is selected as the target image.

[0117] Taking an animal as an example, a photo of a lost pet can be used as a reference image. Road video images captured by devices within a certain distance of the pet owner's community can be used as candidate images to facilitate the search for the lost pet.

[0118] Taking a person as an example, a photograph of the missing person can be used as a reference image. Road video images captured by cameras within a certain distance of the missing person's community can be used as candidate images to facilitate the search for the missing person.

[0119] Taking a vehicle as the target object as an example, a photograph of the vehicle involved in the accident can be used as a reference image. Road video images collected by data acquisition devices within the district, county, or other area where the accident occurred can be used as candidate images to facilitate the tracking of the vehicle.

[0120] Taking clothing and electronic products as examples, photos of these items can be used as reference images. Product images from the corresponding vertical categories on shopping websites can be used as candidate images to enable searching for the same or similar clothing and electronic products.

[0121] S803: Render the candidate image containing the target object onto the display of an augmented reality device and / or a virtual reality device.

[0122] By utilizing augmented reality and / or virtual reality devices, target object identifiers or descriptions can be added to candidate images through rendering. This allows users to intuitively locate the target object within the candidate images or to present a more diverse viewing experience. For example, when the target object is a vehicle involved in an accident, a bounding box can be rendered within the candidate image containing the vehicle, displaying its location, license plate number, and other information. Similarly, bounding box rendering can be applied to people or pets. Furthermore, when the target object is clothing or accessories, the clothing or accessories can be regenerated and combined with a specified user image to demonstrate the effect of the user wearing them. Based on this, a more diverse viewing experience can be provided to users.

[0123] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an object recognition device. For example... Figure 8 The diagram shows an object recognition apparatus according to an embodiment of this application. The object recognition apparatus may include:

[0124] The acquisition module 801 is used to acquire candidate images;

[0125] The recognition module 802 is used to input a reference image containing the target object and a candidate image into a pre-trained object recognition model to identify the candidate image containing the target object.

[0126] In one possible implementation, the object recognition device may further include a related information determination module. This module determines relevant information about the target object using a candidate image containing the target object; the relevant information includes at least one of the target object's identification information and its location information.

[0127] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a training device for an object recognition model. For example... Figure 9 The diagram shows a training apparatus for an object recognition model according to an embodiment of this application. The training apparatus for the object recognition model may include:

[0128] The first-stage training module 901 is used to perform the first-stage training of the initial model using the first set of image samples labeled with pseudo-labels, to obtain the preliminarily trained model; the pseudo-labels are determined using the first set of unlabeled image samples.

[0129] The second-stage training module 902 is used to perform a second-stage training on the initially trained model using a second set of image samples labeled with soft tags, in order to obtain an object recognition model; the soft tags are determined using the unlabeled first set of image samples and the unlabeled second set of image samples.

[0130] In one possible implementation, the first-stage training module 901 may include:

[0131] The first feature extraction submodule is used to extract features from the unlabeled first image sample set using the initial model, and obtain the first feature extraction result.

[0132] The clustering submodule is used to perform clustering processing on the first feature extraction results to obtain at least one feature cluster; the feature cluster is used to characterize the features of candidate objects appearing in each image sample of the unlabeled first image sample set;

[0133] The pseudo-label determination submodule is used to determine pseudo-labels based on feature clusters.

[0134] In one possible implementation, the second-stage training module 902 may include:

[0135] The second feature extraction submodule is used to extract features from the unlabeled second image sample set using the pre-trained model to obtain the second feature extraction result.

[0136] The soft label determination submodule is used to determine soft labels by utilizing the similarity between the second feature extraction results and the class centers; the class centers are determined using the unlabeled first image sample set, and the class centers are used to represent features.

[0137] In one possible implementation, a class center determination submodule is also included. This class center determination module may further include:

[0138] The first feature extraction submodule is used to extract features from the unlabeled first image sample set using the initial model, and obtain the first feature extraction result.

[0139] The clustering submodule is used to cluster the first feature extraction results to obtain at least one feature cluster; the feature cluster is used to characterize the features of candidate objects appearing in the unlabeled first image sample set.

[0140] The class center calculation submodule is used to calculate the class center for each feature cluster and obtain the corresponding class center calculation result.

[0141] One possible implementation also includes a class center update module. The class center update module includes:

[0142] The third feature extraction submodule is used to extract features from the first set of image samples after pseudo-labeling using the pre-trained model, and obtain the third feature extraction result.

[0143] The class center update execution submodule is used to update the class centers using the results of the third feature extraction.

[0144] In one possible implementation, the specified conditions are determined based on the convergence of a loss function of a specified type;

[0145] The loss function is calculated using the model trained in the second stage;

[0146] The specified type of loss function includes at least one of the weighted ternary loss function and the consistency loss function.

[0147] In one possible implementation, the number of samples in the second image sample set is n times the number of samples in the first image sample set; n > 1.

[0148] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an image recognition apparatus. This apparatus can be applied to augmented reality devices and / or virtual reality devices, and the apparatus may include:

[0149] Candidate image acquisition module, used to acquire candidate images;

[0150] The target recognition module is used to input reference images containing target objects and candidate images into a pre-trained object recognition model to identify candidate images containing target objects;

[0151] The display control module is used to render candidate images containing target objects onto the displays of augmented reality devices and / or virtual reality devices.

[0152] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0153] Figure 10 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 10 As shown, the electronic device includes a memory 1010 and a processor 1020. The memory 1010 stores a computer program that can run on the processor 1020. When the processor 1020 executes the computer program, it implements the method described in the above embodiments. The number of memories 1010 and processors 1020 can be one or more.

[0154] The electronic device also includes:

[0155] The communication interface 1030 is used to communicate with external devices and perform data exchange and transmission.

[0156] If the memory 1010, processor 1020, and communication interface 1030 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0157] Optionally, in a specific implementation, if the memory 1010, processor 1020 and communication interface 1030 are integrated on a single chip, then the memory 1010, processor 1020 and communication interface 1030 can communicate with each other through an internal interface.

[0158] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0159] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0160] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0161] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0162] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0163] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0164] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0165] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0166] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0167] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0168] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0170] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an object recognition model, characterized in that, include: The initial model is trained in the first stage using the first set of image samples after pseudo-labeling, resulting in a pre-trained model. The pseudo-labels are determined using an unlabeled first set of image samples; The model after initial training is trained in the second stage using a second set of image samples labeled with soft tags to obtain an object recognition model. The soft label is determined using the unlabeled first image sample set and the unlabeled second image sample set; The method for determining the soft label includes: The model after preliminary training is used to extract features from the unlabeled second image sample set to obtain the second feature extraction result; The soft label is determined by using the similarity between the second feature extraction result and the class center; the class center is determined using the unlabeled first image sample set, and the class center is used to represent the feature.

2. The method according to claim 1, characterized in that, The method for determining the pseudo-tag includes: The initial model is used to extract features from the unlabeled first image sample set to obtain the first feature extraction result; The first feature extraction result is clustered to obtain at least one feature cluster; the feature cluster is used to characterize the features of the candidate object, which is an object that appears in each image sample of the unlabeled first image sample set; The pseudo-label is determined based on the feature cluster.

3. The method according to claim 1, characterized in that, The method for determining the class center includes: The initial model is used to extract features from the unlabeled first image sample set to obtain the first feature extraction result; The first feature extraction result is clustered to obtain at least one feature cluster; the feature cluster is used to characterize the features of the candidate object, which is an object that appears in each image sample of the unlabeled first image sample set; Class center calculation is performed on each of the feature clusters to obtain the corresponding class center calculation results.

4. The method according to claim 1 or 3, characterized in that, It also includes the step of updating the class center: The step of updating the class center includes: The model trained in the preliminary stage is used to extract features from the first set of image samples after pseudo-labeling to obtain the third feature extraction result. The class center is updated using the results of the third feature extraction.

5. The method according to claim 1, characterized in that, During the second stage of training, an object recognition model is obtained when specified conditions are met. The specified conditions are determined based on the convergence of the specified type of loss function; The loss function is calculated by the model during the second stage of training. The specified type of loss function includes at least one of the weighted ternary loss function and the consistency loss function.

6. The method according to claim 1, characterized in that, The number of samples in the second image sample set is n times the number of samples in the first image sample set; where n > 1.

7. A method for object recognition, characterized in that, include: Obtain candidate images; The reference image containing the target object and the candidate image are input into a pre-trained object recognition model to identify the candidate image containing the target object; wherein the object recognition model is trained according to any one of the methods in claims 1 to 6.

8. The method according to claim 7, characterized in that, Also includes: Using the candidate image containing the target object, determine the relevant information of the target object; The relevant information includes at least one of the target object's identification information and the target object's location information.

9. An image recognition method applied to augmented reality devices and / or virtual reality devices, characterized in that, include: Obtain candidate images; The reference image containing the target object and the candidate image are input into a pre-trained object recognition model to identify the candidate image containing the target object; wherein the object recognition model is trained according to any one of the methods in claims 1 to 6; The candidate image containing the target object is rendered onto the display of the augmented reality device and / or virtual reality device.

10. An object recognition device, characterized in that, include: The acquisition module is used to acquire candidate images; The recognition module is used to input a reference image containing the target object and the candidate image into a pre-trained object recognition model to recognize the candidate image containing the target object; wherein the object recognition model is trained according to any one of the methods in claims 1 to 6.

11. A training device for an object recognition model, characterized in that, include: The first-stage training module is used to perform the first-stage training of the initial model using the first set of image samples after pseudo-labeling, so as to obtain the pre-trained model. The pseudo-labels are determined using an unlabeled first set of image samples; The second-stage training module is used to perform a second-stage training on the model after the initial training using a second set of image samples labeled with soft tags, so as to obtain an object recognition model. The soft label is determined using the unlabeled first image sample set and the unlabeled second image sample set; The method for determining the soft label includes: The model after preliminary training is used to extract features from the unlabeled second image sample set to obtain the second feature extraction result; The soft label is determined by using the similarity between the second feature extraction result and the class center; the class center is determined using the unlabeled first image sample set, and the class center is used to represent the feature.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 8.

13. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images