Data processing method, image retrieval method and related apparatus

Abstract

The present disclosure provides a data processing method, an image retrieval method and related devices, and relates to the technical field of image processing, and particularly relates to the technical field of automatic driving, intelligent transportation, computer vision, intelligent search and the like. The specific implementation scheme is as follows: in a plurality of preset sample image acquisition manners, a target acquisition manner is determined; and long-tail image samples are acquired based on the target acquisition manner.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of autonomous driving, intelligent transportation, computer vision, and intelligent search. Background Technology

[0002] In visual recognition tasks, the class distribution of real samples is usually uneven. That is, the head class has more sample instances, while the tail class has fewer samples. The fewer samples in the tail can be called long-tail targets.

[0003] In autonomous driving, the detection and identification of long-tailed targets is both challenging and crucial. This is because long-tailed targets are sparsely distributed in real-world scenarios, making it difficult to acquire a large number of them in a short period of time. Summary of the Invention

[0004] This disclosure provides a data processing method, an image retrieval method, and related apparatus.

[0005] According to one aspect of this disclosure, a data processing method is provided, comprising:

[0006] Among the various preset sample image acquisition methods, the target acquisition method is determined;

[0007] Long-tail image samples are obtained based on the target acquisition method described above.

[0008] According to another aspect of this disclosure, an image retrieval method is provided, comprising:

[0009] Multiple initial images that meet the preset similarity criteria with the target image are selected to obtain an initial image set;

[0010] The initial image set is classified based on the long-tail classification model that has been trained and converged, and the classification confidence of each initial image in the initial image set is obtained.

[0011] Based on the similarity and classification confidence between each preliminary image and the target image, long-tail target images are selected from the preliminary image set.

[0012] According to another aspect of this disclosure, a data processing apparatus is provided, comprising:

[0013] The determination module is used to determine the target acquisition method among a variety of preset sample image acquisition methods;

[0014] The acquisition module is used to acquire long-tail image samples based on the target acquisition method.

[0015] According to another aspect of this disclosure, an image retrieval apparatus is provided, comprising:

[0016] The initial selection module is used to filter out multiple images that meet the preset conditions for similarity with the target image, thus obtaining an initial image set;

[0017] The processing module is used to classify the initial image set based on the long-tail classification model that has been trained and converged, and to obtain the classification confidence of each initial image in the initial image set.

[0018] The selection module is used to filter long-tail target images from the initial image set based on the similarity and classification confidence between each initial image and the target image.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0020] At least one processor; and

[0021] The memory is communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0024] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0025] In this embodiment of the disclosure, since the number of long-tail image samples is relatively small, the target acquisition method can be determined based on the actual situation to obtain a larger number of long-tail image samples.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0028] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of the present disclosure;

[0029] Figure 2 This is a schematic flowchart of a data processing method according to another embodiment of the present disclosure;

[0030] Figure 3 This is a schematic flowchart of a data processing method according to yet another embodiment of the present disclosure;

[0031] Figure 4 This is a schematic diagram of a visual interface according to an embodiment of the present disclosure;

[0032] Figure 5 This is a framework diagram of a training method for a long-tail classification model according to an embodiment of the present disclosure;

[0033] Figure 6 This is a schematic flowchart of an image retrieval method according to an embodiment of the present disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of the present disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of an image retrieval device according to an embodiment of the present disclosure;

[0036] Figure 9 This is a block diagram of an electronic device used to implement the data processing method / image retrieval method of the embodiments of this disclosure. Detailed Implementation

[0037] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0038] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In related technologies, one method for obtaining long-tail targets is image search. This involves first extracting feature vectors from the target in the full dataset, and then finding data similar to the long-tail target based on feature similarity methods. Because image search relies heavily on visual features, its retrieval performance is poor in some scenarios, often requiring manual filtering to obtain more accurate data.

[0040] To improve the efficiency of acquiring long-tail targets, this disclosure proposes a data processing method, which can be implemented as follows: Figure 1 As shown:

[0041] S101, among the preset multiple sample image acquisition methods, determine the target acquisition method.

[0042] S102, Obtain long-tail image samples based on target acquisition method.

[0043] In autonomous driving scenarios, long-tail image samples can generally include pedestrian long-tail targets, vehicle long-tail targets, and road pollution long-tail targets. Pedestrian long-tail targets can include people holding umbrellas, people squatting by the roadside, especially children holding umbrellas and squatting by the roadside. For example, vehicle long-tail targets can include irregularly shaped trucks, large construction vehicles, bicycles carrying goods, and tricycles carrying goods. For road pollution long-tail targets, examples can include plastic bags, stones, and tree branches that affect vehicle movement.

[0044] In this embodiment of the disclosure, since the number of long-tail image samples is relatively small, in order to improve the accuracy of the obtained long-tail image samples, multiple target acquisition methods can be determined based on the actual situation, thereby obtaining more long-tail image samples.

[0045] In some embodiments, the target acquisition method may include at least one of the following three methods:

[0046] Method 1) Obtain long-tailed image samples whose sample accuracy meets the preset accuracy requirements;

[0047] In some embodiments, long-tail image samples that meet preset accuracy requirements can be constructed from a small number of high-quality samples. Preset accuracy requirements may include meeting preset requirements for information such as sharpness and lighting. These long-tail image samples that meet the preset accuracy requirements may only include samples related to long-tail targets, such as positive and negative samples of various long-tail targets. In autonomous driving scenarios, positive samples are at least one of the aforementioned long-tail targets such as pedestrians, vehicles, and road pollution. Negative samples may include normally moving vehicles, pedestrians, and road pollution factors that do not affect driving.

[0048] Furthermore, when training the target model using long-tail image samples, general-class images that meet preset accuracy requirements can also be obtained. These general-class images may include only samples unrelated to the long-tail target, such as positive and negative apple samples. Alternatively, the images meeting the preset accuracy requirements may include both samples related to and unrelated to the long-tail target.

[0049] In this embodiment of the disclosure, high-quality long-tail image samples can be obtained in this way, thereby improving the image quality of the long-tail image samples.

[0050] In some embodiments, to ensure sample quality, long-tailed image samples whose sample accuracy meets preset accuracy requirements may be derived from manually annotated images.

[0051] This target acquisition method can obtain higher quality long-tail target images and improve the data quality of long-tail targets.

[0052] In this embodiment of the disclosure, when training the target model using long-tailed image samples, a small number of manually labeled samples are used, which is easy to implement while ensuring the accuracy of the target model during the training phase.

[0053] Method 2) Obtain long-tail image samples from the task information provided by the relevant tasks for performing long-tail image tasks on the target object;

[0054] In some embodiments, the long-tail image samples obtained in this manner can be samples collected when searching for long-tail targets. The number of long-tail image samples can be expanded by using positive or negative sample images of long-tail targets provided by the target object.

[0055] Method 3) Select images from the image library that are similar to the long-tail reference samples as long-tail image samples.

[0056] In this embodiment of the disclosure, the method of filtering out long-tail image samples can automatically expand the sample and increase the amount of long-tail target data.

[0057] In some embodiments, as described above, the acquired long-tail image samples are used to optimize the target model, which can be any model related to long-tail targets. For example, a long-tail target detection model, a long-tail target classification model, etc.

[0058] In this embodiment of the disclosure, given that the amount of high-precision long-tail target data is relatively small and it is difficult to directly obtain a large number of high-quality samples, when training the target model, different methods can be used to obtain long-tail image samples based on the training stage to meet the training requirements of the target model and gradually optimize the model performance of the target model.

[0059] Specifically, during implementation, based on the training requirements of the target model, the target acquisition method is determined from among several preset sample image acquisition methods, which can be implemented as follows:

[0060] Step A1: Determine the current training stage of the target model;

[0061] Step A2: Among the various sample image acquisition methods, select the target acquisition method corresponding to the current training stage to train the target model.

[0062] During implementation, each training phase can determine the corresponding target acquisition method based on the actual situation, which is used to iteratively optimize and train the target model.

[0063] In this embodiment of the disclosure, the training phase of the target model is divided into multiple phases, and each training phase has a corresponding target acquisition method, so that the target model can acquire different samples in each phase for training iteration and optimization, thereby gradually improving the performance of the target model when long-tail targets are not easy to obtain.

[0064] In this embodiment, the training phase can be an initial training phase, an intermediate training phase, or a final training phase. In actual implementation, training phases can be added or removed as needed.

[0065] In the case where the current training phase is the initial training phase for the target model, the target acquisition method is to acquire images whose sample accuracy meets the preset accuracy requirements. The images with the preset accuracy requirements may include the aforementioned long-tail image samples with preset accuracy requirements and / or general-class images with preset accuracy requirements.

[0066] Once images with sample precision meet the preset precision requirements are obtained, the target model can be trained using the MAML (Model-Agnostic Meta-Learning) method.

[0067] In this embodiment, a small number of readily available high-quality sample images are used to initialize the target model, thereby providing a good sample foundation for the training of the target model, enabling the target model to learn useful knowledge and thus improve the model's performance.

[0068] In some embodiments, when the target model satisfies the first convergence condition, the current training phase of the target model is switched to the intermediate training phase; the target acquisition method corresponding to the intermediate training phase is to obtain long-tail image samples from the task information provided by the target object performing the relevant task of long-tail image.

[0069] In this embodiment of the disclosure, the target model can learn useful knowledge through the initial training stage. At this time, the model is further optimized by obtaining long-tail image samples from the target object. This enables the knowledge learned by the target model in the initial stage to be gradually focused on and transferred to long-tail targets, and to samples that meet the actual operational needs of the target object. This not only expands the long-tail samples, but also improves the model's learning ability on long-tail targets.

[0070] In some embodiments, when the target model satisfies the second convergence condition, the current training stage of the target model is switched to the final training stage; the target acquisition method corresponding to the final training stage is based on long-tail reference samples, which are long-tail image samples selected from multiple images.

[0071] The second convergence condition refers to the convergence of the target model after the intermediate training phase. At this point, the target model's ability on long-tailed targets is further improved after the intermediate training phase. Then, new long-tailed targets are automatically mined from a large number of images through image filtering to optimize the target model and further improve its performance.

[0072] To automatically identify as many long-tail targets as possible, in this embodiment of the disclosure, long-tail image samples are selected from multiple images based on long-tail reference samples, such as... Figure 2 As shown, it can be implemented as follows:

[0073] S201, select multiple similar images that meet the preset conditions for similarity with the long-tail reference samples to obtain a candidate image set.

[0074] During implementation, multiple similar images that meet the preset conditions for similarity with long-tail reference samples can be selected from the image library to obtain a candidate image set.

[0075] Vehicles are often equipped with numerous image acquisition devices, allowing for the collection of comprehensive vehicle data from which images or videos can be extracted. Image frames can be extracted from the video, converting it into multi-frame images. Object detection is then performed on each of the resulting images, yielding bounding boxes for each target. It's important to understand that this object detection doesn't simply target long-tailed targets; it can detect long-tailed targets as well as other targets of various categories, obtaining bounding boxes for each target. Next, based on these bounding boxes, the corresponding targets are segmented from the images. Then, feature extraction is performed on the images of each target; visual features can be extracted to obtain the image features of each target. Based on the images of each target and their image features, the image library described earlier can be constructed.

[0076] When filtering similar images of long-tail reference samples, the image features of the long-tail reference samples are obtained, and the image features of each image in the image library are also obtained. Then, the similarity between the image features in the image library and the image features of the long-tail reference samples is calculated, and multiple similar images that meet the preset conditions with the similarity to the long-tail reference samples are filtered out to obtain a candidate image set.

[0077] Among them, the preset conditions can be that the similarity is greater than a preset threshold or the ranking is within the top preset ranking.

[0078] Cosine similarity can be used to filter similar images of long-tail reference samples. Alternatively, similarity can be calculated using methods such as Pearson correlation coefficient and Euclidean distance. This embodiment of the disclosure does not limit the method of calculating similarity.

[0079] S202, the candidate image set is processed based on the target model used to process long-tailed targets in order to obtain the long-tailed target reference information extracted by the target model.

[0080] S203, based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, sort each similar image in the candidate image set to obtain a candidate image sequence.

[0081] The long-tail target reference information is used to quantify the presence of long-tail targets in similar images. This information can directly or indirectly reflect the probability of long-tail targets being present in similar images. Therefore, a ranking value can be determined based on the positive correlation between similarity, long-tail target reference information, and the probability of long-tail targets being present in similar images. The candidate image sequence is then determined using this ranking value.

[0082] S204: Select long-tailed image samples from the candidate image sequence.

[0083] In this embodiment, the target model can automatically mine more long-tail image samples to expand the sample size. Furthermore, based on the aforementioned multiple model training stages for progressively training the target model, this embodiment can proactively mine more long-tail image samples using a target model that is already mature in long-tail target tasks. The mined long-tail image samples can further optimize the target model itself, thereby efficiently completing model self-optimization and iteration, and improving model performance. Subsequently, further long-tail targets can be mined, improving the automatic mining efficiency of long-tail targets.

[0084] In summary, the training objective of the target model can be to obtain more long-tail image samples. These long-tail image samples can be used to train other models or to further iteratively optimize the target model. The training process for the target model, after three training phases, can include:

[0085] Step B1: Train the target model based on the sample set; the sample set includes a first class of samples, which are the general images that meet the preset accuracy requirements as described above, or long-tail image samples that meet the preset accuracy requirements.

[0086] Step B2: After the target model has been trained on the sample set and meets the first convergence condition, the second type of samples is input into the target model to continue optimizing the target model. This second type of samples is the long-tailed image samples obtained based on the aforementioned method 2.

[0087] By obtaining long-tail image samples in step B2, we can not only enrich the sample size of long-tail image samples, but also enable the target model to be transferred to long-tail target tasks, thereby improving the business capabilities of the target model on long-tail target tasks.

[0088] Step B3 involves using the target model to select long-tailed image samples from multiple images in order to iteratively optimize the target model.

[0089] Therefore, by gradually improving the target model's business capabilities on long-tail targets, the target model can be automatically mined and optimized, thereby improving the efficiency of screening long-tail image samples.

[0090] In this embodiment, the target model can be a long-tail classification model. The long-tail classification model is used to perform binary classification on the input image, classifying it as an image containing a long-tail target (i.e., a long-tail target image) or an image not containing a long-tail target. The classification result is described by a classification confidence score, which measures the confidence level of the binary classification result corresponding to the similar image; it can also be understood as measuring the reliability of the classification result. Therefore, when the target model is a long-tail classification model, the long-tail target reference information extracted by the target model can be this classification confidence score.

[0091] In this embodiment of the disclosure, when the target model is a long-tail classification model, long-tail image samples can be screened based on similarity and classification confidence. Similarity can be understood as measuring the similarity in visual features between long-tail reference samples and similar images, while classification confidence is a description of the category, serving as a supplementary description beyond visual features. Classification confidence differs from visual features and is not a product of the visual feature extraction process; therefore, classification confidence effectively compensates for the shortcomings of solely relying on visual features, thereby enabling more accurate screening of long-tail image samples and expanding the sample size.

[0092] Furthermore, after optimization through the aforementioned initial and intermediate training phases, the classification model possesses a certain ability to classify long-tailed targets. Based on this, the long-tailed classification model's ability to classify long-tailed targets can be used to automatically filter out long-tailed image samples. Therefore, the target acquisition method provided in the final training phase of this disclosure can further and more accurately expand long-tailed target data, thereby improving the training efficiency of the long-tailed classification model.

[0093] In this embodiment of the disclosure, based on the target model being a long-tail classification model, the long-tail classification model can be optimized in the following ways, including as follows: Figure 3 The steps shown are as follows:

[0094] S301, select multiple similar images that meet the preset conditions for similarity with the long-tail reference samples to obtain a candidate image set.

[0095] The specific screening method is similar to the aforementioned screening method, and will not be described again in this embodiment.

[0096] S302, classify the candidate image set based on the long-tail classification model to obtain the classification confidence of each similar image in the candidate image set.

[0097] S303. Based on the similarity and classification confidence between each similar image and the long-tail reference sample, sort the similar images in the candidate image set to obtain the candidate image sequence.

[0098] S304, an optimized long-tail classification model based on candidate image sequences.

[0099] In this embodiment, when there is limited data on long-tail targets, a method is proposed to optimize the long-tail classification model by using similar images of long-tail reference samples. Classification confidence is introduced based on similarity to measure the influence of similar images on the optimization model. Classification confidence compensates for the shortcomings of solely relying on visual features, thereby more accurately measuring the impact of each similar image on the optimization of the long-tail classification model, and further improving the model. Ultimately, this embodiment improves the accuracy of the long-tail classification model in identifying long-tail targets and increases the model's training efficiency. Furthermore, the trained long-tail classification model can further combine similarity to automatically and accurately identify long-tail targets.

[0100] Figure 3 The training method shown can be understood as an implementation method that optimizes the long-tail classification model in the final training stage to facilitate the discovery of long-tail targets. Combining the initial and intermediate training stages can better obtain a long-tail classification model that can discover long-tail image samples. Specifically, it includes the following steps:

[0101] Step C1: Train a long-tail classification model based on the sample set; the sample set includes a first class of samples, which are long-tail image samples that meet the preset accuracy requirements and / or general class images that meet the preset accuracy requirements obtained in the aforementioned method 1).

[0102] In practice, when the sample set is labeled as multi-class, the initial classification model is trained into a multi-class model using the sample set.

[0103] When initializing a long-tail classification model using a sample set, the MAML (Model-Agnostic Meta-Learning) method can be used to train the model. Therefore, when training a multi-class classification task using the sample set, MAML can yield a long-tail classification model that performs well across multiple classification tasks. Subsequently, based on the second class of samples, a long-tail classification model that performs well on the corresponding task can be obtained.

[0104] Step C2: If the initial classification model satisfies the first convergence condition, input the second class of samples into the long-tail classification model to obtain the classification result of the initial classification model for the second class of samples; the classification result is either a long-tail target image or a normal image.

[0105] The classification results of the long-tail classification model for the second type of samples can be obtained; the classification results are either long-tail target images or ordinary images; the second type of samples are long-tail image samples obtained from the task information provided by the task related to the long-tail image of the target object.

[0106] With high-quality general-class images used for the first type of samples, the long-tail classification model can learn general knowledge during the initial training phase. Building on this, further using the second type of samples allows the initial model to be transferred to the task of searching for long-tail targets, improving the long-tail classification model's performance on this task.

[0107] The first convergence condition mainly refers to the training status of the long-tail classification model on the sample set, which can be expressed as the model loss on the sample set no longer changing, or the number of training iterations reaching a preset number.

[0108] Step C3: Based on the classification results and the true classification of the second class of samples, adjust the model parameters of the initial classification model. Under the condition of satisfying the second convergence condition, the long-tail classification model is obtained.

[0109] The second convergence condition can be that the difference between the classification result and the true classification of the second class sample tends to stabilize and / or the preset number of iterations is met.

[0110] In this embodiment, the initial classification model undergoes two-stage training. First, a small, readily available sample set is used to initialize the initial model, providing a good foundation. Then, a second set of samples is used to transfer the initial classification model to a binary classification task, facilitating its adaptation to the current task and ensuring the resulting long-tail classification model meets the task requirements. This two-stage training approach, even with limited long-tail target data, allows for the gradual acquisition of a long-tail classification model that satisfies the task requirements, thereby improving training efficiency.

[0111] In some embodiments, similarity represents the similarity between each similar image and a long-tail reference sample; therefore, the more similar the two, the higher the similarity, resulting in a higher evaluation value. Classification confidence represents the score of each similar image belonging to its corresponding category. The evaluation value is positively correlated with the similarity and classification confidence of each similar image. Based on this, the similar images in the candidate image set are sorted according to their similarity and classification confidence with the long-tail reference samples to obtain a candidate image sequence, which can be implemented as follows:

[0112] Step D1: Add the similarity and classification confidence of each similar image to obtain the evaluation value of each similar image as a long-tail target image.

[0113] In implementation, the similarity of similar images in this disclosure embodiment may include similarity in terms of lighting, contrast, structure, and content within the images. A neural network model can be used to extract image features. This neural network model can extract shallow features such as lighting, color, and texture, as well as deep features, and then fuse the shallow and deep features to obtain the final extracted image features.

[0114] Based on similarity-based image filtering, classification confidence is used as long-tail target reference information and combined with similarity to obtain the evaluation value.

[0115] Before adding the similarity and classification confidence scores, since the physical meanings of similarity and classification confidence scores are different, the similarity and classification confidence scores of similar images can be mapped to the same feature space, and then summed to obtain the evaluation value of each similar image as a long-tailed target image.

[0116] In implementation, a first neural network module can be used to map similarity to this feature space, and a second neural network module can be used to map classification confidence to this feature space. The model parameters of the two neural network modules can be obtained through training. The training loss can be measured by the difference between the candidate image sequence and the expected ranking. In implementation, these two neural network modules can be trained together when adjusting the weights later.

[0117] Step D2: Based on the evaluation values ​​of each similar image, sort the similar images in the candidate image set to obtain a candidate image sequence.

[0118] In this embodiment of the disclosure, an evaluation value is determined by combining multiple dimensions of classification confidence based on similarity, and then the similar images in the candidate image set are sorted to make the final candidate image sequence more accurate.

[0119] In another embodiment, since the influence of similarity and classification confidence may differ for different long-tail targets, in order to more accurately evaluate the evaluation values ​​of each similar image, the similar images in the candidate image set are sorted based on the similarity and classification confidence between each similar image and the long-tail reference sample to obtain a candidate image sequence. This can also be implemented as follows:

[0120] Step E1 involves weighted summation of the similarity and classification confidence of each similar image to obtain an evaluation value for each similar image as a long-tailed target image.

[0121] In implementation, the weights of similarity and classification confidence for similar images can be adaptively adjusted. When classification confidence is given more importance, its weight can be adaptively higher than that of similarity. For example, the weight of classification confidence can be set to 'a', and the weight of similarity to 'b', where 'a' is less than 'b'. Conversely, when similarity is given more importance, its weight can be adaptively higher than that of classification confidence. For example, the weight of similarity can be set to 'b', and the weight of classification confidence to 'a'. The specific numerical values ​​of the weights for similarity and classification confidence are not limited in this embodiment.

[0122] Step E2: Based on the evaluation values ​​of each similar image, sort the similar images in the candidate image set to obtain a candidate image sequence.

[0123] During implementation, similar images in the candidate image set are sorted in descending order of evaluation value to obtain a candidate image sequence.

[0124] In this embodiment of the disclosure, a weighted summation method is used in conjunction with classification confidence based on similarity, which can flexibly adjust the influence of similarity and classification confidence on the evaluation value, thereby making the final candidate image sequence more consistent with the expectations of the target object.

[0125] In addition to the aforementioned methods of addition and weighted summation, similarity and classification confidence can be fitted to the data based on the principles of data analysis. Fitting functions can also be, for example, products or power functions. Regardless of the fitting function used, it should contain adjustable and learnable parameters to ensure that the fitting result closely approximates the desired result.

[0126] In some embodiments, based on the obtained candidate image sequence, similar images in the candidate image sequence can be classified into three categories: low, medium, and high. For example, images with lower evaluation values ​​are classified as low-quality, those with medium evaluation values ​​as medium-quality, and those with higher evaluation values ​​as high-quality. When optimizing the long-tail classification model, optimization can be performed first using similar images from all quality categories, then using similar images from the medium-quality and high-quality categories, and finally using similar images from the high-quality category. This staged optimization approach allows the model to learn general knowledge from a large number of samples first, and then gradually learn the knowledge from high-quality samples.

[0127] In other embodiments, to accelerate optimization and improve training quality, the long-tail classification model can be optimized based on candidate image sequences, which can be implemented as follows:

[0128] Step F1: Output the candidate image sequence to the target object.

[0129] The target audience can be ordinary users who input long-tail reference samples, or professional personnel who train models.

[0130] Step F2, in response to the target object's selection operation of non-desired images in the candidate image sequence, identifies abnormal images in the candidate image sequence.

[0131] During implementation, in the initial stage, the candidate image sequence may contain samples that differ significantly from the long-tail reference samples. Therefore, after automatically filtering out similar images to the long-tail reference samples and appropriately sorting them, the results can be provided to the target object for evaluation. This allows the target object to filter out samples with significant differences, thus achieving precise sample screening. Subsequently, in step C3, the anomalous images can be used as new negative samples to optimize the long-tail classification model.

[0132] In implementation, to improve the accuracy of the long-tail classification model, the output candidate image sequence can be... Figure 4 The visualization is presented in white, where white represents unselected images and gray represents selected images. The target object selects unwanted images from the candidate image sequence to identify abnormal images. These abnormal images are then used as new negative samples and fed into the long-tail classification model to optimize it and improve the accuracy of the long-tail classification model.

[0133] In this embodiment of the disclosure, abnormal images are selected from the candidate image sequence based on the target object. This method can quickly filter out negative samples in a semi-automatic manner, thereby enabling further optimization of the long-tail classification model to improve its accuracy.

[0134] Of course, it is understandable that samples with higher evaluation values ​​in candidate image sequences can be used as positive samples to expand long-tail target data and optimize long-tail classification models.

[0135] This can be summarized as follows: based on candidate image sequences, positive and negative samples of long-tail targets are selected to further optimize the long-tail classification model.

[0136] In other embodiments, besides optimizing the long-tail classification model in the two ways described above, optimizing the long-tail classification model based on candidate image sequences can also be implemented as follows:

[0137] Step G1: Determine the desired order of the candidate image sequences.

[0138] Step G2 optimizes the long-tail classification model based on the difference between the candidate image sequence and the expected ranking.

[0139] In some embodiments, determining the desired order of the candidate image sequence can be implemented by: outputting the candidate image sequence to a target object; and updating the order of similar images in the candidate image sequence in response to the target object's selection operation of non-desired images in the candidate image sequence to obtain the desired order, wherein the non-desired images selected in the desired order are ordered after the unselected images.

[0140] The visual interface is still as Figure 4 As shown, white represents unselected images, and gray represents selected images. After at least one undesired image is selected for the target object, the order of the selected undesired images can be adjusted to follow the unselected images. In the case of multiple undesired images, they can be sorted based on their similarity and classification confidence to obtain the desired ranking.

[0141] In this embodiment, based on the target object selection operation, the output sequence is adaptively adjusted to obtain the adjusted expected ranking. This method can obtain the expected ranking simply and quickly. This expected ranking can accurately express the correlation between the retrieved similar images and the long-tail reference samples, which is beneficial to improving the classification efficiency of the long-tail classification model.

[0142] During implementation, the images corresponding to the expected ranking are used as sample data and input into the long-tail classification model to obtain the predicted ranking. The predicted ranking is compared with the expected ranking, and the long-tail classification model is adjusted based on the difference information. Under the condition of meeting the convergence condition, the optimization of the long-tail classification model is completed.

[0143] In this embodiment of the disclosure, the long-tail classification model is continuously optimized during use, thereby continuously improving its performance. As the number of long-tail target images and ordinary images increases, the long-tail classification model can adapt to various situations, thus improving its accuracy.

[0144] In implementation, one of the three methods for optimizing the long-tail classification model described above can be selected, or multiple methods can be selected for optimization. This disclosure does not limit this approach.

[0145] In some embodiments, in order to improve the accuracy of the evaluation values ​​obtained based on similarity and classification confidence, so as to obtain high-quality retrieval results in the stage of retrieving long-tail targets, the weights of similarity and classification confidence can be optimized based on the difference between the candidate image sequence and the expected ranking, in addition to obtaining the expected ranking.

[0146] In this embodiment of the disclosure, the weights of similarity and classification confidence are adjusted based on the expected ranking, thereby making the subsequent ranking more reasonable and improving the performance of the long-tail classification model.

[0147] In some embodiments, if the difference between the candidate image sequence and the desired ranking is greater than a preset difference, the long-tail classification model is optimized first.

[0148] During implementation, if the difference is greater than the preset difference degree, it indicates that the candidate image sequence and the expected ranking are too far apart, and the long-tail classification model needs to be further optimized to speed up the convergence.

[0149] For example, for each candidate image sequence, the difference between the candidate image sequence and the desired sorting can be represented by the percentage of incorrectly sorted images among all images. For instance, if the candidate image sequence includes 100 candidate images, and 20 images are incorrectly sorted, the percentage is 0.2. The preset difference can also be represented by a percentage threshold.

[0150] In another example, the difference between multiple candidate image sequences and their respective desired rankings can be used to measure whether to optimize the long-tail classification model first. For example, for each candidate image sequence, the difference can be obtained based on the above method, and then the average difference between multiple candidate image sequences and their respective desired rankings can be calculated. This average difference is compared with a preset difference degree to determine whether to prioritize optimizing the long-tail classification model.

[0151] In this embodiment of the disclosure, when the difference between multiple candidate image sequences and their respective desired rankings is greater than a preset difference degree, the long-tail classification model is optimized first, which can quickly adjust the evaluation value so that the ranking based on the evaluation value approaches the desired result as soon as possible, thereby achieving the purpose of fast training convergence.

[0152] Accordingly, if the difference between the candidate image sequence and the desired ranking is less than or equal to the preset difference, the weights of similarity and classification confidence are optimized first.

[0153] In practice, if the difference is not greater than the preset difference level, it means that the candidate image sequence and the expected ranking are not far apart, and the weights of similarity and classification confidence can be appropriately optimized.

[0154] In this embodiment of the disclosure, when the difference between multiple candidate image sequences and their respective desired rankings is less than or equal to a preset difference degree, the weights of similarity and classification confidence are optimized first, so that the final ranking is more accurate.

[0155] For ease of understanding, the framework of the training method for the long-tail classification model proposed in this embodiment includes: training an initial model with multiple manually labeled positive and negative samples from the sample set, and obtaining an initial model that satisfies the first convergence condition after meta-learning inference. When a reference sample is detected as input during the retrieval process for the target object, a second type of sample is constructed based on the input reference sample to optimize the initial model. Under the condition of satisfying the second convergence condition, a long-tail classification model is obtained.

[0156] Subsequently, during the retrieval of long-tail targets, with the target object input as a long-tail reference sample in the third-class sample set, image features of the long-tail reference samples are extracted based on the feature extraction model. The similarity between the image features of the long-tail reference samples and features in the image database is calculated to obtain a candidate image set. This candidate image set is then input into the long-tail classification model to obtain the classification confidence score of each similar image in the candidate image set. The similarity scores and classification confidence scores of each similar image are weighted and summed to obtain an evaluation value for each similar image as a long-tail target image. Based on the magnitude of the evaluation values, the similar images in the candidate image set are ranked to obtain a candidate image sequence. Based on the target object selection operation, the candidate image sequence is optimized to obtain the desired ranking. Images in the desired ranking can be used as supplementary content to the sample set to optimize the converged initial model, resulting in the final long-tail classification model. Alternatively, the long-tail classification model and weights can be jointly trained to optimize both.

[0157] like Figure 5 As shown, the image data collected by the vehicle is first processed by a target detection model to extract the targets in the image, and then each target is segmented from the image data. Each target is then processed by a feature model to extract image features, resulting in a feature vector, which is used to construct the image features in the image database.

[0158] Subsequently, after training the initial long-tail classification model using the sample set, the model's capabilities were improved. The long-tail classification model classifies similar images of the target object's retrieval data, i.e., long-tail reference samples. It calculates evaluation values ​​for similar images based on similarity and classification confidence, ranking the similar images to obtain a candidate image sequence. Then, by confirming the target object, the desired ranking of the candidate image sequence is determined, further obtaining positive and negative samples to optimize the long-tail classification model.

[0159] In summary, in this embodiment, an initial model is first initialized using a sample set with fewer labels, enabling the initial model to perform well on multi-classification tasks and providing a high-quality model foundation. Then, using samples provided during the target object retrieval process (such as second-class samples and long-tail reference samples), the initial model is transferred to the retrieval task, making the long-tail classification model more adaptable to long-tail target classification tasks in retrieval, thereby providing reasonable and reliable classification results and classification confidence. The entire training process continuously optimizes the long-tail classification model at each stage, overcoming the insufficiency of long-tail target data and improving the model's training efficiency and quality.

[0160] Based on the same technical concept, and building upon the long-tail classification model, this disclosure also provides an image retrieval method, such as... Figure 6 As shown, it includes:

[0161] S601, select multiple preliminary images whose similarity to the target image meets the preset conditions to obtain the preliminary image set.

[0162] The preset conditions can be that the similarity is greater than a preset threshold or that the ranking is within the top preset ranking. The method for calculating the similarity has been described above, and will not be repeated here in this embodiment.

[0163] S602, classify the initial image set based on the long-tail classification model that has been trained and converged, and obtain the classification confidence of each initial image in the initial image set.

[0164] S603, based on the similarity and classification confidence between each preliminary image and the target image, select long-tail target images from the preliminary image set.

[0165] In this embodiment, after filtering out similarity values ​​that meet preset conditions, a long-tail classification model is used to obtain the classification confidence score for each image. Long-tail target images are then filtered from the initial image set by combining classification confidence and similarity scores. This method automatically filters out long-tail target images, improving the efficiency of long-tail target acquisition. Furthermore, similarity can be understood as measuring the similarity in visual features between long-tail reference samples and similar images, while classification confidence describes the category of the long-tail target and is a supplementary description beyond visual features. Classification confidence differs from visual features and is not a product of the visual feature extraction process; therefore, classification confidence effectively compensates for the shortcomings of solely using visual features, improving the filtering accuracy of long-tail targets.

[0166] In some embodiments, long-tail target images are selected from the initial image set based on the similarity and classification confidence between each preliminary image and the target image, which can be implemented as follows:

[0167] Step H1: Sort the initial images in the initial image set to obtain the initial image sequence.

[0168] In some embodiments, the preliminary images in the preliminary image set are sorted based on the similarity and classification confidence between each preliminary image and the target image to obtain a preliminary image sequence. This can be implemented by: weighting and summing the similarity and classification confidence of each preliminary image to obtain an evaluation value for each preliminary image as a long-tailed target image; and sorting the preliminary images in the preliminary image set based on the magnitude of the evaluation value of each preliminary image to obtain a preliminary image sequence.

[0169] The method for determining the weights of similarity and classification confidence of each initial image has been explained above, and will not be repeated here in this embodiment.

[0170] In addition to using a weighted summation method to determine the evaluation value, other methods for determining the evaluation value have also been described above and are applicable to the embodiments of this disclosure, and will not be repeated here.

[0171] Step H2: Select images with long tails from the initial image sequence.

[0172] During implementation, if a single long-tail target image is required, the image with the highest evaluation value can be used as the final long-tail target image. If multiple long-tail target images are required, the images with the highest evaluation values ​​within a preset number or images with evaluation values ​​greater than a preset evaluation value can be used as the final long-tail target images.

[0173] In this embodiment of the disclosure, the long-tail target image obtained in this way is more in line with the expectations of the target object, thus improving the experience of the target object.

[0174] Based on the same technical concept, this disclosure proposes a data processing device 700, such as... Figure 7 As shown, it includes:

[0175] The determination module 701 is used to determine the target acquisition method among a variety of preset sample image acquisition methods;

[0176] The acquisition module 702 is used to acquire long-tail image samples based on the target acquisition method.

[0177] In some embodiments, the acquisition module includes any of the following units:

[0178] The first acquisition unit is used to acquire long-tailed image samples whose sample accuracy meets the preset accuracy requirements.

[0179] The second acquisition unit is used to acquire long-tail image samples from the task information provided by the task related to the long-tail image of the target object.

[0180] The third acquisition unit is used to filter out long-tail image samples from multiple images based on long-tail reference samples.

[0181] In some embodiments, the determining module is specifically used for:

[0182] Determine the current training stage of the target model;

[0183] Among various sample image acquisition methods, the target acquisition method corresponding to the current training stage is selected to train the target model.

[0184] In some embodiments, when the current training phase is the initial training phase for the target model, the target acquisition method is to acquire long-tailed image samples whose sample accuracy meets the preset accuracy requirements.

[0185] In some embodiments, if the target model satisfies the first convergence condition, the current training phase of the target model is switched to an intermediate training phase.

[0186] The target acquisition method corresponding to the intermediate training phase is to obtain long-tail image samples from the task information provided by the relevant tasks of performing long-tail image tasks on the target object.

[0187] In some embodiments, if the target model satisfies the second convergence condition, the current training phase of the target model is switched to the final training phase.

[0188] The target acquisition method in the final training stage is based on long-tail reference samples, which are selected from multiple images to obtain long-tail image samples.

[0189] In some embodiments, the third acquisition unit is specifically used for:

[0190] Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set;

[0191] The candidate image set is processed based on the target model used to process long-tailed targets in order to obtain the long-tailed target reference information extracted by the target model;

[0192] Based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, the similar images in the candidate image set are sorted to obtain the candidate image sequence;

[0193] Long-tailed image samples are selected from the candidate image sequences.

[0194] In some embodiments, the target model is a long-tail classification model.

[0195] In some embodiments, it also includes:

[0196] The filtering module is used to filter out multiple similar images that meet the preset conditions of similarity with long-tail reference samples, and obtain a candidate image set;

[0197] The classification module is used to classify the candidate image set based on the long-tail classification model and obtain the classification confidence of each similar image in the candidate image set;

[0198] The sorting module is used to sort the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample, so as to obtain the candidate image sequence.

[0199] The optimization module is used to optimize the long-tail classification model based on candidate image sequences.

[0200] In some embodiments, the sorting module includes:

[0201] The evaluation unit is used to perform a weighted summation of the similarity and classification confidence of each similar image to obtain an evaluation value for each similar image as a long-tailed target image;

[0202] The sorting unit is used to sort the similar images in the candidate image set based on the evaluation value of each similar image, so as to obtain a candidate image sequence.

[0203] In some embodiments, the optimization module includes:

[0204] The output unit is used to output the candidate image sequence to the target object;

[0205] The first determining unit is used to determine the abnormal image in the candidate image sequence in response to the target object's selection operation of the non-desired image in the candidate image sequence;

[0206] The first optimization unit is used to treat anomalous images as new negative samples to optimize the long-tail classification model.

[0207] In some embodiments, the optimization module includes:

[0208] The second determining unit is used to determine the desired order of the candidate image sequence;

[0209] The second optimization unit is used to optimize the long-tail classification model based on the difference between the candidate image sequence and the expected ranking.

[0210] In some embodiments, the second determining unit is specifically used for:

[0211] Output candidate image sequences to the target object;

[0212] In response to the target object's selection operation of non-desired images in the candidate image sequence, the order of each similar image in the candidate image sequence is updated to obtain the desired order, in which the selected non-desired images are ordered after the unselected images.

[0213] In some embodiments, the optimization module is further configured to:

[0214] The weights of similarity and classification confidence are optimized based on the difference between the candidate image sequence and the expected ranking of the candidate image sequence.

[0215] In some embodiments, the optimization module is configured to prioritize optimizing the long-tail classification model when the difference between the candidate image sequence and the desired ranking is greater than a preset difference degree.

[0216] In some embodiments, the optimization module is configured to prioritize optimizing the weights of similarity and classification confidence when the difference between the candidate image sequence and the desired ranking is less than or equal to a preset difference degree.

[0217] In some embodiments, long-tailed image samples whose sample accuracy meets preset accuracy requirements are derived from manually labeled images.

[0218] Based on the same technical concept, this disclosure proposes an image retrieval device 800, such as... Figure 8 As shown, it includes:

[0219] The initial selection module 801 is used to filter out multiple initial selection images that meet the preset conditions of similarity with the target image, and obtain an initial selection image set;

[0220] The processing module 802 is used to classify the initial image set based on the long-tail classification model and obtain the classification confidence of each initial image in the initial image set.

[0221] The selection module 803 is used to select long-tail target images from the initial image set based on the similarity and classification confidence between each initial image and the target image.

[0222] In some embodiments, the selected module is specifically used for:

[0223] Sort the initial images in the initial image set to obtain the initial image sequence;

[0224] Long-tail target images are selected from the initial image sequence.

[0225] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0226] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0227] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0228] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0229] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0230] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as model training methods and image retrieval methods. For example, in some embodiments, the model training methods and image retrieval methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the model training methods and image retrieval methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform a model training method or an image retrieval method by any other suitable means (e.g., by means of firmware).

[0231] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0232] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0233] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0235] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0236] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0237] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0238] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Among the various preset sample image acquisition methods, the target acquisition method is determined; Long-tail image samples are obtained based on the target acquisition method described above; The target acquisition method includes filtering long-tail image samples from multiple images based on long-tail reference samples, including: Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set; The candidate image set is processed based on a long-tail classification model for handling long-tail targets to obtain the classification confidence extracted by the long-tail classification model. Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence. The long-tailed image samples are selected from the candidate image sequence; The long-tail classification model was trained in the following manner: Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set; The candidate image set is classified based on a long-tail classification model to obtain the classification confidence of each similar image in the candidate image set; Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence. Optimizing the long-tail classification model based on the candidate image sequence includes: determining the desired ranking of the candidate image sequence; and optimizing the long-tail classification model based on the difference between the candidate image sequence and the desired ranking.

2. The method according to claim 1, wherein, The target acquisition method also includes at least one of the following: Obtain long-tailed image samples whose sample accuracy meets the preset accuracy requirements; The long-tail image sample is obtained from the task information provided by the task related to performing long-tail image tasks on the target object.

3. The method according to claim 1 or 2, wherein, The step of determining the target acquisition method among preset multiple sample image acquisition methods includes: Determine the current training stage of the target model; Among the various sample image acquisition methods, the target acquisition method corresponding to the current training stage is selected to train the target model.

4. The method according to claim 3, wherein, When the current training phase is the initial training phase for the target model, the target acquisition method is to acquire long-tail image samples whose sample accuracy meets the preset accuracy requirements.

5. The method according to claim 3, wherein, If the target model satisfies the first convergence condition, the current training phase of the target model is switched to the intermediate training phase. The target acquisition method corresponding to the intermediate training stage is to obtain the long-tail image samples from the task information provided by the target object performing the relevant task of the long-tail image.

6. The method according to claim 3, wherein if the target model satisfies the second convergence condition, the current training phase of the target model is switched to the final training phase; The target acquisition method corresponding to the final training stage is to select long-tail image samples from multiple images based on long-tail reference samples.

7. The method according to claim 3, wherein, The target model is a long-tail classification model.

8. The method according to claim 1, wherein, The candidate image sequence is obtained by ranking the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample, including: The similarity and classification confidence of each similar image are weighted and summed to obtain the evaluation value of each similar image as a long-tail target image; Based on the evaluation values ​​of each similar image, the similar images in the candidate image set are sorted to obtain a candidate image sequence.

9. The method according to claim 1, wherein, Optimizing the long-tail classification model based on the candidate image sequence further includes: The candidate image sequence is output to the target object; In response to the target object's selection operation of non-desired images in the candidate image sequence, abnormal images in the candidate image sequence are determined; The abnormal images are used as new negative samples to optimize the long-tail classification model.

10. The method according to claim 1, wherein, Determining the desired order of the candidate image sequence includes: The candidate image sequence is output to the target object; In response to the target object's selection operation of a non-desired image in the candidate image sequence, the order of each similar image in the candidate image sequence is updated to obtain the desired order, wherein the non-desired images selected in the desired order are ordered after the unselected images.

11. The method of claim 8, further comprising: The weights of the similarity and the classification confidence are optimized based on the difference between the candidate image sequence and the expected ranking of the candidate image sequence.

12. The method according to claim 11, wherein, If the difference between the candidate image sequence and the desired ranking is greater than a preset difference degree, the long-tail classification model is optimized.

13. The method according to claim 11, wherein, If the difference between the candidate image sequence and the desired ranking is less than or equal to a preset difference, the weights of the similarity and the classification confidence are optimized.

14. The method according to claim 2, wherein, The long-tailed image samples whose sample accuracy meets the preset accuracy requirements are derived from manually annotated images.

15. An image retrieval method, comprising: Multiple initial images that meet the preset similarity criteria with the target image are selected to obtain an initial image set; The initial image set is classified based on the long-tail classification model that has been trained and converged, and the classification confidence of each initial image in the initial image set is obtained. Based on the similarity and classification confidence between each preliminary image and the target image, long-tail target images are selected from the preliminary image set, including: sorting each preliminary image in the preliminary image set to obtain a preliminary image sequence; and selecting the long-tail target images from the preliminary image sequence. The long-tail classification model was trained in the following manner: Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set; The candidate image set is classified based on a long-tail classification model to obtain the classification confidence of each similar image in the candidate image set; Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence. Optimizing the long-tail classification model based on the candidate image sequence includes: determining the desired ranking of the candidate image sequence; and optimizing the long-tail classification model based on the difference between the candidate image sequence and the desired ranking.

16. A data processing apparatus, comprising: The determination module is used to determine the target acquisition method among a variety of preset sample image acquisition methods; The acquisition module is used to acquire long-tail image samples based on the target acquisition method; The acquisition module includes: a third acquisition unit, used to filter the long-tail image samples from multiple images based on the long-tail reference samples; The third acquisition unit is specifically used for: Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set; The candidate image set is processed based on a long-tail classification model for handling long-tail targets to obtain the classification confidence extracted by the long-tail classification model. Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence. The long-tailed image samples are selected from the candidate image sequence; Also includes: The filtering module is used to filter out multiple similar images that meet the preset conditions of similarity with the long-tail reference sample, and obtain a candidate image set; The classification module is used to classify the candidate image set based on the long-tail classification model and obtain the classification confidence of each similar image in the candidate image set; The sorting module is used to sort the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample, so as to obtain a candidate image sequence. An optimization module is used to optimize the long-tail classification model based on the candidate image sequence; The optimization module includes: The second determining unit is used to determine the desired order of the candidate image sequence; The second optimization unit is used to optimize the long-tail classification model based on the difference between the candidate image sequence and the desired ranking.

17. The apparatus according to claim 16, wherein, The acquisition module includes any one of the following units: The first acquisition unit is used to acquire long-tailed image samples whose sample accuracy meets the preset accuracy requirements. The second acquisition unit is used to acquire the long-tail image sample from the task information provided by the task related to the long-tail image of the target object.

18. The apparatus according to claim 16 or 17, wherein, The determining module is specifically used for: Determine the current training stage of the target model; Among the various sample image acquisition methods, the target acquisition method corresponding to the current training stage is selected to train the target model.

19. The apparatus according to claim 18, wherein, When the current training phase is the initial training phase for the target model, the target acquisition method is to acquire long-tail image samples whose sample accuracy meets the preset accuracy requirements.

20. The apparatus according to claim 18, wherein, If the target model satisfies the first convergence condition, the current training phase of the target model is switched to the intermediate training phase. The target acquisition method corresponding to the intermediate training stage is to obtain the long-tail image samples from the task information provided by the target object performing the relevant task of the long-tail image.

21. The apparatus according to claim 18, wherein if the target model satisfies the second convergence condition, the current training phase of the target model is switched to the final training phase; The target acquisition method corresponding to the final training stage is to select long-tail image samples from multiple images based on long-tail reference samples.

22. The apparatus according to claim 18, wherein, The target model is a long-tail classification model.

23. The apparatus according to claim 16, wherein, The sorting module includes: The evaluation unit is used to perform a weighted summation of the similarity and classification confidence of each similar image to obtain an evaluation value for each similar image as a long-tailed target image; The sorting unit is used to sort the similar images in the candidate image set based on the magnitude of the evaluation value of each similar image to obtain a candidate image sequence.

24. The apparatus according to claim 16, wherein, The optimization module further includes: The output unit is used to output the candidate image sequence to the target object; The first determining unit is configured to determine the abnormal image in the candidate image sequence in response to the target object's selection operation of the non-desired image in the candidate image sequence; The first optimization unit is used to treat the abnormal image as a new negative sample to optimize the long-tail classification model.

25. The apparatus according to claim 16, wherein, The second determining unit is specifically used for: The candidate image sequence is output to the target object; In response to the target object's selection operation of a non-desired image in the candidate image sequence, the order of each similar image in the candidate image sequence is updated to obtain the desired order, wherein the non-desired images selected in the desired order are ordered after the unselected images.

26. The apparatus according to claim 23, wherein the optimization module is further configured to: The weights of the similarity and the classification confidence are optimized based on the difference between the candidate image sequence and the expected ranking of the candidate image sequence.

27. The apparatus according to claim 26, wherein, The optimization module is used to optimize the long-tail classification model when the difference between the candidate image sequence and the desired ranking is greater than a preset difference degree.

28. The apparatus according to claim 26, wherein, The optimization module is used to optimize the weights of the similarity and the classification confidence when the difference between the candidate image sequence and the desired ranking is less than or equal to a preset difference degree.

29. The apparatus according to claim 17, wherein, The long-tailed image samples whose sample accuracy meets the preset accuracy requirements are derived from manually annotated images.

30. An image retrieval device, comprising: The initial selection module is used to filter out multiple images that meet the preset conditions for similarity with the target image, thus obtaining an initial image set; The processing module is used to classify the initial image set based on the long-tail classification model that has been trained and converged, and to obtain the classification confidence of each initial image in the initial image set. The selection module is used to filter long-tail target images from the initial image set based on the similarity and classification confidence between each initial image and the target image; The selected module is specifically used for: The initial images in the initial image set are sorted to obtain the initial image sequence; The long-tailed target images are selected from the initial image sequence; The long-tail classification model was trained in the following manner: Multiple similar images that meet the preset similarity criteria with the long-tail reference samples are selected to obtain a candidate image set; The candidate image set is classified based on a long-tail classification model to obtain the classification confidence of each similar image in the candidate image set; Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence. Optimizing the long-tail classification model based on the candidate image sequence includes: determining the desired ranking of the candidate image sequence; The long-tail classification model is optimized based on the difference between the candidate image sequence and the desired ranking.

31. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

33. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.

Images