Data processing method, image retrieval method and related device

Abstract

The invention provides a data processing method, an image retrieval method and a related device, and relates to the technical field of image processing, in particular to the technical fields of automatic driving, intelligent transportation, computer vision, intelligent search and the like. According to the specific implementation scheme, the method comprises the steps of determining a target acquisition mode in a plurality of preset sample image acquisition modes; and acquiring a long-tail image sample based on the target acquisition mode.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as autonomous driving, intelligent transportation, computer vision, and intelligent search. Background Art

[0002] The category distribution of real samples in visual recognition tasks is usually uneven. That is, the head category has more sample instances, and the tail category has fewer samples. The fewer samples at the tail can be called long-tail targets.

[0003] In autonomous driving, the detection and recognition of long-tail targets is a difficult and important point. This is because long-tail targets are sparsely distributed in actual scenes, and it is difficult to obtain a large number of long-tail targets in a short period of time. Summary of the invention

[0004] The present disclosure provides a data processing method, an image retrieval method and related devices.

[0005] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0006] Determine a target acquisition method among a plurality of preset sample image acquisition methods;

[0007] A long-tail image sample is acquired based on the target acquisition method.

[0008] According to another aspect of the present disclosure, there is provided an image retrieval method, comprising:

[0009] Screen out multiple preliminary images whose similarity with the target image meets preset conditions to obtain a preliminary image set;

[0010] Classify the preliminary image set based on the long-tail classification model with training convergence, and obtain the classification confidence of each preliminary image in the preliminary image set;

[0011] Based on the similarity and classification confidence between each preliminary image and the target image, long-tail target images are screened out from the preliminary image set.

[0012] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0013] A determination module, used to determine a target acquisition method among a plurality of preset sample image acquisition methods;

[0014] An acquisition module is used to acquire long-tail image samples based on the target acquisition method.

[0015] According to another aspect of the present disclosure, there is provided an image retrieval device, comprising:

[0016] A preliminary selection module is used to screen out multiple preliminary selection images whose similarity with the target image meets preset conditions to obtain a preliminary selection image set;

[0017] A processing module, used for classifying the preliminary selected image set based on a long-tail classification model that has converged in training, and obtaining a classification confidence of each preliminary selected image in the preliminary selected image set;

[0018] The selection module is used to select long-tail target images from the set of preliminary selected images based on the similarity and classification confidence between each preliminary selected image and the target image.

[0019] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0025] In the disclosed embodiment, since the number of long-tail image samples is small, the target acquisition method can be determined based on actual conditions to acquire more long-tail image samples.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0028] Figure 1 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0029] Figure 2 is a flowchart of a data processing method according to another embodiment of the present disclosure;

[0030] Figure 3 is a flowchart of a data processing method according to another embodiment of the present disclosure;

[0031] Figure 4 is a schematic diagram of a visualization interface according to an embodiment of the present disclosure;

[0032] Figure 5 is a framework diagram of a training method for a long-tail classification model according to an embodiment of the present disclosure;

[0033] Figure 6 is a flowchart of an image retrieval method according to an embodiment of the present disclosure;

[0034] Figure 7 is a structural schematic diagram of a data processing device according to an embodiment of the present disclosure;

[0035] Figure 8 is a structural schematic diagram of an image retrieval device according to an embodiment of the present disclosure;

[0036] Fig. 9 It is a block diagram of an electronic device used to implement the data processing method\image retrieval method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In related technologies, the way to obtain long-tail targets is to search by image, which means first extracting feature vectors from the targets in the full data, and then finding data similar to the long-tail targets based on feature similarity methods. Since searching by image is more dependent on visual features, the retrieval effect is poor in some scenarios, and usually manual screening is required to obtain more accurate data.

[0040] In order to improve the efficiency of acquiring long-tail targets, a data processing method is proposed in the embodiment of the present disclosure, which can be specifically implemented as follows: Figure 1 As shown:

[0041] S101, determining a target acquisition method among a plurality of preset sample image acquisition methods.

[0042] S102, acquiring long-tail image samples based on a target acquisition method.

[0043] Among them, in the scenario of autonomous driving, long-tail image samples can generally include pedestrian long-tail targets, vehicle long-tail targets and road pollution long-tail targets. Pedestrian long-tail targets can include people holding umbrellas, people squatting on the roadside, especially children holding umbrellas, children squatting on the roadside, etc. Exemplarily, vehicle long-tail targets can include special-shaped trucks, large engineering vehicles, bicycles carrying goods, tricycles carrying goods, etc. For road pollution long-tail targets, exemplary examples include plastic bags, stones, branches, etc. that affect vehicle driving.

[0044] In the disclosed embodiment, since the number of long-tail image samples is small, in order to improve the accuracy of the acquired long-tail image samples, multiple target acquisition methods can be determined based on actual conditions, thereby acquiring more long-tail image samples.

[0045] In some embodiments, the target acquisition method may include at least one of the following three methods:

[0046] Method 1) Obtain long-tail image samples whose sample accuracy meets the preset accuracy requirements;

[0047] In some embodiments, the long-tail image samples whose sample accuracy meets the preset accuracy requirements can be constructed from a small number of high-quality samples. The preset accuracy requirements may include information such as clarity and light meeting the preset requirements. The long-tail image samples that meet the preset accuracy requirements may only include samples related to long-tail targets, such as positive samples and negative samples of multiple long-tail targets. In the autonomous driving scenario, the positive sample is at least one of the aforementioned pedestrian long-tail targets, vehicle long-tail targets, and road pollution long-tail targets. Negative samples can be normal vehicles, pedestrians, road pollution factors that do not affect driving, etc.

[0048] In addition, when the target model is trained using long-tail image samples, general category images that meet the preset accuracy requirements can also be obtained. The general category images that meet the preset accuracy requirements can also only include samples that are not related to the long-tail target, such as positive samples and negative samples of apples. The images that meet the preset accuracy requirements can also include samples related to the long-tail target and samples that are not related to the long-tail target.

[0049] In the disclosed embodiment, high-quality long-tail image samples can be acquired in this way, thereby improving the image quality of the long-tail image samples.

[0050] In some embodiments, in order to ensure sample quality, long-tail image samples whose sample accuracy meets preset accuracy requirements may be derived from manually annotated images.

[0051] This target acquisition method can acquire high-quality long-tail target images and improve the data quality of long-tail targets.

[0052] In the disclosed embodiment, when long-tail image samples are used to train the target model, a small amount of manually annotated samples are used, which is easy to implement and ensures the accuracy of the target model in the training stage.

[0053] Method 2) obtaining long-tail image samples from task information provided by the target object for performing tasks related to long-tail images;

[0054] In some embodiments, the long-tail image samples obtained in this way can be samples collected when searching for long-tail targets. The positive sample images or negative sample images of the long-tail targets provided by the target object can be used to expand the long-tail image sample quantity.

[0055] Method 3) Filter out images that are similar to the long-tail reference samples from the image library as long-tail image samples.

[0056] In the disclosed embodiment, by screening out long-tail image samples, the samples can be automatically expanded to increase the amount of long-tail target data.

[0057] In some embodiments, as described above, the acquired long-tail image samples are used to optimize the target model, which may be any model related to the long-tail target, such as a detection model of the long-tail target, a classification model of the long-tail target, etc.

[0058] In the disclosed embodiment, in view of the fact that the amount of high-precision long-tail target data is small and it is difficult to directly obtain a large number of high-quality samples, when training the target model, different methods can be used to obtain long-tail image samples based on the training stage to meet the training requirements of the target model and gradually optimize the model performance of the target model.

[0059] Specifically, during implementation, based on the training requirements of the target model, the target acquisition method is determined among the preset multiple sample image acquisition methods, which can be implemented as follows:

[0060] Step A1, determining the current training stage of the target model;

[0061] Step A2: among a plurality of sample image acquisition methods, select a target acquisition method corresponding to the current training stage to train the target model.

[0062] During implementation, each training stage can determine the corresponding target acquisition method based on the actual situation, which is used to iteratively optimize the target model.

[0063] In the disclosed embodiment, the training stage of the target model is divided into multiple stages. Each training stage has a corresponding target acquisition method, so that the target model can acquire different samples in each stage to train iterative optimization training, thereby gradually improving the performance of the target model when long-tail targets are difficult to obtain.

[0064] In the disclosed embodiment, the training phase may be an initial training phase, an intermediate training phase, and a final training phase. In actual implementation, the training phase may be increased or decreased according to demand.

[0065] Among them, when the current training stage is the initial training stage for the target model, the target acquisition method is to obtain images whose sample accuracy meets the preset accuracy requirements. The images with the preset accuracy requirements may include the long-tail image samples of the aforementioned preset accuracy requirements and / or common images of the preset accuracy requirements.

[0066] After obtaining images whose sample accuracy meets the preset accuracy requirements, the target model can be trained using the MAML (Model-Agnostic Meta-Learning) method.

[0067] In the disclosed embodiment, a small amount of easily accessible high-quality sample images are used to initialize the target model, thereby providing a good sample basis for target training, so that the target model can learn useful knowledge, thereby improving the performance of the model.

[0068] In some embodiments, when the target model satisfies the first convergence condition, the current training stage of the target model is switched to the intermediate training stage; the target acquisition method corresponding to the intermediate training stage is to obtain long-tail image samples from the task information provided by the target object to perform related tasks of long-tail images.

[0069] In the disclosed embodiment, the target model can learn useful knowledge through the initial stage of training. At this time, the model is further optimized by obtaining long-tail image samples of the target object, so that the knowledge learned by the target model in the initial stage can be gradually focused and migrated to the long-tail targets, and migrated to the samples of the actual operational needs of the target object. This not only expands the long-tail samples, but also improves the model's learning ability on the long-tail targets.

[0070] In some embodiments, when the target model satisfies the second convergence condition, the current training stage of the target model is switched to the final training stage; the target acquisition method corresponding to the final training stage is long-tail image samples screened out from multiple images based on long-tail reference samples.

[0071] The second convergence condition refers to the convergence of the target model after the intermediate training stage. At this time, the target model's ability to perform long-tail targets is further improved after the intermediate training stage. Then, by automatically mining new long-tail targets from a large number of images through image screening, the target model is optimized to further improve the model performance of the target model.

[0072] In order to automatically mine as many long-tail targets as possible, in the embodiment of the present disclosure, based on the long-tail reference samples, long-tail image samples are screened from multiple images, such as Figure 2 As shown, it can be implemented as:

[0073] S201, screening out multiple similar images whose similarity with the long-tail reference sample meets a preset condition, and obtaining a candidate image set.

[0074] During implementation, multiple similar images whose similarity with the long-tail reference samples meets preset conditions can be screened out from the image library to obtain a candidate image set.

[0075] Many image acquisition devices are often installed on vehicles, so the data collected by the entire vehicle can be collected and images or videos can be filtered out. Image frames can be extracted from the video to convert the video into multiple frames. Target detection is performed on the obtained images respectively to obtain the position frames of each target. It can be understood that the target detection here is not simply target detection for long-tail targets, but can be target detection for long-tail targets and other targets of various categories to obtain the position frames of each target. Afterwards, based on the position frame, the corresponding target is segmented from the image. Then, feature extraction is performed on the image of each target, where visual features can be extracted to obtain the image features of each target. Based on the images and image features of each target, the image library described in the previous article can be constructed.

[0076] When screening similar images of long-tail reference samples, the image features of the long-tail reference samples are obtained, and the image features of each image in the image library are obtained at the same time, and then the similarity between the image features in the image library and the image features of the long-tail reference samples is calculated, and multiple similar images whose similarity with the long-tail reference samples meets the preset conditions are screened out to obtain a candidate image set.

[0077] The preset condition may be that the similarity is greater than a preset threshold or the ranking is within a previous preset ranking.

[0078] Among them, cosine similarity can be used to screen out similar images of long-tail reference samples. In addition, the similarity can also be calculated using Pearson Correlation Coefficient, Euclidean Distance, etc. The embodiment of the present disclosure does not limit the method for calculating the similarity.

[0079] S202, processing the candidate image set based on the target model for processing the long-tail target to obtain the long-tail target reference information extracted by the target model.

[0080] S203, based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, sort the similar images in the candidate image set to obtain a candidate image sequence.

[0081] Among them, the long-tail target reference information is used to quantitatively express the situation that the long-tail target is contained in the similar images. The long-tail target reference information can directly or indirectly reflect the probability that the long-tail target is contained in the similar images. Therefore, the ranking value can be determined based on the positive correlation between the similarity and the long-tail target reference information and the probability that the long-tail target is contained in the similar images. Then, the candidate image sequence is determined by the ranking value.

[0082] S204: Filter out long-tail image samples from the candidate image sequence.

[0083] In the disclosed embodiment, the target model can be used to automatically mine more long-tail image samples to expand the number of long-tail image samples. In addition, based on the use of the aforementioned multiple model training stages to gradually train the target model, the disclosed embodiment can use the target model that is already relatively mature in the long-tail target task to actively mine more long-tail image samples. The mined long-tail image samples can further optimize the target model itself, thereby efficiently completing the self-optimization and iteration of the model and improving the model performance. Afterwards, the long-tail targets can be further mined to improve the efficiency of automatic mining of long-tail targets.

[0084] In summary, the training goal of the target model can be to obtain more long-tail image samples. The obtained long-tail image samples can be used to train other models or to further iterate and optimize the target model. After three training stages, the training process of the target model may include:

[0085] Step B1, training a target model based on a sample set; the sample set includes first-category samples, which are general-category images meeting preset accuracy requirements as described above, or long-tail image samples meeting preset accuracy requirements.

[0086] Step B2: When the target model meets the first convergence condition after training with the sample set, the second type of samples are input into the target model to continue optimizing the target model. The second type of samples are the long-tail image samples obtained based on the above-mentioned method 2.

[0087] By obtaining long-tail image samples in step B2, not only can the sample volume of long-tail image samples be enriched, but also the target model can be migrated to the long-tail target task, thereby improving the business capability of the target model on the long-tail target task.

[0088] In step B3, the target model is used to filter out long-tail image samples from multiple images to iteratively optimize the target model.

[0089] Therefore, by gradually improving the business capabilities of the target model on long-tail targets, the target model can be automatically mined and optimized, thereby improving the efficiency of screening out long-tail image samples.

[0090] In the disclosed embodiment, the target model may be a long-tail classification model. The long-tail classification model is used to perform binary classification on the input image to classify the input image into an image containing a long-tail target (i.e., a long-tail target image) or an image not containing a long-tail target. The classification result is described with the aid of classification confidence, which measures the confidence of the binary classification result corresponding to the similar image, and can also be understood as the classification confidence used to measure the credibility of the classification result. Therefore, in the case where the target model is a long-tail classification model, the long-tail target reference information extracted by the target model can be the classification confidence.

[0091] In the disclosed embodiment, when the target model is a long-tail classification model, the long-tail image samples can be screened based on similarity and classification confidence. Among them, similarity can be understood as a measure of the similarity in visual features between the long-tail reference sample and the similar image, and classification confidence is a description of the category, which is a supplementary description outside the visual features. Classification confidence is different from visual features, and is not a product of the visual feature extraction process. Therefore, classification confidence can well make up for the shortcomings of simply using visual features, so that long-tail image samples can be more accurately screened to expand the number of long-tail image samples.

[0092] In addition, after the classification model is optimized in the aforementioned initial training stage and the intermediate training stage, it has a certain ability to classify long-tail targets. On this basis, the long-tail classification model can be used to automatically filter out long-tail image samples through its ability to classify long-tail targets. Therefore, through the target acquisition method of the final training stage provided by the embodiment of the present disclosure, the long-tail target data can be further accurately expanded, thereby improving the training efficiency of the long-tail classification model.

[0093] In the embodiment of the present disclosure, based on the target model being a long-tail classification model, the long-tail classification model can be optimized by the following methods, including: Figure 3 Steps shown:

[0094] S301, screening out multiple similar images whose similarity with the long-tail reference sample meets a preset condition, and obtaining a candidate image set.

[0095] The specific screening method is similar to the aforementioned screening method, and will not be described in detail in the embodiments of the present disclosure.

[0096] S302, classifying the candidate image set based on the long-tail classification model to obtain the classification confidence of each similar image in the candidate image set.

[0097] S303, sorting the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample to obtain a candidate image sequence.

[0098] S304, optimizing the long-tail classification model based on the candidate image sequence.

[0099] In the embodiment of the present disclosure, when there is less long-tail target data, in order to optimize the long-tail classification model, it is proposed to use similar images of long-tail reference samples to optimize the long-tail classification model. Based on the similarity, the classification confidence is introduced to measure the influence of similar images on the optimization model. The use of classification confidence makes up for the shortcomings of simply using visual features, so that it is possible to more accurately measure the influence of each similar image on the optimization of the long-tail classification model, and then further improve the long-tail classification model. Finally, by using the embodiment of the present disclosure, it is possible to improve the accuracy of the long-tail classification model in identifying long-tail targets and improve the training efficiency of the model. Moreover, the trained long-tail classification model can further combine the similarity to automatically and accurately mine the long-tail targets.

[0100] Figure 3 The training method shown can be understood as an implementation method for optimizing the long-tail classification model in the final training stage in order to facilitate the mining of long-tail targets. Combining the methods of the initial training stage and the intermediate training stage can better obtain a long-tail classification model that can mine long-tail image samples. Specifically, the following steps are included:

[0101] Step C1, training a long-tail classification model based on a sample set; the sample set includes a first category of samples, which are long-tail image samples and / or general category images that meet preset accuracy requirements obtained in the aforementioned method 1).

[0102] During implementation, when the sample set is labeled as multi-category, the initial classification model is trained as a multi-classification model through the sample set.

[0103] Among them, when the long-tail classification model is initialized with a sample set, the MAML (Model-Agnostic Meta-Learning) method can be used to train the long-tail classification model. Therefore, when the sample set is used for multi-classification task training, MAML can be used to obtain a long-tail classification model with relatively good results in multi-classification tasks. Then, based on the second type of samples, a long-tail classification model with better results in the corresponding task can be obtained.

[0104] Step C2, when the initial classification model meets the first convergence condition, input the second category samples into the long-tail classification model to obtain the classification result of the initial classification model for the second category samples; the classification result is a long-tail target image or a common image.

[0105] Among them, the classification result of the long-tail classification model for the second type of samples can be obtained; the classification result is a long-tail target image or a common image; among them, the second type of samples are long-tail image samples obtained from the task information provided by the target object to perform related tasks of the long-tail image.

[0106] When the first type of samples uses high-quality general images, the long-tail classification model can learn general knowledge in the initial training stage. On this basis, the second type of samples are further used to enable the initial model to migrate to the task of searching for long-tail targets, thereby improving the business capabilities of the long-tail classification model in the long-tail target classification task.

[0107] Among them, the first convergence condition mainly refers to the training condition of the long-tail classification model on the sample set, which can be expressed by the model loss on the sample set no longer changing, or the number of training times reaching the preset number.

[0108] Step C3, based on the classification results and the true classification of the second class of samples, the model parameters of the initial classification model are adjusted to obtain a long-tail classification model when the second convergence condition is met.

[0109] The second convergence condition may be that the gap information between the classification result and the true classification of the second type of samples tends to be stable and / or meets a preset number of iterations.

[0110] In the disclosed embodiment, the initial classification model is trained in two stages. One is to use a small amount of easily accessible sample sets to initialize the initial classification model, thereby providing a good model foundation. Then, the initial classification model is migrated to the binary classification task using the second type of sample, so as to facilitate the migration of the initial classification model to the current task, so that the obtained long-tail classification model can adapt to the current task requirements. The two-stage training can obtain a long-tail classification model that can meet the task requirements through step-by-step training when there is less long-tail target data, thereby improving the training efficiency of the model.

[0111] In some embodiments, the similarity represents the similarity between each similar image and the long-tail reference sample, so the more similar the two are, the higher the similarity is, resulting in a higher evaluation value. The classification confidence represents the score of each similar image belonging to the corresponding category. The evaluation value is positively correlated with the similarity and classification confidence of each similar image. Based on this, based on the similarity and classification confidence between each similar image and the long-tail reference sample, each similar image in the candidate image set is sorted to obtain a candidate image sequence, which can be implemented as follows:

[0112] Step D1, adding the similarity and classification confidence of each similar image to obtain an evaluation value of each similar image being a long-tail target image.

[0113] In implementation, the similarity of each similar image in the embodiment of the present disclosure may include the similarity of light, contrast, structure, and content in the image. A neural network model may be used to extract image features. The neural network model may extract shallow features such as light, color, and texture, and may also extract deep features, and then fuse the shallow features and the deep features to obtain the final extracted image features.

[0114] On the basis of screening out similar images based on similarity, the classification confidence is used as the long-tail target reference information and combined with the similarity to obtain the evaluation value.

[0115] Before adding the similarity and classification confidence, since the physical meanings of similarity and classification confidence are different, the similarity and classification confidence of similar images can be mapped to the same feature space, and then added to obtain the evaluation value of each similar image as a long-tail target image.

[0116] In implementation, the first neural network module can be used to map the similarity to the feature space, and the second neural network module can be used to map the classification confidence to the feature space. The model parameters of the two neural network modules can be obtained through training. The loss of training can be measured by the difference between the candidate image sequence and the expected sorting. In implementation, the two neural network modules can be trained together when adjusting the weights later.

[0117] Step D2: sorting the similar images in the candidate image set based on the evaluation value of each similar image to obtain a candidate image sequence.

[0118] In the disclosed embodiment, based on the similarity, the evaluation value is determined by combining multiple dimensions of classification confidence, and then the similar images in the candidate image set are sorted, so that the final candidate image sequence is more accurate.

[0119] In another embodiment, since the influence of similarity and classification confidence may be different for different long-tail targets, in order to more accurately evaluate the evaluation value of each similar image, based on the similarity and classification confidence between each similar image and the long-tail reference sample, each similar image in the candidate image set is sorted to obtain a candidate image sequence, which can also be implemented as follows:

[0120] Step E1, performing weighted summation on the similarity and classification confidence of each similar image to obtain an evaluation value of each similar image being a long-tail target image.

[0121] During implementation, the weights of the similarity and classification confidence of each similar image can be adaptively adjusted. When the classification confidence is given more importance, the weight of the classification confidence can be adaptively higher than the weight of the similarity. Exemplarily, the weight of the classification confidence is set to a, and the weight of the similarity is set to b, where a is less than b. When the similarity is given more importance, the weight of the similarity can be adaptively higher than the weight of the classification confidence. Exemplarily, the weight of the similarity is set to b, and the weight of the classification confidence is set to a. The specific numerical values ​​of the weights of the similarity and the classification confidence are not limited in the embodiments of the present disclosure.

[0122] Step E2: sorting the similar images in the candidate image set based on the evaluation value of each similar image to obtain a candidate image sequence.

[0123] During implementation, similar images in the candidate image set are sorted in descending order based on the evaluation values ​​to obtain a candidate image sequence.

[0124] In the disclosed embodiment, based on the similarity, a weighted sum method is used in combination with the classification confidence, so that the influence of the similarity and the classification confidence on the evaluation value can be flexibly adjusted, so that the final candidate image sequence is more in line with the expectations of the target object.

[0125] In addition, in addition to the aforementioned addition and weighted summation methods, data fitting can be performed on similarity and classification confidence according to the principle of data analysis. The fitting function can also be used, for example, by using a product, a power function, etc. Regardless of the fitting function used, the fitting function has adjustable and learnable parameter items to make the fitting result of the fitting function close to the expected result.

[0126] In some embodiments, based on the candidate image sequence, similar images in the candidate image sequence can be classified into three categories: low, medium, and high. For example, images with lower evaluation values ​​are classified into low-quality categories, images with medium evaluation values ​​are classified into medium-quality categories, and images with higher evaluation values ​​are classified into high-quality categories. When optimizing the long-tail classification model, similar images of all quality categories can be first optimized, and then similar images of medium-quality categories and high-quality categories can be optimized, and finally similar images of high-quality categories can be optimized. In this way, the stage-by-stage optimization method can facilitate the model to first learn general knowledge through a large number of samples, and then gradually learn the knowledge of high-quality samples.

[0127] In other embodiments, in order to speed up the optimization speed and improve the training quality, the long-tail classification model is optimized based on the candidate image sequence, which can also be implemented as follows:

[0128] Step F1, outputting a candidate image sequence to the target object.

[0129] The target object can be an ordinary user who inputs long-tail reference samples, or a professional person who trains models.

[0130] Step F2, in response to the target object's selection operation on an undesired image in the candidate image sequence, determining an abnormal image in the candidate image sequence.

[0131] During implementation, in the initial stage, there may be samples in the candidate image sequence that are significantly different from the long-tail reference samples. Therefore, after automatically screening out similar images of the long-tail reference samples and sorting them appropriately, they can be provided to the target object to evaluate the sorting results. This allows the target object to screen out samples with significant differences, thereby achieving fine screening of samples. Afterwards, in step C3, the abnormal images can be used as newly added negative samples to optimize the long-tail classification model.

[0132] In implementation, in order to improve the accuracy of the long-tail classification model, the output candidate image sequence can be Figure 4 A visual display is performed, with white representing unselected images and gray representing selected images. The target object selects non-desired images in the candidate image sequence, determines abnormal images in the candidate image sequence, and uses the abnormal images as new negative samples, which are then input into the long-tail classification model to optimize it, thereby improving the accuracy of the long-tail classification model.

[0133] In the disclosed embodiment, abnormal images are selected from a candidate image sequence based on the target object. This method can quickly screen out negative samples in a semi-automatic manner, thereby further optimizing the long-tail classification model to improve the accuracy of the long-tail classification model.

[0134] Of course, it is understandable that samples with higher evaluation values ​​in the candidate image sequence can be used as positive samples to expand the long-tail target data to optimize the long-tail classification model.

[0135] It can be summarized as follows: based on the candidate image sequence, positive and negative samples of long-tail targets are screened out to further optimize the long-tail classification model.

[0136] In some other embodiments, in addition to optimizing the long-tail classification model in the above two ways, optimizing the long-tail classification model based on the candidate image sequence may also be implemented as follows:

[0137] Step G1, determining the desired ordering of the candidate image sequence.

[0138] Step G2, optimizing the long-tail classification model based on the difference between the candidate image sequence and the expected ranking.

[0139] In some embodiments, determining a desired ordering of a candidate image sequence may be implemented as follows: outputting the candidate image sequence to a target object; in response to the target object's selection operation of a non-desired image in the candidate image sequence, updating the ordering of similar images in the candidate image sequence to obtain a desired ordering, wherein the selected non-desired image in the desired ordering is ordered after the unselected image.

[0140] The visual interface is still Figure 4 As shown, white represents unselected images and gray represents selected images. After the target object selects at least one undesired image, the order of the selected undesired images can be adjusted to be after the unselected images. In the case of multiple undesired images, they can be sorted based on the similarity and classification confidence of each undesired image to obtain a desired order.

[0141] In the disclosed embodiment, based on the selection operation of the target object, the output sequence is adaptively adjusted to obtain the adjusted expected ordering, which can simply and quickly obtain the expected ordering. The expected ordering can accurately express the correlation between the retrieved similar images and the long-tail reference samples, which is conducive to improving the classification efficiency of the long-tail classification model.

[0142] During implementation, the image corresponding to the expected sorting is used as sample data and input into the long-tail classification model to obtain the predicted sorting. The predicted sorting is compared with the expected sorting, and the long-tail classification model is adjusted based on the gap information. When the convergence conditions are met, the optimization of the long-tail classification model is completed.

[0143] In the disclosed embodiments, the long-tail classification model is continuously optimized during use, so that the performance of the long-tail classification model is continuously improved, and as the number of long-tail target images and ordinary images increases, the long-tail classification model can adapt to various situations to improve accuracy.

[0144] During implementation, the above three methods of optimizing the long-tail classification model may be executed one by one, or multiple methods may be selected for optimization, which is not limited in the embodiments of the present disclosure.

[0145] In some embodiments, in order to improve the accuracy of the evaluation values ​​obtained based on similarity and classification confidence, so as to obtain high-quality retrieval results in the stage of retrieving long-tail targets, on the basis of obtaining the desired ranking, the weights of similarity and classification confidence can also be optimized based on the difference between the candidate image sequence and the desired ranking.

[0146] In the disclosed embodiment, based on the desired sorting, the weights of similarity and classification confidence are adjusted, so that the sorting obtained later is more reasonable, thereby improving the performance of the long-tail classification model.

[0147] In some embodiments, when the difference between the candidate image sequence and the desired sorting is greater than a preset difference, the long-tail classification model is optimized preferentially.

[0148] During implementation, when the difference is greater than the preset difference, it means that the candidate image sequence is too far away from the expected ordering, and the long-tail classification model needs to be further optimized to speed up the convergence.

[0149] Exemplarily, for each candidate image sequence, the difference between the candidate image sequence and the expected order can be represented by the proportion of images with incorrect ordering in all images. For example, if the candidate image sequence includes 100 candidate images, and 20 images are incorrectly ordered, the proportion is 0.2. The preset difference can also be represented by a proportion threshold.

[0150] In another example, the difference between multiple candidate image sequences and their respective expected rankings can also be used to measure whether to optimize the long-tail classification model first. For example, each candidate image sequence can obtain a difference based on the above method, and then calculate the mean difference between the multiple candidate image sequences and their respective expected rankings, and use the mean difference and the preset difference degree to compare, and then decide whether to optimize the long-tail classification model first.

[0151] In the disclosed embodiment, when the difference between multiple candidate image sequences and their respective desired rankings is greater than a preset difference, the long-tail classification model is optimized preferentially, and the evaluation value can be quickly adjusted so that the ranking based on the evaluation value approaches the desired result as quickly as possible, thereby achieving the purpose of rapid convergence of training.

[0152] Accordingly, when the difference between the candidate image sequence and the desired sorting is less than or equal to the preset difference, the weights of the similarity and the classification confidence are optimized preferentially.

[0153] During implementation, when the difference is not greater than the preset difference, it means that the candidate image sequence is not far from the expected order, and the weights of the similarity and classification confidence can be appropriately optimized.

[0154] In the disclosed embodiment, when the difference between a plurality of candidate image sequences and their respective desired rankings is less than or equal to a preset difference, the weights of similarity and classification confidence are preferentially optimized to make the final ranking more accurate.

[0155] For ease of understanding, the framework of the training method for the long-tail classification model proposed in the embodiment of the present disclosure includes: training the initial model with multiple groups of manually annotated positive and negative samples in the sample set, so that after meta-learning reasoning, an initial model that satisfies the first convergence condition is obtained. When the reference sample input by the target object during the retrieval process is detected, a second type of sample is constructed based on the input reference sample to optimize the initial model, and a long-tail classification model is obtained when the second convergence condition is met.

[0156] Afterwards, in the process of retrieving long-tail targets, when the target object is input into the long-tail reference sample of the third class sample set, the image features of the long-tail reference sample are extracted based on the feature extraction model, and the image features of the long-tail reference sample are calculated with the features in the image library to obtain the candidate image set. The candidate image set is input into the long-tail classification model to obtain the classification confidence of each similar image in the candidate image set. The similarity and classification confidence of each similar image are weighted summed to obtain the evaluation value of each similar image as the long-tail target image; based on the size of the evaluation value of each similar image, the similar images in the candidate image set are sorted to obtain the candidate image sequence. Based on the selection operation of the target object, the candidate image sequence is optimized to obtain the expected sorting. The images in the expected sorting can be used as supplementary content of the sample set to optimize the converged initial model to obtain the final long-tail classification model. The long-tail classification model and weights can also be jointly trained to optimize the long-tail classification model and weights.

[0157] like Figure 5 As shown in the figure, firstly, the image data collected by the vehicle is extracted through the target detection model to extract each target in the image, and then each target is segmented from the image data. Each target is extracted with the feature model to obtain the feature vector, thereby constructing the image features in the image library.

[0158] Subsequently, after the initial long-tail classification model is trained with the sample set, the business capabilities of the long-tail classification model are prompted. The long-tail classification model classifies the similar images of the target object's retrieval data, that is, the long-tail reference sample. Thus, the evaluation value of the similar images is calculated by similarity and classification confidence, and the similar images are sorted to obtain a candidate image sequence. Then, the desired sorting of the candidate image sequence is determined by confirming the target object, so as to further obtain positive and negative samples, and sort and optimize the long-tail classification model.

[0159] In summary, in the disclosed embodiment, the initial model is first initialized with a sample set with few annotations, so that the initial model can have good performance in multi-classification tasks, providing a high-quality model foundation. Then, through the samples provided in the target object retrieval process (such as second-category samples and long-tail reference samples), the initial model is migrated to the retrieval task, so that the long-tail classification model can be more adapted to the long-tail target classification task in the retrieval task, thereby providing reasonable and reliable classification results and classification confidence. The entire training process optimizes the long-tail classification model in continuous stages, which can overcome the shortcomings of long-tail target data and improve the training efficiency and quality of the model.

[0160] Based on the same technical concept, on the basis of obtaining the long-tail classification model, the embodiment of the present disclosure also provides an image retrieval method, such as Figure 6 As shown, including:

[0161] S601, screening out a plurality of preliminary selected images whose similarity with the target image meets a preset condition, to obtain a preliminary selected image set.

[0162] The preset condition may be that the similarity is greater than a preset threshold or the ranking is within a preset ranking. The method for calculating the similarity has been described above, and the embodiments of the present disclosure will not be described in detail.

[0163] S602, classifying the preliminary selected image set based on the long-tail classification model that has converged in training, and obtaining the classification confidence of each preliminary selected image in the preliminary selected image set.

[0164] S603: based on the similarity and classification confidence between each of the preliminary selected images and the target image, a long-tail target image is screened out from the preliminary selected image set.

[0165] In the disclosed embodiment, on the basis of screening out similarities that meet preset conditions, a long-tail classification model is used to obtain the classification confidence of each image, and long-tail target images are screened out from the preliminary image set in a manner combining classification confidence and similarity. Based on this method, the long-tail target images that can be automatically screened out improve the efficiency of acquiring long-tail targets. In addition, similarity can be understood as a measure of the similarity between long-tail reference samples and similar images in visual features, and classification confidence is a description of the long-tail target category, which is a supplementary description beyond visual features. Classification confidence is different from visual features, and is not a product of the visual feature extraction process. Therefore, classification confidence can well make up for the shortcomings of using visual features alone and improve the screening accuracy of long-tail targets.

[0166] In some embodiments, based on the similarity and classification confidence between each preliminary selected image and the target image, long-tail target images are screened out from the preliminary selected image set, which can be implemented as follows:

[0167] Step H1, sorting the preliminary selected images in the preliminary selected image set to obtain a preliminary selected image sequence.

[0168] In some embodiments, based on the similarity and classification confidence between each preliminary image and the target image, each preliminary image in the preliminary image set is sorted to obtain a preliminary image sequence, which can be implemented as follows: weighted summation of the similarity and classification confidence of each preliminary image is performed to obtain an evaluation value of each preliminary image as a long-tail target image; based on the size of the evaluation value of each preliminary image, each preliminary image in the preliminary image set is sorted to obtain a preliminary image sequence.

[0169] The method for determining the weights of the similarity and classification confidence of each preliminary selected image has been described above, and the embodiment of the present disclosure will not be repeated here.

[0170] In addition, in addition to using the weighted summation method to determine the evaluation value, other methods for determining the evaluation value are also described above, which are also applicable to the embodiments of the present disclosure and will not be repeated here.

[0171] Step H2, filter out the long-tail target images from the preliminary image sequence.

[0172] During implementation, when one long-tail target image is needed, the image with the highest evaluation value can be used as the final long-tail target image; when multiple long-tail target images are needed, images with evaluation values ​​ranking within the top preset number or images greater than the preset evaluation value can be used as the final long-tail target image.

[0173] In the disclosed embodiment, the long-tail target image obtained in this manner better meets the expectations of the target object, thereby improving the experience of the target object.

[0174] Based on the same technical concept, a data processing device 700 is proposed in the embodiment of the present disclosure, such as Figure 7 As shown, including:

[0175] A determination module 701 is used to determine a target acquisition method among a plurality of preset sample image acquisition methods;

[0176] The acquisition module 702 is used to acquire long-tail image samples based on the target acquisition method.

[0177] In some embodiments, the acquisition module includes any of the following units:

[0178] A first acquisition unit, used to acquire long-tail image samples whose sample accuracy meets a preset accuracy requirement;

[0179] A second acquisition unit is used to acquire long-tail image samples from task information provided by the target object for performing tasks related to long-tail images;

[0180] The third acquisition unit is used to filter out long-tail image samples from multiple images based on the long-tail reference samples.

[0181] In some embodiments, the determination module is specifically configured to:

[0182] Determine the current training stage of the target model;

[0183] Among a plurality of sample image acquisition methods, a target acquisition method corresponding to a current training stage is selected to train a target model.

[0184] In some embodiments, when the current training stage is an initial training stage for a target model, the target acquisition method is to acquire long-tail image samples whose sample accuracy meets a preset accuracy requirement.

[0185] In some embodiments, when the target model satisfies the first convergence condition, the current training stage of the target model is switched to the intermediate training stage;

[0186] The target acquisition method corresponding to the intermediate training stage is to obtain long-tail image samples from the task information provided by the target object performing the relevant tasks of the long-tail image.

[0187] In some embodiments, when the target model satisfies the second convergence condition, the current training phase of the target model is switched to the final training phase;

[0188] The target acquisition method corresponding to the final training stage is to filter out long-tail image samples from multiple images based on long-tail reference samples.

[0189] In some embodiments, the third acquisition unit is specifically configured to:

[0190] Filter out multiple similar images whose similarity with the long-tail reference samples meets the preset conditions to obtain a candidate image set;

[0191] Processing the candidate image set based on the target model for processing the long-tail target to obtain the long-tail target reference information extracted by the target model;

[0192] Based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, the similar images in the candidate image set are sorted to obtain a candidate image sequence;

[0193] From the candidate image sequence, long-tail image samples are screened out.

[0194] In some embodiments, the target model is a long-tail classification model.

[0195] In some embodiments, it also includes:

[0196] A screening module is used to screen out multiple similar images whose similarity with the long-tail reference sample meets preset conditions to obtain a candidate image set;

[0197] A classification module is used to classify the candidate image set based on a long-tail classification model to obtain the classification confidence of each similar image in the candidate image set;

[0198] A sorting module is used to sort the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample to obtain a candidate image sequence;

[0199] The optimization module is used to optimize the long-tail classification model based on the candidate image sequence.

[0200] In some embodiments, the sorting module includes:

[0201] An evaluation unit, used for performing weighted summation of the similarity and classification confidence of each similar image to obtain an evaluation value of each similar image being a long-tail target image;

[0202] The sorting unit is used to sort the similar images in the candidate image set based on the evaluation value of each similar image to obtain a candidate image sequence.

[0203] In some embodiments, the optimization module includes:

[0204] An output unit, used for outputting a candidate image sequence to a target object;

[0205] A first determining unit, configured to determine an abnormal image in the candidate image sequence in response to a selection operation of a target object on an undesired image in the candidate image sequence;

[0206] The first optimization unit is used to take the abnormal image as a newly added negative sample to optimize the long-tail classification model.

[0207] In some embodiments, the optimization module includes:

[0208] a second determining unit, configured to determine a desired ordering of the candidate image sequence;

[0209] The second optimization unit is used to optimize the long-tail classification model based on the difference between the candidate image sequence and the expected ranking.

[0210] In some embodiments, the second determining unit is specifically configured to:

[0211] Output candidate image sequence to the target object;

[0212] In response to the target object's selection operation on an undesired image in the candidate image sequence, the order of similar images in the candidate image sequence is updated to obtain a desired order, in which the selected undesired image is ordered after the unselected image.

[0213] In some embodiments, the optimization module is further configured to:

[0214] The weights of similarity and classification confidence are optimized based on the difference between the candidate image sequence and the desired ordering of the candidate image sequence.

[0215] In some embodiments, the optimization module is used to prioritize the optimization of the long-tail classification model when the difference between the candidate image sequence and the desired sorting is greater than a preset difference.

[0216] In some embodiments, the optimization module is used to prioritize the weights of similarity and classification confidence when the difference between the candidate image sequence and the desired sorting is less than or equal to a preset difference.

[0217] In some embodiments, the long-tail image samples whose sample accuracy meets the preset accuracy requirement come from manually annotated images.

[0218] Based on the same technical concept, an image retrieval device 800 is proposed in the embodiment of the present disclosure, such as Figure 8 As shown, including:

[0219] A preliminary selection module 801 is used to screen out a plurality of preliminary selection images whose similarity with the target image meets a preset condition, and obtain a preliminary selection image set;

[0220] The processing module 802 is used to classify the preliminary selected image set based on the long-tail classification model to obtain the classification confidence of each preliminary selected image in the preliminary selected image set;

[0221] The selection module 803 is used to select long-tail target images from the set of preliminary selected images based on the similarity and classification confidence between each preliminary selected image and the target image.

[0222] In some embodiments, the selection module is specifically used to:

[0223] Sort the preliminary selected images in the preliminary selected image set to obtain a preliminary selected image sequence;

[0224] Filter out long-tail target images from the preliminary image sequence.

[0225] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0226] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0227] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0228] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0229] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0230] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a model training method\image retrieval method. For example, in some embodiments, the model training method\image retrieval method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the model training method\image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the model training method\image retrieval method in any other suitable manner (for example, by means of firmware).

[0231] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0232] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0233] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0235] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0236] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0237] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0238] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Determine a target acquisition method among a plurality of preset sample image acquisition methods; A long-tail image sample is acquired based on the target acquisition method.

2. The method according to claim 1, wherein: The target acquisition method includes at least one of the following: Obtain long-tail image samples whose sample accuracy meets the preset accuracy requirements; Acquire the long-tail image sample from task information provided by the target object for performing a task related to the long-tail image; Based on the long-tail reference samples, the long-tail image samples are screened out from multiple images.

3. The method according to claim 1 or 2, wherein: The target acquisition method is determined among the preset multiple sample image acquisition methods, including: Determine the current training stage of the target model; Among the multiple sample image acquisition methods, a target acquisition method corresponding to the current training stage is selected to train the target model.

4. The method according to claim 3, wherein: In the case where the current training stage is an initial training stage for the target model, the target acquisition method is to acquire images whose sample accuracy meets a preset accuracy requirement.

5. The method according to claim 3, wherein: When the target model satisfies the first convergence condition, the current training phase of the target model is switched to the intermediate training phase; The target acquisition method corresponding to the intermediate training stage is to obtain the long-tail image samples from the task information provided by the target object when performing the task related to the long-tail image.

6. The method according to claim 3, wherein when the target model satisfies the second convergence condition, the current training stage of the target model is switched to the final training stage; The target acquisition method corresponding to the final training stage is to screen out the long-tail image samples from multiple images based on the long-tail reference samples.

7. The method according to claim 2 or 6, wherein: The step of selecting the long-tail image sample from a plurality of images based on the long-tail reference sample includes: Screening out multiple similar images whose similarity to the long-tail reference sample meets a preset condition to obtain a candidate image set; Processing the candidate image set based on a target model for processing long-tail targets to obtain long-tail target reference information extracted by the target model; Based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, the similar images in the candidate image set are sorted to obtain a candidate image sequence; The long-tail image samples are screened out from the candidate image sequence.

8. The method according to claim 7, wherein: The target model is a long-tail classification model.

9. The method according to any one of claims 2, 7, and 8, further comprising: Screening out multiple similar images whose similarity to the long-tail reference sample meets a preset condition to obtain a candidate image set; Classifying the candidate image set based on a long-tail classification model to obtain a classification confidence of each similar image in the candidate image set; Based on the similarity and classification confidence between each similar image and the long-tail reference sample, the similar images in the candidate image set are sorted to obtain a candidate image sequence; Based on the candidate image sequence, the long-tail classification model is optimized.

10. The method according to claim 9, wherein: The step of sorting the similar images in the candidate image set based on the similarity and classification confidence between the similar images and the long-tail reference samples to obtain a candidate image sequence includes: The similarity and classification confidence of each similar image are weightedly summed to obtain the evaluation value of each similar image being a long-tail target image; Based on the evaluation value of each similar image, the similar images in the candidate image set are sorted to obtain a candidate image sequence.

11. The method according to claim 9, wherein: Based on the candidate image sequence, optimizing the long-tail classification model includes: Outputting the candidate image sequence to a target object; In response to a selection operation of a target subject on an undesired image in the candidate image sequence, determining an abnormal image in the candidate image sequence; The abnormal images are used as new negative samples to optimize the long-tail classification model.

12. The method according to claim 9, wherein: Based on the candidate image sequence, optimizing the long-tail classification model includes: determining a desired ordering of the candidate image sequence; The long-tail classification model is optimized based on the difference between the candidate image sequence and the desired ranking.

13. The method according to claim 12, wherein: The determining of the desired order of the candidate image sequence comprises: Outputting the candidate image sequence to a target object; In response to the target object's selection operation on an undesired image in the candidate image sequence, the order of similar images in the candidate image sequence is updated to obtain the desired order, in which the selected undesired image is ordered after the unselected image.

14. The method according to claim 10, further comprising: The weights of the similarity and the classification confidence are optimized based on the difference between the candidate image sequence and the expected ordering of the candidate image sequence.

15. The method according to claim 14, wherein: When the difference between the candidate image sequence and the expected sorting is greater than a preset difference, the long-tail classification model is optimized preferentially.

16. The method according to claim 14, wherein: When the difference between the candidate image sequence and the desired sorting is less than or equal to a preset difference, the weights of the similarity and the classification confidence are preferentially optimized.

17. The method according to claim 2, wherein: The long-tail image samples whose sample accuracy meets the preset accuracy requirement come from manually annotated images.

18. An image retrieval method, comprising: Screen out multiple preliminary images whose similarity with the target image meets preset conditions to obtain a preliminary image set; Classifying the preliminary selected image set based on the long-tail classification model with converged training, and obtaining the classification confidence of each preliminary selected image in the preliminary selected image set; Based on the similarity and classification confidence between each preliminary selected image and the target image, a long-tail target image is screened out from the preliminary selected image set.

19. The method according to claim 18, wherein: The step of selecting a long-tail target image from the set of preliminary selected images based on the similarity and classification confidence between each preliminary selected image and the target image comprises: Sorting the preliminary selected images in the preliminary selected image set to obtain a preliminary selected image sequence; The long-tail target image is screened out from the preliminary selected image sequence.

20. A data processing device, comprising: A determination module, used to determine a target acquisition method among a plurality of preset sample image acquisition methods; An acquisition module is used to acquire long-tail image samples based on the target acquisition method.

21. The device according to claim 20, wherein: The acquisition module includes any of the following units: A first acquisition unit, used to acquire long-tail image samples whose sample accuracy meets a preset accuracy requirement; A second acquisition unit is used to acquire the long-tail image sample from task information provided by the target object when performing a task related to the long-tail image; The third acquisition unit is used to filter out the long-tail image samples from multiple images based on the long-tail reference samples.

22. The device according to claim 20 or 21, wherein: The determination module is specifically used for: Determine the current training stage of the target model; Among the multiple sample image acquisition methods, a target acquisition method corresponding to the current training stage is selected to train the target model.

23. The device according to claim 22, wherein: In the case where the current training stage is an initial training stage for the target model, the target acquisition method is to acquire long-tail image samples whose sample accuracy meets a preset accuracy requirement.

24. The device according to claim 22, wherein: When the target model satisfies the first convergence condition, the current training phase of the target model is switched to the intermediate training phase; The target acquisition method corresponding to the intermediate training stage is to obtain the long-tail image samples from the task information provided by the target object when performing the task related to the long-tail image.

25. The apparatus according to claim 22, wherein when the target model satisfies the second convergence condition, the current training stage of the target model is switched to the final training stage; The target acquisition method corresponding to the final training stage is to screen out the long-tail image samples from multiple images based on the long-tail reference samples.

26. The device according to claim 21 or 25, wherein: The third acquisition unit is specifically used for: Screening out multiple similar images whose similarity to the long-tail reference sample meets a preset condition to obtain a candidate image set; Processing the candidate image set based on a target model for processing long-tail targets to obtain long-tail target reference information extracted by the target model; Based on the similarity between each similar image and the long-tail reference sample and the long-tail target reference information, the similar images in the candidate image set are sorted to obtain a candidate image sequence; The long-tail image samples are screened out from the candidate image sequence.

27. The device according to claim 26, wherein: The target model is a long-tail classification model.

28. The apparatus according to any one of claims 21, 26, and 27, further comprising: A screening module is used to screen out a plurality of similar images whose similarity with the long-tail reference sample meets a preset condition to obtain a candidate image set; A classification module, used to classify the candidate image set based on a long-tail classification model to obtain a classification confidence of each similar image in the candidate image set; A sorting module, used to sort the similar images in the candidate image set based on the similarity and classification confidence between each similar image and the long-tail reference sample to obtain a candidate image sequence; An optimization module is used to optimize the long-tail classification model based on the candidate image sequence.

29. The device according to claim 28, wherein The sorting module comprises: An evaluation unit, used for performing weighted summation of the similarity and classification confidence of each similar image to obtain an evaluation value of each similar image being a long-tail target image; The sorting unit is used to sort the similar images in the candidate image set based on the evaluation value of each similar image to obtain a candidate image sequence.

30. The device according to claim 28, wherein The optimization module comprises: An output unit, used for outputting the candidate image sequence to a target object; A first determining unit, configured to determine an abnormal image in the candidate image sequence in response to a selection operation of a target object on an undesired image in the candidate image sequence; The first optimization unit is used to use the abnormal image as a newly added negative sample to optimize the long-tail classification model.

31. The apparatus of claim 28, wherein: The optimization module comprises: A second determining unit, configured to determine a desired ordering of the candidate image sequence; The second optimization unit is used to optimize the long-tail classification model based on the difference between the candidate image sequence and the expected ranking.

32. The device according to claim 31, wherein The second determining unit is specifically configured to: Outputting the candidate image sequence to a target object; In response to the target object's selection operation on an undesired image in the candidate image sequence, the order of similar images in the candidate image sequence is updated to obtain the desired order, in which the selected undesired image is ordered after the unselected image.

33. The apparatus according to claim 29, wherein the optimization module is further configured to: The weights of the similarity and the classification confidence are optimized based on the difference between the candidate image sequence and the expected ordering of the candidate image sequence.

34. The device according to claim 33, wherein The optimization module is used to preferentially optimize the long-tail classification model when the difference between the candidate image sequence and the expected sorting is greater than a preset difference.

35. The device according to claim 33, wherein The optimization module is used to preferentially optimize the weights of the similarity and the classification confidence when the difference between the candidate image sequence and the expected sorting is less than or equal to a preset difference.

36. The apparatus of claim 21, wherein: The long-tail image samples whose sample accuracy meets the preset accuracy requirement come from manually annotated images.

37. An image retrieval device, comprising: A preliminary selection module is used to screen out multiple preliminary selection images whose similarity with the target image meets preset conditions to obtain a preliminary selection image set; A processing module, used for classifying the preliminary selected image set based on a long-tail classification model that has converged in training, and obtaining a classification confidence of each preliminary selected image in the preliminary selected image set; The selection module is used to select long-tail target images from the set of preliminary selected images based on the similarity and classification confidence between each preliminary selected image and the target image.

38. The device according to claim 37, wherein The selected modules are specifically used for: Sorting the preliminary selected images in the preliminary selected image set to obtain a preliminary selected image sequence; The long-tail target image is screened out from the preliminary selected image sequence.

39. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.

40. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-19.

41. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 19.

Images