A method, apparatus and storage medium for filtering long-tailed image data

Abstract

This application relates to a method, apparatus, and storage medium for filtering long-tail image data. The method includes acquiring a set of long-tail sample images and a pre-selected image set including candidate images; extracting image features from each sample image and each candidate image to obtain a feature vector for each sample image and a candidate image feature vector for each candidate image; determining the feature similarity and structural similarity between each pair of feature vectors of sample images and candidate images based on a preset similarity algorithm; weighting the feature similarity and structural similarity to obtain the target similarity between each candidate image and each sample image; and identifying the candidate images whose target similarity meets the preset similarity conditions as long-tail image data. This application determines long-tail image data based on the weighted fusion result of feature similarity and structural similarity, which can improve the efficiency and accuracy of long-tail image data collection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and storage medium for filtering long-tailed image data. Background Technology

[0002] Long-tail data refers to a small portion of data that exists only in most categories, often representing anomalies, new occurrences, or potentially dangerous situations. This type of data is infrequent and is often referred to as a corner case or bad case in English. In the field of autonomous driving, the vehicle's intelligent perception module acquires information about the vehicle itself and its surrounding environment. The planning and decision-making module makes decisions based on the perception results and system-defined rules. The execution structure executes the decision-making module's actions regarding vehicle acceleration, deceleration, and steering. Currently, the intelligent perception modules of autonomous vehicles are mostly based on deep neural networks for visual perception. However, training deep neural network models often requires a large amount of sample image data. After the model has been trained to a certain stage, the technical indicators of visual perception do not significantly improve with the increase in data volume. One reason for this is that the number of long-tail sample images in the sample image data is too small.

[0003] Long-tailed data is crucial for training the generalization ability of deep neural network models. How to filter out long-tailed image data from massive amounts of data and give full play to its important role in the iteration of intelligent sensing systems is one of the key research directions at present. Summary of the Invention

[0004] In view of this, this application proposes a method, apparatus, and storage medium for filtering long-tail image data. This method filters image data collected from the real physical world, thereby increasing the amount of known types of long-tail image data and enabling it to play an important role in the iteration of intelligent sensing systems.

[0005] According to one aspect of this application, a method for filtering long-tail image data is provided, the method comprising:

[0006] Obtain a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category;

[0007] Image features are extracted from each sample image to obtain the sample image feature vector for each sample image;

[0008] Image features are extracted from each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image.

[0009] Based on a preset similarity algorithm, the feature similarity between each sample image feature vector and each candidate image feature vector is determined pairwise;

[0010] Based on the structural information of the images, determine the structural similarity between each sample image and each candidate image pairwise;

[0011] The feature similarity and the structural similarity are weighted to obtain the target similarity between each candidate image and each sample image;

[0012] The candidate images corresponding to the target similarity that meet the preset similarity conditions are determined as long-tail image data.

[0013] Further, after determining the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data, the method includes:

[0014] Obtain the long-tail category and candidate image of the sample image corresponding to the target similarity that meets the preset similarity conditions;

[0015] Based on the correspondence between the long-tail category of the sample image corresponding to the target similarity that meets the preset similarity condition and the candidate image, the long-tail category of the candidate image is determined.

[0016] Furthermore, the candidate image includes a preset category label, and the category label and the long-tail category satisfy a preset first association relationship;

[0017] Before obtaining the long-tailed sample image set and the pre-selected image set including the candidate images, the following steps are included:

[0018] Obtain an initial image set; the initial image set stores the collected candidate images, and each candidate image includes a category label obtained based on visual feature classification processing;

[0019] Obtain the long-tail categories corresponding to the sample image set;

[0020] Based on the first association between the category label and the long-tail category, candidate images associated with each long-tail category of the sample image set are selected from the initial image set;

[0021] The candidate images associated with each long-tail category of the sample image set are used to generate a pre-selected image set.

[0022] Preferably, each sample image includes a preset long-tail feature label; wherein, the long-tail feature label has a preset second association relationship with the long-tail category, and the long-tail feature label is associated with at least one of the long-tail categories;

[0023] After determining the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data, the method includes:

[0024] Obtain the candidate images corresponding to the target similarity that meet the preset similarity conditions;

[0025] Obtain the long-tail feature labels of the sample images corresponding to the target similarity that meets the preset similarity conditions;

[0026] Based on the second association between the long-tail feature label and the long-tail category, the long-tail category of the long-tail image data is determined.

[0027] Optionally, the step of extracting image features from each sample image to obtain a sample image feature vector for each sample image includes:

[0028] A preset neural network model is invoked to extract image features from each sample image, resulting in a sample image feature vector for each sample image; wherein...

[0029] The sample image feature vector includes mapping parameters for at least one of local image information, global image information, and context-based semantic information.

[0030] Optionally, the step of extracting image features from each candidate image in the pre-selected image set to obtain a candidate image feature vector for each candidate image includes:

[0031] A pre-defined neural network model is invoked to extract image features from each candidate image in the pre-selected image set, resulting in a candidate image feature vector for each candidate image; where,

[0032] The candidate image feature vector includes mapping parameters of at least one of local image information, global image information, and context-based semantic information.

[0033] Further, determining the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image based on a preset similarity algorithm includes:

[0034] The information entropy of the feature vector of each sample image and the feature vector of each candidate image are determined respectively, so as to obtain the first information entropy corresponding to each sample image and the second information entropy corresponding to each candidate image.

[0035] Based on the first information entropy and the second information entropy, similarity processing is performed to obtain the feature similarity between each sample image feature vector and each candidate image feature vector.

[0036] Further, the weighted processing of the feature similarity and the structural similarity to obtain the target similarity between each candidate image and each sample image includes:

[0037] The feature similarity and structural similarity are weighted and averaged based on a preset scaling factor to obtain the target similarity between each candidate image and each sample image.

[0038] According to another aspect of this application, a long-tail image data filtering device is provided, comprising:

[0039] The sample acquisition module is used to acquire a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category;

[0040] The feature vector extraction module is used to extract image features from each sample image to obtain the individual sample image feature vectors; and,

[0041] Image features are extracted from each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image.

[0042] The similarity calculation module is used to determine the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image, based on a preset similarity algorithm; and,

[0043] Based on the structural information of the images, determine the structural similarity between each sample image and each candidate image pairwise;

[0044] The similarity evaluation module is used to perform weighted processing on the feature similarity and the structural similarity to obtain the target similarity between each candidate image and each sample image;

[0045] The long-tail image data filtering module is used to determine the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data.

[0046] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the long-tail image data filtering method as described above.

[0047] The long-tail image data filtering method, apparatus, and storage medium provided in this application have the following beneficial effects:

[0048] This application extracts image features from each sample image to obtain a feature vector for each sample image; it also extracts image features from each candidate image in a pre-selected image set to obtain a candidate image feature vector for each candidate image; based on a preset similarity algorithm, it determines the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image; based on image structure information, it determines the pairwise structural similarity between each sample image and each candidate image; it then performs weighted processing on feature similarity and structural similarity to obtain the target similarity between each candidate image and each sample image; and finally, it identifies the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data. By weighting feature similarity and structural similarity, a target similarity that integrates image features and image quality-related structural information is obtained, and long-tail image data is determined based on this target similarity, thus improving the accuracy of the selection process. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0050] Figure 1 A schematic diagram illustrating an application environment provided in an embodiment of this application;

[0051] Figure 2 A flowchart illustrating a long-tail image data filtering method provided in an embodiment of this application;

[0052] Figure 3 A flowchart illustrating another long-tail image data filtering method provided in this application embodiment;

[0053] Figure 4 A schematic diagram illustrating the process of obtaining a pre-selected image set for candidate images provided in an embodiment of this application;

[0054] Figure 5 A flowchart illustrating the process of determining the category of long-tail image data is provided for an embodiment of this application;

[0055] Figure 6 A block diagram of a long-tail image data filtering device provided in an embodiment of this application;

[0056] Figure 7 A block diagram of another long-tail image data filtering device provided in the embodiments of this application;

[0057] Figure 8 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0060] Furthermore, to better illustrate the technical solutions of this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented even without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art are not described in detail in order to highlight the main points of this application.

[0061] This application applies a long-tail image filtering method to the field of autonomous driving, filtering out known long-tail category image data from massive amounts of collected autonomous driving image data. The long-tail category images filtered out by this application can enrich the long-tail category image data for driving-related scenarios. As training material for the intelligent perception module, these images can be used for the generalization training of deep neural network models, further improving the learning ability of the deep neural network models in the intelligent perception module, thereby better handling visual tasks in autonomous driving scenarios.

[0062] Currently, there are several common methods for collecting and processing long-tail data. One is to collect long-tail data in the real world; another is to generate or reconstruct long-tail data based on a simulation environment. Methods for collecting long-tail data in the real physical world are generally divided into online and offline methods. Online methods typically involve using in-vehicle devices or fixed-location edge networks. Limited by the amount of data transmitted and transmission efficiency, these methods are often used under specific conditions to collect data and upload it to a cloud computing center. Offline methods, on the other hand, typically take place in a cloud computing center, where long-tail data is aggregated and analyzed by analyzing large volumes of already collected and stored data and integrating overall information.

[0063] The proposed solution can serve as an offline method to filter image data of the real physical world collected from the cloud, identifying long-tail data to enrich the volume of known types of long-tail data and leverage its important role in the iteration of intelligent sensing systems. Simultaneously, it can more efficiently promote data flow and the continuous upgrading and iteration of application scenarios.

[0064] The following examples illustrate this in detail:

[0065] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application, such as... Figure 1 As shown, the application environment includes at least server 01 and client 02. In this embodiment, server 01 can be used to store image data of the real physical world collected by the autonomous vehicle based on the intelligent perception module. Server 01 can also deploy the algorithm model of the long-tail image data filtering method of this application to filter out long-tail image data from the image data. Specifically, server 01 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0066] In this embodiment, client 02 can be used to monitor image data collected by server 01 and control the implementation process of long-tail image data filtering. In one embodiment, server 01 receives images uploaded by autonomous vehicles and stores them centrally in an image library; server 01 also deploys the algorithm model of the long-tail image data filtering method of this application and reserves an interface for calling the algorithm model to provide image data filtering services. Client 02 can call the algorithm model through this interface to execute the long-tail image filtering method of this application to filter the required long-tail image data from server 01.

[0067] In another embodiment, both client 02 and server 01 can be used independently to execute the long-tail image data filtering method provided in this application. Client 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, and may also include software running on the physical device, such as applications, web pages, and mini-programs. The operating system running on the electronic device in this embodiment may include, but is not limited to, Android, iOS, Linux, and Windows systems.

[0068] In the embodiments described in this specification, the server 01 and the client 02 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0069] The following describes a method for filtering long-tail image data according to this application. Figure 2 This is a flowchart illustrating a long-tail image data filtering method provided in this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0070] S210, obtain a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category.

[0071] In the embodiments described in this specification, the long-tail sample image set includes sample images of at least one long-tail category. Each long-tail category includes at least one pre-collected sample image. Each sample image is associated with corresponding long-tail category information.

[0072] Specifically, long-tail categories can be based on the visual features of sample images, categorized according to the causes of the relevant visual features and their potential impact on visual tasks. For example, they can be based on visual features such as objects in the image, their outlines, colors, textures, and shapes. For instance, in a scene of a vehicle driving on a bridge on a rainy night, the captured sample image includes information such as raindrops reflecting weather conditions, the outline and texture of the bridge, and image brightness reflecting time information. Based on these image features, the long-tail category of the sample image could be category A (driving in inclement weather) or category B (driving in low visibility conditions in inclement weather), etc. In other scenarios, long-tail categories can also be based on image features such as damaged, obstructed, or abnormally exposed camera lenses, categorizing them into category C (abnormal camera viewpoint) or category D (abnormal equipment exposure).

[0073] S220, perform image feature extraction on each sample image to obtain the sample image feature vector of each sample image.

[0074] In the embodiments of this specification, the feature vector can be obtained based on a trained image feature extraction module. The image feature extraction module can extract visual features from the image. Here, feature extraction includes both general features for images and features extracted differently for different scenes. Through feature extraction, a high-dimensional feature vector is obtained, and these features are then fused.

[0075] Specifically, feature vectors can be obtained based on a trained convolutional neural network model. Each sample image is input into the convolutional neural network model for feature extraction, resulting in its own sample image feature vector. Similarly, each candidate image is input into the convolutional neural network model for feature extraction, resulting in its own candidate image feature vector. For example, it can be an image visual feature extraction model based on a VGG network structure. Through training with pre-samples, it extracts multi-dimensional image features from the input image and, based on the learned mapping relationship between image features at different levels and vector parameters, fuses the relevant features and outputs the corresponding high-dimensional feature vector. For example, the feature vector can reflect at least one of the following image information: local image information, global image information, and context-based semantic information.

[0076] In one embodiment, step S220 may include calling a preset neural network model to extract image features from each sample image to obtain a sample image feature vector for each sample image; wherein the sample image feature vector includes a mapping parameter of at least one of local image information, global image information, and context-based semantic information.

[0077] S230, perform image feature extraction on each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image.

[0078] In one embodiment, step S220 may include calling a preset neural network model to extract image features from each candidate image in the pre-selected image set to obtain a candidate image feature vector for each candidate image; wherein the candidate image feature vector includes a mapping parameter of at least one of local image information, global image information, and context-based semantic information.

[0079] S240, based on a preset similarity algorithm, determines the feature similarity between each sample image feature vector and each candidate image feature vector pairwise.

[0080] In this embodiment, the preset similarity algorithm can be a similarity algorithm that measures the distance between different feature vectors based on information entropy, or a similarity algorithm that measures the distance between different feature vectors based on cosine similarity can be selected as needed. Those skilled in the art can choose an appropriate distance metric as needed.

[0081] Furthermore, the aforementioned method of determining the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image, based on a preset similarity algorithm, may include:

[0082] S2401, determine the information entropy of the feature vector of each sample image and the feature vector of each candidate image respectively, and obtain the first information entropy corresponding to each sample image and the second information entropy corresponding to each candidate image respectively;

[0083] S2402, similarity processing is performed based on the first information entropy and the second information entropy to obtain the feature similarity between each sample image feature vector and each candidate image feature vector.

[0084] Specifically, the probability of each gray level of the features in the image is statistically determined, and then the information entropy of the image features is calculated using formula (1).

[0085] Formula (1):

[0086]

[0087] The calculation process is explained in detail below. Based on the feature vector of the sample image, the first information entropy H(X1) of each sample image is obtained by the above formula (1). Based on the feature vector of the candidate image, the second information entropy H(X2) of each candidate image is obtained by the above formula. The difference between H(X1) and H(X2) is calculated to obtain the feature similarity between each sample image feature vector and each candidate image feature vector.

[0088] S250, based on the structural information of the images, determines the structural similarity between each sample image and each candidate image pairwise.

[0089] In the embodiments of this specification, the structural information of the image represents the brightness and contrast related to the object structure. Structural similarity (SSIM) is a full-reference image quality evaluation metric that measures image similarity between two images x and y from three dimensions: luminance l(x,y), contrast c(x,y), and structure s(x,y). The following provides an algorithm for obtaining the structural similarity (SSIM) between two images:

[0090] Structural similarity calculation, formula (2):

[0091] SSIM(X,Y)=l(X,Y)·c(X,Y)·s(X,Y)

[0092] The SSIM value ranges from [0,1]. The larger the value, the smaller the image distortion, i.e., the more similar the image.

[0093] The calculation of the brightness dimension is given by formula (3):

[0094]

[0095] Where μx and μy represent the mean values ​​of images X and Y, respectively.

[0096] The contrast dimension is calculated using formula (4):

[0097]

[0098] The calculation of structural dimensions is given by formula (5):

[0099]

[0100] Where σ X σ Y σ represents the variance of images X and Y, respectively; XY This represents the covariance of images X and Y. C1, C2, and C3 are constants to avoid the denominator being zero.

[0101] In practical applications, a sliding window can be used to divide the image into blocks. Assuming the total number of blocks is N, the structural similarity of the corresponding blocks is calculated, and then the average value is used as a measure of the image's structural similarity. Specifically,

[0102] Calculate the average structural similarity using formula (6):

[0103]

[0104] Where N is the total number of blocks in the image.

[0105] S260, weighted processing of feature similarity and structural similarity is performed to obtain the target similarity between each candidate image and each sample image pairwise.

[0106] In one specific embodiment, step S260 includes:

[0107] Based on a preset scaling factor, a weighted average of feature similarity and structural similarity is calculated to obtain the target similarity between each candidate image and each sample image.

[0108] Target similarity can be obtained using the following formula (7):

[0109]

[0110] Similarity represents the similarity between the targets;

[0111] λ is a set proportionality coefficient, and its value ranges from (0,1). In practice, the value is determined by combining the project and experience.

[0112] Feature_similarity is the result of feature similarity calculation.

[0113] S270, the candidate images corresponding to the target similarity that meet the preset similarity conditions are determined as long-tail image data.

[0114] The similarity condition can be a set threshold. When the target similarity value exceeds this threshold, the similarity condition is considered to be met, and the candidate image corresponding to the target similarity is determined as long-tail image data.

[0115] In some other embodiments, the similarity condition can also be a set threshold range. When the value of the target similarity meets the set threshold range, the similarity condition is considered to be met, and the candidate image corresponding to the target similarity is determined as long-tail image data.

[0116] In one embodiment of this application, such as Figure 3 As shown, the method may further include, after S270 above:

[0117] S280, respectively obtain the long-tail category and candidate image of the sample image corresponding to the target similarity that meets the preset similarity conditions;

[0118] S290, Based on the correspondence between the long-tail category of the sample image corresponding to the target similarity that meets the preset similarity conditions and the candidate image, determine the long-tail category of the candidate image.

[0119] In this embodiment, after identifying long-tail image data through target similarity, the long-tail image data is further classified and managed through steps S280 and S290. In practical applications, the selected long-tail image data can be used as training samples for intelligent perception models. The classification and storage based on long-tail categories in this embodiment can provide targeted training samples for subsequent model training, improving model training efficiency.

[0120] In other embodiments, such as Figure 4 As shown, obtaining a pre-selected image set for candidate images may include:

[0121] S211, Obtain an initial image set; the initial image set stores the collected candidate images, and each candidate image includes a category label obtained based on visual feature classification processing, and the category label and the long-tail category satisfy a preset first association relationship.

[0122] Specifically, category labels can be based on visual features of spatiotemporal and lighting information on the collected candidate images. For example, they can include different category labels that reflect road conditions, weather conditions, and time information.

[0123] In one embodiment, category labels can be assigned by processing each candidate image in an initial image set using a pre-trained convolutional neural network model. This convolutional neural network model, based on its training capabilities, is able to identify basic image features and generate category labels.

[0124] In practical applications, the initial image set can be stored on server 01, and the pre-trained convolutional neural network model can also be set up on server 01. After acquiring the images uploaded by the autonomous vehicle, they are first processed by the convolutional neural network model to obtain the assigned category labels.

[0125] S212, obtain the long-tail categories corresponding to the sample image set.

[0126] S213, based on the first association between the category label and the long-tail category, select candidate images from the initial image set that are associated with each long-tail category of the sample image set.

[0127] Long-tail categories can be based on the visual features of sample images, categorized according to the causes of related visual features and their potential impact on visual tasks. Based on this, a preliminary association between long-tail categories and category labels can be established using road conditions, weather conditions, and time information that may be associated with long-tail categories.

[0128] S214 generates a pre-selected image set from the candidate images associated with each long-tail category of the sample image set.

[0129] In this embodiment, after the candidate images in the initial image set are uploaded to server O1, long-tail categories and classification labels are assigned based on visual image features, and a first association relationship between the classification labels and long-tail categories is pre-established, thereby establishing structured data of candidate images in the initial image library. A pre-selected image set is then generated based on this structured data, excluding obviously irrelevant candidate images from the initial image library. The first association relationship between the classification labels and long-tail categories reduces the data processing workload for determining long-tail image data from the candidate images in the pre-selected image set. Furthermore, this application uses a weighted processing method based on feature similarity and structural similarity for long-tail image data filtering, allowing for adjustment of the filtering accuracy based on the weighting coefficients. Therefore, the method of this application helps improve the efficiency and accuracy of long-tail image data filtering.

[0130] In one embodiment of this application, each sample image includes a preset long-tail feature label. Specifically, such as... Figure 5 As shown, after step S270, the method includes:

[0131] S510, Obtain the candidate image corresponding to the target similarity that meets the preset similarity conditions;

[0132] S520, Obtain the long-tail feature labels of the sample images corresponding to the target similarity that meet the preset similarity conditions;

[0133] S530 determines the long-tail category of long-tail image data based on the second association between long-tail feature labels and long-tail categories.

[0134] By using the above method, a pre-established second association between long-tail feature labels and long-tail categories can be further used to manage the selected long-tail image data in a two-level classification manner.

[0135] Specifically, in the mapping of the second association, the long-tail feature label is associated with at least one long-tail category. This allows for a more comprehensive evaluation of the visual image features in the candidate images. For example, when a candidate image contains information related to multiple long-tail categories, it can be fully evaluated based on the two-level classification management described above to determine whether it belongs to the corresponding multiple long-tail categories, thus enabling a more comprehensive evaluation of the candidate images.

[0136] The long-tail image data filtering method provided in this application can be trained into a model and deployed in Service 02 to filter long-tail image data from candidate images collected by stored autonomous vehicles. Pre-screening using structured data constructed from candidate images effectively improves filtering efficiency. Weighted fusion processing of structural and feature similarity further enhances filtering efficiency, effectively addressing the issue of long-tail data collection in massive datasets. Constructing the association between long-tail feature labels and long-tail categories of sample images enables a more comprehensive feature similarity analysis of candidate images, thereby improving filtering effectiveness to some extent. Furthermore, the long-tail image data filtering method provided in this application can also be deployed in other electronic devices to filter long-tail category images from image data stored locally or on a server. This embodiment provides an illustrative example of the implementation of the long-tail image data filtering method in conjunction with the application environment and does not constitute a specific limitation on the method. Those skilled in the art can make adaptive adjustments based on the same or different application environments.

[0137] This application embodiment further provides a long-tail image data filtering device, including:

[0138] The sample acquisition module 110 is used to acquire a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category.

[0139] Feature vector extraction module 120 is used to extract image features from each sample image to obtain the sample image feature vector for each sample image; and,

[0140] Image features are extracted from each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image.

[0141] The similarity calculation module 130 is used to determine the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image based on a preset similarity algorithm; and,

[0142] Based on the structural information of the images, determine the structural similarity between each sample image and each candidate image pairwise;

[0143] The similarity evaluation module 140 is used to perform weighted processing on feature similarity and structural similarity to obtain the target similarity between each candidate image and each sample image;

[0144] The long-tail image data filtering module 150 is used to determine the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data.

[0145] In one embodiment, the long-tail image data filtering device further includes:

[0146] The preprocessing module 100 is used to acquire an initial image set; wherein the initial image set stores the acquired candidate images, each candidate image including a category label obtained from visual feature classification processing, the candidate images including a preset category label, the category label and the long-tail category satisfying a preset first association relationship; acquire each long-tail category corresponding to the sample image set; based on the first association relationship between the category label and the long-tail category, select candidate images associated with each long-tail category of the sample image set from the initial image set; generate a preselected image set from the candidate images associated with each long-tail category of the sample image set.

[0147] In one embodiment, the long-tail image screening device further includes:

[0148] The long-tail image data classification module 160 is used to obtain the long-tail category of the sample image corresponding to the target similarity that meets the preset similarity conditions and the candidate image respectively; based on the correspondence between the long-tail category of the sample image corresponding to the target similarity that meets the preset similarity conditions and the candidate image, the long-tail category of the candidate image is determined.

[0149] In a preferred embodiment, the long-tail image data classification module 160 is configured to: acquire candidate images corresponding to target similarities that meet preset similarity conditions; acquire long-tail feature labels of sample images corresponding to target similarities that meet preset similarity conditions; and determine the long-tail category of the long-tail image data based on a second association relationship between the long-tail feature labels and long-tail categories. Each sample image includes a preset long-tail feature label, the long-tail feature label has a preset second association relationship with the long-tail category, and the long-tail feature label is associated with at least one long-tail category.

[0150] In one embodiment, the feature vector extraction module 120 is used to call a preset neural network model to extract image features from each sample image to obtain a sample image feature vector for each sample image; wherein, the sample image feature vector includes a mapping parameter of at least one of local image information, global image information and context-based semantic information.

[0151] In one embodiment, the feature vector extraction module 120 is used to call a preset neural network model to extract image features from each candidate image in the pre-selected image set, thereby obtaining a candidate image feature vector for each candidate image; wherein, the candidate image feature vector includes a mapping parameter of at least one of local image information, global image information, and context-based semantic information.

[0152] In one embodiment, the similarity calculation module 130 is used to determine the information entropy of the feature vectors of each sample image and the feature vectors of each candidate image, respectively, to obtain the first information entropy corresponding to each sample image and the second information entropy corresponding to each candidate image; and to perform similarity processing based on the first information entropy and the second information entropy to obtain the feature similarity between each pair of feature vectors of each sample image and feature vectors of each candidate image.

[0153] In one embodiment, the similarity evaluation module 140 is used to perform a weighted average of feature similarity and structural similarity based on a preset ratio coefficient to obtain the target similarity between each candidate image and each sample image.

[0154] The vehicle tracking data processing method of this application can be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0155] This application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the long-tail image data filtering method provided in the above-described control method embodiments. Figure 8 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. For example... Figure 8 As shown, the electronic device 800 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 810 (CPUs 810 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 830 for storing data, and one or more storage media 820 (e.g., one or more mass storage devices) for storing application programs 823 or data 822. The memory 830 and storage media 820 may be temporary or persistent storage. The program stored in the storage media 820 may include one or more modules, each module may include a series of instruction operations on a server. Furthermore, the CPU 810 may be configured to communicate with the storage media 820 and execute the series of instruction operations in the storage media 820 on the electronic device 800. Electronic device 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, etc.

[0156] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 800. In one example, the input / output interface 840 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 840 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0157] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 800 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0158] Memory is used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in memory. Memory can primarily include a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functions, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory may also include a memory controller to provide the processor with access to the memory.

[0159] The message processing method provided in this application embodiment can be executed in a mobile terminal, computer terminal, server or similar computing device.

[0160] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing the message processing method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the above-described long-tail image data filtering method.

[0161] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0162] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0163] The embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0164] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0165] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0166] The above are merely preferred embodiments of this application and are not intended to limit the application. It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of this application is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0167] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for filtering long-tailed image data, characterized in that, include: Obtain a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category; Image features are extracted from each sample image to obtain the sample image feature vector for each sample image; Image features are extracted from each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image. Based on a preset similarity algorithm, the feature similarity between each sample image feature vector and each candidate image feature vector is determined pairwise; Based on image structure information, determine the structural similarity between each sample image and each candidate image in pairs; The feature similarity and the structural similarity are weighted to obtain the target similarity between each candidate image and each sample image; The candidate images corresponding to the target similarity that meet the preset similarity conditions are determined as long-tail image data; Before obtaining the long-tailed sample image set and the pre-selected image set including the candidate images, the following steps are included: An initial image set is obtained; the initial image set stores the collected candidate images, and each candidate image includes a category label obtained based on visual feature classification processing; the category label and the long-tail category satisfy a preset first association relationship; Obtain the long-tail categories corresponding to the sample image set; Based on the first association between the category label and the long-tail category, candidate images associated with each long-tail category of the sample image set are selected from the initial image set; The candidate images associated with each long-tail category of the sample image set are used to generate a pre-selected image set.

2. The method according to claim 1, characterized in that, After determining the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data, the method includes: Obtain the long-tail category and candidate image of the sample image corresponding to the target similarity that meets the preset similarity conditions; Based on the correspondence between the long-tail category of the sample image corresponding to the target similarity that meets the preset similarity condition and the candidate image, the long-tail category of the candidate image is determined.

3. The method according to claim 1, characterized in that, Each sample image includes a preset long-tail feature label; wherein, the long-tail feature label has a preset second association relationship with the long-tail category, and the long-tail feature label is associated with at least one of the long-tail categories; After determining the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data, the method includes: Obtain the candidate images corresponding to the target similarity that meet the preset similarity conditions; Obtain the long-tail feature labels of the sample images corresponding to the target similarity that meet the preset similarity conditions; Based on the second association between the long-tail feature label and the long-tail category, the long-tail category of the long-tail image data is determined.

4. The method according to claim 1, characterized in that, The step of extracting image features from each sample image to obtain the sample image feature vector for each sample image includes: A preset neural network model is invoked to extract image features from each sample image, resulting in a sample image feature vector for each sample image; wherein... The sample image feature vector includes mapping parameters for at least one of local image information, global image information, and context-based semantic information.

5. The method according to claim 4, characterized in that, The step of extracting image features from each candidate image in the pre-selected image set to obtain a candidate image feature vector for each candidate image includes: A pre-defined neural network model is invoked to extract image features from each candidate image in the pre-selected image set, resulting in a candidate image feature vector for each candidate image; where, The candidate image feature vector includes mapping parameters of at least one of local image information, global image information, and context-based semantic information.

6. The method according to claim 1, characterized in that, The step of determining the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image based on a preset similarity algorithm includes: The information entropy of the feature vector of each sample image and the feature vector of each candidate image are determined respectively, so as to obtain the first information entropy corresponding to each sample image and the second information entropy corresponding to each candidate image. Based on the first information entropy and the second information entropy, similarity processing is performed to obtain the feature similarity between each sample image feature vector and each candidate image feature vector.

7. The method according to claim 1, characterized in that, The weighted processing of the feature similarity and the structural similarity to obtain the target similarity between each candidate image and each sample image includes: The feature similarity and structural similarity are weighted and averaged based on a preset scaling factor to obtain the target similarity between each candidate image and each sample image.

8. A long-tail image data filtering device, characterized in that, include: The sample acquisition module is used to acquire a long-tail sample image set and a pre-selected image set including candidate images, wherein the long-tail sample image set includes sample images of at least one long-tail category; The feature vector extraction module is used to extract image features from each sample image to obtain the individual sample image feature vectors; and, Image features are extracted from each candidate image in the pre-selected image set to obtain the candidate image feature vector for each candidate image. The similarity calculation module is used to determine the pairwise feature similarity between the feature vectors of each sample image and the feature vectors of each candidate image, based on a preset similarity algorithm; and, Based on the structural information of the images, determine the structural similarity between each sample image and each candidate image pairwise; The similarity evaluation module is used to perform weighted processing on the feature similarity and the structural similarity to obtain the target similarity between each candidate image and each sample image; The long-tail image data filtering module is used to determine the candidate images corresponding to the target similarity that meet the preset similarity conditions as long-tail image data; A preprocessing module is used to acquire an initial image set before acquiring a long-tail sample image set and a pre-selected image set including candidate images; the initial image set stores the acquired candidate images, each candidate image including a category label obtained based on visual feature classification processing; the category label and the long-tail category satisfy a preset first association relationship; acquire each long-tail category corresponding to the sample image set; based on the first association relationship between the category label and the long-tail category, filter candidate images associated with each long-tail category of the sample image set from the initial image set; and generate a pre-selected image set from the candidate images associated with each long-tail category of the sample image set.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement claim 1. The long-tail image data filtering method described in any one of the 7.

Images