Data screening method, device and equipment and storage medium

Abstract

Embodiments of the present application provide a data screening method, device and equipment and a storage medium to screen data. The source data and the target data are decoupled to determine the features of the source data and the target data; the screening condition of the target data is determined according to the features of the source data and the target data; and the target data is screened based on the screening condition to determine the screened target data. The source data and the target data are decoupled to obtain the features of the source data and the target data, and then the target data is screened based on the source data according to the corresponding features of the source data, so that the target data more suitable for subsequent processing requirements is obtained. The screened target data can retain the different features based on the source data, and provide more diverse data basis for subsequent processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data filtering method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] During data processing, various reasons necessitate the generation of new data. This generated data is based on the features of the original data. For example, new images may be generated from some original images, or new image data may be used to expand the training data of image-related models.

[0003] However, some generated data may have various problems, making it unsuitable for subsequent processing. For example, the features in the data may differ significantly from the processing requirements, affecting the processing quality. Summary of the Invention

[0004] This application provides a data filtering method for filtering data.

[0005] Accordingly, embodiments of this application also provide a data filtering device, an electronic device, and a storage medium to ensure the implementation and application of the above method.

[0006] To address the aforementioned issues, this application discloses a data filtering method, comprising: decoupling source data and target data by features to determine the features of the source data and the target data; determining filtering conditions for the target data based on the features of the source data and the target data; and filtering the target data based on the filtering conditions to determine the filtered target data.

[0007] This application also discloses a data filtering method, comprising: decoupling the features of source image data and target image data to determine the features of the source image data and the target image data; determining filtering conditions for the target image data based on the features of the source image data and the target image data; and filtering the target image data based on the filtering conditions to determine the filtered target image data.

[0008] This application also discloses a data filtering method, including: acquiring source data and target data corresponding to an unmanned vending system; performing feature decoupling on the source data and target data to determine the features of the source data and the target data; determining filtering conditions for the target data based on the features of the source data and the target data; and filtering the target data based on the filtering conditions to determine the filtered target data.

[0009] This application also discloses a data processing apparatus, characterized in that the apparatus includes: a feature decoupling module for decoupling the features of source data and target data to determine the features of the source data and the features of the target data; a condition determination module for determining the filtering conditions of the target data based on the features of the source data and the features of the target data; and a filtering module for filtering the target data based on the filtering conditions to determine the filtered target data.

[0010] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in this application.

[0011] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more methods as described in this application.

[0012] Compared with the prior art, the embodiments of this application have the following advantages:

[0013] In this embodiment of the application, the source data and target data are decoupled by feature to obtain the features of the source data and the features of the target data. Then, based on the corresponding features of the source data, the target data is filtered to obtain target data that better meets the needs of subsequent processing. The filtered target data can retain the different features on the basis of the source data, providing a more diverse data foundation for subsequent processing. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating a data filtering example according to an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of the interaction between a user terminal and a server according to an embodiment of this application;

[0016] Figure 3 This is a schematic diagram illustrating another data filtering example according to an embodiment of this application;

[0017] Figure 4 This is a flowchart illustrating the steps of an embodiment of a data filtering method according to this application;

[0018] Figure 5 This is a flowchart illustrating the steps of an embodiment of an image data filtering method according to this application;

[0019] Figure 6 This is a flowchart illustrating the steps of another embodiment of the image data filtering method of this application;

[0020] Figure 7This is a flowchart illustrating the steps of another embodiment of the data filtering method of this application;

[0021] Figure 8 This is a flowchart illustrating the steps of a data filtering service method according to an embodiment of this application;

[0022] Figure 9 This is a structural block diagram of an embodiment of a data filtering device according to this application;

[0023] Figure 10 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] This application can be applied to the field of data processing. In data processing, it is sometimes necessary to generate data based on existing raw data, thereby performing data augmentation and subsequent training and processing based on the raw and generated data. For example, in the Person Re-Identification (ReID) scenario, images can be generated based on the original image for data augmentation. ReID, also known as pedestrian re-identification, is a technique that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. In cross-domain ReID scenarios, pedestrian images belonging to image domain A can be transferred to image domain B while maintaining the consistency of the pedestrian ID. For example, an image of pedestrian A taken outdoors during the day can be processed to generate an image of the same person A indoors in a large shopping mall, or an image of pedestrian A taken by camera 1 can be transferred to generate an image of A taken by camera 2, and so on. Generating data based on raw data can be used as a means of data augmentation. The generated data can be mixed with the raw data for subsequent data processing. However, some of the generated data may not be suitable for subsequent processing, so data screening is required to improve the quality of subsequent processing.

[0026] This application's embodiments can be applied to cross-data-domain scenarios. During the migration of source data from a source data domain to a target data domain, target data in the target data domain can also be generated based on the source data. A data domain can be understood as a dataset composed of data with certain characteristics, where the data share the same features. For example, for data of the same type, such as image data, it can be determined whether the data belongs to the same data domain based on time period, acquisition device, or region. Similarly, for multiple types of data, it can be determined whether they belong to the same data domain based on the data source, such as data from the same type of website belonging to the same data domain. Specifically, different data domains can be distinguished based on the application scenario.

[0027] Taking cross-domain pedestrian re-identification scenarios as an example, different data domains can be divided in various ways, including different times and different geographical locations. For instance, video data collected by the same camera in different seasons belongs to different data domains; similarly, video data collected in different cities or other geographical locations can also belong to different data domains; and indoor (e.g., shopping malls, office buildings) and outdoor (e.g., streets) environments can also belong to different data domains. Pedestrian re-identification refers to the ability to identify pedestrians (or users) from image, video, and other data, and to determine whether the pedestrian is the same person in different images or videos.

[0028] Taking website data analysis as an example, data from e-commerce websites, social networking sites, lifestyle sharing websites, and video websites can be from different data domains. In one scenario, user reviews of certain products from social networking sites and lifestyle sharing websites can be transferred to e-commerce scenarios. Furthermore, target review data can be generated based on this source review data, such as combining product objects with different functions, prices, and captured images to form target review data.

[0029] It can also be applied to other scenarios, such as logistics and finance. The specific data required for each scenario can be determined, and the data from the e-commerce platform can be filtered based on the source data. Subsequent processing can then be performed, such as unsupervised learning training or other data analysis processes.

[0030] The source data and target data in this application embodiment can be various types of data, including various types of data such as images, text, and audio. Target data can be generated based on the source data for each type of data; for example, a target image can be generated based on a source image, target text can be generated based on source text, and target audio data can be generated based on source audio data.

[0031] Reference Figure 1 The diagram illustrates a data filtering example according to an embodiment of this application.

[0032] Step 102: Decouple the source data and target data by feature decoupling to determine the features of the source data and the target data.

[0033] In this embodiment, target data can be generated based on source data. Of course, target data can also be generated without being based on source data, provided the source data meets the scenario requirements. Therefore, using source data as a benchmark to filter target data improves processing efficiency by filtering data before subsequent processing. Taking the generation of target data based on source data as an example, the features of the source data can be analyzed, and then the features can be split and combined to form the target data. In a cross-domain scenario, source data can be migrated from the source data domain (or first data domain) to the target data domain (or second data domain). Correspondingly, target data, also known as augmented data, can be generated based on the source data in the target data domain. This involves feature analysis of the source data to extract the required target features and determine the domain features corresponding to the target data domain. The target features and domain features are then combined to obtain the target data corresponding to the target data domain. In other scenarios, source data can be processed using models to extract and combine its features to generate target data. Taking images as an example, features such as background features, pose features, shape features, and texture features can be extracted from various source images. These features are then combined to obtain the corresponding target image. For source data containing a target object, other features can be combined while retaining the target object to construct the target data. In a ReID scenario, where the target object is a user (pedestrian), and the source images correspond to the user's identifier, features can be extracted and combined from the source images corresponding to that identifier. For example, the color and style of a user's clothing in one source image can be combined with the user's features in another source image. Similarly, scene features from different source images can be combined to obtain the target image. In cross-domain ReID scenarios, features from a new data domain can be combined with user features from the source images. For instance, in a scene transitioning from indoors to outdoors, features of the outdoor scene, such as different lighting conditions and seasons, can be combined with user features to obtain the target image.

[0034] In one optional embodiment, the source data is input into a data generation model, which outputs corresponding target data. The data generation model extracts features from the source data and combines these features to obtain the target data. The data generation model can be various neural network models, deep learning models, etc. Therefore, multiple source data can be input into the data generation model, which performs feature extraction, combination, and other processing to output at least one generated target data. For example, the data generation model can be a Generative Adversarial Network (GAN) model, a Variational Auto-encoder (VAE) model, a Glow model, etc.

[0035] In one example, the data generation model is a GAN model, a deep learning model comprising at least two modules: a generative model and a discriminative model. The discriminative model extracts features from the data and identifies the target object corresponding to those features. The generative model generates new data containing the target object based on a set of target object data. In this adversarial generative network, the two models can learn from each other through a game to produce output. For example, when source data is input into the adversarial generative network model, the model can perform a series of processes, including feature extraction, target object identification, and generating target data containing the target object, before outputting the target data.

[0036] In one optional embodiment, in a cross-domain processing scenario, data from a first data domain is migrated to a second data domain. The source data includes image data from the first data domain; the target data includes image data from the second data domain. The step of inputting the source data into a data generation model and outputting the corresponding target data includes: inputting the image data from the first data domain into the data generation model and outputting the image data from the second data domain. This data generation model can be used to generate data in a cross-domain scenario. Image data from the first data domain can be input into the data generation model, and through processing by the model, image data from the second data domain can be obtained as the target data. The data generation model can be trained on data corresponding to the cross-domain scenario. For example, in a cross-domain ReID scenario, pedestrian images belonging to image domain A can be used as source data and migrated to image domain B while maintaining the consistency of the pedestrian ID. This pedestrian is the target object, and the resulting pedestrian image in data domain B can be used as the target data. Multiple source data can be input into a generative adversarial network (GAN) model, and through processing by the GAN model, one or more target data can also be obtained. For example, taking text data as an example, the source text data can be input into the GAN model for semantic segmentation, extraction, and combination, and the target text data can be output.

[0037] In some optional embodiments of this application, the source data described above can be obtained from the user terminal; in other scenarios, the target data can also be provided by the user terminal. For example... Figure 2 The example interface shown illustrates a first page for uploading data. Users can log in and display this first page via various devices. The first page includes an upload control, such as... Figure 2The upload controls can include source data upload controls and target data upload controls. Users can upload source data using the source data upload control, for example, by selecting or entering the address of the source data, and then click the "Upload" control to upload it. After the source data is uploaded to the server, the server can also generate target data based on the source data. If the user already has target data, they can also select or enter the address of the source data using the target data upload control, and then upload the target data.

[0038] For example, in a Software-as-a-Service (SaaS) scenario, data filtering and processing can be encapsulated as a data filtering service, providing an external service interface. Users can subscribe to this data filtering service to obtain service results and thus use the corresponding service. Correspondingly, a service page can be provided based on this service, such as... Figure 2 The page shown allows users to access the service. After selecting source data, target data, and other data, users can upload data using the upload control and call the service interface to filter the data. The filtered results are then fed back to the user and displayed on the second page.

[0039] After obtaining the source data and target data, feature decoupling can be performed on the source data and target data respectively to determine the features of the source data and the target data. Feature decoupling can be understood as feature separation, that is, performing feature analysis on the data and then extracting various types of features. Various feature extraction methods can be used to perform feature decoupling processing on the source data and target data. In the embodiments of this application, the required features can be decoupled from the source data and target data respectively, and a feature space can be constructed based on the extracted features.

[0040] In one optional embodiment of this application, the step of decoupling the source data and target data to determine the features of the source data and the target data includes: obtaining a source data set and a target data set; decoupling the features of the source data in the source data set to determine at least one feature of the source data; and decoupling the features of the target data in the target data set to determine at least one feature of the target data. Specifically, the source data in the source data set can be decoupled to obtain corresponding first and second features; and the target data in the target data set can be decoupled to obtain corresponding first and second features.

[0041] The extracted features can be further categorized into different types based on class. These features include a first feature and a second feature. The first feature is a consistency feature, which is a feature strongly correlated with the target object in the data. It can be understood as a feature that can be identified as the same target object in different scenarios. For example, in a ReID scenario, a pedestrian's clothing, gender, shoes, facial features, and posture features can be used as consistency features. The second feature is a diversity feature, which can be understood as a feature that enhances the diversity of the data. For example, environmental features, data source features, and other features independent of the target object in the data. For example, in a ReID scenario, different lighting, camera conditions, backgrounds, such as different geographical features indoors and outdoors, different time features during the day and night, illumination features, and posture features of different cameras, etc.

[0042] Feature decoupling can be achieved in various ways. For example, after generating target data based on GAN, the features of the source and target data can be decoupled based on the Inception Score (IS). Similarly, target data generated by other methods can also be decoupled based on the IS metric to obtain features. Unlike the training process where consistency features are obtained based on the IS metric, this embodiment of the application can obtain both consistency and diversity features through decoupling based on the IS metric during the data filtering stage. The source and target data can be input into the Inception model, which acts like a classifier, extracting multi-dimensional label vectors from the data (source and target data). Each dimension of the vector represents the probability that the input sample belongs to a certain category, thus allowing the acquisition of consistency and diversity features based on the label vectors. In other examples, models with random initialization or those trained using ImageNet can also be used to decouple the source and target data, separating the desired features.

[0043] Step 104: Determine the filtering conditions for the target data based on the characteristics of the source data and the characteristics of the target data.

[0044] Step 106: Filter the target data based on the filtering conditions to determine the target data to be filtered.

[0045] After obtaining the features of the source data and the target data through feature decoupling, the features of the source data can be used as a benchmark to filter the features of the target data. Specifically, the distribution of features can be analyzed based on the features of the source data, and then corresponding filtering conditions can be determined. The target data can then be filtered based on these filtering conditions.

[0046] In one optional embodiment, the target data is filtered based on the characteristics of the source data to determine the target data to be filtered. This includes: determining filtering conditions based on the characteristics of the source data; and filtering the target data based on the filtering conditions to determine the target data to be filtered. The characteristics of the source data can be analyzed, for example, to determine the distribution of the characteristics of the source data, or to cluster the source data to determine the center of the features, etc., thereby determining corresponding filtering conditions based on the characteristics of the source data. Different types of features can correspond to different filtering conditions. Then, the characteristics of the target data can be filtered based on the filtering conditions to determine the filtered target data. For example, based on the feature analysis of the source data, the feature center can be determined, and then a distance can be set as the radius. A circular feature region is formed based on the feature center and the radius, and filtering conditions are determined based on this feature region, etc.

[0047] In a further optional embodiment, determining the filtering conditions based on the features of the source data includes: constructing a feature space based on the features of the source data and the features of the target data; analyzing the feature distribution information of the corresponding features of the source data in the feature space; and determining the filtering conditions based on the feature distribution information.

[0048] After decoupling and obtaining the features of the source data and the target data, a corresponding feature space can be determined based on features of the same type, within which each feature is distributed. The feature space can be understood as the space in which all feature vectors exist.

[0049] The aforementioned features include a first feature and a second feature, and correspondingly, a first feature space corresponding to the first feature and a second feature space corresponding to the second feature can be determined. Specifically, constructing the feature space based on the features of the source data and the features of the target data includes: constructing a first feature space based on the first feature of the source data and the first feature of the target data; and constructing a second feature space based on the second feature of the source data and the second feature of the target data. The first feature space may be the space containing the consistency feature, and the second feature space may be the space containing the diversity feature.

[0050] Then, statistical analysis can be performed on the corresponding features of the source data in the feature space to determine the feature distribution information of the corresponding features of the source data. Based on this feature distribution information, the screening criteria for the target data can then be determined. Here, feature distribution information refers to information describing the distribution of the corresponding features of the source data.

[0051] In one optional embodiment, analyzing the feature distribution information of the corresponding features of the source data in the feature space includes: analyzing the features of the source data in the feature space to determine feature centers and density information. The feature centers can be determined by analyzing the features of the source data in the feature space, wherein the mean of each feature of each source data can be calculated to determine the feature centers, or cluster analysis can be performed on each feature of each source data to determine the feature centers.

[0052] After determining the feature center, the density information of the corresponding features in the source data can also be determined. This density information characterizes the tightness of the feature distribution. In some examples, the density information can be represented based on the distance to the feature center. Therefore, the distance to the center point can be adaptively determined as the density information based on the distribution of features in the source data in the feature space. This density information and the feature center can define a feature region that covers most of the corresponding features in the source data, such as 80%-90% or more. The feature region can be a region with a fixed shape, such as a circular region or a rectangular region, or it can be adaptively adjusted based on the distribution of the corresponding features in the source data. In other scenarios, some distances can be pre-set to characterize the density information; this embodiment does not impose any limitations on this.

[0053] The step of determining the filtering conditions based on the feature distribution information includes: determining the feature region based on the feature center and density information; and determining the filtering conditions based on the feature region. After determining the feature center and density information, a feature region for filtering the features of the target data in the feature space can be determined. This feature region can be determined based on the area formed by the feature center and density information. Taking a circular region as an example, the feature center can be used as the center, and the radius can be determined based on the density information. The distance representing the density information can be used as the radius. A circular region can be determined based on the circle and the radius, which serves as the feature region. Corresponding filtering conditions can be generated based on this feature region. For example, the filtering conditions may be that the features of the target data are within the feature region, and / or that the features of the target data are outside the feature region. In other examples, if the feature region is rectangular, the length, width, and other information of the rectangle can be determined based on the density information, and the corresponding rectangular region can be determined with the feature center as the center of the rectangle, thereby obtaining the corresponding feature region.

[0054] In this embodiment, the features include a first feature and a second feature, which correspondingly determine a first feature space and a second feature space. For the first feature space: in the first feature space, the feature distribution information of the first feature of the source data is determined; based on the feature distribution information, a first feature region of the source data is determined; and a first filtering condition is determined as the first feature being within the first feature region. Specifically, the first feature of the source data in the first feature space can be analyzed to determine a first feature center and first density information; based on the first feature center as the first circle center and the first density information, a first radius is determined; based on the first circle center and the first radius, a first circular region is determined, and the area containing the first circular region is determined as the first feature region. For the second feature space: in the second feature space, the feature distribution information of the second feature of the source data is determined; based on the feature distribution information, a second feature region of the source data is determined; and a second filtering condition is determined as the second feature being outside the second feature region. Specifically, the second feature of the source data in the second feature space is analyzed to determine the second feature center and the second density information; the second feature center is used as the second circle center, and the second density information is used to determine the second radius; the second circle center and the second radius are used to determine the second circular region, and the range of the second circular region is defined as the second feature region.

[0055] Then, features of the target data can be filtered in the feature space based on the filtering conditions to determine the target data whose features meet the filtering conditions. Specifically, the features of the target data can be compared with the feature region to determine whether the filtering conditions are met. For example, if the filtering condition is that the feature of the target data is within the feature region, then the target data whose feature is within the feature region is determined to meet the filtering conditions; conversely, if the filtering condition is that the feature of the target data is outside the feature region, then the target data whose feature is outside the feature region is determined to meet the filtering conditions.

[0056] The features include a first feature and a second feature. The first feature is distributed in a first feature space, and the second feature is distributed in a second feature space. The first feature space corresponds to a first filtering condition, and the second feature space corresponds to a second filtering condition. In an optional embodiment, the step of filtering the target data based on the filtering condition to determine the target data includes: filtering target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition.

[0057] If the first filtering condition is that the first feature of the target data is within the first feature region, and the second filtering condition is that the second feature of the target data is outside the second feature region, then the first feature of the target data can be compared with the first feature region in the first feature space; and the second feature of the target data can be compared with the second feature region in the second feature space; thus, target data whose first feature is within the first feature region and whose second feature is outside the second feature region can be determined. For example, in one instance, the target data in the first feature space is first filtered to identify target data that meets the first filtering condition. Then, it is determined whether the target data that meets the first filtering condition also meets the second filtering condition. If it does, then the target data can be identified as the target data to be filtered. Alternatively, the target data can be first filtered in the second feature space to obtain target data that meets the second filtering condition. Then, it is determined in the first feature space whether the features of the target data that meets the second filtering condition also meet the first filtering condition. If they do, then the feature data is identified as the target data to be filtered.

[0058] In this embodiment, target data can be generated by analyzing real data (source data). Then, the data features are decoupled to obtain consistency and diversity features. Next, statistical analysis is performed on the features of the real data in the two feature spaces to obtain the feature centers of the real data, which serve as benchmarks for the distribution of suitable samples (target data). By calculating the distances of the generated samples to the centers of the real samples in both feature spaces, samples that meet the given requirements in both spaces are selected, resulting in the final selected target data.

[0059] Based on the data uploaded by the user, the filtered target data can also be fed back to the user. In other embodiments, filtering results can be generated based on the filtered target data and then sent. In one example, the filtered target data can be added to the filtering results; in other examples, abnormal data during the filtering process can be identified and added to the filtering results. The filtering results can then be fed back to the user and displayed on the user's device. Figure 2 As shown, after the data is uploaded and processed, the server can provide the filtering results, which are then displayed on the user's device. In other examples, this anomalous data can also provide feedback to the user in generating target data. For instance, the loss function can be determined based on the anomalous data to adjust the parameters of the model used by the user, enabling model training so that subsequent models can generate target data that better meets the requirements.

[0060] In addition, after the filtering results are displayed, users can also review them, such as through manual review, and then confirm the final filtering results. This includes removing unsatisfactory target data or adding abnormal data back to the target data. Users can also provide feedback on their adjustments to the filtered target data to the server, allowing the server to adjust the filtering process accordingly.

[0061] In summary, feature decoupling of source and target data yields the features of both data. Then, based on the corresponding features of the source data, the target data is filtered to obtain target data that better meets the needs of subsequent processing. The filtered target data retains the different features compared to the source data, providing a more diverse data foundation for subsequent processing.

[0062] Optionally, the features obtained from the data decoupled include consistency features and diversity features. Based on the consistency features of the source data, target data that matches the features of the target object in the source data can be selected. Based on the diversity features, features that differ from other features of the source data can be selected, thus obtaining target data that simultaneously satisfies the above two conditions, providing diverse data for subsequent processing.

[0063] In this embodiment of the application, after the target data is selected, the selected target data and source data can be used for subsequent processing, such as using the selected target data and source data for model training.

[0064] Taking website data analysis as an example, data from e-commerce websites, social networking sites, lifestyle sharing websites, and video websites can be from different data domains. In one scenario, user reviews of certain products from social networking and lifestyle sharing websites can be transferred to the e-commerce scenario. Furthermore, target review data can be generated based on this source review data, for example, by combining product objects with different functions, prices, and photographic images. Then, other review data can be filtered using the above method, decoupling the consistency and diversity features of the source and target review data. The consistency features are those related to the product object, such as functions and hardware parameters for electronic products, and scent and ingredients for skincare products. The diversity features are other related characteristics, such as user experience for electronic products, and experiences with different skin types for skincare products. Thus, in the consistency and diversity feature spaces, the target review data is filtered based on the filtering conditions determined by the source review data to obtain the desired features.

[0065] This application's embodiments can be applied to cross-data domain data migration and processing scenarios. Data from a first data source can be migrated to a second data domain, and target data can be generated in the second data domain as augmented data for training and other processing within the second data domain. The data domain can be determined based on the scenario information of the application. For example, in a cross-domain ReID scenario, the data domain can be determined by the cross-domain scenario information, such as determining the first data domain as outdoor and the second data domain as indoor. Similarly, in scenarios such as data migration from social networking sites to e-commerce websites, or data migration from e-commerce websites to video websites, the data domain can be determined by combining the specific scenario information. Scenario information can include scenario information for enterprise security, unmanned supermarkets, and vending machines, etc. This scenario information can determine the corresponding data domain, and based on the scenario information, corresponding data can be obtained, thus conveniently determining the data corresponding to various scenarios in the corresponding data domain, and performing data augmentation, filtering, and subsequent processing based on the data.

[0066] The data filtering method described above can also be applied to other scenarios, such as logistics and finance. The specific data to be generated can be determined by combining the data in the specific scenario, and the data of the mall can be filtered by combining the source data. Then, subsequent processing can be performed, such as unsupervised learning training, or other data analysis processes.

[0067] Based on the above embodiments, this application also provides a data filtering method that can extract a first feature and a second feature from source data and target data respectively, thereby filtering the target data based on the first feature space and the second feature space to determine the target data that meets the requirements.

[0068] Reference Figure 3 The diagram illustrates another example of data filtering according to an embodiment of this application.

[0069] Reference Figure 4 The diagram shows a flowchart of the steps of an embodiment of a data filtering method according to this application.

[0070] Step 402: Input the source data into the data generation model and output the corresponding target data. Multiple source data sets can be input into a data generation model such as a GAN model to obtain multiple target data sets, which can constitute a target data set.

[0071] Step 404: Decouple the source data in the source data set by features to obtain the corresponding first feature and second feature.

[0072] Step 406: Decouple the target data in the target data set by features to obtain the corresponding first feature and second feature.

[0073] Step 408: Determine the first feature space based on the first feature of the source data and the first feature of the target data.

[0074] Step 410: Determine the second feature space based on the second features of the source data and the second features of the target data.

[0075] Step 412: In the first feature space, determine the first feature distribution information of the first feature of the source data.

[0076] Step 414: Determine the first screening condition of the first feature space based on the first feature distribution information.

[0077] Specifically, the first feature of the source data in the first feature space is analyzed to determine the first feature center and the first density information, and the first feature center and the first density information are used as the first feature distribution information; the first feature region of the source data is determined based on the first feature distribution information; and based on the first feature region, the first screening condition is determined to be that the first feature of the target data is within the first feature region.

[0078] Furthermore, the first feature center can be used as the first circle center, and the first radius can be determined by the first distance corresponding to the first density information; a first circular region can be determined based on the first circle center and the first radius, and the range where the first circular region is located can be determined as the first feature region.

[0079] Step 416: In the second feature space, determine the second feature distribution information of the second feature of the source data.

[0080] Step 418: Determine the second screening conditions for the second feature space based on the second feature distribution information.

[0081] Specifically, the second feature of the source data in the second feature space is analyzed to determine the second feature center and the second density information, and the second feature center and the second density information are used as the second feature distribution information; the second feature region of the source data is determined based on the second feature distribution information; and based on the second feature region, the second screening condition is determined to be that the second feature of the target data is within the second feature region.

[0082] Furthermore, the second feature center can be used as the second circle center, and the second radius can be determined by the second distance corresponding to the second density information; the second circular region can be determined based on the second circle center and the second radius, and the range where the second circular region is located can be determined as the second feature region.

[0083] Step 420: Determine whether the target data meets the first filtering condition. Specifically, in the first feature space, the first feature of the target data is compared with the first feature region to determine whether the first feature of the target data is within the first feature region.

[0084] Step 422: Determine whether the target data meets the second filtering condition. Specifically, in the second feature space, the second feature of the target data is compared with the second feature region to determine whether the second feature of the target data is within the second feature region.

[0085] Target data is determined where the first feature is within the first feature region and the second feature is outside the second feature region.

[0086] Step 424: Determine the target data for screening.

[0087] This application's embodiments can generate target data based on source data. The target data can be used as augmented data in subsequent processing. To improve data quality, features can be extracted from the data through feature decoupling. Suitable information is selected from the extracted features to filter out suitable target data. From a large amount of target data, suitable target data is selected for data augmentation. Suitable data for subsequent processing is filtered out through joint measurement of two feature spaces.

[0088] The above embodiments can be applied to ReID scenarios, where the data type is an image, the source data includes source image data, and the target data includes target image data.

[0089] Reference Figure 5 The diagram shows a flowchart of an embodiment of an image data filtering method according to this application.

[0090] Step 502: Decouple the source image data and target image data by feature decoupling to determine the features of the source image data and the features of the target image data.

[0091] Step 504: Determine the filtering conditions for the target image data based on the characteristics of the source image data and the characteristics of the target image data.

[0092] Step 506: Filter the target image data based on the filtering conditions to determine the filtered target image data.

[0093] The step of decoupling the source image data and the target image data to determine the features of the source image data and the target image data includes: obtaining a set of source image data and a set of target image data; decoupling the features of the source image data in the source image data set to determine at least one feature of the source image data; and decoupling the features of the target image data in the target image data set to determine at least one feature of the target image data.

[0094] The step of determining the filtering conditions for the target image data based on the features of the source image data and the target image data includes: constructing a feature space based on the features of the source image data and the target image data; analyzing the feature distribution information of the corresponding features of the source image data in the feature space; and determining the filtering conditions for the target image data based on the feature distribution information.

[0095] Analyzing the feature distribution information of the corresponding features of the source image data in the feature space includes: analyzing the features of the source image data in the feature space to determine the feature center and density information.

[0096] The step of determining the filtering conditions for target image data based on the feature distribution information includes: determining feature regions based on the feature center and density information; and determining the filtering conditions for target image data based on the feature regions.

[0097] Before performing feature decoupling on the source image data and target image data, the process further includes: obtaining the source image data and target image data from the user terminal. Correspondingly, a filtering result can be generated based on the filtered target image data, and the filtering result can be sent. The filtering conditions can also be returned to the user terminal so that the user terminal can adjust the filtering conditions.

[0098] Taking the cross-domain ReID scenario as an example, the source image data of the first data domain can be migrated to the second data domain. The target image data in the second data domain can also be generated based on the source image data of the first data domain. The image data of the first data domain can be input into the data generation model and the image data of the second data domain can be output.

[0099] The features include consistency features and diversity features. Based on the above embodiments, this application also provides a data filtering method applied in the ReID scenario. It can extract consistency features and diversity features from source image data and target image data respectively, thereby filtering the target image data based on the consistency feature space and the diversity feature space to determine the target image data that meets the requirements.

[0100] Reference Figure 6 The diagram illustrates a flowchart of another embodiment of the image data filtering method of this application.

[0101] Step 602: Input the source image data into the data generation model and output the corresponding target image data. Multiple source image data sets can be input into a data generation model such as a GAN model to obtain multiple target image data sets, which can constitute a target image data set.

[0102] Step 604: Decouple the source image data in the source image dataset to obtain the corresponding consistency features and diversity features.

[0103] Step 606: Decouple the target image data in the target image dataset to obtain the corresponding consistency features and diversity features.

[0104] Step 608: Determine the consistency feature space based on the consistency features of the source image data and the consistency features of the target image data.

[0105] Step 610: Determine the diversity feature space based on the diversity features of the source image data and the diversity features of the target image data.

[0106] Step 612: In the consistency feature space, determine the consistency feature distribution information of the consistency features of the source image data.

[0107] Step 614: Determine the first screening condition of the consistency feature space based on the consistency feature distribution information.

[0108] Specifically, the consistency features of the source image data in the consistency feature space are analyzed to determine the first feature center and the first density information, and the first feature center and the first density information are used as the first feature distribution information; the first feature region of the source image data is determined based on the first feature distribution information; and based on the first feature region, the first screening condition is determined to be that the consistency features of the target image data are within the first feature region.

[0109] Furthermore, the first feature center can be used as the first circle center, and the first radius can be determined by the first distance corresponding to the first density information; a first circular region can be determined based on the first circle center and the first radius, and the range where the first circular region is located can be determined as the first feature region.

[0110] Step 616: In the diversity feature space, determine the diversity feature distribution information of the source image data.

[0111] Step 618: Determine the second screening condition for the diversity feature space based on the diversity feature distribution information.

[0112] Specifically, the diversity features of the source image data in the diversity feature space are analyzed to determine the second feature center and the second density information, and the second feature center and the second density information are used as the second feature distribution information; the second feature region of the source image data is determined based on the second feature distribution information; and the second screening condition is determined based on the second feature region, which is that the diversity features of the target image data are within the second feature region.

[0113] Furthermore, the second feature center can be used as the second circle center, and the second radius can be determined by the second distance corresponding to the second density information; the second circular region can be determined based on the second circle center and the second radius, and the range where the second circular region is located can be determined as the second feature region.

[0114] Step 620: Determine whether the target image data meets the first screening condition. Specifically, in the consistency feature space, the consistency features of the target image data are compared with the first feature region to determine whether the consistency features of the target image data are within the first feature region.

[0115] Step 622: Determine whether the target image data meets the second screening condition. Specifically, in the diversity feature space, the diversity features of the target image data are compared with the second feature region to determine whether the diversity features of the target image data are within the second feature region.

[0116] Target image data whose consistency features are located within the first feature region and whose diversity features are located outside the second feature region.

[0117] Step 624: Determine the target image data to be screened.

[0118] This application embodiment can generate target image data based on source image data. The target image data can be used as augmented data in subsequent processing. To improve data quality, features can be extracted from the image data through feature decoupling. Suitable information is selected from the extracted features to filter out suitable target image data. From a large amount of target image data, suitable target image data is selected for data augmentation. Suitable data for subsequent processing is filtered out through joint measurement of two feature spaces. This application embodiment, based on feature separation, can filter data and select high-quality data for subsequent processing.

[0119] Based on the above embodiments, the embodiments of this application can also be applied to scenarios where data of unmanned vending systems are processed. The unmanned vending system may include unmanned supermarkets, unmanned vending machines (or automatic vending machines), gift distribution machines, sample distribution machines (machines that automatically distribute samples), coupon distribution machines, etc.

[0120] Reference Figure 7 The diagram illustrates a flowchart of another embodiment of the data filtering method of this application.

[0121] Step 702: Obtain the source data and target data corresponding to the unmanned vending system.

[0122] The data from these machines can include product information, such as the product name, identifier, quantity, and information of the user who purchased or received the product. It may also include images or videos of the usage process. The corresponding source and target data can be identified. This data can be uploaded to the server from the user's end, such as through... Figure 2 The example uploads data via the interface. Data can also be obtained by the server through other means, such as from a data collection device; this embodiment does not impose any restrictions on this. In some scenarios, the target data can also be generated based on the source data.

[0123] In other embodiments, the aforementioned data can also be cross-domain data. For example, the source data may be image data collected from a first data source, while the target data may be image data collected from a second data source. If the first data source is the database of an unmanned vending system, and the second data source is the database of an e-commerce system, then the product object information corresponding to the e-commerce system can be filtered based on the product object information of the unmanned vending system. Furthermore, in cross-domain scenarios, the first data source can be a data source other than the database of the unmanned vending system, such as the database of an e-commerce system, a social networking site, or a lifestyle sharing website, while the second data source is the database of the e-commerce system. Alternatively, image data from the second data source can be generated based on the image data from the first data source and used as the target data.

[0124] Specifically, source data and target data can be categorized based on data type. For source data and target data of the same type, such as text-type source data and target data, and for image-type source data and target data, the following processing is performed:

[0125] Step 704: Decouple the source data and target data by feature decoupling to determine the features of the source data and the features of the target data.

[0126] The step of decoupling the source data and target data by features to determine the features of the source data and the target data includes: obtaining a source data set and a target data set; decoupling the source data in the source data set by features to determine at least one feature of the source data; and decoupling the target data in the target data set by features to determine at least one feature of the target data.

[0127] Step 706: Determine the filtering conditions for the target data based on the characteristics of the source data and the characteristics of the target data.

[0128] The step of determining the screening conditions for the target data based on the features of the source data and the target data includes: constructing a feature space based on the features of the source data and the target data; analyzing the feature distribution information of the corresponding features of the source data in the feature space; and determining the screening conditions for the target data based on the feature distribution information.

[0129] The step of analyzing the feature distribution information of the source data corresponding to the features in the feature space includes: analyzing the features of the source data in the feature space to determine the feature center and density information. The step of determining the screening conditions for the target data based on the feature distribution information includes: determining the feature region based on the feature center and density information; and determining the screening conditions for the target data based on the feature region.

[0130] The features include a first feature and a second feature, such as the first feature being a consistency feature and the second feature being a diversity feature. A feature space is constructed based on the features of the source data and the features of the target data, including: constructing a first feature space based on the first feature of the source data and the first feature of the target data; and constructing a second feature space based on the second feature of the source data and the second feature of the target data.

[0131] The first feature space corresponds to a first feature region, and the second feature space corresponds to a second feature region; the step of determining the filtering conditions for target data based on the feature regions includes: determining the first filtering condition based on the first feature region as a first feature of the target data being within the first feature region; and determining the second filtering condition based on the second feature region as a second feature of the target data being outside the second feature region.

[0132] In some optional embodiments of this application, the filtering conditions are returned to the user terminal so that the user terminal can adjust the filtering conditions. This allows users to adjust the filtering conditions according to their needs and obtain target data that better meets their requirements.

[0133] Step 708: Filter the target data based on the filtering conditions to determine the target data to be filtered.

[0134] The step of filtering the target data based on the filtering conditions to determine the target data includes: filtering target data whose first feature meets the first filtering conditions and whose second feature meets the second filtering conditions.

[0135] The step of filtering target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition includes: in the first feature space, comparing the first feature of the target data with the first feature region; in the second feature space, comparing the second feature of the target data with the second feature region; and determining target data whose first feature is within the first feature region and whose second feature is outside the second feature region.

[0136] In some optional embodiments of this application, a filtering result is generated based on the target data to be filtered, and the filtering result is sent. The filtering result can be displayed on the user side, such as... Figure 2 As shown in the interface, if the user is not satisfied with the results, they can adjust the filter criteria and filter again.

[0137] After determining the source data and the selected target data, other processing required for the scenario can be performed, such as identifying users of the unmanned vending system, such as customers and maintenance personnel. Therefore, users of the unmanned vending system can be identified based on the source data and the selected target data. This could involve user identification based on images or based on product information. Specific identification processing can be determined based on the data; for example, for product objects, it can be determined based on user information, facial recognition, or pedestrian re-identification.

[0138] The above example uses an unmanned vending system. In actual processing, the source and target data can also be data from various scenarios such as e-commerce systems, logistics systems, financial systems, social networking sites, and lifestyle sharing websites. Based on the source data corresponding to the scenario, the target data is filtered to select the data that meets the requirements for subsequent processing, such as target object identification.

[0139] Based on the above embodiments, the embodiments of this application can also be applied to SaaS scenarios, where data filtering and processing can be encapsulated into a data filtering service, thereby providing a service interface to the outside world. Users can obtain service results by subscribing to the data filtering service and thus use the corresponding service.

[0140] Reference Figure 8 The diagram illustrates a flowchart of the steps of a data filtering service method according to an embodiment of this application.

[0141] Step 802: Provide the first interface.

[0142] The service provides a first interface for users to use, allowing them to upload data. Users can subscribe to the service on the server side to obtain the first interface, or the server can check user permissions when using the first interface; this application does not impose any limitations on this.

[0143] The server can also provide corresponding service pages based on this service. For example... Figure 2 The page shown allows users to access the service through the first page. After selecting source data, target data, and other data, users can upload data using the upload control and call the first interface to filter the data. The filtering results are then fed back to the user and displayed on the second page.

[0144] Step 804: Obtain source data and target data through the first interface.

[0145] Step 806: Decouple the source data and target data by feature decoupling to determine the features of the source data and the target data.

[0146] The step of decoupling the source data and target data by features to determine the features of the source data and the target data includes: obtaining a source data set and a target data set; decoupling the source data in the source data set by features to determine at least one feature of the source data; and decoupling the target data in the target data set by features to determine at least one feature of the target data.

[0147] Step 808: Determine the filtering conditions for the target data based on the characteristics of the source data and the characteristics of the target data.

[0148] The step of determining the screening conditions for the target data based on the features of the source data and the target data includes: constructing a feature space based on the features of the source data and the target data; analyzing the feature distribution information of the corresponding features of the source data in the feature space; and determining the screening conditions for the target data based on the feature distribution information.

[0149] The step of analyzing the feature distribution information of the source data corresponding to the features in the feature space includes: analyzing the features of the source data in the feature space to determine the feature center and density information. The step of determining the screening conditions for the target data based on the feature distribution information includes: determining the feature region based on the feature center and density information; and determining the screening conditions for the target data based on the feature region.

[0150] The features include a first feature and a second feature, such as the first feature being a consistency feature and the second feature being a diversity feature. A feature space is constructed based on the features of the source data and the features of the target data, including: constructing a first feature space based on the first feature of the source data and the first feature of the target data; and constructing a second feature space based on the second feature of the source data and the second feature of the target data.

[0151] The first feature space corresponds to a first feature region, and the second feature space corresponds to a second feature region; the step of determining the filtering conditions for target data based on the feature regions includes: determining the first filtering condition based on the first feature region as a first feature of the target data being within the first feature region; and determining the second filtering condition based on the second feature region as a second feature of the target data being outside the second feature region.

[0152] In some optional embodiments of this application, the filtering conditions are returned to the user terminal so that the user terminal can adjust the filtering conditions. This allows users to adjust the filtering conditions according to their needs and obtain target data that better meets their requirements.

[0153] Step 810: Filter the target data based on the filtering conditions to determine the target data to be filtered.

[0154] The step of filtering the target data based on the filtering conditions to determine the target data includes: filtering target data whose first feature meets the first filtering conditions and whose second feature meets the second filtering conditions.

[0155] The step of filtering target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition includes: in the first feature space, comparing the first feature of the target data with the first feature region; in the second feature space, comparing the second feature of the target data with the second feature region; and determining target data whose first feature is within the first feature region and whose second feature is outside the second feature region.

[0156] Step 812: Generate filtering results based on the target data to be filtered, and send the filtering results to the user terminal through the second interface.

[0157] The first and second interfaces can be the same or different interfaces. After data filtering is completed, the filtering results can be fed back to the user through the second interface. For example... Figure 2 As shown, the filtering results can be viewed on the second page.

[0158] In some examples, the above filtering conditions can also be fed back to the user end through a third interface, allowing the user to adjust the filtering conditions based on their needs. In other examples, if the user end determines that the filtering results do not meet their needs, they can also obtain the filtering conditions through the third interface, adjust them, and then feed them back, before re-filtering the target data and obtaining the filtering results again. The third result can be the same as or different from the first and second interfaces.

[0159] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0160] Based on the above embodiments, this embodiment also provides a data processing device that is applied in electronic devices such as terminal devices and servers.

[0161] Reference Figure 9 The diagram shows a structural block diagram of an embodiment of a data processing apparatus according to this application, which may specifically include the following modules:

[0162] The feature decoupling module 902 is used to decouple the source data and target data to determine the features of the source data and the target data. The target data is generated based on the source data.

[0163] The condition determination module 904 is used to determine the filtering conditions of the target data based on the characteristics of the source data and the characteristics of the target data.

[0164] The filtering module 906 is used to filter the target data based on the filtering conditions to determine the target data to be filtered.

[0165] The condition determination module includes: a spatial determination unit and a filtering condition determination unit, wherein:

[0166] The spatial determination unit is used to construct a feature space based on the features of the source data and the features of the target data.

[0167] The filtering condition determination unit is used to analyze the feature distribution information of the corresponding features of the source data in the feature space, and determine the filtering conditions of the target data based on the feature distribution information.

[0168] The feature decoupling module 502 is used to acquire a source data set and a target data set; perform feature decoupling on the source data in the source data set to determine at least one feature of the source data; and perform feature decoupling on the target data in the target data set to determine at least one feature of the target data.

[0169] The filtering condition determination unit is used to analyze the features of the source data in the feature space, determine the feature center and density information; determine the feature region based on the feature center and density information; and determine the filtering conditions based on the feature region.

[0170] In one optional embodiment, the features include a first feature and a second feature; the space determination unit is configured to construct a first feature space based on the first feature of the source data and the first feature of the target data; and to construct a second feature space based on the second feature of the source data and the second feature of the target data.

[0171] The first feature space corresponds to the first feature region, and the second feature space corresponds to the second feature region; the filtering condition determination unit is used to determine, based on the first feature region, that the first filtering condition is that the first feature of the target data is within the first feature region; and based on the second feature region, to determine, based on the second filtering condition, that the second feature of the target data is outside the second feature region.

[0172] The filtering module 906 is used to filter target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition.

[0173] The filtering module 906 is used to compare a first feature of the target data with a first feature region in the first feature space; and to compare a second feature of the target data with a second feature region in the second feature space; and to determine target data in which the first feature is within the first feature region and the second feature is outside the second feature region.

[0174] Also includes:

[0175] The post-processing module is used for training with the selected target data and source data.

[0176] The data generation module is used to input the source data into the data generation model and output the corresponding target data.

[0177] The data acquisition module is used to acquire source data and target data from the user terminal. The target data can be generated based on the source data.

[0178] The data feedback module is used to generate filtering results based on the target data being filtered, and to send the filtering results.

[0179] The data feedback module is also used to return the filtering conditions to the user terminal so that the user terminal can adjust the filtering conditions.

[0180] The source data includes image data in a first data domain; the target data includes image data in a second data domain; and the data generation module is used to input the image data in the first data domain into the data generation model and output the image data in the second data domain.

[0181] The data domain is determined based on the scenario information of the application scenario.

[0182] In some scenarios, the source data includes source image data, and the target data includes target image data;

[0183] The feature decoupling module 902 is used to decouple the source image data and the target image data to determine the features of the source image data and the target image data. The target image data is generated based on the source image data.

[0184] The condition determination module 904 is used to determine the filtering conditions of the target image data based on the characteristics of the source image data and the characteristics of the target image data.

[0185] The filtering module 906 is used to filter the target image data based on the filtering conditions to determine the target image data to be filtered.

[0186] The condition determination module includes: a spatial determination unit and a filtering condition determination unit, wherein:

[0187] The spatial determination unit is used to construct a feature space based on the features of the source image data and the features of the target image data.

[0188] The filtering condition determination unit is used to analyze the feature distribution information of the corresponding features of the source image data in the feature space, and determine the filtering conditions of the target image data based on the feature distribution information.

[0189] The feature decoupling module 502 is used to acquire a source image data set and a target image data set; perform feature decoupling on the source image data in the source image data set to determine at least one feature of the source image data; and perform feature decoupling on the target image data in the target image data set to determine at least one feature of the target image data.

[0190] The filtering condition determination unit is used to analyze the features of the source image data in the feature space, determine the feature center and density information; determine the feature region based on the feature center and density information; and determine the filtering conditions based on the feature region.

[0191] In one alternative embodiment, the feature includes a first feature and a second feature; wherein the first feature includes a consistency feature and the second feature includes a diversity feature.

[0192] The spatial determination unit is used to construct a first feature space based on the first feature of the source image data and the first feature of the target image data; and to construct a second feature space based on the second feature of the source image data and the second feature of the target image data.

[0193] The first feature space corresponds to the first feature region, and the second feature space corresponds to the second feature region; the filtering condition determination unit is used to determine, based on the first feature region, that the first filtering condition is that the first feature of the target image data is within the first feature region; and based on the second feature region, to determine, the second filtering condition is that the second feature of the target image data is outside the second feature region.

[0194] The filtering module 906 is used to filter target image data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition.

[0195] The filtering module 906 is used to compare a first feature of the target image data with a first feature region in the first feature space; and to compare a second feature of the target image data with a second feature region in the second feature space; and to determine target image data in which the first feature is within the first feature region and the second feature is outside the second feature region.

[0196] Also includes:

[0197] The post-processing module is used for training using the filtered target image data and source image data.

[0198] The data generation module is used to input the source image data into the data generation model and output the corresponding target image data.

[0199] The data acquisition module is used to acquire source image data and target image data from the user terminal.

[0200] The data feedback module is used to generate filtering results based on the filtered target image data and send the filtering results.

[0201] The data feedback module is also used to return the filtering conditions to the user terminal so that the user terminal can adjust the filtering conditions.

[0202] The above embodiments can also be applied to unmanned vending systems.

[0203] The data acquisition module is used to acquire the source data and target data corresponding to the unmanned vending system.

[0204] The feature decoupling module 902 is used to decouple the source data and target data to determine the features of the source data and the target data. The target data is generated based on the source data.

[0205] The condition determination module 904 is used to determine the filtering conditions of the target data based on the characteristics of the source data and the characteristics of the target data.

[0206] The filtering module 906 is used to filter the target data based on the filtering conditions to determine the target data to be filtered.

[0207] The condition determination module includes: a spatial determination unit and a filtering condition determination unit, wherein:

[0208] The spatial determination unit is used to construct a feature space based on the features of the source data and the features of the target data.

[0209] The filtering condition determination unit is used to analyze the feature distribution information of the corresponding features of the source data in the feature space, and determine the filtering conditions of the target data based on the feature distribution information.

[0210] The feature decoupling module 902 is used to acquire a source data set and a target data set; perform feature decoupling on the source data in the source data set to determine at least one feature of the source data; and perform feature decoupling on the target data in the target data set to determine at least one feature of the target data.

[0211] The filtering condition determination unit is used to analyze the features of the source data in the feature space, determine the feature center and density information; determine the feature region based on the feature center and density information; and determine the filtering conditions based on the feature region.

[0212] In one optional embodiment, the features include a first feature and a second feature; the space determination unit is configured to construct a first feature space based on the first feature of the source data and the first feature of the target data; and to construct a second feature space based on the second feature of the source data and the second feature of the target data.

[0213] The first feature space corresponds to the first feature region, and the second feature space corresponds to the second feature region; the filtering condition determination unit is used to determine, based on the first feature region, that the first filtering condition is that the first feature of the target data is within the first feature region; and based on the second feature region, to determine, based on the second filtering condition, that the second feature of the target data is outside the second feature region.

[0214] The filtering module 906 is used to filter target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition.

[0215] The filtering module 906 is used to compare a first feature of the target data with a first feature region in the first feature space; and to compare a second feature of the target data with a second feature region in the second feature space; and to determine target data in which the first feature is within the first feature region and the second feature is outside the second feature region.

[0216] Also includes:

[0217] The post-processing module is used for training using the filtered target data and source data. The post-processing module is also used to identify and determine the users of the unmanned vending system based on the source data and the filtered target data.

[0218] The data generation module is used to input the source data into the data generation model and output the corresponding target data.

[0219] The data acquisition module is used to acquire source data and target data of the unmanned vending system from the user terminal. Specifically, the data acquisition module collects image data from a first data source as source data; and determines image data from a second data source as target data; the first and second data sources belong to different data domains.

[0220] The data feedback module is used to generate filtering results based on the target data being filtered, and to send the filtering results.

[0221] The data feedback module is also used to return the filtering conditions to the user terminal so that the user terminal can adjust the filtering conditions.

[0222] The source data includes image data in a first data domain; the target data includes image data in a second data domain; and the data generation module is used to input the image data in the first data domain into the data generation model and output the image data in the second data domain.

[0223] Based on the above embodiments, the embodiments of this application can also be applied to SaaS scenarios.

[0224] The interface provider module is used to provide the first interface.

[0225] The interface call module is used to obtain source data and target data through the first interface.

[0226] The service processing module is used to decouple the source data and target data by feature, determine the features of the source data and the target data; determine the filtering conditions for the target data based on the features of the source data and the target data; filter the target data based on the filtering conditions to determine the filtered target data; and determine the filtering result based on the filtered target data.

[0227] The result feedback module is used to provide a second interface to feedback the filtering results.

[0228] The service processing module includes a feature decoupling submodule, a condition determination submodule, and a filtering submodule.

[0229] The feature decoupling submodule is used to decouple the source data and target data by determining the features of the source data and the target data. The target data is generated based on the source data.

[0230] The condition determination submodule is used to determine the filtering conditions for the target data based on the characteristics of the source data and the characteristics of the target data.

[0231] The filtering submodule is used to filter the target data based on the filtering conditions to determine the target data to be filtered.

[0232] The condition determination submodule includes: a spatial determination unit and a filtering condition determination unit, wherein:

[0233] The spatial determination unit is used to construct a feature space based on the features of the source data and the features of the target data.

[0234] The filtering condition determination unit is used to analyze the feature distribution information of the corresponding features of the source data in the feature space, and determine the filtering conditions of the target data based on the feature distribution information.

[0235] The feature decoupling submodule is used to acquire a source data set and a target data set; perform feature decoupling on the source data in the source data set to determine at least one feature of the source data; and perform feature decoupling on the target data in the target data set to determine at least one feature of the target data.

[0236] The filtering condition determination unit is used to analyze the features of the source data in the feature space, determine the feature center and density information; determine the feature region based on the feature center and density information; and determine the filtering conditions based on the feature region.

[0237] In one optional embodiment, the features include a first feature and a second feature; the space determination unit is configured to construct a first feature space based on the first feature of the source data and the first feature of the target data; and to construct a second feature space based on the second feature of the source data and the second feature of the target data.

[0238] The first feature space corresponds to the first feature region, and the second feature space corresponds to the second feature region; the filtering condition determination unit is used to determine, based on the first feature region, that the first filtering condition is that the first feature of the target data is within the first feature region; and based on the second feature region, to determine, based on the second filtering condition, that the second feature of the target data is outside the second feature region.

[0239] The filtering submodule is used to filter target data whose first feature meets the first filtering condition and whose second feature meets the second filtering condition.

[0240] The filtering submodule is used to compare a first feature of the target data with a first feature region in the first feature space; and to compare a second feature of the target data with a second feature region in the second feature space; and to determine target data whose first feature is within the first feature region and whose second feature is outside the second feature region.

[0241] Also includes:

[0242] The post-processing module is used for training using the selected target data and source data. This training can also be executed by calling the corresponding data training service, thus allowing the corresponding interface to be called to perform training processing.

[0243] The data generation module is used to input the source data into the data generation model and output the corresponding target data. The above data generation process can be implemented based on a data generation service, and therefore, the corresponding interface can be called to perform the data generation processing.

[0244] The data acquisition module is used to acquire source data and target data from the user terminal. The target data can be generated based on the source data.

[0245] The data feedback module is used to generate filtering results based on the target data being filtered, and to send the filtering results.

[0246] The data feedback module is also used to return the filtering conditions to the user terminal so that the user terminal can adjust the filtering conditions.

[0247] The source data includes image data in a first data domain; the target data includes image data in a second data domain; and the data generation module is used to input the image data in the first data domain into the data generation model and output the image data in the second data domain.

[0248] Compared to existing methods that only consider consistency features for subsequent training, where the target data used may not be suitable for subsequent training, this application selects appropriate information from the extracted features through feature decoupling, thereby ultimately selecting suitable target data from a large amount of target data for data augmentation.

[0249] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0250] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0251] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 10 An exemplary apparatus 1000 is schematically shown that can be used to implement the various embodiments described in this application.

[0252] In one embodiment, Figure 10 An exemplary device 1000 is shown, which includes one or more processors 1002, a control module (chipset) 1004 coupled to at least one of the processors 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0253] Processor 1002 may include one or more single-core or multi-core processors, and processor 1002 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1000 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0254] In some embodiments, apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 that are combined with the one or more computer-readable media and configured to execute instructions 1014 to implement modules and thereby perform the actions described in this disclosure.

[0255] In one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component communicating with the control module 1004.

[0256] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0257] Memory 1006 may be used, for example, to load and store data and / or instructions 1014 for device 1000. In one embodiment, memory 1006 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1006 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0258] In one embodiment, the control module 1004 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1008 and (one or more) input / output devices 1010.

[0259] For example, NVM / storage device 1008 may be used to store data and / or instructions 1014. NVM / storage device 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0260] NVM / storage device 1008 may include storage resources that are physically part of a device on which device 1000 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 1008 may be accessed via a network via one or more input / output devices 1010.

[0261] One or more input / output devices 1010 may provide an interface for device 1000 to communicate with any other suitable device. Input / output devices 1010 may include communication components, audio components, sensor components, etc. Network interface 1012 may provide an interface for device 1000 to communicate via one or more networks. Device 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0262] In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1004. In one embodiment, at least one of the processors 1002 may be logically packaged with one or more controllers of the control module 1004 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die. In one embodiment, at least one of the processors 1002 may be integrated with the logic of one or more controllers of the control module 1004 on the same die to form a system-on-a-chip (SoC).

[0263] In various embodiments, device 1000 may be, but is not limited to, a terminal device such as a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0264] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0265] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0266] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0267] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0268] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0269] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0270] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0271] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0272] The above provides a detailed description of a data filtering method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of data screening, characterized by, The method comprises: characteristic decoupling of source data and target data to determine characteristics of the source data and characteristics of the target data, wherein the source data and the target data are various types of data, and the data types include at least one of the following: images, texts, audio; constructing a feature space according to the characteristics of the source data and the characteristics of the target data; analyzing the characteristics of the source data in the feature space to determine a feature center and density information as feature distribution information; determining a screening condition for the target data according to the feature distribution information; screening the target data based on the screening condition to determine screened target data.

2. The method of claim 1, wherein, The characteristic decoupling of source data and target data to determine characteristics of the source data and characteristics of the target data comprises: obtaining a source data set and a target data set; characteristic decoupling of source data in the source data set to determine at least one characteristic of the source data; characteristic decoupling of target data in the target data set to determine at least one characteristic of the target data.

3. The method of claim 1, wherein, The determination of a screening condition for the target data according to the feature distribution information comprises: determining a feature region according to the feature center and density information; determining a screening condition for the target data based on the feature region.

4. The method of claim 3, wherein, The characteristics include first characteristics and second characteristics, the first characteristics are characteristics strongly related to target objects in data, and the second characteristics are characteristics for enhancing data diversity; The construction of a feature space according to the characteristics of the source data and the characteristics of the target data comprises: constructing a first feature space according to the first characteristics of the source data and the first characteristics of the target data; constructing a second feature space according to the second characteristics of the source data and the second characteristics of the target data.

5. The method of claim 4, wherein, The first feature space corresponds to a first feature region, and the second feature space corresponds to a second feature region; the determination of a screening condition for the target data based on the feature region comprises: based on the first feature region, determining that a first screening condition is that the first characteristics of the target data are within the first feature region; based on the second feature region, determining that a second screening condition is that the second characteristics of the target data are outside the second feature region.

6. The method of claim 5, wherein, The screening of the target data based on the screening condition to determine screened target data comprises: screening target data whose first characteristics meet the first screening condition and whose second characteristics meet the second screening condition.

7. The method of claim 6, wherein, The screening of target data whose first characteristics meet the first screening condition and whose second characteristics meet the second screening condition comprises: in the first feature space, comparing the first characteristics of the target data with the first feature region; in the second feature space, comparing the second characteristics of the target data with the second feature region; determining target data whose first characteristics are within the first feature region and whose second characteristics are outside the second feature region.

8. The method according to any one of claims 1 to 7, characterized in that, Before the characteristic decoupling of source data and target data, the method further comprises: obtaining source data and target data from a user end.

9. The method of claim 8, wherein, The method further comprises: generating a screening result according to the screened target data and sending the screening result.

10. The method of claim 8, wherein, The method further comprises: returning the screening condition to the user end so that the user end adjusts the screening condition.

11. The method of claim 8, wherein, The source data includes image data of a first data domain; the target data includes image data of a second data domain; further comprising: Inputting the image data of the first data domain into a data generation model to output image data of the second data domain.

12. A method of data screening, characterized by, Comprising: Decoupling features of source image data and target image data, determining features of the source image data and features of the target image data, the features including: consistency features and diversity features, the consistency features being features strongly related to target objects in the data, and the diversity features being features enhancing data diversity; Forming a feature space according to the features of the source image data and the features of the target image data; Analyzing the features of the source image data in the feature space to determine feature centers and density information as feature distribution information; Determining a screening condition for the target image data according to the feature distribution information; Screening the target image data based on the screening condition to determine screened target image data.

13. The method of claim 12, wherein, The screening condition includes a first screening condition corresponding to the consistency features and a second screening condition corresponding to the diversity features; The screening of the target image data based on the screening condition to determine the screened target image data includes: Screening target image data whose first feature meets the first screening condition and whose second feature meets the second screening condition.

14. A method of data screening, characterized by, Comprising: Obtaining source data and target data corresponding to an unmanned vending system, wherein the source data and the target data are various types of data, and the data types include at least one of the following: image, text, and audio; Decoupling features of the source data and the target data to determine features of the source data and features of the target data; Forming a feature space according to the features of the source data and the features of the target data; Analyzing the features of the source data in the feature space to determine feature centers and density information as feature distribution information; Determining a screening condition for the target data according to the feature distribution information; Screening the target data based on the screening condition to determine screened target data.

15. The method of claim 14, wherein, Further comprising: Identifying the source data and the screened target data to determine a user of the unmanned vending system.

16. The method of claim 14, wherein, Further comprising: Collecting image data from a first data source as source data; Determining image data of a second data source as target data; The first data source and the second data source belong to different data domains.

17. A data processing apparatus, characterized by The device comprises: A feature decoupling module for decoupling features of source data and target data to determine features of the source data and features of the target data, wherein the source data and the target data are various types of data, and the data types include at least one of the following: image, text, and audio; A condition determination module for forming a feature space according to the features of the source data and the features of the target data; analyzing the features of the source data in the feature space to determine feature centers and density information as feature distribution information; and determining a screening condition for the target data according to the feature distribution information; A screening module for screening the target data based on the screening condition to determine screened target data.

18. An electronic device, comprising: Comprising: A processor; And a memory having stored thereon executable code that, when executed, is adapted to cause the processor to perform any one or more of the methods described in any one of claims 1-16.

19. One or more machine -readable media having stored thereon executable code that, when executed, is adapted to cause a processor to perform any one or more of the methods described in any one of claims 1-16.

Images