A data equalization method, device, apparatus and storage medium

Abstract

The application discloses a data balancing method and device, equipment and storage medium, and relates to the technical field of computers. The method comprises the following steps: performing hash processing on first samples and second samples in a data set, and mapping the first samples and the second samples into first hash buckets and second hash buckets; determining second samples matched with the first samples in the first hash buckets from second hash buckets adjacent to first hash buckets in the first hash buckets; searching in adjacent hash buckets in a counterfactual search process, reducing a search range, reducing a search amount, and improving search efficiency. Then, based on a feature relationship between the first samples and the second samples, third samples in the second samples are processed to obtain fourth samples. The labels of the fourth samples are consistent with the labels of the first samples, and data balancing is realized. Since the time required by the search process is shortened, the efficiency of data balancing is improved.

Description

A data balancing method, apparatus, device, and storage medium Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data balancing method, apparatus, device, and storage medium. Background Technology

[0002] The development of randomized computer technology allows for the use of machine learning algorithms to train diagnostic models and to predict certain diseases based on these trained models. For example, by inputting physical examination data into a diagnostic model, a prediction can be obtained as to whether a user has diabetes.

[0003] Training a diagnostic model for diabetes requires a large amount of physical examination data, but most people in these data do not have diabetes; only a minority do. This imbalance in data leads to poor accuracy in the resulting diagnostic model.

[0004] Therefore, how to achieve data balancing is a technical problem that the industry needs to solve. Summary of the Invention

[0005] This application provides a data balancing method, apparatus, device, and storage medium, which can achieve efficient data balancing.

[0006] Firstly, this application provides a data balancing method, including:

[0007] Obtain a dataset in which the number of samples in the first class is less than the number of samples in the second class, and the labels of the samples in the first class are different from the labels of the samples in the second class.

[0008] The dataset is hashed to map the first type of samples to a first type of hash bucket and the second type of samples to a second type of hash bucket.

[0009] From the second type of hash buckets adjacent to the first hash bucket in the first type of hash buckets, determine the second sample that matches the first sample in the first hash bucket;

[0010] Based on the feature relationship between the first sample and the second sample, the third sample in the second type of samples is processed to generate the fourth sample; the label of the fourth sample is consistent with the label of the first type of sample.

[0011] In some possible implementations, determining a second sample that matches a first sample in the first hash bucket from a second type of hash bucket adjacent to the first hash bucket in the first type of hash bucket includes:

[0012] Calculate the similarity between the first sample in the first hash bucket and the samples in the second type of hash bucket that are adjacent to the first hash bucket;

[0013] Based on the similarity between the first sample and the samples in the second type of hash bucket, the sample that matches the first sample is determined to be the second sample.

[0014] In some possible implementations, determining the sample matching the first sample as the second sample based on the similarity between the first sample and samples in the second type of hash bucket includes:

[0015] Identify the top S samples in terms of similarity between the first sample and the samples in the second type of hash bucket;

[0016] Take any one of the top S samples as the second sample to match the first sample, where S is a positive integer; or,

[0017] Identify at least one sample in which the similarity between the first sample and samples in the second type of hash bucket is greater than a first similarity threshold;

[0018] At least one of the samples is used as a second sample that matches the first sample.

[0019] In some possible implementations, calculating the similarity between a first sample in the first hash bucket and samples in a second type of hash bucket adjacent to the first hash bucket includes:

[0020]

[0021] Where, F(x) i ,b i ) represents the similarity score between the first sample and samples in the second type of hash bucket, where the similarity score is negatively correlated with similarity; f(p) represents the score calculated for the p-th time between the first sample and samples in the second type of hash bucket; dim p (b i ) represents the i-th dimension feature of the first sample corresponding to the p-th calculation; dim p (x i Let be the i-th dimension feature of the sample in the second type of hash bucket corresponding to the p-th calculation.

[0022] In some possible implementations, the step of processing the third sample in the second class of samples to generate the fourth sample based on the feature relationship between the first sample and the second sample includes:

[0023] Based on the feature relationship between the first sample and the second sample, determine the difference dimension corresponding to the difference features between the first sample and the second sample;

[0024] Using the aforementioned difference features, replace the features corresponding to the difference dimension in the third sample of the second type of samples to generate a fourth sample.

[0025] In some possible implementations, the similarity between the third sample in the second type of samples and the first sample is greater than a second similarity threshold; or, the similarity between the third sample and the first sample ranks among the top M in terms of similarity between the third sample and the samples in the second type of samples matched with the first type of samples, where M is a positive integer.

[0026] In some possible implementations, the dataset is a physical examination dataset, where the first type of samples is labeled as having the disease and the second type of samples is labeled as not having the disease.

[0027] Secondly, this application provides a data equalization device, comprising:

[0028] The acquisition module is used to acquire a dataset in which the number of samples of the first class is less than the number of samples of the second class, and the labels of the samples of the first class are different from the labels of the samples of the second class.

[0029] The hash module is used to perform hash processing on the dataset, mapping the first type of samples to a first type of hash bucket and the second type of samples to a second type of hash bucket;

[0030] A matching module is used to determine a second sample that matches the first sample in the first hash bucket from a second type of hash bucket that is adjacent to the first hash bucket in the first type of hash bucket;

[0031] The equalization module is used to process the third sample in the second type of samples according to the feature relationship between the first sample and the second sample to generate a fourth sample; the label of the fourth sample is consistent with the label of the first type of sample.

[0032] In some possible implementations, the matching module is specifically used to calculate the similarity between a first sample in the first hash bucket and a sample in a second type of hash bucket adjacent to the first hash bucket; and to determine the sample that matches the first sample as the second sample based on the similarity between the first sample and the sample in the second type of hash bucket.

[0033] In some possible implementations, the matching module is specifically used to determine the top S samples in terms of similarity between the first sample and samples in the second type of hash bucket;

[0034] Take any one of the top S samples as the second sample to match the first sample, where S is a positive integer; or,

[0035] Identify at least one sample in which the similarity between the first sample and samples in the second type of hash bucket is greater than a first similarity threshold;

[0036] At least one of the samples is used as a second sample that matches the first sample.

[0037] In some possible implementations, the matching module is specifically used to calculate the similarity between a first sample in the first hash bucket and a sample in a second type of hash bucket adjacent to the first hash bucket using the following formula:

[0038]

[0039] Where, F(x) i ,b i ) represents the similarity score between the first sample and samples in the second type of hash bucket, where the similarity score is negatively correlated with similarity; f(p) represents the score calculated for the p-th time between the first sample and samples in the second type of hash bucket; dim p (b i ) represents the i-th dimension feature of the first sample corresponding to the p-th calculation; dim p (x i Let be the i-th dimension feature of the sample in the second type of hash bucket corresponding to the p-th calculation.

[0040] In some possible implementations, the equalization module is specifically used to determine the difference dimension corresponding to the difference feature between the first sample and the second sample based on the feature relationship between the first sample and the second sample; and to use the difference feature to replace the feature corresponding to the difference dimension in the third sample of the second type of sample to generate a fourth sample.

[0041] In some possible implementations, the similarity between the third sample in the second type of samples and the first sample is greater than a second similarity threshold; or, the similarity between the third sample and the first sample ranks among the top M in terms of similarity between the third sample and the samples in the second type of samples matched with the first type of samples, where M is a positive integer.

[0042] In some possible implementations, the dataset is a physical examination dataset, where the first type of samples is labeled as having the disease and the second type of samples is labeled as not having the disease.

[0043] Thirdly, this application provides a computing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of the first aspects.

[0044] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of the first aspects.

[0045] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform any of the optional methods described in the first aspect.

[0046] To achieve the above objectives, this application adopts the following technical solution:

[0047] This application provides a data balancing method, which includes: a processing device acquiring a dataset in which the number of first-class samples is less than the number of second-class samples, and the labels of the first-class samples are different from the labels of the second-class samples; then performing hash processing on the dataset, mapping the first-class samples to a first-class hash bucket, and mapping the second-class samples to a second-class hash bucket; then determining a second sample that matches the first sample in the first hash bucket from the second-class hash bucket adjacent to the first hash bucket in the first-class hash bucket; and processing a third sample in the second-class samples according to the feature relationship between the first sample and the second sample to generate a fourth sample, the label of which is consistent with the label of the first-class samples, thereby increasing the number of the relatively small number of first-class samples and achieving data balancing.

[0048] In this method, different types of hash buckets are assigned to samples of different categories. Then, during the counterfactual pair search process, the search can be performed from the second type of hash bucket that is adjacent to the first hash bucket in the first type of hash bucket. This narrows the search range and search volume, improves search efficiency, and thus improves the efficiency of data balancing.

[0049] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0050] Figure 1 is a flowchart of a data balancing method provided in an embodiment of this application;

[0051] Figure 2 is a schematic diagram of hash processing of a dataset provided in an embodiment of this application;

[0052] Figure 3 is a schematic diagram of sample matching provided in an embodiment of this application;

[0053] Figure 4 is a schematic diagram of generating a fourth sample according to an embodiment of this application;

[0054] Figure 5 is a schematic diagram of a data balancing device provided in an embodiment of this application. Detailed Implementation

[0055] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0056] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0058] Data balancing, in machine learning or data mining tasks, refers to the process of adjusting the number of samples in each category of a dataset to make the sample sizes of different categories equal or similar. Data balancing helps machine learning algorithms better learn each category in the dataset, thereby improving the algorithm's performance and accuracy.

[0059] In some examples, data balance can be achieved by reducing the number of majority class samples or increasing the number of minority class samples. Specifically, the number of majority class samples can be reduced by deleting majority class samples or by random sampling, thus making the sample numbers of each class equal or similar. Conversely, the number of minority class samples can be increased by copying, combining, or sampling minority class samples, thus making the sample numbers of each class equal or similar.

[0060] Taking a physical examination dataset as an example, the number of samples with diabetes is relatively small, while the number of samples without diabetes is relatively large. Therefore, data balance can be achieved by increasing the number of minority class samples as described above. For example, counterfactual methods can be used to achieve data balance.

[0061] In data balancing, counterfactual pairs are a technique that generates data samples that differ from the original data but are still plausible. These samples can be used to increase the size of the dataset, thereby improving the performance and robustness of machine learning models. Counterfactual pairs are new data samples generated by changing one or more features of the original data samples. These new data samples are similar to the original data samples but belong to different categories. For example, for a photograph, counterfactual pairs can be generated by changing the colors, angles, or adding noise. However, the current process of finding counterfactual pairs is complex, computationally intensive, and slow, resulting in poor efficiency in data balancing.

[0062] In view of this, embodiments of this application provide a data balancing method. This method can be applied to a processing device, which can be a terminal device or a server. In other embodiments, the method can also be executed by other devices, such as a data balancing device. Taking the method being executed by a processing device as an example, the method includes:

[0063] The processing device acquires a dataset in which the number of samples in the first class is less than the number of samples in the second class, and the labels of the samples in the first class are different from those of the samples in the second class. The dataset is then hashed, mapping the samples in the first class to first-class hash buckets and the samples in the second class to second-class hash buckets. Next, from the second-class hash buckets adjacent to the first-class hash buckets in the first class, a second sample matching the first sample in the first hash bucket is determined. Based on the feature relationship between the first and second samples, a third sample in the second class is processed to generate a fourth sample. This fourth sample has the same label as the first sample, thus increasing the number of samples in the first class and achieving data balance.

[0064] In this method, different types of hash buckets are assigned to samples of different categories. Then, during the counterfactual pair search process, the search can be performed from the second type of hash bucket that is adjacent to the first hash bucket in the first type of hash bucket. This narrows the search range and search volume, improves search efficiency, and thus improves the efficiency of data balancing.

[0065] To make the technical solution of this application clearer and easier to understand, the application scenarios provided by the embodiments of this application are described below.

[0066] In this application scenario, the dataset can be a physical examination dataset. The first type of sample can be samples with diabetes, and the second type of sample can be samples without diabetes. However, for the physical examination dataset, most people do not have diabetes, so the data imbalance problem is quite serious.

[0067] The diagnostic model trained using an imbalanced physical examination dataset will suffer from prediction bias due to the imbalance of the samples, resulting in poor accuracy and usability of the diagnostic model.

[0068] To address this, this application provides a data balancing method. This method generates a first type of sample (i.e., samples of individuals with diabetes) by using counterfactual pairs, thereby increasing the number of first-type samples in the physical examination dataset and achieving data balancing. Specifically, the first and second types of samples in the dataset are first hashed and mapped to first and second type hash buckets, respectively. Then, a second sample matching the first sample in the first hash bucket is determined from the adjacent second type hash buckets within the first type hash bucket. During the counterfactual pair search process, the search is performed from neighboring hash buckets, narrowing the search range, reducing the search volume, and improving search efficiency. Next, based on the feature relationship between the first and second samples, a third sample in the second type is processed to obtain a fourth sample. The label of this fourth sample is consistent with the label of the first type sample, thus achieving data balancing. Since the search process time is shortened, the efficiency of data balancing is improved.

[0069] The data balancing method provided in the embodiments of this application will be described below with reference to the accompanying drawings. As shown in Figure 1, this figure is a flowchart of a data balancing method provided in the embodiments of this application, which includes:

[0070] S101. Obtain the dataset.

[0071] The dataset includes a first class of samples and a second class of samples, wherein the number of first class samples is less than the number of second class samples, and the labels of the first class samples are different from the labels of the second class samples.

[0072] It should be noted that the embodiments of this application are only examples of datasets including two types of samples. In other embodiments, the dataset may include more samples, such as three or five types.

[0073] In some examples, the dataset can be a physical examination dataset, where the first type of sample can be samples with diabetes and the second type of sample can be samples without diabetes. In other examples, the first type of sample can be samples with respiratory diseases and the second type of sample can be samples without respiratory diseases.

[0074] In some examples, the physical examination dataset can be represented by T, where the i-th data point can be [t]. i =(dim1(t i ),dim2(t i ),dim3(t i ),…,dim n (t i ))], where dim n (t i () represents the nth dimension feature of the i-th data. The first class of samples can be represented by X = {x1, x2, ..., x...}min} represents, where x min Let B be the min-th sample in the first class of samples, meaning the number of samples in the first class is min. The second class of samples can be represented by B = {b1, b2, ..., b}. max} indicates that, where b max It represents the max-th sample in the second class of samples, meaning the number of samples in the second class is max.

[0075] For example, physical examination data may include the following characteristics: fasting glucose (GLU-G), glucose (GLU), age (AGE), systolic blood pressure (SBP), waist circumference (Waistline), serum triglycerides (TG), serum gamma-glutamyl transferase (GGT), carcinoembryonic antigen (CEA), body mass index (BMI), serum high-density lipoprotein (HDL-C), serum alanine aminotransferase (ALT), specific gravity (sp.gr), weight (weight), diastolic blood pressure (DBP), serum alkaline phosphatase (ALP), sex (Sex), pH, hemoglobin (HGB), serum bile acid (TBA), blood urea nitrogen (BUN), heparin (HEP), hematocrit (PCV), bacteria (Bacteria), occult blood (RBC / HPF), total cholesterol (TC), Helicobacter pylori (Bacteria / HPF), red blood cells (RBC), mean corpuscular hemoglobin concentration (MCHC), etc.

[0076] It should be noted that the features described above are merely illustrative examples, and in other examples, the medical examination data may include more or fewer features.

[0077] In some embodiments, the aforementioned dataset can be obtained through a medical examination device.

[0078] S102. Perform hash processing on the dataset, mapping the first type of samples to the first type of hash bucket and the second type of samples to the second type of hash bucket.

[0079] After obtaining the dataset, it can be hashed to map samples of the same type to hash buckets of the same type. For example, the first type of samples can be mapped to the first type of hash bucket, and the second type of samples can be mapped to the second type of hash bucket.

[0080] In some embodiments, before hashing the dataset, the individual features in the dataset can be weighted. For example, a random forest algorithm can be used to determine the importance of each feature and obtain an importance score v = {v1, v2, v3, ..., v...}. nNext, the importance scores are normalized to obtain the weights corresponding to each feature after normalization. For example, the weights corresponding to each feature can be determined using the following formula:

[0081]

[0082] Among them, w i v represents the weight corresponding to the i-th feature. i The score representing the i-th feature, where n represents the number of features.

[0083] It should be noted that the embodiments in this application only use the random forest algorithm as an example to determine the importance of features. In other examples, other algorithms, such as mutual information, can also be used.

[0084] After obtaining the weights corresponding to each feature, the first and second class samples can be weighted separately to obtain the weighted first and second class samples.

[0085] Among them, the first type of sample after weighting is b i =(w1×dim1(b i ),w2×

[0086] dim2(b i ),…,w n ×dim n (b i )), b i w represents the i-th sample in the first class of samples. n The weight of the nth dimension feature, dim n (b i Let be the nth dimension feature of the i-th sample in the first class of samples. The weighted sample in the second class is x. i =(w1×dim1(x i ),w2×dim2(x i ),…,w n ×dim n (x i ), where x i w represents the i-th sample in the second class of samples. n The weight of the nth dimension feature, dim n (x i ) represents the nth dimension feature of the i-th sample in the second class of samples.

[0087] After weighting the first and second types of samples, hash mapping can be performed on the weighted samples to obtain hash tables. The samples in the first and second types are then distributed into hash buckets, with the first type of samples distributed into the first type of hash bucket and the second type of samples into the second type of hash bucket. Samples in each hash bucket are adjacent in the original data space, meaning they have high similarity.

[0088] In some embodiments, the Locality-Sensitive Hashing (LSH) algorithm can be used to hash the dataset. LSH is a technique for quickly searching for similar data in a high-dimensional space. Its principle is to map high-dimensional data to a low-dimensional space, making similar data closer in the low-dimensional space. Therefore, the similarity of data in the high-dimensional space can be determined by comparing the distances in the low-dimensional space.

[0089] As shown in Figure 2, this figure is a schematic diagram of hash processing of a dataset provided in an embodiment of this application.

[0090] After hashing dataset 210, hash value 220 is obtained. Then, adjacent hash values ​​of the same type within hash value 220 are placed into the same type of hash bucket. For example, hash values ​​221 and 222 are placed into hash bucket 231, where hash values ​​221 and 222 are both first-type hash values; hash values ​​223 and 224 are placed into hash bucket 232, where hash values ​​223 and 224 are both second-type hash values.

[0091] Among them, hash bucket 231 is a first-type hash bucket, and hash buckets 232 and 233 are second-type hash buckets. Hash bucket 231 is adjacent to hash buckets 232 and 233. The data in each hash bucket includes parameters such as key, value, and next. The key indicates the index of the data entry, and the value is the content of that data entry, for example, it could be sample b. i , next represents the address of the next node to be stored.

[0092] S103. From the second type of hash buckets that are adjacent to the first hash bucket in the first type of hash buckets, determine the second sample that matches the first sample in the first hash bucket.

[0093] The first hash bucket refers to a hash bucket within the first type of hash buckets, such as hash bucket 231 in Figure 2. The second type of hash buckets adjacent to this first hash bucket can be hash buckets 232 and 233 in Figure 2. The first sample can be the sample corresponding to the hash value 221 mentioned above, for example, sample b1. After determining the search range (hash buckets 232 and 233) and the search object (the sample matching the first sample), the search can begin, thus obtaining the second sample matching the first sample, for example, the second sample could be x1.

[0094] In this method, during the search for counterfactual pairs, the search is only performed in adjacent hash buckets, which reduces the search scope and search volume, thereby shortening the search time and improving efficiency.

[0095] In some embodiments, the similarity between the first sample in the first hash bucket and the samples in the second type of hash bucket adjacent to the first hash bucket can be calculated, and then the sample matching the first sample can be determined as the second sample based on the similarity between the first sample and the samples in the second type of hash bucket.

[0096] For example, the similarity between the first sample b1 and the second-class samples corresponding to each hash value in hash bucket 232 can be calculated. Similarly, the similarity between the first sample b1 and the second-class samples corresponding to each hash value in hash bucket 233 can be calculated, thus obtaining the similarity between the first sample and the samples in the second-class hash buckets. In some examples, the similarity between the first sample and the samples in the second-class hash buckets can be calculated based on the following formula:

[0097]

[0098] Where, F(x) i ,b i The similarity score is the score between the first sample and the samples in the second type of hash bucket. The similarity score is negatively correlated with the similarity degree; that is, the lower the similarity score, the higher the similarity. Similarity can be determined based on the similarity score. f(p) is the score calculated for the p-th time between the first sample and the samples in the second type of hash bucket. p (b i ) represents the i-th dimension feature of the first sample corresponding to the p-th calculation; dim p (x i Let be the i-th dimension feature of the sample in the second type of hash bucket corresponding to the p-th calculation.

[0099] It should be noted that the above formula is only one way to determine the similarity level. In other embodiments, other formulas can also be used to determine the similarity level.

[0100] In some embodiments, after determining the similarity between the first sample and the samples in the second type of hash bucket, the similarity can be ranked to obtain the top S samples, where S is a positive integer. Then, any one of the top S samples is used as the second sample to match the first sample, where S is a positive integer, for example, S can be 1, 2, or other values. Preferably, the sample with the highest similarity ranking can be used as the second sample to match the first sample.

[0101] In other embodiments, after determining the similarity between the first sample and samples in the second type of hash bucket, at least one sample whose similarity to the first sample and samples in the second type of hash bucket is greater than a first similarity threshold can be identified. Then, any one of these at least one samples is used as a second sample that matches the first sample. In some examples, the first similarity threshold can be characterized by a similarity score threshold. Since the similarity score is negatively correlated with the similarity, a similarity greater than the first similarity threshold can be achieved by a similarity score less than the similarity score threshold. Thus, samples with similarity scores less than the similarity score threshold can be determined based on the above formula (1), thereby obtaining samples with similarity greater than the first similarity threshold.

[0102] Figure 3 illustrates a sample matching method according to an embodiment of this application. The figure includes a first type of sample 310 and a second type of sample 320. Sample 311 (e.g., the first sample) in the first type of sample 310 matches sample 321 (e.g., the second sample) in the second type of sample 320; sample 312 in the first type of sample 310 matches sample 322 in the second type of sample 320. For ease of description, the successfully matched second type of sample will be denoted as P, and the unmatched second type of sample will be denoted as Q.

[0103] In some embodiments, based on a similar principle, each sample in the first class of samples can be matched with a sample in the second class of samples to obtain min counterfactual pairs. Based on this, dataset P contains min samples, and dataset Q contains max-min samples.

[0104] S104. Based on the characteristic relationship between the first sample and the second sample, process the third sample in the second type of sample to generate the fourth sample.

[0105] After obtaining the second sample that matches the first sample, the feature relationship between the first and second samples can be determined. Then, based on the feature relationship, the third sample in the second category is processed to generate the fourth sample, whose label is consistent with the label of the first category sample.

[0106] In some embodiments, after obtaining the aforementioned min counterfactual pairs, the samples in dataset Q can be processed to transform them into first-class samples. Specifically, firstly, any unmatched Q, such as the third sample, is identified from the second-class hash bucket. Then, from the hash bucket containing the third sample, the sample with the highest similarity to the third sample, such as the first sample, is searched.

[0107] In some embodiments, the sample with the highest similarity to the third sample can be determined based on cosine similarity, specifically, it can be calculated using the following formula:

[0108]

[0109] Where, f(p) i ,q i ) represents sample p in P i With sample q in Q i similarity, dim k (p i ) represents sample p in P i The k-th feature is dim k (q i ) represents the sample q in Q. i The k-th feature is denoted as n, and n is the dimension of the feature.

[0110] In some examples, the similarity between sample p1 (such as the third sample) in P and each sample in Q of the hash bucket containing sample p1 can be determined using the above formula (2), thereby obtaining the samples q that rank among the top M in similarity to p1. i Preferably, the sample q1 (e.g., the first sample) with the highest similarity is determined. Here, M is a positive integer, which can be 1, 2, or other values. In other words, the similarity between the third sample and the first sample ranks among the top M similarities between the third sample and the samples in the second category that match the first category of samples.

[0111] In other examples, the similarity between sample p1 (such as the third sample) in P and each sample in Q of the hash bucket containing sample p1 can be determined using the above formula (2), and samples q with similarity greater than the second similarity threshold can be identified. i From samples q with a similarity greater than the second similarity threshold i We determine sample q1 (e.g., the first sample), which means that the similarity between the third sample in the second class and the first sample is greater than the second similarity threshold.

[0112] Next, based on the characteristic relationship between sample q1 (e.g., the first sample) and sample x1 (e.g., the second sample), sample p1 (e.g., the third sample) is processed to obtain the fourth sample. The label of the newly generated fourth sample is consistent with the label of the first type of sample, thereby increasing the number of first type of samples.

[0113] In some embodiments, based on the feature relationship between the first sample and the second sample, the difference dimension corresponding to the difference feature between the first sample and the second sample can be determined. Then, the feature corresponding to the difference dimension in the third sample of the second class of samples is replaced by the difference feature, thereby generating a fourth sample. The fourth sample has a high similarity to the third sample, and the label of the fourth sample is opposite to the label of the third sample. The label of the fourth sample is the same as the label of the first class of samples, thereby increasing the number of first class of samples.

[0114] As shown in Figure 4, this figure is a schematic diagram of generating a fourth sample according to an embodiment of this application.

[0115] Wherein, sample p i With sample x i For matching counterfactual pairs, such as sample p i It can be the first sample mentioned above, sample x i It can be the second sample mentioned above, sample p i With sample x i The difference features can be dim(k) and dim(k+1), with dimensions corresponding to the k-th and (k+1)-th dimensions, respectively. The sample q is replaced using the difference feature dim(k) of the k-th dimension. i The feature in the k-th dimension is used to replace sample q with the difference feature dim(k+1) in the (k+1)-th dimension. i The features of the (k+1)th dimension are used to obtain the sample n. i , where sample q i It can be a third sample, sample n i It could be the fourth sample.

[0116] It should be noted that the above example only uses two differential features as an example. In other embodiments, the number of differential features can be one or more, such as three or four.

[0117] Based on the above description, this application provides a data balancing method. This method generates a first type of sample, namely, samples of people with diabetes, by using counterfactual pairs, thereby increasing the number of first type samples in the physical examination dataset and achieving data balancing. Specifically, the first and second type samples in the dataset are first hashed and mapped to first and second type hash buckets, respectively. Then, a second sample matching the first sample in the first hash bucket is determined from the second type hash bucket adjacent to the first hash bucket. During the counterfactual pair search process, the search is performed from neighboring hash buckets, narrowing the search range, reducing the search volume, and improving search efficiency. Next, based on the feature relationship between the first and second samples, a third sample in the second type is processed to obtain a fourth sample. The label of this fourth sample is consistent with the label of the first type sample, thereby achieving data balancing. Since the time required for the search process is shortened, the efficiency of data balancing is improved.

[0118] This application also provides a data balancing device, as shown in Figure 5. This figure is a schematic diagram of a data balancing device provided in this application embodiment. The device includes:

[0119] The acquisition module 501 is used to acquire a dataset in which the number of first-class samples is less than the number of second-class samples, and the labels of the first-class samples are different from the labels of the second-class samples.

[0120] The hash module 502 is used to perform hash processing on the dataset, mapping the first type of samples to a first type of hash bucket and the second type of samples to a second type of hash bucket;

[0121] The matching module 503 is used to determine a second sample that matches the first sample in the first hash bucket from a second type of hash bucket that is adjacent to the first hash bucket in the first type of hash bucket;

[0122] The equalization module 504 is used to process the third sample in the second type of samples according to the feature relationship between the first sample and the second sample to generate a fourth sample; the label of the fourth sample is consistent with the label of the first type of sample.

[0123] In some possible implementations, the matching module 503 is specifically used to calculate the similarity between the first sample in the first hash bucket and the sample in the second type of hash bucket adjacent to the first hash bucket; and to determine the sample that matches the first sample as the second sample based on the similarity between the first sample and the sample in the second type of hash bucket.

[0124] In some possible implementations, the matching module 503 is specifically used to determine the top S samples in terms of similarity between the first sample and the samples in the second type of hash bucket;

[0125] Take any one of the top S samples as the second sample to match the first sample, where S is a positive integer; or,

[0126] Identify at least one sample in which the similarity between the first sample and samples in the second type of hash bucket is greater than a first similarity threshold;

[0127] At least one of the samples is used as a second sample that matches the first sample.

[0128] In some possible implementations, the matching module 503 is specifically used to calculate the similarity between a first sample in the first hash bucket and a sample in a second type of hash bucket adjacent to the first hash bucket using the following formula:

[0129]

[0130] Where, F(x) i ,b i ) represents the similarity score between the first sample and samples in the second type of hash bucket, where the similarity score is negatively correlated with similarity; f(p) represents the score calculated for the p-th time between the first sample and samples in the second type of hash bucket; dim p (b i ) represents the i-th dimension feature of the first sample corresponding to the p-th calculation; dim p (x i Let be the i-th dimension feature of the sample in the second type of hash bucket corresponding to the p-th calculation.

[0131] In some possible implementations, the equalization module 504 is specifically used to determine the difference dimension corresponding to the difference feature between the first sample and the second sample based on the feature relationship between the first sample and the second sample; and use the difference feature to replace the feature corresponding to the difference dimension in the third sample of the second type of sample to generate a fourth sample.

[0132] In some possible implementations, the similarity between the third sample in the second type of samples and the first sample is greater than a second similarity threshold; or, the similarity between the third sample and the first sample ranks among the top M in terms of similarity between the third sample and the samples in the second type of samples matched with the first type of samples, where M is a positive integer.

[0133] In some possible implementations, the dataset is a physical examination dataset, where the first type of samples is labeled as having the disease and the second type of samples is labeled as not having the disease.

[0134] This application provides a computing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method as described in any of the above method embodiments.

[0135] This application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform any of the methods described in the above-described method embodiments.

[0136] This application provides a computer program product that, when run on a computer, causes the computer to perform any of the optional methods described in the above method embodiments.

[0137] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this example, it should be understood that the disclosed processing apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, in each embodiment of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data balancing method, characterized in that, include: A dataset is obtained where the number of samples in the first class is less than the number of samples in the second class, and the labels of the samples in the first class are different from those of the samples in the second class. The dataset is a physical examination dataset, where the samples in the first class are those with the target disease, and the samples in the second class are those without the target disease. A Locality Sensitive Hashing (LSH) algorithm is used to hash the dataset, mapping the samples in the first class to a first-class hash bucket and the samples in the second class to a second-class hash bucket. From the second-class hash buckets adjacent to the first-class hash buckets, a second sample matching the first sample in the first-class hash bucket is determined. Based on the feature relationship between the first and second samples, a difference dimension corresponding to the difference feature between the first and second samples is determined. Using the difference feature, the feature corresponding to the difference dimension in the third sample of the second class is replaced to generate a fourth sample. The label of the fourth sample is consistent with the label of the first class samples.

2. The method according to claim 1, characterized in that, The step of determining a second sample that matches a first sample in the first hash bucket from a second type of hash bucket adjacent to the first hash bucket includes: calculating the similarity between the first sample in the first hash bucket and a sample in the second type of hash bucket adjacent to the first hash bucket; and determining the sample that matches the first sample as the second sample based on the similarity between the first sample and the sample in the second type of hash bucket.

3. The method according to claim 2, characterized in that, The step of determining the sample matching the first sample as the second sample based on the similarity between the first sample and samples in the second type of hash bucket includes: determining the top S samples in terms of similarity between the first sample and samples in the second type of hash bucket; taking any one of the top S samples as the second sample matching the first sample, where S is a positive integer; or, determining at least one sample in which the similarity between the first sample and samples in the second type of hash bucket is greater than a first similarity threshold; taking any one of the at least one sample as the second sample matching the first sample.

4. The method according to claim 2, characterized in that, The calculation of the similarity between a first sample in the first hash bucket and a sample in a second type of hash bucket adjacent to the first hash bucket includes: in, The similarity score is the similarity score between the first sample and the samples in the second type of hash bucket, and the similarity score is negatively correlated with the similarity. The score calculated for the p-th time for the first sample and the samples in the second type of hash bucket; The i-th dimension feature of the first sample is calculated for the p-th calculation; Let i be the i-th dimension feature of the sample in the second type of hash bucket corresponding to the p-th calculation.

5. The method according to claim 1, characterized in that, The similarity between the third sample in the second type of samples and the first sample is greater than the second similarity threshold; or, the similarity between the third sample and the first sample ranks among the top M in the similarity between the third sample and the samples in the second type of samples matched with the first type of samples, where M is a positive integer.

6. The method according to any one of claims 1-5, characterized in that, The dataset is a physical examination dataset, where the first type of samples is labeled as having the disease, and the second type of samples is labeled as not having the disease.

7. A data equalization device, characterized in that, include: The acquisition module is used to acquire a dataset in which the number of first-class samples is less than the number of second-class samples, the labels of the first-class samples are different from the labels of the second-class samples, the dataset is a physical examination dataset, the first-class samples are samples with the target disease, and the second-class samples are samples without the target disease; The hash module is used to perform hash processing on the dataset using the locality-sensitive hashing algorithm, mapping the first type of samples to a first type of hash bucket and the second type of samples to a second type of hash bucket; A matching module is used to determine a second sample that matches the first sample in the first hash bucket from a second type of hash bucket that is adjacent to the first hash bucket in the first type of hash bucket; The balancing module is used to determine the difference dimension corresponding to the difference feature between the first sample and the second sample based on the feature relationship between the first sample and the second sample; using the difference feature, the feature corresponding to the difference dimension in the third sample in the second type of sample is replaced to generate a fourth sample; the label of the fourth sample is consistent with the label of the first type of sample.

8. A computing device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 6.

Images