A recursive sampling method and device for dynamically adjusting a sampling rate, equipment and medium

Abstract

The application relates to a recursive sampling method and device for dynamically adjusting a sampling rate, equipment and a medium, wherein the method comprises the following steps: obtaining an original data set and performing pretreatment, identifying a keyword, and calculating the distribution density of the keyword in samples; keyword density interval adaptive screening: dividing the sample interval that hits the keyword into density intervals, and adaptively screening the intervals according to the comparison result of the proportion of negative samples in each density interval and a threshold value; sampling the samples after the keyword density interval adaptive screening; judging whether the number of the samples after sampling is greater than a target sampling number, if yes, recursively performing the keyword density interval adaptive screening step; otherwise, randomly sampling from the samples after the keyword density interval adaptive screening in the current round to obtain the target sampling number of samples. Compared with the prior art, the application has the advantages of being capable of balancing the positive and negative sample proportions and the difficult example proportion, dynamically adjusting the sampling process and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data sampling, and in particular to a recursive sampling method and device based on sample keyword density dynamic adjustment of sampling rate, equipment and medium. BACKGROUND

[0002] When training a multi-label text detection model, a certain amount of label-related data needs to be collected as training data. The source of the training data is through keyword table hitting the text in the Internet. If the original data file contains content matching the keywords in the keyword table, it will be hit and recorded, and the data source and the hit density of the keywords (i.e. the proportion of the number of keywords hit in the entire text) will also be recorded. In application, samples with one or more negative labels are called "negative samples", and samples without any labels are called "positive samples". When training a multi-label text detection model, the existing technology mainly has the following defects:

[0003] 1. Unbalanced ratio of positive and negative samples. The existing model is difficult to effectively capture a sufficient proportion of negative samples in the data set obtained by sampling in the face of the problem of diversified data set sources and sparse keyword distribution. This defect will lead to an uncoordinated ratio of positive and negative samples in the data set used for training, and the model lacks negative samples for learning the underlying detection logic, affecting the model training and detection effect.

[0004] 2. Insufficient proportion of difficult examples. A large number of data hitting keywords are from a single related website, which will further lead to a high proportion of simple cases in the training set, lack of ambiguous "difficult examples", and make the model unable to learn the underlying internal logic of classification, affecting the model training efficiency and accuracy.

[0005] 3. Insufficient sample sources. Due to its own characteristics, a large number of samples are naturally obtained from these websites. If the traditional sampling method is used for random sampling, a large number of samples will come from these data sources, and it is impossible to fully sample samples from different data sources and different keyword hit density intervals. Even if the naive resampling method is used to increase the number of negative samples to be comparable to the number of positive samples, it will affect the multi-source nature of the data set, and the overall quality of the data set will have limited improvement effect, which will also affect the model training and detection effect.

[0006] In summary, the existing multi-label text detection model usually relies on a fixed sampling strategy and cannot dynamically adjust the sampling weight according to the characteristics of the data source and the local density of the keyword, resulting in an imbalance between the positive and negative samples in the sample collection process. At the same time, the easy-to-know random sampling cannot collect difficult examples specifically and cannot make the model learn the underlying paradigm efficiently, affecting the model training effect and the performance of the model in handling difficult cases. Although some models introduce data enhancement and sampling balancing techniques, they will significantly increase the required computing power and sampling time complexity, and have not been finely adjusted in combination with the keyword density and the characteristics of different data sources.

[0007] The currently commonly used negative sample dynamic sampling method DNS (Dynamic Negative Sampling) mainly solves the problem of balancing the sampling probability between high-frequency words and low-frequency words in the traditional negative sampling method. In word vector models such as Word2Vec, it is usually necessary to select positive sample words and negative sample words from the context for training, and the sampling probability in the traditional negative sampling method is fixed, which makes it difficult to handle the trade-off relationship between high-frequency words and low-frequency words during training. The DNS method dynamically adjusts the negative sampling probability, so that low-frequency words are more likely to be selected as negative samples, while high-frequency words are relatively less likely to be selected, thereby better balancing the importance of different words in training. This dynamic adjustment method can improve the learning effect of low-frequency words by the model, while avoiding the excessive influence of high-frequency words on training, thereby improving the training effect and the quality of the word vector. However, this method still has the following problems: (1) the sampling of negative samples is insufficient; (2) the proportion of difficult examples is insufficient; (3) the sampling rate is difficult to remain consistent in multiple data sources: the difference in data distribution between different data sources will have a greater impact on the sampling result; (4) the sampling process is fixed: unable to dynamically adjust according to the keyword distribution, unable to explore the relationship between the data source and the keyword density. SUMMARY

[0008] The purpose of the present application is to overcome the defects of the prior art and provide a recursive sampling method, device, equipment and medium based on dynamic adjustment of sampling rate according to sample keyword density, which realizes a more intelligent and dynamic sampling process without consuming a large amount of computing power, improves the quality of the data set used to train the model, and improves the generalization ability of the model.

[0009] The purpose of the present application can be achieved by the following technical solutions:

[0010] According to a first aspect of the present application, a recursive sampling method based on dynamic adjustment of sampling rate according to sample keyword density is provided, which comprises the following steps:

[0011] Keyword density calculation and data preprocessing: obtain the original data set and preprocess it, identify the keywords, and calculate the distribution density of the keywords in the sample;

[0012] Keyword density interval adaptive screening: divide the sample interval that hits the keyword into density intervals, and adaptively screen the interval according to the comparison result of the proportion of negative samples in each density interval and the threshold value;

[0013] Data source balanced sampling: by presetting and gradually reducing the maximum sample number threshold in each data source, sampling the samples after keyword density interval adaptive screening until the remaining sample number is the product of the total number of samples after keyword density interval adaptive screening and the preset retention rate;

[0014] Recursive adjustment sampling: determine whether the sample number after data source balanced sampling is greater than the target sample number. If the sample number is greater than the target sample number, recursively go to the keyword density interval adaptive screening step for the next round of density interval division and adaptive screening. In each recursion, the negative sample proportion threshold for screening is reduced. If the sample number is less than or equal to the target sample number, randomly sample from the samples after keyword density interval adaptive screening in this round to obtain the target sample number of samples.

[0015] As a preferred technical solution, the keyword density interval adaptive screening includes the following steps:

[0016] Initialization of uniform sampling: divide the sample interval that hits the keyword into a first preset number of intervals of equal length, and randomly extract a second preset number of samples in each interval without repetition. If the number of samples in the interval is less than the second preset number, discard the density interval directly;

[0017] Statistical keyword density sampling window negative sample proportion: label the extracted samples by machine or manual labeling. Samples labeled with one or more labels are negative samples, and samples without any label are positive samples. Calculate the proportion of negative samples in each density interval, and retain the intervals with a negative sample proportion within a preset percentage threshold range;

[0018] The magnifying glass expands the window sampling: for the interval in which the negative sample proportion is not within the preset percentage threshold range, each density interval is evenly divided into a third preset number of sub-windows with equal length, if the total number of samples in the sub-window is less than the second preset number, the sub-window is expanded to the left and right adjacent sub-windows until the total number of samples reaches the second preset number, the proportion of negative samples in each expanded sub-window is calculated, the sub-window in which the negative sample proportion is within the preset percentage threshold range is retained, and it is ensured that the retained sub-windows do not overlap the density interval, and all retained sub-windows are merged.

[0019] As a preferred technical solution, the preset percentage threshold range is set according to the principle of increasing the sampling weight of the density area in which the positive and negative sample proportions are close to 1:1.

[0020] As a preferred technical solution, the preset percentage threshold range is set to [50%-m, 50%+m], wherein m is a controllable percentage range threshold.

[0021] As a preferred technical solution, the initial value of the controllable percentage range threshold m is set according to business requirements and data conditions.

[0022] As a preferred technical solution, in the process of recursive sampling, the new controllable percentage range threshold in each round of update is b times the controllable percentage range threshold in the previous round, 0<b<1, and the interval of the selected negative sample proportion is continuously reduced.

[0023] As a preferred technical solution, in the data source balanced sampling step, the total number of samples of each interval obtained after keyword density interval screening is recorded as sum, the maximum sample number in the current data source is taken as the initial value of the sample number threshold max_length, and the preset retention rate is r. In the sampling process, the sampling proportion of different data sources is adjusted based on the histogram height limit, that is, the data is updated by continuously reducing the sample number threshold max_length to ensure that the sample number of each data source is less than or equal to max_length, until the total number of retained samples reaches r×sum.

[0024] According to the second aspect of the present application, a recursive sampling device based on dynamic adjustment of sampling rate of sample keywords density is provided, comprising:

[0025] The keyword density calculation and data preprocessing module: acquires the original data set and performs preprocessing, identifies keywords, and calculates the distribution density of the keywords in the samples;

[0026] The keyword density interval self-adaptive screening module: divides the sample interval that hits the keyword into density intervals, and performs self-adaptive screening on the interval according to the comparison result of the proportion of negative samples in each density interval and the threshold.

[0027] data source balanced sampling module: by presetting and gradually reducing the maximum sample quantity threshold in each data source, the samples after the keyword density interval adaptive screening are sampled until the remaining sample quantity is the product of the total sample quantity after the keyword density interval adaptive screening and the preset retention rate;

[0028] recursive adjustment sampling module: judging whether the sample quantity after the data source balanced sampling is greater than the target sample quantity, if the sample quantity is greater than the target sample quantity, recursively going to the keyword density interval adaptive screening module to perform the next round of density interval division and adaptive screening, wherein the proportion threshold of negative samples used for screening is reduced and adjusted each time; if the sample quantity is less than or equal to the target sample quantity, randomly sampling from the samples after the keyword density interval adaptive screening in this round to obtain the target sample quantity samples.

[0029] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, the memory has a computer program stored thereon, and the processor implements the method when executing the program.

[0030] According to a fourth aspect of the present application, a computer readable storage medium is provided, which has a computer program stored thereon, and the program is executed by a processor to implement the method.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] (1) Balance the proportion of positive and negative samples: the recursive sampling technology adopted by the present application can sample a balanced data set of positive and negative sample proportions in a data source with uneven distribution of positive and negative samples for training the model, ensuring sufficient sampling of positive and negative samples.

[0033] (2) Balance the sampling rate of each data source: the recursive sampling technology adopted by the present application can dynamically adjust the sampling rate between different data sources, so that data in high-frequency data sources and low-frequency data sources can be fully sampled, reducing the side effect of high-frequency data sources having a large proportion under the naive sampling method.

[0034] (3) Improve the proportion of difficult examples: the recursive sampling technology adopted by the present application can collect more difficult-to-distinguish positive and negative samples, ensuring that the sampling process pays attention to all keyword hit density samples, improving the model's ability to recognize weak signal samples, especially difficult examples, providing data support for the model to learn and induce the underlying logic of different labels from the training set.

[0035] (4) Dynamic adjustment of sampling process: the recursive sampling technology adopted by the present application can dynamically adjust the sampling proportion of different data sources and the sampling proportion of different keyword hit densities. This mechanism can reduce the impact of the keyword density distribution of a specific data source on the sampling process and effectively avoid sample bias.

[0036] (5) Balance sampling diversity and time complexity: the recursive sampling technology adopted by the present application can ensure the diversity of the sampling results without causing a sharp rise in sampling time complexity.

[0037] In summary, the present application has the advantages of improving the quality of the model training set, which is significantly better than the best prior art. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of the method of the present application;

[0039] Figure 2 is a flowchart of the keyword density interval adaptive screening. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0041] The present application is based on the naive sampling method of keyword hit density. To solve the problem that the existing method cannot fully utilize the underlying correlation between the sampling points and the data sources and the keyword hit density, an innovative adaptive sampling method is proposed. It is a recursive sampling technology that can dynamically adjust the sampling rate of different keyword hit density intervals and different data sources according to the correlation between the keyword hit density and different data sources, so as to achieve the expected goal of coordinating the positive and negative sample proportions of the collected results, making the sample sources more diverse, and increasing the proportion of difficult cases.

[0042] First, the key contents involved in the present embodiment are defined as follows:

[0043] 1. Recognition model: use an executable deep learning model to score documents. The scores of documents that meet certain common characteristics show clustering characteristics, so they can be distinguished from the score distribution of random documents. For specific business needs, these characteristics can be pornography, gambling, drugs, violence, etc. That is, the model has the ability to recognize specific types of text.

[0044] 2、Key word: The key word is a word or phrase that appears in the corpus that the target category classification model needs to identify, and its frequency is significantly higher than that in other texts. A key word collection (i.e., a key word table) is collected and prepared. The key word can be in the form of a single word or phrase in multiple languages.

[0045] 3、Key word category: The target category to which the key word is strongly associated. For example, the key word in the key word table mentioned above is matched with the text data in the data source. The text that hits the key word may have categories such as pornography, gambling, drugs, violence, etc. Each hit text may contain multiple categories or multiple key words of the same category. This field is stored in the data to be sampled together with the text ID, text content, and other fields.

[0046] 4、Key word density: The ratio of the number of key words in a certain segment of text identified as toxic to the total number of characters in the segment, which is called key word density. This field is also stored in the data to be sampled together with the corresponding text.

[0047] Accordingly, we propose a recursive sampling technique. In the sampling process, the proportion of negative samples in the key word density interval is calculated, and the sampling rate of each data source is dynamically adjusted according to the number of negative samples of different data sources to ensure that the training set sampled can cover more difficult examples and maintain the balance of positive and negative samples under different data sources and key word densities, providing more balanced, diverse, and rich training data for model training.

[0048] Specifically, as shown in Figure 1 , the method comprises the following steps:

[0049] S1, key word density calculation and data preprocessing: obtaining and preprocessing the original data set, identifying the key word, and calculating the distribution density of the key word in the sample.

[0050] By preprocessing the original data set, identifying and labeling the key word and its category, and calculating its distribution density in the sample, the number of each sample, the text content, the key word label category, the key word density, and the data source are stored in the same json object. Assuming that the number of labels of the multi-label model is n, in the ideal training data set, the proportion of positive samples and negative samples with different labels is balanced, i.e., 1:1:...:1.

[0051] S2, key word density interval adaptive screening: dividing the key word hit sample interval into density intervals, and adaptively screening the interval according to the comparison result of the proportion of negative samples in each density interval and the threshold.

[0052] As shown in Figure 2 , the key word density interval adaptive screening comprises the following steps:

[0053] S21, initialization of uniform sampling: evenly divide the sample interval that hits the keyword into x intervals of equal length, randomly select a samples in each interval without repetition, and discard the density interval if the number of samples in the interval is less than a.

[0054] S22, statistic negative sample proportion of keyword density sampling window: label the extracted samples by machine or manual labeling, the samples labeled with one or more labels are negative samples, and the samples without any label are positive samples. Calculate the proportion f of negative samples in each density interval i (where i is the interval number, 1≤i≤x), and keep the interval with a negative sample proportion between [50%-m, 50%+m], where m is a controllable percentage range threshold. The initial value of the controllable percentage range threshold m is set according to business requirements and data conditions, which is 20% in this embodiment. In the process of recursive sampling, the new controllable percentage range threshold is 0.9 times the controllable percentage range threshold of the previous round in each round of update, and the interval of the selected negative sample proportion is constantly reduced.

[0055] By increasing the sampling weight of the keyword density area corresponding to the positive and negative sample proportion close to 1:1, the proportion of positive samples and different types of negative samples in the data obtained by sampling is more balanced. At this time, most of the negative samples of each type belong to the ambiguous type, so there is a higher proportion of difficult examples.

[0056] S23, magnifying glass inflation window sampling: for intervals with a negative sample proportion not between [50%-m, 50%+m], evenly divide each density interval into y sub-windows of equal length, and if the total number of samples in a sub-window is less than a, expand and expand the left and right adjacent sub-windows until the total number of samples reaches a. Calculate the proportion p of negative samples in each expanded sub-window i (where i is the sliding window number, 1≤i≤y), keep the sub-window with a negative sample proportion between [50%-m, 50%+m], and ensure that the retained sub-windows do not overlap the density interval. Merge all retained sub-windows.

[0057] S3, data source balanced sampling: by presetting and gradually reducing the maximum sample number threshold in each data source, sample the samples after keyword density interval adaptive screening until the remaining sample number is the product of the total number of samples after keyword density interval adaptive screening and the preset retention rate.

[0058] Let the total number of samples in each interval after keyword density interval screening be sum, and assume that there are q data sources, then sum=n1+n2+...+n i +...+n q , n iThe sample quantity of the i-th data source is represented. The maximum sample quantity in the current data source is taken as the initial value of the sample quantity threshold max_length, that is, the initial value of max_length is set to n1-n q The maximum value in n1-n

[0059] The preset retention rate is r. In the sampling process, the sampling proportion of different data sources is adjusted based on the highest limit of the histogram, that is, the data is updated by continuously reducing the sample quantity threshold max_length (max_length can be reduced by 1 each time), when n i > max_length, only max_length samples in the data source i are randomly extracted and reserved, so that the sample quantity of each data source is less than or equal to max_length, until the total number of retained samples reaches r×sum.

[0060] S4, recursive adjustment of sampling: it is judged whether the sample quantity after the data source balanced sampling is greater than the target sample quantity, if the sample quantity is greater than the target sample quantity, recursion is performed to the keyword density interval adaptive screening step, the next round of density interval division and adaptive screening is performed, wherein the proportion threshold of negative samples for screening is reduced each time; if the sample quantity is less than or equal to the target sample quantity, random sampling is performed from the samples after the keyword density interval adaptive screening in this round, and the target sample quantity samples are obtained.

[0061] The main innovation of the present application is:

[0062] The keyword density window sampling rate automatic adjustment mechanism: the negative sample proportion in the local interval of the keyword density in the sample is calculated, and the sampling interval is dynamically adjusted. Through the magnifying glass expansion step, the negative sample proportion of the reserved keyword density interval is finely adjusted to meet the set range.

[0063] Multi-source data sampling rate automatic adjustment mechanism: the data source data quantity histogram is used, the maximum sample quantity of the data source is controlled, the proportion of high-frequency data sources in the sample is inhibited, and the target sample quantity is accurately collected.

[0064] Recursive adjustment mechanism: the window quantity of the average sampling is proportional to the retained sample, so as to ensure the granularity consistency of the sampling window. The user can adjust the parameter to control the recursion times, and the sampling strategy has high flexibility.

[0065] Through experiments on a multi-language toxicity detection model, the recursive sampling technology of the present application effectively improves the richness and difficult example proportion of the training data set, so that the quality of the training data is higher, and the model trained has 10-20% improvement in precision, recall and other aspects. Especially in the context of the Internet data, the negative sample density is relatively sparse, and the sample data source is more diverse. Compared with the model trained by the previously obtained data set through naive random sampling, the overall detection accuracy of the model trained by the data obtained by the sampling method of the present application is higher.

[0066] In addition to being applicable to training multi-label text detection models, the present application can also be used in the training data sampling process of multi-language, multi-modal multi-label detection models. It can solve most of the potential problems of unbalanced positive and negative sample ratio, inconsistent sampling rate in multiple data sources, and fixed sampling process at a low computing power cost, providing help and support for obtaining higher quality training sets.

[0067] The above is an introduction to the method embodiment. The following describes the scheme of the present application through a device embodiment.

[0068] The present embodiment provides a recursive sampling device based on dynamic adjustment of sampling rate according to sample keyword density, which is used to implement the above method. The device comprises:

[0069] A keyword density calculation and data preprocessing module: acquires the original data set and performs preprocessing, identifies keywords, and calculates the distribution density of the keywords in the sample;

[0070] A keyword density interval adaptive screening module: divides the sample interval that hits the keyword into density intervals, and performs adaptive screening on the interval according to the comparison result of the proportion of negative samples in each density interval and the threshold value;

[0071] A data source balanced sampling module: by presetting and gradually reducing the maximum sample number threshold in each data source, samples the samples that have passed the keyword density interval adaptive screening until the remaining sample number is the product of the total number of samples that have passed the keyword density interval adaptive screening and the preset retention rate;

[0072] A recursive adjustment sampling module: determines whether the sample number after data source balanced sampling is greater than the target sampling number. If the sample number is greater than the target sampling number, recursively go to the keyword density interval adaptive screening module for the next round of density interval division and adaptive screening. In each recursion, the negative sample proportion threshold used for screening is adjusted. If the sample number is less than or equal to the target sampling number, randomly sample from the samples that have passed the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0074] The electronic device of the present application includes a central processing unit (CPU) that can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0075] A plurality of components in the device are connected to the I / O interface, including: an input unit such as a keyboard, a mouse, etc.; an output unit such as various types of displays, a speaker, etc.; a storage unit such as a magnetic disk, an optical disk, etc.; and a communication unit such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0076] The processing unit performs the various methods and processes described above, such as methods S1-S4. For example, in some embodiments, methods S1-S4 can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via the ROM and / or the communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1-S4 described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform methods S1-S4 by any other appropriate means (e.g., by means of firmware).

[0077] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0078] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowchart diagrams and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0079] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0080] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density, characterized in that: The method comprises the following steps: Keyword density calculation and data preprocessing: Obtain the original data set and preprocess it, identify keywords, and calculate the distribution density of keywords in the sample; Keyword density interval adaptive screening: The sample interval of the hit keyword is divided into density intervals, and the intervals are adaptively screened based on the comparison result of the proportion of negative samples in each density interval and the threshold; Data source balanced sampling: By presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after the keyword density interval adaptive screening, until the remaining number of samples is the product of the total number of samples after the keyword density interval adaptive screening and the preset retention rate; Recursively adjust sampling: Determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening step to perform the next round of density interval division and adaptive screening. Each recursion adjusts the proportion threshold of negative samples used for screening; if the number of samples is less than or equal to the target sampling number, random sampling is performed from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

2. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 1, characterized in that: The keyword density interval adaptive screening includes the following steps: Initialize uniform sampling: divide the sample interval of the keyword hit into a first preset number of intervals of equal length, randomly extract a second preset number of samples in each interval without duplication, and directly discard the density interval if the number of samples in the interval is less than the second preset number; Count the proportion of negative samples in the keyword density sampling window: Classify the extracted samples by labeling them through machine or manual labeling. Samples with one or more labels are negative samples, and samples without any labels are positive samples. Calculate the proportion of negative samples in each density interval and retain the intervals where the negative sample ratio falls within the preset percentage threshold. Magnifying glass expansion window sampling: For intervals where the proportion of negative samples is not within the preset percentage threshold range, each density interval is evenly divided into a third preset number of sub-windows of equal length. If the total number of samples in the sub-window is less than the second preset number, it is expanded to the left and right adjacent sub-windows until the total number of samples reaches the second preset number. The proportion of negative samples in each expanded sub-window is calculated, and the sub-windows whose negative sample proportion is within the preset percentage threshold range are retained. It is ensured that the retained sub-windows have no overlapping density intervals, and all retained sub-windows are merged.

3. The recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 2, characterized in that: The preset percentage threshold range is set based on the principle of increasing the sampling weight for density areas where the ratio of positive to negative samples is close to 1:

1.

4. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 3, characterized in that: The preset percentage threshold range is set to [50%-m, 50%+m], where m is a controllable percentage range threshold.

5. The recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 4, characterized in that: The initial value of the controllable percentage range threshold m is set according to business requirements and data conditions.

6. The recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 4, characterized in that: During the recursive sampling process, the new controllable percentage range threshold for each round of update is b times the controllable percentage range threshold of the previous round, where 0 < b < 1, and the interval of the negative sample ratio selected is continuously narrowed.

7. The recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 1, characterized in that: In the data source balanced sampling step, the total number of samples in each interval obtained after screening by the keyword density interval is denoted as sum, the maximum number of samples in the current data source is used as the initial value of the sample number threshold max_length, and the preset retention rate is r. During the sampling process, the sampling ratio of different data sources is adjusted based on the highest limit of the histogram, that is, the sample number threshold max_length is continuously reduced to update the data, ensuring that the number of samples in each data source is less than or equal to max_length until the total number of remaining samples reaches r × sum.

8. A recursive sampling device that dynamically adjusts the sampling rate based on sample keyword density, characterized in that: Including: Keyword density calculation and data preprocessing module: Obtain the original data set and perform preprocessing, identify keywords, and calculate the distribution density of keywords in the samples. Keyword density interval adaptive screening module: Divide the density intervals of the sample intervals where keywords are hit, and adaptively screen the intervals according to the comparison results of the ratio of negative samples in each density interval and the threshold. Data source balanced sampling module: Sample the samples after keyword density interval adaptive screening by presetting and gradually reducing the maximum sample number threshold in each data source until the remaining number of samples is the product of the total number of samples after keyword density interval adaptive screening and the preset retention rate. Recursive adjustment sampling module: Determine whether the number of samples after data source balanced sampling is greater than the target sampling number. If the number of samples is greater than the target sampling number, recursively go to the keyword density interval adaptive screening module for the next round of density interval division and adaptive screening. Among them, the ratio threshold of negative samples used for screening is reduced and adjusted each time of recursion. If the number of samples is less than or equal to the target sampling number, randomly sample from the samples after keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the method described in any one of claims 1 to 7 is implemented.

Images