Recursive sampling method and device for dynamically adjusting sampling rate, equipment and medium

Abstract

The invention relates to a recursive sampling method and device for dynamically adjusting a sampling rate, equipment and a medium, and the method comprises the following steps: obtaining an original data set, carrying out the preprocessing, recognizing keywords, and calculating the distribution density of the keywords in a sample; adaptive screening of keyword density intervals: carrying out density interval division on sample intervals hitting keywords, and carrying out adaptive screening on the intervals according to a comparison result of the proportion of negative samples in each density interval and a threshold value; sampling is carried out on the samples subjected to keyword density interval self-adaptive screening; judging whether the number of samples after sampling is greater than a target sampling number or not, and if so, recurding to a keyword density interval adaptive screening step; otherwise, random sampling is conducted on the samples subjected to keyword density interval self-adaptive screening in the current round, and the samples of the target sampling number are obtained. Compared with the prior art, the method has the advantages that the positive and negative sample proportion and the difficult sample proportion can be balanced, and the sampling process is dynamically adjusted.

Description

Technical Field

[0001] The present invention relates to the field of data sampling technology, and in particular to a recursive sampling method, device, equipment and medium for dynamically adjusting a sampling rate based on sample keyword density. Background Art

[0002] When training a multi-label text detection model, a certain amount of label-related data needs to be collected as training data. The source of the training data is the text on the Internet hit by the keyword table. If the original data file contains content that matches the keywords in the keyword table, it will be hit and recorded. At the same time, the data source and the hit density of the keywords will be recorded and calculated (that is, the proportion of the number of hit keywords in the entire text). In the application, samples that are marked with one or more negative labels are called "negative samples", and those without any labels are called "positive samples". When training a multi-label text detection model, the existing technology mainly has the following defects:

[0003] 1. Imbalance in the ratio of positive and negative samples. When facing the problem of diverse data sets and sparse keyword distribution, existing models find it difficult to effectively capture a sufficient proportion of negative samples in the sampled data sets. This defect will lead to an imbalance in the ratio of positive and negative samples in the training data sets, and the model lacks negative samples for learning the underlying detection logic, which affects the model training and detection effects.

[0004] 2. Insufficient proportion of difficult examples. A large amount of data of hit keywords comes from a single related website, which will further lead to a high proportion of simple cases in the training set. The lack of ambiguous "difficult examples" makes it impossible for the model to learn the underlying internal logic of classification, affecting the efficiency and accuracy of model training.

[0005] 3. The sample sources are not rich enough. Due to their own characteristics, some data sources naturally have a large number of samples collected from these websites. If traditional sampling methods are used for random sampling, a large number of samples will come from these data sources, and it will not be possible to fully sample samples from different data sources and different keyword hit density intervals. Even if a simple resampling method is used to increase the number of negative samples to the same level as positive samples, the multi-source nature of the data set will be affected, which will have limited effect on improving the overall quality of the data set and will also affect the model training and detection results.

[0006] In summary, existing multi-label text detection models usually rely on fixed sampling strategies and cannot dynamically adjust sampling weights according to the characteristics of the data source and the local density of keywords, resulting in an imbalance in the ratio of positive and negative samples during sample collection. At the same time, it is easy to see that random sampling cannot collect difficult examples in a targeted manner, and cannot enable the model to efficiently learn the underlying paradigm, affecting the model training effect and the performance of the model in handling difficult cases. Although some models have introduced data enhancement and sampling balance techniques, they will significantly increase the required computing power and sampling time complexity, and have not been fine-tuned in combination with keyword density and the characteristics of different data sources.

[0007] The commonly used negative sample dynamic sampling method DNS (Dynamic Negative Sampling) mainly solves the problem of difficulty in balancing the sampling probabilities between high-frequency words and low-frequency words in traditional negative sampling methods. In word vector models such as Word2Vec, it is usually necessary to select positive sample words and negative sample words from the context for training. However, the sampling probability in traditional negative sampling methods is fixed, which makes it difficult to handle the trade-off relationship between high-frequency words and low-frequency words well during training. The DNS method dynamically adjusts the negative sampling probability, making low-frequency words more likely to be selected as negative samples, while high-frequency words are relatively less likely to be selected, thereby better balancing the importance of different words in training. This dynamic adjustment method can improve the model's learning effect on low-frequency words, while avoiding the excessive influence of high-frequency words on training, thereby improving the training effect and the quality of word vectors. However, this method still has the following problems: (1) the sampling of negative samples is insufficient; (2) the proportion of difficult examples is insufficient; (3) the sampling rate is difficult to maintain consistency among multiple data sources: the difference in data distribution between different data sources will have a significant impact on the sampling results; (4) the sampling process is fixed: it fails to dynamically adjust according to the keyword distribution and fails to explore the relationship between the data source and the keyword density. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a recursive sampling method, device, equipment and medium for dynamically adjusting the sampling rate based on the sample keyword density, so as to realize a more intelligent and dynamically adjusted sampling process without consuming a lot of computing power, improve the quality of the data set used to train the model, and improve the generalization ability of the model.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] According to a first aspect of the present invention, there is provided a recursive sampling method for dynamically adjusting a sampling rate based on sample keyword density, the method comprising the following steps:

[0011] Keyword density calculation and data preprocessing: obtain the original data set and preprocess it, identify keywords, and calculate the distribution density of keywords in the sample;

[0012] Adaptive screening of keyword density intervals: The sample intervals of the hit keywords are divided into density intervals, and the intervals are adaptively screened based on the comparison between the proportion of negative samples in each density interval and the threshold;

[0013] Data source balanced sampling: by presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after the keyword density interval adaptive screening, until the remaining number of samples is the product of the total number of samples after the keyword density interval adaptive screening and the preset retention rate;

[0014] Recursively adjust sampling: Determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening step to perform the next round of density interval division and adaptive screening. In each recursion, the proportion threshold of negative samples used for screening is reduced and adjusted; if the number of samples is less than or equal to the target sampling number, random sampling is performed from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

[0015] As a preferred technical solution, the keyword density interval adaptive screening includes the following steps:

[0016] Initialize uniform sampling: divide the sample interval of the hit keyword evenly into a first preset number of intervals of equal length, randomly extract a second preset number of samples in each interval without repetition, and directly discard the density interval if the number of samples in the interval is less than the second preset number;

[0017] Count the proportion of negative samples in the keyword density sampling window: classify the extracted samples by labeling them through machine or manual labeling. Samples with one or more labels are negative samples, and samples without any labels are positive samples. Calculate the proportion of negative samples in each density interval, and retain the intervals in which the proportion of negative samples in the density interval is within the preset percentage threshold range.

[0018] Magnifying glass expansion window sampling: For intervals where the negative sample ratio is not within the preset percentage threshold range, each density interval is evenly divided into a third preset number of sub-windows of equal length. If the total number of samples in a sub-window is less than the second preset number, it expands to the adjacent left and right sub-windows until the total number of samples reaches the second preset number. Calculate the negative sample ratio in each expanded sub-window, retain the sub-windows with negative sample ratios within the preset percentage threshold range, and ensure that the retained sub-windows have no overlapping density intervals. Merge all the retained sub-windows.

[0019] As a preferred technical solution, the preset percentage threshold range is set according to the principle of increasing the sampling weight for density regions where the positive and negative sample ratios are close to 1:1.

[0020] As a preferred technical solution, the preset percentage threshold range is set to [50% - m, 50% + m], where m is a controllable percentage range threshold.

[0021] As a preferred technical solution, the initial value of the controllable percentage range threshold m is set according to business requirements and data conditions.

[0022] As a preferred technical solution, during the recursive sampling process, the new controllable percentage range threshold for each round of update is b times the controllable percentage range threshold of the previous round, where 0 < b < 1, continuously narrowing the interval of the selected negative sample ratio.

[0023] As a preferred technical solution, in the data source balance sampling step, the total number of samples in each interval obtained after screening by keyword density interval is denoted as sum, the maximum number of samples in the current data source is used as the initial value of the sample quantity threshold max_length, and the preset retention rate is r. During the sampling process, the sampling ratio of different data sources is adjusted based on the highest limit of the histogram, that is, by continuously reducing the sample quantity threshold max_length to update the data, ensuring that the number of samples in each data source is less than or equal to max_length until the total number of retained samples reaches r × sum.

[0024] According to the second aspect of the present invention, a recursive sampling device for dynamically adjusting the sampling rate based on the keyword density of samples is provided, including:

[0025] Keyword density calculation and data preprocessing module: Obtain the original data set and perform preprocessing, identify keywords, and calculate the distribution density of keywords in the samples;

[0026] Keyword density interval adaptive screening module: Divide the sample intervals where keywords are hit into density intervals, and adaptively screen the intervals according to the comparison results of the negative sample ratio and the threshold in each density interval;

[0027] Data source balanced sampling module: by presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after the keyword density interval adaptive screening, until the remaining number of samples is the product of the total number of samples after the keyword density interval adaptive screening and the preset retention rate;

[0028] Recursively adjust the sampling module: determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening module to perform the next round of density interval division and adaptive screening. In each recursion, the proportion threshold of negative samples used for screening is reduced and adjusted; if the number of samples is less than or equal to the target sampling number, random sampling is performed from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

[0029] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0030] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) Balancing the ratio of positive and negative samples: The recursive sampling technology adopted in the present invention can sample a data set with a balanced ratio of positive and negative samples from a data source with an uneven distribution of positive and negative samples to obtain a data set for training the model, thereby ensuring sufficient sampling of positive and negative samples.

[0033] (2) Balancing the sampling rates of various data sources: The recursive sampling technology adopted in the present invention can dynamically adjust the sampling rates between different data sources, so that the data in both high-frequency data sources and low-frequency data sources can be fully sampled, thereby reducing the side effect of excessively large proportion of high-frequency data sources under naive sampling methods.

[0034] (3) Improving the proportion of difficult examples: The recursive sampling technology used in the present invention can collect more positive and negative samples that are difficult to distinguish, ensuring that the sampling process pays attention to samples with all keyword hit densities, thereby improving the model's ability to recognize weak signal samples, especially difficult examples, and providing data support for the model to learn from the training set to summarize the underlying logic of different labels.

[0035] (4) Dynamic adjustment of the sampling process: The recursive sampling technology adopted by the present invention can dynamically adjust the sampling ratios of different data sources and the sampling ratios of different keyword hit densities. This mechanism can reduce the impact of the keyword density distribution of a specific data source on the sampling process and effectively avoid sample bias.

[0036] (5) Balancing sampling diversity and time complexity: The recursive sampling technology adopted in the present invention can ensure the diversity of the sampling results without causing a sharp increase in the sampling time complexity.

[0037] Taking the above advantages into consideration, the present invention has a significant effect on improving the quality of model training sets compared with the best existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 Flowchart for adaptive screening of keyword density intervals. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] Based on the naive sampling method based only on keyword hit density, the present invention proposes an innovative adaptive sampling method to address the problem that existing methods cannot fully utilize the data sources corresponding to sampling points and the underlying associations of keyword hit density. The method is a recursive sampling technology that can dynamically adjust different keyword hit density intervals and sampling rates of different data sources according to the relationship between keyword hit density and different data sources, so as to achieve the desired goals of coordinating the ratio of positive and negative samples in the collection results, making the sample sources more diverse, and having a higher proportion of difficult examples.

[0042] First, this embodiment defines the key contents involved as follows:

[0043] 1. Recognition model: Use an executable deep learning model to score documents. The scores of documents that meet certain common characteristics show clustering characteristics, which can be distinguished from the score distribution of random documents. For specific business needs, these characteristics can be pornography, gambling, drugs, violence, etc., that is, the model has the ability to recognize specific categories of text.

[0044] 2. Keywords: In the corpus that the classification model of the target category needs to identify, the frequency of keywords appearing is significantly higher than the frequency of appearing in other texts. Collect and compile a collection of keywords (i.e., keyword table). Keywords are presented in the form of words or phrases in multiple languages.

[0045] 3. Keyword category: target category with strong association with keywords. For example, the keywords in the keyword table mentioned above are matched with the text data in the data source. The texts with the hit keywords may have categories such as pornography, gambling, drugs, violence, etc. Each hit text may contain multiple categories or multiple keywords of the same category. This field will be stored in the data to be sampled together with fields such as text ID and text content.

[0046] 4. Keyword density: The ratio of the number of keywords to the total number of characters in a certain paragraph of text that is identified as toxic is recorded as keyword density. This field will also be stored in the data to be sampled with the corresponding text.

[0047] Based on this, we proposed a recursive sampling technology. During the sampling process, this technology calculates the proportion of negative samples within the keyword density range, and dynamically adjusts the sampling rate of each data source according to the number of negative samples from different data sources to ensure that the sampled training set can cover more difficult examples and maintain a balance between positive and negative samples under different data sources and keyword densities, providing more balanced, diverse, and rich training data for model training.

[0048] Specifically, Figure 1 As shown, the method comprises the following steps:

[0049] S1, keyword density calculation and data preprocessing: obtain the original data set and preprocess it, identify keywords, and calculate the distribution density of keywords in the sample.

[0050] By preprocessing the original data set, identifying and annotating keywords and their categories, calculating their distribution density in the samples, the number, text content, keyword label category, keyword density and data source of each sample are stored in the same json object. Assuming that the number of labels of the multi-label model is n, in the ideal training data set, the ratio of positive samples to negative samples with different labels is balanced, which is 1:1:...:1.

[0051] S2, keyword density interval adaptive screening: the sample interval of the hit keyword is divided into density intervals, and the intervals are adaptively screened based on the comparison result of the proportion of negative samples in each density interval and the threshold.

[0052] like Figure 2 As shown, the keyword density interval adaptive screening includes the following steps:

[0053] S21, initialize uniform sampling: divide the sample interval of the hit keyword into x intervals of equal length evenly, randomly select a samples without repetition in each interval, and directly discard the density interval if the number of samples in the interval is less than a.

[0054] S22, count the proportion of negative samples in the keyword density sampling window: classify the extracted samples by labeling them through machine or manual labeling. Samples with one or more labels are negative samples, and samples without any labels are positive samples. Calculate the proportion of negative samples in each density interval f i (where i is the interval number, 1≤i≤x), the negative sample ratio in the reserved density interval is between [50%-m, 50%+m], where m is the controllable percentage range threshold. The initial value of the controllable percentage range threshold m is set according to business needs and data conditions, and is set to 20% in this embodiment. In the process of recursive sampling, each round of updating takes a new controllable percentage range threshold as 0.9 times the controllable percentage range threshold of the previous round, and continuously narrows the interval of the selected negative sample ratio.

[0055] By increasing the sampling weight for keyword density areas where the positive-negative sample ratio is close to 1:1, the ratio of positive samples to different types of negative samples in the sampled data is more balanced. At this time, most of the negative samples of each type are ambiguous, so there is a higher proportion of difficult examples.

[0056] S23, magnifying glass expansion window sampling: For the interval where the negative sample ratio is not between [50%-m, 50%+m], divide each density interval into y sub-windows of equal length. If the total number of samples in a sub-window is less than a, expand it to the left and right adjacent sub-windows until the total number of samples reaches a. Calculate the proportion of negative samples p in each expanded sub-window i (where i is the sliding window number, 1≤i≤y), retain the sub-windows with negative sample ratios between [50%-m, 50%+m], ensure that the retained sub-windows have no overlapping density intervals, and merge all retained sub-windows.

[0057] S3, balanced sampling of data sources: by presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after adaptive screening of keyword density intervals until the number of remaining samples is the product of the total number of samples after adaptive screening of keyword density intervals and the preset retention rate.

[0058] The total number of samples in each interval obtained after keyword density interval screening is recorded as sum. Assuming there are q data sources, sum = n1 + n2 + ... + n i +...+n q , n iIndicates the number of samples in the i-th data source. The maximum number of samples in the current data source is used as the initial value of the sample number threshold max_length, that is, the initial value of max_length is set to n1-n q The maximum value in .

[0059] The preset retention rate is r. During the sampling process, the sampling ratio of different data sources is adjusted based on the maximum limit of the histogram, that is, the data is updated by continuously reducing the sample quantity threshold max_length (max_length can be reduced by 1 each time it is updated). i > max_length, only max_length samples are randomly selected from data source i and retained, ensuring that the number of samples from each data source is less than or equal to max_length, until the total number of retained samples reaches r×sum.

[0060] S4, recursively adjust sampling: determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening step to perform the next round of density interval division and adaptive screening, wherein each recursion reduces the proportion threshold of negative samples used for screening; if the number of samples is less than or equal to the target sampling number, randomly sample from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

[0061] The main innovative features of the present invention are:

[0062] Automatic adjustment mechanism for keyword density window sampling rate: Sampling is performed in the local interval of keyword density in the sample to calculate the negative sample ratio, and the sampling interval is adjusted dynamically. Fine adjustment is performed through the magnifying glass expansion step to make the negative sample ratio of the retained keyword density interval meet the set range.

[0063] Automatic adjustment mechanism for multi-source data sampling rate: By utilizing the data source data volume histogram and controlling the maximum number of samples from the data source, the proportion of high-frequency data sources in the samples is suppressed, and the target number of samples is collected more accurately.

[0064] Recursive adjustment mechanism: The number of average sampling windows is proportional to the retained samples, thus ensuring the consistency of the granularity of the sampling windows. Users can adjust the parameters to control the number of recursions, and this sampling strategy is highly flexible.

[0065] Through experiments on multilingual toxicity detection models, the recursive sampling technology of the present invention effectively improves the richness of the training data set and the proportion of difficult examples, making the training data of higher quality, and the trained model has a 10% to 20% improvement in detection precision and recall. In particular, for Internet data, where the density of negative samples is relatively sparse and the sample data sources are more diverse, the overall detection accuracy of the model trained with the data obtained by the sampling method of the present invention is higher than that of the model previously trained by obtaining the data set through naive random sampling.

[0066] In addition to being applied to training multi-label text detection models, the present invention can also be applied to the training data sampling process of multi-language, multi-modal multi-label detection models. It can solve potential problems such as imbalanced ratios of positive and negative samples in most data sets, difficulty in maintaining consistent sampling rates in multiple data sources, and fixed sampling processes at a relatively low computing cost, providing help and support for obtaining higher quality training sets.

[0067] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.

[0068] This embodiment provides a recursive sampling device for dynamically adjusting the sampling rate based on sample keyword density, which is used to implement the above method. The device includes:

[0069] Keyword density calculation and data preprocessing module: obtain the original data set and preprocess it, identify keywords, and calculate the distribution density of keywords in the sample;

[0070] Keyword density interval adaptive screening module: divides the sample interval of the hit keyword into density intervals, and adaptively screens the intervals based on the comparison result between the proportion of negative samples in each density interval and the threshold;

[0071] Data source balanced sampling module: by presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after the keyword density interval adaptive screening, until the remaining number of samples is the product of the total number of samples after the keyword density interval adaptive screening and the preset retention rate;

[0072] Recursively adjust the sampling module: determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening module to perform the next round of density interval division and adaptive screening. In each recursion, the proportion threshold of negative samples used for screening is reduced and adjusted; if the number of samples is less than or equal to the target sampling number, random sampling is performed from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0074] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0075] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0076] The processing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 in any other appropriate manner (e.g., by means of firmware).

[0077] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0078] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0079] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0080] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density, characterized in that: The method comprises the following steps: Keyword density calculation and data preprocessing: obtain the original data set and preprocess it, identify keywords, and calculate the distribution density of keywords in the sample; Adaptive screening of keyword density intervals: The sample intervals of the hit keywords are divided into density intervals, and the intervals are adaptively screened based on the comparison between the proportion of negative samples in each density interval and the threshold; Data source balanced sampling: by presetting and gradually reducing the maximum sample number threshold in each data source, samples are sampled after the keyword density interval adaptive screening, until the remaining number of samples is the product of the total number of samples after the keyword density interval adaptive screening and the preset retention rate; Recursively adjust sampling: Determine whether the number of samples after balanced sampling of the data source is greater than the target sampling number. If the number of samples is greater than the target sampling number, recurse to the keyword density interval adaptive screening step to perform the next round of density interval division and adaptive screening. In each recursion, the proportion threshold of negative samples used for screening is reduced and adjusted; if the number of samples is less than or equal to the target sampling number, random sampling is performed from the samples after the keyword density interval adaptive screening in this round to obtain the target sampling number of samples.

2. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 1, characterized in that: The keyword density interval adaptive screening includes the following steps: Initialize uniform sampling: divide the sample interval of the hit keyword evenly into a first preset number of intervals of equal length, randomly extract a second preset number of samples in each interval without repetition, and directly discard the density interval if the number of samples in the interval is less than the second preset number; Count the proportion of negative samples in the keyword density sampling window: classify the extracted samples by labeling them through machine or manual labeling. Samples with one or more labels are negative samples, and samples without any labels are positive samples. Calculate the proportion of negative samples in each density interval, and retain the intervals in which the proportion of negative samples in the density interval is within the preset percentage threshold range. Magnifying glass expansion window sampling: For intervals where the proportion of negative samples is not within the preset percentage threshold range, each density interval is evenly divided into a third preset number of sub-windows of equal length. If the total number of samples in a sub-window is less than the second preset number, the sub-windows adjacent to the left and right are expanded until the total number of samples reaches the second preset number. The proportion of negative samples in each expanded sub-window is calculated, and the sub-windows whose negative sample proportion is within the preset percentage threshold range are retained. It is ensured that the retained sub-windows have no overlapping density intervals, and all retained sub-windows are merged.

3. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 2, characterized in that: The preset percentage threshold range is set according to the principle of increasing the sampling weight for density areas where the ratio of positive to negative samples is close to 1:

1.

4. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 3, characterized in that: The preset percentage threshold range is set to [50%-m, 50%+m], where m is a controllable percentage range threshold.

5. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 4, characterized in that: The initial value of the controllable percentage range threshold m is set according to business requirements and data conditions.

6. A recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 4, characterized in that: During the recursive sampling process, the new controllable percentage range threshold for each round of update is b times the controllable percentage range threshold of the previous round, where 0 < b < 1, continuously narrowing the interval of the selected negative sample ratio.

7. The recursive sampling method for dynamically adjusting the sampling rate based on sample keyword density according to claim 1, characterized in that: In the data source balanced sampling step, the total number of samples in each interval obtained after screening by the keyword density interval is denoted as sum, the maximum sample quantity in the current data source is used as the initial value of the sample quantity threshold max_length, and the preset retention rate is r. During the sampling process, the sampling ratio of different data sources is adjusted based on the highest limit of the histogram, that is, by continuously reducing the sample quantity threshold max_length to update the data, ensuring that the sample quantity of each data source is less than or equal to max_length until the total number of remaining samples reaches r × sum.

8. A recursive sampling device for dynamically adjusting the sampling rate based on sample keyword density, characterized in that: Including: Keyword density calculation and data preprocessing module: Obtain the original data set and perform preprocessing, identify keywords, and calculate the distribution density of keywords in the samples; Keyword density interval adaptive screening module: Divide the density interval of the sample intervals that hit the keywords, and perform adaptive screening on the intervals according to the comparison results of the ratio of negative samples in each density interval and the threshold; Data source balanced sampling module: Sample the samples after keyword density interval adaptive screening by presetting and gradually reducing the maximum sample quantity threshold in each data source until the remaining sample quantity is the product of the total number of samples after keyword density interval adaptive screening and the preset retention rate; Recursive adjustment sampling module: Determine whether the sample quantity after data source balanced sampling is greater than the target sampling quantity. If the sample quantity is greater than the target sampling quantity, recursively go to the keyword density interval adaptive screening module for the next round of density interval division and adaptive screening, where the ratio threshold of negative samples used for screening is reduced and adjusted each time; if the sample quantity is less than or equal to the target sampling quantity, randomly sample from the samples after keyword density interval adaptive screening in this round to obtain the target sampling quantity of samples.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the method described in any one of claims 1 to 7 is implemented.

Images