Distribution-based pre-training data automatic screening method and electronic equipment

Abstract

The invention discloses a distribution-based pre-training data automatic screening method and electronic equipment, and belongs to the technical field of large models. The method comprises the steps that an original data set is received through a data input module, and data in the original data set has multiple formats; converting data in the original data set into original feature distribution by using a first algorithm; wherein the first algorithm comprises a Hash algorithm and an n-gram algorithm; obtaining a reference data set corresponding to a target task, and constructing target task distribution corresponding to the reference data set; calculating the cosine similarity of each piece of data in the original feature distribution based on the target task distribution and the original feature distribution; and determining a target data set based on the cosine similarity. The method can automatically complete the screening of the original data set, reduces the cost, and achieves the technical effects of efficient and accurate data screening and high adaptability.

Description

Technical Field

[0001] The present application belongs to the field of large model technology, and in particular relates to a distribution-based automatic screening method for pre-training data and an electronic device. Background Art

[0002] In the development and application of large-scale language models (LLM), the scale and quality of pre-training data are key factors in determining model performance. However, the current methods of obtaining and screening pre-training data have the following problems when facing ultra-large-scale, multi-domain, and multi-format data sets: 1) Inefficiency: Traditional screening methods based on manual annotation or rules cannot cope with the processing needs of large-scale data; 2) Inaccurate selection: Lack of refined analysis of target tasks and inability to identify data that contributes significantly to target tasks; 3) Lack of dynamic adaptability: Existing methods cannot adaptively adjust screening strategies according to changes in task requirements.

[0003] In order to solve the above problems, the present invention proposes a distribution-based automatic screening method for pre-training data and an electronic device. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the present application provides a distribution-based automatic screening method for pre-training data and an electronic device, which can solve the problems of low efficiency, inaccurate selection, and lack of dynamic adaptability encountered in the prior art when acquiring and screening pre-training data.

[0005] The technical effects to be achieved by this application are achieved through the following solutions: In a first aspect, the present application provides a distribution-based automatic screening method for pre-training data, comprising: Using a data input module to receive an original data set, wherein the data in the original data set has multiple formats; Using a first algorithm to convert the data in the original data set into an original feature distribution; wherein the first algorithm includes a hash algorithm and an n-gram algorithm; Obtain a reference data set corresponding to the target task, and construct a target task distribution corresponding to the reference data set; Based on the target task distribution and the original feature distribution, calculating the cosine similarity of each piece of data in the original feature distribution; A target data set is determined based on the cosine similarity.

[0006] In some embodiments, the method further comprises: According to the training result of the target data set in the model training, the threshold parameter used in the process of determining the target data set based on the cosine similarity is adjusted.

[0007] In some embodiments, it is characterized in that the step of receiving the original data set by using a data input module comprises: Use the data input module to load the original data set through file stream processing; A batch import strategy is used to implement the input of dynamic data streams captured from the network in real time.

[0008] In some embodiments, the original data set includes: Common datasets; Domain datasets; and User-defined dataset.

[0009] In some embodiments, after converting the data in the original data set into the original feature distribution using the first algorithm, the method includes: The original feature distribution is mapped to a vector space of fixed length through a hash function.

[0010] In some embodiments, the cosine similarity of each piece of data in the original feature distribution is calculated according to the following formula: Cosine similarity = ; Among them, x represents any word x in each data in the original feature distribution, P(x) represents the probability of any word x in the original feature distribution, and Q(x) represents the probability of any word x in the target task distribution.

[0011] In some embodiments, determining the target data set based on the cosine similarity includes: Determine the set threshold; The data whose cosine similarity is less than the set threshold are deleted, and the data whose cosine similarity is greater than or equal to the set threshold are retained to form a target data set.

[0012] In some embodiments, the multiple formats include JSON, CSV, and / or plain text.

[0013] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the aforementioned methods when executing the computer program.

[0014] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the aforementioned methods.

[0015] Through the distribution-based automatic data screening method and electronic device of the embodiments of the present application, the method extracts features from the original data set and the reference data set corresponding to the target task, and calculates the cosine similarity of the corresponding feature distribution, and then screens the original data set based on the cosine similarity to obtain the target data set. The screening of the original data set can be automatically completed, the cost is reduced, and the technical effect of efficient and accurate data screening with strong adaptability is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 Flow chart of a method for automatic screening of pre-training data based on distribution in one embodiment of the present application; Figure 2 Schematic diagram of the structure of a distribution-based automatic screening device for pre-training data in one embodiment of the present application; Figure 3 It is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0019] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] In the related technologies, the current methods of obtaining and screening pre-training data face the following challenges when facing ultra-large-scale, multi-domain, and multi-format data sets: 1. Huge data scale: The volume of public data sets such as The Pile and Common Crawl reaches TB level, which makes it difficult to use all of them for pre-training.

[0021] 2. Data quality varies: Data comes from a variety of sources (such as social media, news articles, and encyclopedia content), which inevitably contain erroneous information, noisy data, or low-quality content.

[0022] 3. Insufficient matching between target tasks: Existing random sampling or rule filtering strategies are difficult to meet the data requirements of specific tasks (such as medicine, law, etc.), resulting in limited model training effects.

[0023] This application is based on the automated screening technology of data distribution, which automatically completes efficient and accurate data screening by evaluating the matching of feature distribution between the original data set and the reference data set corresponding to the target task.

[0024] The core technologies of this application include: 1. Data feature modeling: Use the n-gram feature method to convert text into an efficient distribution representation.

[0025] 2. Distribution similarity calculation: Use KL divergence, cosine similarity and other measurement methods to quantify the similarity between the data distribution and the target task distribution.

[0026] 3. Weight calculation and screening: According to the similarity calculation results, each piece of data is assigned a different importance weight, and the data that contributes more to the target task is retained.

[0027] 4. Feedback optimization mechanism: Based on the performance feedback of the model after training, the screening strategy is dynamically adjusted to ensure long-term optimization.

[0028] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0029] First, refer to Figure 1 , the distribution-based automatic screening method of pre-training data of the present application is described in detail. The present application provides a distribution-based automatic screening method of pre-training data, comprising: S1: using a data input module to receive an original data set, wherein the data in the original data set has multiple formats; S2: using a first algorithm to convert the data in the original data set into an original feature distribution; wherein the first algorithm includes a hash algorithm and an n-gram algorithm; S3: Obtain a reference data set corresponding to the target task, and construct a target task distribution corresponding to the reference data set; S4: Based on the target task distribution and the original feature distribution, calculate the cosine similarity of each data in the original feature distribution; S5: Determine a target data set based on the cosine similarity.

[0030] The distribution-based automatic data screening method of the present application extracts features from the original data set and the reference data set corresponding to the target task, calculates the cosine similarity of the corresponding feature distribution, and then screens the original data set based on the cosine similarity to obtain the target data set. The screening of the original data set can be automatically completed, reducing costs and achieving the technical effect of efficient and accurate data screening with strong adaptability.

[0031] In some embodiments, the method further comprises: According to the training result of the target data set in the model training, the threshold parameter used in the process of determining the target data set based on the cosine similarity is adjusted. Exemplarily, the threshold parameter can be a set threshold for determining the size of the cosine similarity. Those skilled in the art can determine the size of the set threshold according to actual conditions, and no specific limitation is made here.

[0032] For example, the screening strategy and parameter configuration can be optimized according to the performance of the target dataset in the downstream task. For example, the threshold value can be adjusted according to the performance of the target dataset in the downstream task. The threshold value can be increased or decreased, thereby obtaining a more accurate target dataset.

[0033] Specifically, the implementation steps include: Test the model performance on downstream tasks (such as text classification) and provide feedback on: 1) Accuracy

[0034] 2) Mean absolute error (MAE).

[0035] 3) Target task completion rate.

[0036] Exemplarily, the screening strategy is adjusted according to the feedback results, for example: 1) Adjust the set threshold or increase the screening threshold to improve data quality.

[0037] 2) Adjust the feature extraction method to improve the data distribution matching accuracy.

[0038] The key technologies involved in adjusting the threshold parameters used in the process of determining the target data set based on the cosine similarity in this application include: 1) Use a Bayesian optimization framework such as Hyperopt for policy tuning.

[0039] 2) Data resampling mechanism: dynamically expand the sample space based on the feedback results, and then adjust the threshold parameters, etc.

[0040] In some embodiments, it is characterized in that the step of receiving the original data set by using a data input module comprises: The data input module is used to load the original data set through file stream processing to avoid memory overflow caused by loading too much data at one time.

[0041] A batch import strategy is used to implement the input of dynamic data streams captured in real time from the network. For example, 1 GB of data is processed each time, and the amount of data processed each time can be determined based on actual needs.

[0042] Exemplarily, the data input module supports dynamic import of large-scale streaming data.

[0043] Specifically, streaming data may include pandas.read_csv and the chunk_size parameter in Python.

[0044] The data input module improves the preprocessing speed of large-scale data through multi-threaded data import.

[0045] In some embodiments, the original data set includes: Common datasets; Domain datasets; and User-defined dataset.

[0046] In some embodiments, after converting the data in the original data set into the original feature distribution using the first algorithm, the method includes: The original feature distribution is mapped to a vector space of fixed length through a hash function.

[0047] Exemplarily, each piece of data is converted into a feature vector using an n-gram algorithm, and storage and computational overheads are reduced through hash mapping.

[0048] For example, the text is segmented into n-gram features in units of sentences. For example, the input text is: "The future of deep learning", then the hash representation of n-gram is: {hash ("deep"), hash ("learning"), hash ("future"), hash ("deep learning"), hash ("learning the future")}.

[0049] The above n-grams are mapped to a vector space of fixed length through a hash function to avoid directly storing the original features and reduce storage occupancy.

[0050] For example, feature hashing can be performed using sklearn.feature_extraction.text.HashingVectorizer.

[0051] Exemplarily, deep semantic features can also be extracted based on the BERT or GPT embedding model for high-quality feature modeling.

[0052] Exemplarily, the reference data set corresponding to the target task may include medical literature, financial reports, and legal documents.

[0053] Specifically, we can extract the word frequency distribution, n-gram distribution and other information of the reference dataset to construct the target task distribution. For example, the target task distribution or feature distribution of medical terms is: {"disease": 0.15, "drug": 0.1, "treatment": 0.2, ...}.

[0054] The above target task distribution is stored in vectorized form for subsequent calculations.

[0055] In some embodiments, the cosine similarity of each piece of data in the original feature distribution is calculated according to the following formula: Cosine similarity = ; Among them, x represents any word x in each data in the original feature distribution, P(x) represents the probability of any word x in the original feature distribution, and Q(x) represents the probability of any word x in the target task distribution.

[0056] Specifically, for example, if a piece of data contains 5 words, then for each word, Afterwards, the values ​​corresponding to the five words are summed up to obtain the cosine similarity of the data.

[0057] Illustratively, after obtaining the cosine similarity, the cosine similarity may be normalized to obtain the normalized cosine similarity, which is convenient for subsequent comparison with a threshold.

[0058] In some embodiments, determining the target data set based on the cosine similarity includes: Determine the set threshold; The data whose cosine similarity is less than the set threshold are deleted, and the data whose cosine similarity is greater than or equal to the set threshold are retained to form a target data set.

[0059] Exemplarily, if the cosine similarity is lower than a set threshold, the corresponding data is marked as having low relevance for subsequent screening.

[0060] When calculating cosine similarity, you can use an efficient computing framework (such as NumPy or PyTorch, etc.) to achieve parallel processing of large-scale distribution comparison. Among them, the data distribution dynamic update mechanism: supports rebuilding the target distribution model according to new tasks.

[0061] In some embodiments, the data may be sorted according to cosine similarity, such as retaining the top 5% of the data.

[0062] On the basis of ensuring relevance, we can add diversity constraints to avoid filtering data that is too single. We can also set diversity indicators, that is, use the following methods as filtering conditions: Filter condition = cosine similarity > set threshold, and diversity index > diversity threshold.

[0063] Output high-quality datasets that meet the screening criteria for pre-training.

[0064] Exemplarily, the diversity indicator may include an entropy value of the TF-IDF weight distribution.

[0065] Exemplarily, the present application supports multiple output file formats (such as TFRecord, CSV, etc.).

[0066] In some embodiments, the multiple formats include JSON, CSV, plain text, and / or Parquet.

[0067] The distribution-based automatic screening method of pre-training data in this application has the following advantages: 1. High efficiency: Reduce redundant calculations on massive data and achieve fast screening through feature hashing.

[0068] 2. High precision: Accurately match data and task distribution to improve the target relevance of screened data.

[0069] 3. Multi-field applicability: Supports dynamic requirements of different fields and tasks without manual adjustment.

[0070] 4. Full automation: realize full process automation and reduce labor costs.

[0071] 5. Strong adaptability: Through the feedback optimization mechanism, the screening strategy can be adjusted in real time according to different tasks.

[0072] 6. Controllable costs: Reduce unnecessary data loading and training costs and improve resource utilization.

[0073] like Figure 2 As shown, the present application also provides a distribution-based automatic screening device for pre-trained data, which includes: a data input module, a feature extraction module, a task modeling module, a similarity calculation module, a screening output module and a feedback optimization module.

[0074] Exemplarily, the data input module is used to receive raw data sets in multiple formats (such as JSON, CSV, plain text, etc.); and supports dynamic import of large-scale streaming data.

[0075] Exemplarily, the feature extraction module uses hashing or n-gram technology to convert text into a fixed-length feature representation vector. N-gram is a language model commonly used in large-vocabulary continuous speech recognition. For Chinese, we call it the Chinese Language Model (CLM). The Chinese language model uses the collocation information between adjacent words in the context to achieve automatic conversion of Chinese characters.

[0076] For example, convert “pretrainingdata” to {“pre”, “training”, “data”, “pretraining”, “trainingdata”}.

[0077] Exemplarily, the task modeling module is used to input feature samples or domain knowledge corresponding to the target task and to calculate the distribution of the target task (such as term distribution in the medical field).

[0078] Exemplarily, the similarity calculation module uses a cosine similarity calculation method. In addition, a statistical method (such as KL divergence) or a deep learning model can be used to calculate the similarity between data and task distribution and generate weights.

[0079] Exemplarily, the screening output module sorts and screens the data according to weights, outputs a highly relevant data subset, and optionally imposes diversity constraints to ensure that the screened data covers different sub-fields.

[0080] Exemplarily, the feedback optimization module adjusts the distribution similarity calculation method and weight strategy according to the training results of the downstream task model.

[0081] The above-mentioned distribution-based automatic screening device for pre-trained data can implement any of the above-mentioned distribution-based automatic screening methods for pre-trained data, and obtain all the technical effects achieved by the above-mentioned distribution-based automatic screening method for pre-trained data, which will not be repeated here.

[0082] The distribution-based automatic screening method, device and electronic device of the present application can achieve the following technical effects.

[0083] 1) Improve efficiency: Through feature hashing and efficient distribution matching, the time consumption of the screening process is reduced by more than 50%.

[0084] 2) Improved accuracy: The relevance of the selected data subset to the target task is increased by 30%.

[0085] 3) Resource saving: significantly reduce the loading and training time of redundant data and reduce training costs.

[0086] 4) High degree of automation: no manual intervention required, strong adaptability, suitable for a variety of tasks.

[0087] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0088] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also discloses an electronic device; Specifically, Figure 3 The hardware structure diagram of an electronic device for automatic screening of distributed pre-trained data provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other in communication within the device through the bus 450.

[0090] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0091] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0092] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0093] The communication interface 440 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (for example, USB, network cable, etc.) or a wireless manner (for example, mobile network, Wi-Fi, Bluetooth, etc.).

[0094] The bus 450 includes a path that transmits information between the various components of the device (eg, the processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 ).

[0095] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0096] The electronic device of the above embodiment is used to implement the corresponding distribution-based automatic screening method of pre-trained data in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0097] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the distribution-based automatic screening method for pre-training data as described in any of the above embodiments.

[0098] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0099] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the distribution-based automatic screening method for pre-training data as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0100] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0101] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the well-known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0102] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.

[0103] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A distribution-based automatic screening method for pre-training data, characterized in that: include: Using a data input module to receive an original data set, wherein the data in the original data set has multiple formats; Using a first algorithm to convert the data in the original data set into an original feature distribution; wherein the first algorithm includes a hash algorithm and an n-gram algorithm; Obtain a reference data set corresponding to the target task, and construct a target task distribution corresponding to the reference data set; Based on the target task distribution and the original feature distribution, calculating the cosine similarity of each piece of data in the original feature distribution; A target data set is determined based on the cosine similarity.

2. The method for automatically screening pre-training data based on distribution according to claim 1, characterized in that: The method further comprises: According to the training result of the target data set in the model training, the threshold parameter used in the process of determining the target data set based on the cosine similarity is adjusted.

3. The method for automatic screening of pre-training data based on distribution according to claim 1 or 2, characterized in that: The step of receiving the original data set by using a data input module comprises: Use the data input module to load the original data set through file stream processing; A batch import strategy is used to implement the input of dynamic data streams captured from the network in real time.

4. The method for automatically screening pre-training data based on distribution according to claim 3, characterized in that: The original data set includes: Common datasets; Domain datasets; and User-defined dataset.

5. The method for automatic screening of pre-training data based on distribution according to claim 1, characterized in that: After the data in the original data set is converted into the original feature distribution by using the first algorithm, the method includes: The original feature distribution is mapped to a vector space of fixed length through a hash function.

6. The method for automatic screening of pre-training data based on distribution according to claim 1, characterized in that: The cosine similarity of each piece of data in the original feature distribution is calculated according to the following formula: Cosine similarity = ; Among them, x represents any word x in each data in the original feature distribution, P(x) represents the probability of any word x in the original feature distribution, and Q(x) represents the probability of any word x in the target task distribution.

7. The method for automatic screening of pre-training data based on distribution according to claim 1, characterized in that: The determining of the target data set based on the cosine similarity comprises: Determine the set threshold; The data whose cosine similarity is less than the set threshold are deleted, and the data whose cosine similarity is greater than or equal to the set threshold are retained to form a target data set.

8. The method for automatic screening of pre-training data based on distribution according to claim 1, characterized in that: The multiple formats include JSON, CSV, and / or plain text.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the distribution-based automatic screening method for pre-training data as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the distribution-based automatic screening method for pre-training data as described in any one of claims 1 to 8.

Images