A data processing method and related products

Abstract

The application discloses a data processing method and related products, applied to the computer technology field. In the application, at least one to-be-processed data set is acquired, and the to-be-processed data set comprises a plurality of text samples; semantic clustering is performed based on the plurality of text samples to obtain at least one semantic data cluster; semantic condensation weights of the text samples are determined based on generation probabilities and spatial distance weights of the text samples, the generation probability is a degree of credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is a distance relationship of the text sample with a cluster center point in the corresponding semantic data cluster; global semantic probability distributions of the to-be-processed data sets are determined based on the semantic condensation weights; and data processing is performed on the to-be-processed data sets based on the global semantic probability distributions to obtain a target data set. Through construction of the global semantic probability distribution capable of reflecting effective semantic diversity of the data set, the accuracy of data processing is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and related products. Background Technology

[0002] With the development of large-scale language model technology, instruction fine-tuning has become a crucial training stage for improving the model's instruction compliance, task generalization ability, and human-computer interaction capabilities. Instruction fine-tuning refers to further training a pre-trained language model using data samples consisting of user instructions, input content, the model's expected output, or manually labeled responses. This enables the model to better understand the intent of natural language tasks and generate responses that meet human needs. Therefore, how to obtain data for instruction fine-tuning from large datasets has become a key issue in the training and practical application of large-scale language models.

[0003] Generally, fine-tuning data can be obtained by evaluating and filtering instruction data. This can be achieved through manual rules, strong model scoring, preference models, or gradient signals to assess the quality of individual task data; alternatively, semantic analysis can be used to measure whether the data covers more task types and semantic regions. However, this data processing method is prone to selecting data with overly concentrated semantic distributions and low sample value, thus reducing the accuracy of data processing. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a data processing method and related products that can improve the accuracy of data processing.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a data processing method, which may include: Obtain at least one dataset to be processed, which includes multiple text samples; Semantic clustering is performed on multiple text samples to obtain at least one semantic data cluster; Based on the generation probability and spatial distance weight of each text sample, the semantic cohesion weight of each text sample is determined. The generation probability is the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster. Based on the semantic cohesion weights, the global semantic probability distribution of each dataset to be processed is determined; Based on the global semantic probability distributions described above, data processing is performed on each of the datasets to be processed to obtain the target dataset.

[0006] Optionally, the step of semantic clustering based on multiple text samples to obtain at least one semantic data cluster includes: Semantic extraction is performed on each of the text samples to obtain semantic embedding vectors; Semantic clustering is performed based on each of the semantic embedding vectors to obtain at least one semantic data cluster.

[0007] Optionally, the step of semantic extraction from each of the text samples to obtain a semantic embedding vector includes: The text samples are analyzed using a large language model to obtain the hidden state information of each text sample at the word position; The hidden state information is pooled to obtain the semantic embedding vector.

[0008] Optionally, the step of performing semantic clustering based on each of the semantic embedding vectors to obtain at least one of the semantic data clusters includes: The semantic embedding vectors are subjected to dimensionality reduction processing to obtain low-dimensional semantic information. The dimensionality reduction processing includes at least one of the following: unified manifold approximation and projection dimensionality reduction algorithm. The low-dimensional semantic information is clustered to obtain at least one semantic data cluster, and the clustering process includes at least one of the following: density-based hierarchical clustering.

[0009] Optionally, determining the semantic cohesion weight of each text sample based on its generation probability and spatial distance weight includes: The generation probability of each text sample is generated using a large language model; Based on each semantic embedding vector, determine the cluster center point of each semantic data cluster; Based on each cluster center point, determine the spatial distance weight of each text sample relative to the cluster center point; The semantic cohesion weight is obtained by calculating the product of the generation probability and the spatial distance.

[0010] Optionally, the generation probability of each text sample using a large language model includes: Determine the word sequence of each text sample, wherein the word sequence represents the text sample based on multiple words; The generation probability of each word element is calculated using the large language model, where the generation probability of the target word element is the generation probability based on the preceding word elements. The generation probability of the text sample is obtained by calculating based on the generation probability of each lexical unit.

[0011] Optionally, determining the global semantic probability distribution of each of the datasets to be processed based on each of the semantic cohesion weights includes: The semantic condensation weights of all text samples in each semantic data cluster are summed to obtain the semantic condensation parameters of each semantic data cluster. The global semantic probability distribution is obtained by normalizing all the semantic condensed state parameters.

[0012] Optionally, the step of processing each of the datasets to be processed based on each of the global semantic probability distributions to obtain the target dataset includes: Based on the global semantic probability distribution, the diversity parameters of each dataset to be processed are determined; the diversity parameters are used to characterize the effective semantic diversity after reducing the influence of low-quality text samples and semantic outlier text samples. The datasets to be processed are sorted according to the diversity parameters to obtain the target dataset, which is used to fine-tune the model instructions of the large language model.

[0013] Optionally, the step of processing each of the datasets to be processed based on each of the global semantic probability distributions to obtain the target dataset includes: Based on the global semantic probability distributions, determine the sample selection parameters for each semantic data cluster; For each of the semantic data clusters, target text samples are obtained by filtering according to the semantic cohesion weights and sample filtering parameters of each text sample. The target dataset is constructed based on the target text samples in each of the semantic data clusters.

[0014] Optionally, obtaining the dataset to be processed includes: Obtain the raw dataset to be processed; The original dataset is formatted to obtain the dataset to be processed; the dataset to be processed is used to fine-tune the model instructions of the large language model.

[0015] Optionally, the step of formatting the original dataset to obtain the dataset to be processed includes: The original dataset was format-converted to obtain multiple initial text samples; The initial text samples are lexicalized using a large language model to obtain multiple lexical samples. The text samples are obtained by segmenting each word sample based on a preset word length. The dataset to be processed is constructed based on multiple text samples.

[0016] Secondly, embodiments of this application provide a data processing apparatus, which may include: An acquisition module is used to acquire at least one dataset to be processed, the dataset to be processed including multiple text samples; The clustering module is used to perform semantic clustering based on multiple text samples to obtain at least one semantic data cluster; The first determining module is used to determine the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample. The generation probability is the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster. The second determining module is used to determine the global semantic probability distribution of each of the datasets to be processed based on the semantic agglomeration weights. The processing module is used to process each of the datasets to be processed based on the global semantic probability distributions to obtain the target dataset.

[0017] Thirdly, embodiments of this application provide a control device, including a processor and a memory, wherein the memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to perform the data processing method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which is loaded by a processor to execute the data processing method as described in the first aspect.

[0019] This application involves obtaining at least one dataset to be processed, which includes multiple text samples; performing semantic clustering based on the multiple text samples to obtain at least one semantic data cluster; determining the semantic cohesion weight of each text sample based on its generation probability and spatial distance weight, where the generation probability represents the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight represents the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster; determining the global semantic probability distribution of each dataset to be processed based on each semantic cohesion weight; and processing each dataset to be processed based on each global semantic probability distribution to obtain the target dataset. This approach combines the inherent generation quality of the text itself with the spatial cohesion relationship of the text within semantic clusters, avoiding the problem of incorrectly assigning high value to anomalous samples due to judging data diversity solely based on geometric distance. By constructing a global semantic probability distribution that reflects the effective semantic diversity of the dataset, the accuracy of data selection during the fine-tuning stage of large-scale language model instructions is improved. Therefore, the accuracy of data processing can be enhanced. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation

[0022] As described earlier, with the development of large-scale language model technology, instruction fine-tuning has become a crucial training stage for improving a model's instruction compliance, task generalization, and human-computer interaction capabilities. Instruction fine-tuning refers to further training a pre-trained language model using data samples consisting of user instructions, input content, the model's expected output, or manually labeled responses. This allows the model to better understand the intent of natural language tasks and generate responses that meet human needs. Related research indicates that the effectiveness of instruction fine-tuning does not solely depend on the scale of the data; data quality, semantic coverage, and the complementarity between data sets have a significant impact on the final model performance. Furthermore, a small, well-selected, high-quality dataset can often achieve better training results than a large-scale, low-quality dataset. In practical applications, open-source or self-built instruction fine-tuning data often comes from diverse sources, potentially including various task formats such as question answering, summarizing, translation, reasoning, code generation, and multi-turn dialogue. While such data is massive in scale, it often suffers from problems such as incorrect answers, grammatical anomalies, formatting issues, topic deviations, redundancy, and low-information samples. Using such data directly for model training without selection not only increases training costs but may also weaken the model's instruction compliance, factual consistency, and generalization performance. Therefore, how to extract data for fine-tuning instructions from a large amount of data has become a key issue in the training and application of large-scale language models.

[0023] Generally, data for fine-tuning can be obtained by evaluating and filtering instruction data, using quality scoring methods, diversity measurement methods, or a combination of both. Specifically, quality assessment of individual data can be achieved through manual rules, strong model scoring, preference models, or gradient signals; alternatively, semantic analysis can be performed based on lexical statistics, topic distribution, semantic embedding distance, cluster coverage, or inter-sample similarity to measure whether the data covers more task types and semantic regions. These methods can alleviate the problems of low-quality and duplicate data to some extent, but they are prone to selecting data with overly concentrated semantic distributions and low sample value. Especially in data diversity assessment methods based on semantic embedding spatial distance, a common practice is to assume that data far from existing samples and located in sparse regions can bring new information increments. However, for natural language data, "far from the center" in the vector space does not necessarily equate to truly effective semantic diversity. Low-quality or out-of-distribution data such as grammatically incorrect text, garbled text, incorrect answers, and irrelevant topic text may also appear as outliers in the embedding space, thus being assigned high diversity value by traditional geometric sparsity indicators. This kind of "pseudo-diversity" can lead to discrepancies between evaluation metrics and the actual training gains of the model, causing the selection results to deviate from the true need for high-quality instruction fine-tuning data. Therefore, it reduces the accuracy of data processing.

[0024] Based on this, this application provides an inventive concept that combines the inherent generation quality of the text itself with the spatial cohesion of the text within semantic clusters, avoiding the problem of incorrectly assigning high value to anomalous samples due to judging data diversity solely based on geometric distance. In other words, it can both identify effective coverage between different semantic clusters and reduce the spurious boosting effect of low-quality samples, marginal outliers, and topic-irrelevant samples on diversity scores. By constructing a global semantic probability distribution that reflects the effective semantic diversity of the dataset, the accuracy of data selection during the fine-tuning stage of large-scale language model instructions is improved. Therefore, the accuracy of data processing can be enhanced.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0026] The collection and processing of relevant data (including but not limited to experimental data, test data, simulation data, user data, etc.) involved in this application shall strictly comply with the requirements of national laws and regulations when applied in the following embodiments, obtain the informed consent or separate consent of the subject obtaining the data information, and carry out data use and processing within the scope of laws and regulations and the authorization of the subject.

[0027] See Figure 1 The figure is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0028] Combination Figure 1 As shown, the data processing method provided in this application embodiment may include: S11: Obtain at least one dataset to be processed.

[0029] Specifically, the data to be processed refers to the raw instruction fine-tuning data set aggregated from multiple channels such as open source communities, business-accumulated data, and public corpora, which has not yet undergone systematic quality verification and diversity screening. It is used to carry out subsequent data processing operations throughout the entire process. Among them, the dataset to be processed includes multiple text samples, and each text sample is a single independent data unit in the dataset to be processed.

[0030] S12: Perform semantic clustering based on multiple text samples to obtain at least one semantic data cluster.

[0031] Specifically, semantic clustering is an unsupervised computation process that automatically groups all samples according to semantic similarity based on the semantic embedding vector features of text samples. It is used to group samples with highly similar semantic themes and task types into the same group, breaking the logical defects of traditional global space computation.

[0032] In this embodiment, a pre-trained general semantic embedding model is invoked to generate a corresponding high-dimensional semantic vector for each text sample, mapping the natural language text into a unified vector space. Then, based on the total size of the full sample, an adaptive method or an ablation experiment is used to select a cluster number threshold, and an appropriate clustering algorithm is selected to group all samples.

[0033] It achieves the segmentation of the same semantic subspace in the global mixed semantic space, thereby completely eliminating the interference of distance calculation across semantic topics, reducing the probability of "pseudo-diversity" samples being misjudged from the root cause, and at the same time, it pre-collects and removes a large number of invalid low-quality samples outside the distribution, improving the computational efficiency of subsequent steps.

[0034] S13: Determine the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample.

[0035] Specifically, the generation probability is a comprehensive credibility score of a single text sample across three dimensions: linguistic fluency, semantic logical rationality, and factual accuracy, calculated by a pre-trained language model; it represents the credibility of the text sample in terms of language quality and semantic rationality. Spatial distance weight refers to the normalized distance between the semantic embedding vector of a single text sample and the cluster centroid vector within the same semantic data cluster; it represents the distance relationship between the text sample and the cluster centroid within the corresponding semantic data cluster.

[0036] In this embodiment, for all text samples within each semantic data cluster, a unified quality assessment model is invoked to calculate the generation probability of each sample. The model verifies the samples for grammatical errors and incoherent sentences based on language fluency, checks for matching instructions and outputs and logical contradictions based on semantic logic, and verifies the accuracy of the output content based on factual information. Finally, a generation probability score normalized to the 0-1 interval is output. Then, for all samples within the same cluster, the spatial distance from the sample vector to the cluster center vector is calculated, and all distance values ​​are normalized to obtain spatial distance weights within the 0-1 interval. Finally, the generation probability and spatial distance weight of the same sample are multiplied to obtain the final semantic cohesion weight for that sample.

[0037] By deeply binding the quality and diversity attributes of a single sample, a comprehensive weight that can truly reflect the overall value of the sample is obtained. This changes the logic of traditional techniques that only use a single geometric distance to judge the value of a sample, accurately distinguishes between "effective diversity samples" and "pseudo-diversity outliers", and provides a reliable comprehensive value indicator for subsequent global distribution calculation.

[0038] S14: Determine the global semantic probability distribution of each dataset to be processed based on the semantic cohesion weights.

[0039] Specifically, the global semantic probability distribution refers to the probability density result obtained by statistically analyzing the semantic cohesion weights of all text samples, which can truly reflect the overall semantic value distribution of the dataset. It is used to replace the false distribution results obtained by traditional schemes that rely solely on vector geometric distance calculations, and accurately represents the true and effective semantic coverage of the entire dataset.

[0040] In this embodiment, the semantic cohesion weights of all text samples are globally normalized and converted into the sampling probability of each sample in the global distribution. Then, in the semantic embedding space of the entire dataset, the normalized semantic cohesion weights are used as the contribution of each sample to perform weighted kernel density estimation calculation, and finally the global semantic probability distribution of the entire dataset is generated.

[0041] This method significantly masks the contribution of low-value samples to the global distribution, thereby obtaining a true semantic distribution result dominated by high-quality and effective samples. It corrects the distortion problem of traditional unweighted distribution statistics and provides an accurate distribution reference for subsequent dataset optimization.

[0042] S15: Based on the global semantic probability distribution, perform data processing on each dataset to be processed to obtain the target dataset.

[0043] Specifically, based on the characteristics of the global semantic probability distribution, targeted screening and evaluation are performed on all samples to be processed, ultimately resulting in a high-quality target dataset that meets the requirements of instruction fine-tuning. This achieves priority screening of high-value samples and balanced coverage of the global semantic distribution, thereby obtaining a target dataset that simultaneously considers the quality of individual samples and the diversity of real data. Ultimately, this significantly improves the processing accuracy of instruction fine-tuning data, allowing the fine-tuned large model to achieve better instruction compliance and generalization performance.

[0044] This application involves obtaining at least one dataset to be processed, which includes multiple text samples; performing semantic clustering based on the multiple text samples to obtain at least one semantic data cluster; determining the semantic cohesion weight of each text sample based on its generation probability and spatial distance weight, where the generation probability represents the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight represents the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster; determining the global semantic probability distribution of each dataset to be processed based on each semantic cohesion weight; and processing each dataset to be processed based on each global semantic probability distribution to obtain the target dataset. This approach combines the inherent generation quality of the text itself with the spatial cohesion relationship of the text within semantic clusters, avoiding the problem of incorrectly assigning high value to anomalous samples due to judging data diversity solely based on geometric distance. By constructing a global semantic probability distribution that reflects the effective semantic diversity of the dataset, the accuracy of data selection during the fine-tuning stage of large-scale language model instructions is improved. Therefore, the accuracy of data processing can be enhanced.

[0045] In one possible implementation, the dataset to be processed is obtained, including: Obtain the original dataset to be processed; perform formatting processing on the original dataset to obtain the dataset to be processed.

[0046] Specifically, first, obtain the original dataset to be processed and denote it as... The original dataset is a collection of instruction fine-tuning data. Among them, For the first Sample text This represents the total number of text samples contained in the original dataset. The original dataset may include question-and-answer data, dialogue data, summary data, translation data, inference data, code generation data, or other text data used for fine-tuning instructions for large language models.

[0047] The original dataset is format-converted to obtain multiple initial text samples. For data samples with different organizational formats, the instructions, input content, response content, or multi-turn dialogue content are converted into a unified text field, i.e., initial text samples, so that each sample can serve as the basic object for subsequent semantic representation extraction and probability calculation. Further, the large language model is used to perform lexicalization on each initial text sample, obtaining multiple word samples. These word samples are then segmented based on a preset word length to obtain text samples. Based on these multiple text samples, a dataset to be processed is constructed. For example, the preset word length can be 256 words to reduce the bias caused by excessively long texts in semantic embedding extraction and probability calculation. The dataset to be processed is used to fine-tune the model instructions of the large language model.

[0048] In one possible implementation, an exemplary method for obtaining at least one semantic data cluster based on multiple text samples includes: A1: Semantic extraction is performed on each text sample to obtain a semantic embedding vector.

[0049] Specifically, after data preprocessing, each text sample is input into a large language model or a semantic encoding model built based on a large language model to obtain its semantic embedding representation, i.e., a semantic embedding vector. The semantic embedding vector is used to characterize the semantic position of the text sample in a high-dimensional semantic space.

[0050] For example, semantic extraction is performed on each text sample to obtain a semantic embedding vector, including: The large language model is used to analyze each text sample to obtain the hidden state information of each text sample at the word position; the hidden state information is pooled to obtain the semantic embedding vector.

[0051] Specifically, text samples Input a large language model or a semantic encoding model built based on a large language model to the text sample. Encode the text to obtain the sample text. The hidden state information output at each lexical position is the hidden state representation. Then, average pooling is performed on the hidden state information at all lexical positions to obtain the text sample. The corresponding semantic embedding vector.

[0052]

[0053] in, For text samples semantic embedding vector, This refers to the semantic embedding extraction function corresponding to a large language model or semantic encoding model. This is a text sample.

[0054] A2: Perform semantic clustering based on each semantic embedding vector to obtain at least one semantic data cluster.

[0055] Specifically, after obtaining the semantic embedding vectors of all text samples, the semantic embedding vectors are dimensionality reduced to preserve the main semantic structure of the dataset.

[0056] For example, semantic clustering is performed based on each semantic embedding vector to obtain at least one semantic data cluster, including: The semantic embedding vectors are dimensionality reduced to obtain low-dimensional semantic information; the low-dimensional semantic information is then clustered to obtain at least one semantic data cluster.

[0057] Specifically, in this embodiment, the dimensionality reduction processing includes at least one of the following: a unified manifold approximation and projection dimensionality reduction algorithm, and the clustering processing includes at least one of the following: density-based hierarchical clustering processing. The unified manifold approximation and projection algorithm can be used to reduce the dimensionality of high-dimensional semantic embedding vectors to obtain low-dimensional semantic information, i.e., low-dimensional semantic representation. Then, density-based hierarchical clustering processing is performed on the low-dimensional semantic information to divide the dataset into several disjoint semantic data clusters. This clustering process does not require pre-specifying the number of clusters; instead, it adaptively identifies potential semantic clusters based on the semantic distribution structure of the samples in the dataset. The density-based hierarchical clustering processing has a minimum clustering size parameter, which controls the minimum number of text samples required to form effective semantic data clusters, thereby obtaining relatively stable semantic clustering results. In this embodiment, the minimum clustering size parameter is preferably 30. It is understood that this application does not impose specific limitations on this.

[0058] The following is an example expression for a semantic data cluster:

[0059] in, For the first A semantic data cluster, This represents the number of semantic data clusters obtained through clustering.

[0060] In one possible implementation, an exemplary method for determining the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample includes: B1: The generation probability of each text sample is generated using a large language model.

[0061] Specifically, each text sample can be represented as a sequence of multiple lexical units, and the generation probability of each lexical unit is calculated in an autoregressive manner using a large language model.

[0062] For example, the generation probability of each text sample is generated using a large language model, including: Determine the word sequence of each text sample; calculate the word generation probability of each word using a large language model; and calculate the generation probability of the text sample based on the word generation probability of each word.

[0063] Specifically, for each text sample , representing it as by A sequence of lexical units. The lexical sequence represents a text sample based on multiple lexical units. The following is an example expression for the lexical sequence of a text sample:

[0064] in, For the first Sample text The number of lexical units, For the first The probability of a text sample.

[0065] Among them, the word generation probability is the generation probability of the target word based on the preceding word conditions.

[0066] For the Given a set of lexical units, calculate the probability of generating each unit. The following is an example formula for determining the probability of generating a lexical unit:

[0067] in, For large language models in the first The generated score output at each word position. For normalized exponential functions, To select true words from the normalized probability distribution The corresponding probability value.

[0068] To avoid the conditional probabilities of long texts multiplying together to result in excessively small values, this embodiment uses the geometric mean of the conditional generation probabilities of word units as the text sample. The generation probability is used to characterize the linguistic rationality and semantic coherence of the text sample. Generally, texts with grammatical errors, garbled characters, illogical statements, or other low-quality text have a low generation probability, thus their contribution to subsequent diversity calculations is reduced. The following describes how to determine text samples. An example formula for the generation probability:

[0069] in, , For generation probability, The number of lexical units, This represents the probability of word generation.

[0070] B2: Determine the cluster center point of each semantic data cluster based on each semantic embedding vector.

[0071] Specifically, for any semantic data cluster The cluster centroid of the semantic data cluster is calculated based on the semantic embedding vectors of all text samples within the cluster. The following is an example formula for calculating the cluster centroid:

[0072] in, For semantic data clusters The cluster center point, The number of text samples in this semantic data cluster. This is a semantic embedding vector.

[0073] B3: Determine the spatial distance weight of each text sample relative to the cluster center point based on the cluster center point.

[0074] Specifically, computational semantic data clusters The semantic embedding vector of each text sample and cluster center point The Euclidean distance between the text samples is calculated, and this distance is converted into spatial distance weights using a Gaussian kernel function. Text samples closer to the cluster center have higher spatial distance weights, indicating they better represent the core semantics of the semantic data cluster; text samples farther from the cluster center have lower spatial distance weights, indicating they may be marginal or outlier samples. The following is an example formula for calculating spatial distance weights:

[0075] in, For text samples Spatial distance weight, For semantic data clusters The cluster center point, This represents the number of text samples in this semantic data cluster. , This is a semantic embedding vector.

[0076] B4: Calculate the product of the generation probability and the spatial distance to obtain the semantic cohesion weight.

[0077] Specifically, after obtaining the generation probability and spatial distance weight of the text sample, the two are multiplied together to obtain the semantic cohesion weight of the text sample. The following is an example formula for calculating the semantic cohesion weight:

[0078] in, For text samples Semantic cohesion weight; For text samples Spatial distance weight, The probability of generation.

[0079] As can be seen from the above exemplary formula, when a text sample has both a high generation probability and a high spatial distance weight, its semantic cohesion weight will be high; if the text sample itself is of low quality, or although the text quality is high, it is far from the center point in its semantic data cluster, its semantic cohesion weight will be reduced accordingly. In this way, this embodiment can simultaneously suppress two types of samples that are prone to causing "pseudo-diversity": one is inherently low-quality text such as grammatical errors, garbled characters, and illogical statements; the other is marginal samples or out-of-distribution samples that, although the text itself is readable, have a weak correlation with the semantic distribution of the main dataset.

[0080] In one possible implementation, an exemplary method for determining the global semantic probability distribution of each dataset to be processed based on each semantic cohesion weight includes: C1: The semantic condensed state parameters of each semantic data cluster are obtained by summing the semantic condensation weights of all text samples in each semantic data cluster.

[0081] Specifically, for each semantic data cluster The semantic condensation weights of all text samples within the cluster are summed to obtain the semantic condensation parameters of the semantic data cluster. These parameters characterize the effective semantic contribution of the semantic data cluster after excluding the influence of low-quality samples and semantic outliers. The following is an example formula for calculating the semantic condensation parameters:

[0082] in, For semantic data clusters semantic condensed state parameters, For text samples Semantic cohesion weights.

[0083] C2: Normalize all semantic condensed state parameters to obtain the global semantic probability distribution.

[0084] Specifically, the semantic condensed state scores of all semantic data clusters are normalized to obtain the global semantic probability distribution of the dataset. This global semantic probability distribution reflects the effective contribution proportion of different semantic data clusters in the entire dataset. The following is an example formula for calculating the global semantic probability distribution:

[0085] in, For semantic data clusters Global semantic probability distribution in the dataset, For semantic data clusters The semantic condensed state parameters.

[0086] In one possible implementation, an exemplary method for processing each dataset to obtain the target dataset based on each global semantic probability distribution includes: D1: Determine the diversity parameters of each dataset to be processed based on the global semantic probability distribution.

[0087] Specifically, after obtaining the global semantic probability distribution, the Shannon entropy formula is applied to it to obtain the diversity parameter of the dataset to be processed. The diversity parameter characterizes the effective semantic diversity after reducing the influence of low-quality text samples and semantically outlier text samples. A higher diversity parameter indicates that the dataset to be processed has more comprehensive coverage among multiple effective semantic clusters; a lower diversity parameter indicates that the effective semantic coverage of the dataset to be processed is narrower or the semantic distribution is more concentrated. The following is an example formula for calculating the diversity parameter:

[0088] in, The diversity parameter of the dataset to be processed; For semantic data clusters Global semantic probability distribution in the dataset.

[0089] D2: Sort each dataset according to its diversity parameters to obtain the target dataset.

[0090] Specifically, when there are multiple datasets or subsets of data to be processed, diversity parameters are calculated for each dataset or subset. These diversity parameters are then ranked, and the datasets with higher diversity parameters are identified as target datasets with high effective semantic diversity. These target datasets are used to fine-tune the model instructions for the large language model.

[0091] In one possible implementation, an exemplary method for processing each dataset to obtain the target dataset based on each global semantic probability distribution includes: E1: Determine the sample selection parameters for each semantic data cluster based on the global semantic probability distribution.

[0092] Specifically, when it is necessary to select a training subset from a large-scale candidate instruction data pool, which is the dataset to be processed, the sample selection parameters for each semantic data cluster, i.e., the sample selection quota, can be determined based on the global semantic probability distribution of each semantic data cluster. In this embodiment, the sample selection parameters can be represented by a ratio.

[0093] E2: For each semantic data cluster, the target text sample is obtained by filtering according to the semantic cohesion weight of each text sample and the sample filtering parameters.

[0094] Specifically, within each semantic data cluster, text samples are sorted or weighted according to their semantic cohesion weights, and text samples with higher scores are selected to form the filtered target text samples.

[0095] E3: Construct the target dataset based on the target text samples in each semantic data cluster.

[0096] Specifically, the filtered target text samples can form a target dataset for fine-tuning training of large language models, thereby reducing the negative impact of low-quality text, redundant text, and semantically outlier text on the model training effect.

[0097] Based on the data processing method provided in the above embodiments, see [link to relevant documentation]. Figure 2 This application also provides a schematic diagram of the structure of a data processing device.

[0098] Combination Figure 2 As shown, the data processing device 20 provided in this application embodiment includes: an acquisition module 21, a clustering module 22, a first determination module 23, a second determination module 24, and a processing module 25.

[0099] Acquisition module 21 is used to acquire at least one dataset to be processed, the dataset to be processed including multiple text samples; Clustering module 22 is used to perform semantic clustering based on multiple text samples to obtain at least one semantic data cluster; The first determining module 23 is used to determine the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample. The generation probability is the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster. The second determining module 24 is used to determine the global semantic probability distribution of each of the datasets to be processed based on the semantic agglomeration weights. The processing module 25 is used to process each of the datasets to be processed based on the global semantic probability distributions to obtain the target dataset.

[0100] In one possible implementation, clustering module 22 includes: Extraction subunits are used to extract semantics from each of the text samples to obtain semantic embedding vectors; The clustering subunit is used to perform semantic clustering based on each of the semantic embedding vectors to obtain at least one of the semantic data clusters.

[0101] In one possible implementation, sub-units are extracted for: The text samples are analyzed using a large language model to obtain the hidden state information of each text sample at the word position; The hidden state information is pooled to obtain the semantic embedding vector.

[0102] In one possible implementation, the clustering subunit is used for: The semantic embedding vectors are subjected to dimensionality reduction processing to obtain low-dimensional semantic information. The dimensionality reduction processing includes at least one of the following: unified manifold approximation and projection dimensionality reduction algorithm. The low-dimensional semantic information is clustered to obtain at least one semantic data cluster, and the clustering process includes at least one of the following: density-based hierarchical clustering.

[0103] In one possible implementation, the first determining module 23 includes: A generation subunit is used to generate the generation probability of each text sample using a large language model; The first determining subunit is used to determine the cluster center point of each semantic data cluster based on each semantic embedding vector; The second determining subunit is used to determine the spatial distance weight of each text sample relative to the cluster center point based on each cluster center point. The computational subunit is used to calculate the product of the generation probability and the spatial distance to obtain the semantic cohesion weight.

[0104] In one possible implementation, sub-units are generated for: Determine the word sequence of each text sample, wherein the word sequence represents the text sample based on multiple words; The generation probability of each word element is calculated using the large language model, where the generation probability of the target word element is the generation probability based on the preceding word elements. The generation probability of the text sample is obtained by calculating based on the generation probability of each lexical unit.

[0105] In one possible implementation, the second determining module 24 is used for: The semantic condensation weights of all text samples in each semantic data cluster are summed to obtain the semantic condensation parameters of each semantic data cluster. The global semantic probability distribution is obtained by normalizing all the semantic condensed state parameters.

[0106] In one possible implementation, the processing module 25 is used for: Based on the global semantic probability distribution, the diversity parameters of each dataset to be processed are determined; the diversity parameters are used to characterize the effective semantic diversity after reducing the influence of low-quality text samples and semantic outlier text samples. The datasets to be processed are sorted according to the diversity parameters to obtain the target dataset, which is used to fine-tune the model instructions of the large language model.

[0107] In one possible implementation, the processing module 25 is used for: Based on the global semantic probability distributions, determine the sample selection parameters for each semantic data cluster; For each of the semantic data clusters, target text samples are obtained by filtering according to the semantic cohesion weights and sample filtering parameters of each text sample. The target dataset is constructed based on the target text samples in each of the semantic data clusters.

[0108] It should be noted that the data processing apparatus provided in this application embodiment has the same beneficial effects as the data processing method provided in the above embodiments, and therefore will not be described again.

[0109] In one possible implementation, see Figure 3 The figure is a schematic diagram of a control device provided in an embodiment of this application.

[0110] The control device may include a memory 311 and a processor 312. For example... Figure 3 As shown, the memory can be random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (Electronic Programmable ROM), registers, hard disks, removable disks, etc.

[0111] The memory 311 can store computer instructions. When the computer instructions stored in the memory 311 are executed by the processor 312, the processor 312 can be used to perform data processing methods. The memory 311 can also store data.

[0112] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or a semiconductor medium (e.g., solid-state disk (SSD)).

[0113] This application also provides a readable storage medium for storing the methods provided in the above embodiments. Examples include random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (EPROM), registers, hard disks, removable disks, or any other form of storage medium in the art.

[0114] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0115] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the methods disclosed in the embodiments, since they correspond to the product embodiments disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the description of the product embodiments.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, include: Obtain at least one dataset to be processed, which includes multiple text samples; Semantic clustering is performed on multiple text samples to obtain at least one semantic data cluster; Based on the generation probability and spatial distance weight of each text sample, the semantic cohesion weight of each text sample is determined. The generation probability is the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster. Based on the semantic cohesion weights, the global semantic probability distribution of each dataset to be processed is determined; Based on the global semantic probability distributions described above, data processing is performed on each of the datasets to be processed to obtain the target dataset.

2. The method according to claim 1, characterized in that, The step of semantic clustering based on multiple text samples to obtain at least one semantic data cluster includes: Semantic extraction is performed on each of the text samples to obtain semantic embedding vectors; Semantic clustering is performed based on each of the semantic embedding vectors to obtain at least one semantic data cluster.

3. The method according to claim 2, characterized in that, The step of extracting semantics from each of the text samples to obtain a semantic embedding vector includes: The text samples are analyzed using a large language model to obtain the hidden state information of each text sample at the word position; The hidden state information is pooled to obtain the semantic embedding vector.

4. The method according to claim 2, characterized in that, The step of performing semantic clustering based on each of the semantic embedding vectors to obtain at least one of the semantic data clusters includes: The semantic embedding vectors are subjected to dimensionality reduction processing to obtain low-dimensional semantic information. The dimensionality reduction processing includes at least one of the following: unified manifold approximation and projection dimensionality reduction algorithm. The low-dimensional semantic information is clustered to obtain at least one semantic data cluster, and the clustering process includes at least one of the following: density-based hierarchical clustering.

5. The method according to claim 1, characterized in that, The determination of the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample includes: The generation probability of each text sample is generated using a large language model; Based on each semantic embedding vector, determine the cluster center point of each semantic data cluster; Based on each cluster center point, determine the spatial distance weight of each text sample relative to the cluster center point; The semantic cohesion weight is obtained by calculating the product of the generation probability and the spatial distance.

6. The method according to claim 5, characterized in that, The generation probability of each text sample generated using a large language model includes: Determine the word sequence of each text sample, wherein the word sequence represents the text sample based on multiple words; The generation probability of each word element is calculated using the large language model, where the generation probability of the target word element is the generation probability based on the preceding word elements. The generation probability of the text sample is obtained by calculating based on the generation probability of each lexical unit.

7. The method according to claim 1, characterized in that, The step of determining the global semantic probability distribution of each dataset to be processed based on each of the semantic cohesion weights includes: The semantic condensation weights of all text samples in each semantic data cluster are summed to obtain the semantic condensation parameters of each semantic data cluster. The global semantic probability distribution is obtained by normalizing all the semantic condensed state parameters.

8. The method according to claim 1, characterized in that, The process of processing each of the datasets to be processed based on the global semantic probability distributions to obtain the target dataset includes: Based on the global semantic probability distribution, the diversity parameters of each dataset to be processed are determined; the diversity parameters are used to characterize the effective semantic diversity after reducing the influence of low-quality text samples and semantic outlier text samples. The datasets to be processed are sorted according to the diversity parameters to obtain the target dataset, which is used to fine-tune the model instructions of the large language model.

9. The method according to claim 1, characterized in that, The process of processing each of the datasets to be processed based on the global semantic probability distributions to obtain the target dataset includes: Based on the global semantic probability distributions, determine the sample selection parameters for each semantic data cluster; For each of the semantic data clusters, target text samples are obtained by filtering according to the semantic cohesion weights and sample filtering parameters of each text sample. The target dataset is constructed based on the target text samples in each of the semantic data clusters.

10. A data processing apparatus, characterized in that, include: An acquisition module is used to acquire at least one dataset to be processed, the dataset to be processed including multiple text samples; The clustering module is used to perform semantic clustering based on multiple text samples to obtain at least one semantic data cluster; The first determining module is used to determine the semantic cohesion weight of each text sample based on the generation probability and spatial distance weight of each text sample. The generation probability is the credibility of the text sample in terms of language quality and semantic rationality, and the spatial distance weight is the distance relationship between the text sample and the cluster center point in the corresponding semantic data cluster. The second determining module is used to determine the global semantic probability distribution of each of the datasets to be processed based on the semantic agglomeration weights. The processing module is used to process each of the datasets to be processed based on the global semantic probability distributions to obtain the target dataset.

Images