Large model data set construction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a large model data set construction method and device, electronic equipment and a storage medium, and belongs to the technical field of large models, and the method comprises the steps: obtaining a text data set for large model training, preprocessing the text data set to obtain a plurality of long texts, carrying out splitting processing and vectorization processing on the long texts, and obtaining a large model data set; the method comprises the following steps of: obtaining split texts and corresponding text vectors, mapping the text vectors meeting a preset similarity condition into the same hash bucket through a locality sensitive hash algorithm, and performing similarity calculation on the text vectors in each hash bucket to obtain a similarity value between every two split texts; constructing a text similarity graph according to the split texts in the hash bucket and the similarity value between the split texts, selecting the split texts in a preset proportion as important texts in the text similarity graph based on the similarity value between the split texts, calculating the importance score of the important texts, determining a target data set according to the importance score, and storing the target data set in the hash bucket. And high-value data can be identified.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a method, apparatus, electronic device and storage medium for constructing large model datasets. Background Technology

[0002] With the development of information technology and the widespread application of large language models, various industries are choosing to train their own vertical large models based on large language model foundations. During the training process of these large models, the sources of relevant data are diverse, and the data suffers from textual redundancy and low quality. Therefore, deduplication of the data to ensure coverage across different dimensions and to identify high-value text data is crucial.

[0003] In related technologies, important text is usually identified manually using annotation tools, but this is inefficient and relatively costly. Alternatively, text deduplication can be performed using string matching algorithms, which can find similarities and differences between two texts, but the time and space complexity is high, making it unsuitable for repeated comparisons of large amounts of data. This results in messy and inefficient data, failing to solve the problem of identifying high-value data. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for constructing large model datasets to solve the problem of being unable to identify high-value data.

[0005] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows: In a first aspect, embodiments of this application provide a method for constructing a large model dataset, comprising: acquiring a text dataset for training a large model; preprocessing the text dataset to obtain multiple long texts; splitting and vectorizing the long texts to obtain split texts and corresponding text vectors; mapping the text vectors that meet preset similarity conditions to the same hash bucket using a locality-sensitive hashing algorithm; calculating the similarity of the text vectors in each hash bucket to obtain a similarity value between every two split texts in the hash bucket; constructing a text similarity graph based on the split texts in the hash bucket and the similarity values ​​between the split texts; selecting a preset proportion of the split texts as important texts based on the similarity values ​​between the split texts in the text similarity graph; calculating the importance score of the important texts; and determining a target dataset based on the importance score, wherein the target dataset is used for training the large model.

[0006] Secondly, embodiments of this application provide a large model dataset construction apparatus, comprising: a preprocessing module, configured to acquire a text dataset for training a large model, preprocess the text dataset to obtain multiple long texts, and split and vectorize the long texts to obtain split texts and corresponding text vectors; a similarity module, configured to map text vectors that meet preset similarity conditions to the same hash bucket using a locality-sensitive hashing algorithm, calculate the similarity of the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket; a filtering module, configured to construct a text similarity graph based on the split texts in the hash bucket and the similarity values ​​between the split texts, and select a preset proportion of the split texts as important texts based on the similarity values ​​between the split texts in the text similarity graph; and a format conversion module, configured to calculate the importance score of the important texts, and determine a target dataset based on the importance score, the target dataset being used for training the large model.

[0007] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-described method for constructing a large model dataset.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that can be executed by a processor to implement the above-described method for constructing a large model dataset.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the above-mentioned method for constructing a large model dataset.

[0010] Sixthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned method for constructing a large model dataset.

[0011] The technical solution of this application embodiment obtains a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained. These long texts are then split and vectorized to obtain split texts and corresponding text vectors. Using a locality-sensitive hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. Similarity is calculated for the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket. Based on the split texts in the hash bucket and the similarity values ​​between them, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. The importance score of the important texts is calculated, and based on the importance score, a target dataset is determined. This target dataset is used for training the large model. Therefore, splitting and vectorizing the long texts obtained after preprocessing the text dataset for large model training to obtain split texts and text vectors improves the efficiency of long text processing. Text vectors that meet preset similarity conditions are mapped to the same hash bucket. This is used to calculate the similarity value between the split texts in each hash bucket and construct a text similarity graph. This allows for mapping relatively similar split texts that meet preset similarity conditions to the same hash bucket to form a text similarity graph. By selecting a preset proportion of split texts as important texts from the text similarity graph, the split texts with high similarity in the text similarity graph can be centrally deduplicated to obtain important texts. Then, by scoring the important texts, an importance score is obtained, which improves the quality and diversity of the data in the acquired target dataset. Finally, a high-value target dataset is obtained, which can quickly identify high-value data for use in training large models. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in one or more embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the structure of a large model dataset construction system according to an embodiment of this application; Figure 2 This is a schematic flowchart of a method for constructing a large model dataset according to an embodiment of this application; Figure 3 This is a flowchart of a method for constructing a large model dataset according to an embodiment of this application; Figure 4This is a schematic flowchart of a method for constructing a large model dataset according to another embodiment of this application; Figure 5 This is a schematic block diagram of a large model dataset construction apparatus according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of a large model dataset construction device according to an embodiment of this application. Detailed Implementation

[0014] This application provides a method, apparatus, electronic device, and storage medium for constructing large model datasets to solve the problem of being unable to identify high-value data.

[0015] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0016] The method for constructing large model datasets provided in this application can be executed by an electronic device or by software installed in an electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include smartphones, laptops, smart wearable devices, vehicle terminals, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0017] The following detailed description, in conjunction with the accompanying drawings, of a method for constructing a large model dataset provided in this application, through specific embodiments and application scenarios, will be provided in detail.

[0018] Figure 1 This is a schematic diagram of the structure of a large model dataset construction system according to an embodiment of this application. Figure 1As shown, the system includes: a data preprocessing module, a Map data cleaning module, a data distribution module, a Reduce data cleaning module, and a data format conversion module. Specifically, the system uses the distributed framework MapReduce to perform parallel computation on large-scale text datasets. The data preprocessing module filters and identifies the text dataset to obtain long text. The Map data cleaning module, at the Map stage of MapReduce, splits the long text and performs vector conversion processing to obtain split text and text vectors. The data distribution module uses a locality-sensitive hashing algorithm to map text vectors that meet preset similarity conditions to the same hash bucket. The Reduce data cleaning module, at the Reduce stage of MapReduce, receives the split text and text vectors sent from the Map stage and stores them in hash buckets. It then performs text deduplication processing on the split text to obtain deduplicated text data. The data format conversion module converts the deduplicated text data according to a preset format to obtain a target dataset in a preset format for training a large model.

[0019] Figure 2 This diagram illustrates a schematic flowchart of a method for constructing a large model dataset according to an embodiment of the present invention, the method comprising the following steps: S202, Obtain the text dataset for training the large model. Preprocess the text dataset to obtain multiple long texts, and then split and vectorize the long texts to obtain the split texts and their corresponding text vectors.

[0020] The large model includes: a large government affairs model; the text dataset includes: text data from multiple sources, such as government affairs text data from different times and regions.

[0021] Preprocessing includes filtering sensitive information from the acquired text dataset and identifying entity information within the text dataset. This includes filtering for ID numbers, phone numbers, etc., and identifying entity information such as time, region level, and extraction region.

[0022] Obtain the preprocessed text dataset and generate multiple long texts. In the distributed MapReduce framework, the long texts can be split into fixed-length subtexts through the MAP side.

[0023] The split text is vectorized to obtain the text vector corresponding to each split text.

[0024] Because it uses a distributed MapReduce framework, it can simultaneously split multiple long texts to obtain split fragments and text vectors, thus improving processing efficiency.

[0025] S204 uses the Locality Sensitive Hash algorithm to map text vectors that meet the preset similarity conditions into the same hash bucket, calculates the similarity of text vectors in each hash bucket, and obtains the similarity value between every two split texts in the hash bucket.

[0026] Locality-Sensitive Hashing (LSH) refers to a hash function that maps high-dimensional data to hash buckets in a low-dimensional space. This increases the probability of similar data being mapped to the same hash bucket, while dissimilar data is mapped to different hash buckets. Specifically, in the Map layer of a distributed framework, each independent text vector is mapped to a different hash bucket using LSH. For example, text vectors that meet a predefined similarity condition are mapped to the same hash bucket, while those that do not meet the predefined similarity condition are mapped to a different hash bucket.

[0027] The preset similarity condition refers to the hash value obtained after the corresponding text vector in the hash bucket is hashed using the locality-sensitive hashing algorithm. Meeting the preset similarity condition means that, based on the hash value of the target text vector, multiple text vectors with the same or similar hash values ​​are obtained, and these multiple text vectors are mapped to the same hash bucket as the target text vector. The target text vector can be any text vector corresponding to a split text.

[0028] The hash buckets in the mapping can be understood as the reduce side of the distributed framework. Different hash buckets exist on the reduce side to receive the text vectors mapped by the Map side.

[0029] On the reduce side, the similarity of text vectors in each hash bucket is calculated to obtain the similarity value between every two split texts in the hash bucket. For example, by calculating the cosine similarity value between text vectors, the cosine similarity value is used as the similarity value between the split texts corresponding to the text vectors.

[0030] S206. Based on the similarity values ​​between the split texts in the hash bucket, construct a text similarity graph. In the text similarity graph, select a preset proportion of split texts as important texts based on the similarity values ​​between the split texts.

[0031] By using the split text in the hash bucket as nodes and the similarity values ​​between the split texts as edges, a text similarity graph is constructed. The constructed text similarity graph can filter out the nodes corresponding to the edges whose similarity values ​​between the split texts do not reach a preset threshold, and can also construct a connected subgraph for the nodes corresponding to the edges whose similarity values ​​between the split texts reach a preset threshold.

[0032] For example, in a text similarity graph, by calculating the similarity values ​​between the split texts, the split text with the highest similarity value in a connected subgraph of the text similarity graph is selected as the center text of the connected subgraph. From the split texts with the highest similarity values ​​in each connected subgraph, a predetermined proportion of the split texts are selected as important texts. For example, based on the Pareto principle, the similarity values ​​between the split texts in the text similarity graph can be obtained, and the top 20% of the split texts with the highest similarity values ​​can be selected as important texts. The predetermined proportion for selection can be set based on experience and is not specifically stipulated.

[0033] S208 calculates the importance score of important texts, and determines the target dataset based on the importance score. The target dataset is used for training the large model.

[0034] The importance scores of the important texts obtained from S206 are calculated, such as based on the time and extracted region of each important text. Based on these importance scores, various types of important texts are selected, resulting in diverse target datasets. This increases the diversity and timeliness of the target datasets, making them suitable for training large models.

[0035] The technical solution of this application embodiment obtains a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained. These long texts are then split and vectorized to obtain split texts and corresponding text vectors. Using a locality-sensitive hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. Similarity is calculated for the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket. Based on the split texts in the hash bucket and the similarity values ​​between them, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. The importance score of the important texts is calculated, and based on the importance score, a target dataset is determined. This target dataset is used for training the large model. Therefore, splitting and vectorizing the long texts obtained after preprocessing the text dataset for large model training to obtain split texts and text vectors improves the efficiency of long text processing. Text vectors that meet preset similarity conditions are mapped to the same hash bucket. This is used to calculate the similarity value between the split texts in each hash bucket and construct a text similarity graph. This allows for mapping relatively similar split texts that meet preset similarity conditions to the same hash bucket to form a text similarity graph. By selecting a preset proportion of split texts as important texts from the text similarity graph, the split texts with high similarity in the text similarity graph can be centrally deduplicated to obtain important texts. Then, by scoring the important texts, an importance score is obtained, which improves the quality and diversity of the data in the acquired target dataset. Finally, a high-value target dataset is obtained, which can quickly identify high-value data for use in training large models.

[0036] In one embodiment, by using the Locality Sensitive Hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. The similarity of the text vectors in each hash bucket is calculated to obtain the similarity value between every two split texts in the hash bucket (i.e., S204). The following steps A1-A2 can be performed: Step A1: Map the text vectors according to the Locality Sensitive Hash algorithm, and map text vectors whose hash values ​​meet the preset similarity conditions to the same hash bucket, resulting in multiple hash buckets containing multiple text vectors. The preset similarity conditions include the hash values ​​of the text vectors in the hash bucket after hash processing.

[0037] In a distributed framework, text vectors are retrieved from the Map phase and each text vector is mapped to the Reduce phase using a Locality Sensitive Hashing (LSH) algorithm. The Reduce phase acts like a hash bucket, receiving the distributed text vectors. LSH uses a specific set of hash functions to map similar text vectors to the same or similar hash values, thus placing them in the same hash bucket.

[0038] The preset similarity conditions specifically include the hash value of the text vectors in the hash bucket after hashing. Meeting the preset similarity conditions means that the hash value of the text vector after hashing is the same as or similar to the hash value of the text vectors in the hash bucket. Specifically, if the hash value obtained after hashing a text vector using the Locality Sensitive Hash algorithm is the same as or similar to the hash value of the text vectors in the hash bucket, then that text vector is mapped to that hash bucket. It is also possible to obtain multiple text vectors with the same or similar hash values ​​and map them to the same hash bucket. This allows text vectors that meet the preset similarity conditions to be mapped to the same hash bucket, ensuring that similar split text is mapped to the same hash bucket with a higher probability, while dissimilar split text is mapped to the same hash bucket with a lower probability.

[0039] It should be noted that, in order to ensure the uniformity of the split text distributed to the hash bucket, when the hash bucket reaches its capacity limit, a bucket overflow strategy is adopted, and the newly received text vector is placed into the adjacent hash bucket.

[0040] Step A2: Calculate the cosine similarity value between every two text vectors in each hash bucket, and determine the similarity value between every two split texts in the hash bucket based on the cosine similarity value.

[0041] The cosine similarity value includes the cosine angle between two text vectors. The cosine angle is used to evaluate the similarity between two text vectors; therefore, the cosine similarity value is used to determine the similarity between split texts.

[0042] Calculate the cosine similarity value between every two text vectors in each hash bucket, and use the cosine similarity value as the similarity value between every two split texts.

[0043] In this embodiment, by employing a distributed framework, multiple text vectors can be mapped simultaneously. Using a locality-sensitive hashing algorithm, the hash values ​​of the text vectors are obtained. Text vectors whose hash values ​​meet preset similarity conditions are mapped to the same hash bucket. This allows for the mapping of split texts with high probability of similarity to the same hash bucket, thus performing preliminary classification of the split texts. Furthermore, the cosine similarity value of the text vectors in each hash bucket is calculated as the similarity value between any two split texts.

[0044] In one embodiment, a text similarity graph is constructed based on the split texts in the hash bucket and the similarity values ​​between the split texts (i.e., S206), which can be achieved by performing the following steps B1-B3: Step B1: Determine graph nodes based on the split text in each hash bucket, determine graph feature edges based on the similarity values ​​between the split texts, and filter out graph feature edges whose similarity values ​​between the split texts do not reach a preset threshold based on the similarity values ​​between the split texts.

[0045] Each hash bucket contains a split text as a graph node, and the similarity value between split texts is used as a graph feature edge. The preset threshold is a preset similarity value. When the similarity value between split texts does not reach the preset similarity value, the similarity value between the split texts is filtered out, which means deleting the graph feature edge between the split texts.

[0046] Step B2: Obtain the remaining feature edges in the hash bucket after filtering out the graph feature edges whose similarity values ​​between the split texts do not reach the preset threshold, and the graph nodes corresponding to the remaining feature edges, and construct a similarity graph.

[0047] Specifically, in each hash bucket, the graph feature edges whose similarity values ​​between the split texts that have not reached the preset threshold are obtained. The remaining feature edges after filtering are determined. The remaining feature edges and the graph nodes corresponding to the remaining feature edges are constructed into a similar graph. Not all graph nodes in the similar graph are connected.

[0048] Step B3: Based on the similarity graph, generate multiple connected subgraphs using the connected component algorithm, and construct a text similarity graph by treating the multiple connected subgraphs as multiple communities using the community detection algorithm.

[0049] Connected component analysis (CFI) algorithms refer to decomposing a similar graph into multiple connected subgraphs. Based on the constructed similar graph, multiple connected subgraphs are obtained, where the nodes in each subgraph are connected. Multiple connected subgraphs can be represented as { , ... }

[0050] Community detection algorithms refer to a set of tightly connected, interconnected graph nodes that form a structure with strong internal connections and sparse external connections. Specifically, multiple connected subgraphs are treated as internal parts, representing multiple communities, and the connections between these subgraphs are treated as external parts, representing the connections between communities. Calculations are performed on these connected subgraphs to obtain a text similarity graph with strong internal connections and sparse external connections. This can be represented as { , ... }

[0051] In this embodiment, the similarity values ​​between the split texts in each hash bucket are obtained. By filtering out graph feature edges whose similarity values ​​between split texts do not reach a preset threshold, a similarity graph is constructed. This allows for the initial screening of split texts from massive datasets to build a similarity graph. Multiple connected subgraphs are obtained using a connected component algorithm, enabling the acquisition of connected subgraphs between multiple similar split texts. This facilitates subsequent analysis of each connected subgraph. Then, a community detection algorithm is used to construct a text similarity graph. Through this text similarity graph, the density relationship between each community is clearly obtained, improving the filtering of highly similar split texts to achieve deduplication.

[0052] In one embodiment, in the text similarity graph, based on the similarity values ​​between the split texts, a preset proportion of the split texts are selected as important texts (i.e., S206), and the following steps C1-C3 can be performed: Step C1: Based on the similarity values ​​between the split texts in the text similarity graph, determine the split text with the highest similarity value in the connected subgraph of each community corresponding to the text similarity graph, and use it as the center text.

[0053] Based on the text similarity graph, multiple communities corresponding to the text similarity graph are obtained. Based on the similarity values ​​between the split texts in the text similarity graph, connected subgraphs are determined within these communities. The split text with the highest similarity value in each connected subgraph is then selected as the center text of that subgraph. It should be understood that a text similarity graph contains multiple communities, each community contains a connected subgraph, and each connected subgraph contains multiple split texts but only one center text. Since all split texts in the connected subgraph have high similarity, selecting the split text with the highest similarity value as the Chinese text filters out other highly similar split texts, retaining only the center text and removing other highly similar and duplicate split texts.

[0054] Step C2: Sort the center text of each connected subgraph according to the similarity value between the corresponding split texts to obtain the sorted center text.

[0055] The central text of each connected subgraph is sorted according to the similarity value between its subtexts, from highest to lowest, resulting in multiple sorted subtexts. For example, each connected subgraph in a community corresponds to one central text, and the central texts corresponding to multiple connected subgraphs are sorted according to the similarity value between their subtexts. It should be understood that the central text is also one of the subtexts in the connected subgraph.

[0056] Step C3: Select a predetermined proportion of split text from the sorted central text as important text.

[0057] Based on the sorted central text obtained in step C2, obtain the split texts of a preset proportion in the sorted central text. For example, according to the Pareto principle, determine the top 20% of central texts by calculating the similarity value between the split texts and sorting the central texts according to the similarity value between the split texts. The top 20% of split texts, i.e., the central texts, are regarded as important texts.

[0058] In this embodiment, based on the connected subgraphs of multiple communities in the text similarity graph, the split text with the highest similarity value among the split texts in each connected subgraph is selected. This filters out other split texts with high similarity in the connected subgraphs, obtaining the split text with the highest similarity value as the center text, thus achieving the purpose of deduplication of split texts in the connected subgraphs. By sorting the split texts with the highest similarity values ​​among the split texts in multiple connected subgraphs, i.e., the center texts, a preset proportion of split texts is obtained. Using the preset proportion of split texts as important texts can improve the efficiency of obtaining representative text data.

[0059] In one embodiment, importance scores of important texts are calculated, and the target dataset (i.e., S208) is determined based on these importance scores. This can be achieved by performing the following steps D1-D3: Step D1: Determine the importance score of each important text in the text similarity graph based on the occurrence time of the important text, the current time, the number of times the important text appears at the region level, and the extraction region of the important text.

[0060] Based on the acquired multiple important texts, the occurrence time, current time, frequency of different regional levels appearing in each important text, and extraction region for each important text are determined. Specifically, the occurrence time includes the year the important text appeared, the current time includes the current year, and the frequency of regional levels appearing in the important text includes the number of times different regional levels appear in the important text. Different regional levels, such as national, provincial, municipal, and county levels, may also correspond to different policies. The extraction region for important texts includes the region to which the important texts belong, where the extraction region is 100 if it is within the jurisdiction and -100 if it is outside the jurisdiction.

[0061] The formula for calculating the importance score S is as follows:

[0062] in, This represents the initial recommendation, which can be 100. The longer the time frame, the higher the recommendation level. This indicates that the values ​​corresponding to the initial recommendations decrease sequentially. For example, if an important text is no longer among the initial recommendations after a long period of time, then the important text has been present for a longer period of time. The indicated value needs to be reduced, such as to 90, and the specific value should be reduced sequentially without specific restrictions; This indicates the regional level. The regional policies corresponding to this important text are at different regional levels, which can include: national, provincial / municipal, and district / county levels, etc. National-level policies grant... The value is 10 for provincial and municipal levels, 5 for districts and counties, and 1 for others. The values ​​represent a region-level relationship without specific limitations; This indicates 100, meaning within the jurisdiction. -100 indicates outside the jurisdiction; now indicates the current time; year indicates the time when important text appeared; Indicates the number of times a region level appears in important text; This indicates the extracted region, and ∑ represents the summation operation, calculating the sum of the occurrences of different region levels in important text. It represents multiplication.

[0063] Step D2: Based on the importance score, determine the text data in the long text corresponding to the important text.

[0064] Based on the importance score calculated in step D1, the reduce end outputs a preset number of important texts. By calculating the importance score, a variety of important texts can be obtained. The preset number can be the number obtained based on historical experience.

[0065] It should be noted that if the reduce side outputs a preset number of important texts but has not yet achieved global deduplication, then it is necessary to obtain the remaining text vectors from the map side or obtain all text vectors again, and map them to the reduce side using the locality-sensitive hashing algorithm for deduplication again.

[0066] Find the corresponding text data in the preprocessed long text from the preset number of important texts.

[0067] Step D3: Convert the text data in the long text according to the preset format to obtain the target dataset in the preset format.

[0068] For example, based on a specific large model, text data in long texts is converted according to a preset format to obtain a target dataset that conforms to the preset format of the large model. This target dataset is then used to train the large model. Preset formats include specific object notation (JavaScript Object Notation, JSON) formats or binary file TFRecord formats.

[0069] In this embodiment, by assigning weights to important texts, an importance score is obtained for each important text. Based on the importance score, a preset number of important texts are determined in the corresponding long text. By obtaining the source data, the deduplicated source data is converted into the target dataset required by the large model according to a preset format, which can improve the accuracy of obtaining the target dataset.

[0070] In one embodiment, a text dataset for training a large model is obtained. This dataset is preprocessed to obtain multiple long texts. These long texts are then split and vectorized to obtain the split texts and their corresponding text vectors (i.e., S202). The following steps E1-E3 can be performed: Step E1: Obtain a multi-source text dataset for training a large model, and filter the text in the text dataset using a preset filtering method to obtain the filtered text in the text dataset.

[0071] Large-scale models are generally government affairs models, and the obtained text datasets are generally government-related text data.

[0072] The filtering methods include using regular expressions, setting rules, and other methods to clean individual information such as ID numbers, mobile phone numbers, and postal addresses from the text dataset, filter sensitive information in the individual information, and also clean and filter special symbols in the text dataset.

[0073] Step E2 involves performing entity recognition on the text in the filtered text dataset to obtain key attribute information of the text in the text dataset, and identifying the text in the text dataset with key attribute information as long text.

[0074] Key attribute information is used to describe the text in the text dataset.

[0075] Obtain the text dataset after data filtering in step E1, and use entity recognition methods to identify key attribute information such as year, region level, region, time, person, and position in the text dataset.

[0076] Entity recognition is an important task in the field of natural language processing, capable of identifying key attribute information in text.

[0077] Because government-related data is geographically and temporally specific, when performing semantic comparisons, we avoid the influence of entities on semantics. The time of text appearance is represented as year, the text region level is identified as national, provincial, municipal, district / county, etc., the extracted region is represented as area, the time is represented as date, the person is represented as person, and the position is represented as position, etc.

[0078] The text dataset, after filtering and entity recognition, is identified as long text.

[0079] Step E3: Split the long text to obtain split text, and then vectorize the split text to obtain text vectors.

[0080] The long text from step E2 is split into smaller segments of length n. These segments share a common length of m, and the lengths of the first and last characters of this common length are preserved to ensure semantic coherence. Based on current requirements, n is typically chosen to be 1024 and m to be 100. By ensuring the common content within the segments, semantic coherence is guaranteed.

[0081] The Map side can simultaneously read multiple split texts after splitting, embed the split texts into vectors, obtain the vector information of each split text, and use it as a text vector.

[0082] In this embodiment, by filtering and entity recognition on multi-source text datasets, a preprocessed text dataset, i.e., long text, is obtained. This allows for the acquisition of various types of text data, ensuring the reliability and efficiency of obtaining long text. Distributed splitting and vectorization of the long text yields split text, which improves the efficiency of subsequent data processing within the long text.

[0083] In conclusion, Figure 3 This is a flowchart of a method for constructing a large model dataset according to an embodiment of this application, as follows: Figure 3As shown, the process involves acquiring a text dataset and preprocessing it, including using regular expressions and other filtering methods to remove personal information such as ID numbers, phone numbers, and postal addresses, as well as sensitive information; and performing entity recognition on the text dataset to identify key attributes such as the time of occurrence, region level, and extraction region of the text. Map-side data processing includes splitting the preprocessed long text into segments of a specific length and vectorizing these segments into text vectors. Data distribution involves using Locality Sensitive Hashing (LSH) and bucket overflow strategies to distribute the text vectors from the Map side to the Reduce side. Reduce-side data cleaning includes calculating the similarity values ​​between the split texts, constructing a similarity graph by using the split texts as nodes and those with similarity values ​​exceeding a preset threshold as edges, using a community detection algorithm to identify the central texts, and selecting the top 20% of central texts as important texts. An importance score is then obtained using a weighted evaluation scoring algorithm to identify high-quality data and extract a preset number of important texts with high importance scores. Determine whether the global deduplication effect meets the preset effect. If it does not meet the preset effect, continue to deduplicatize the text vectors from the data distribution stage. If it meets the preset effect, obtain the text data corresponding to the preset number of important texts with the highest importance scores in the long text, and perform format conversion on the text data to finally obtain the target dataset.

[0084] Figure 4 This is a schematic flowchart illustrating a method for constructing a large model dataset according to another embodiment of this application, such as... Figure 4 As shown, the method includes the following steps: S401: Obtain a multi-source text dataset for training a large government affairs model, and filter the text dataset using a filtering method to obtain the text in the filtered text dataset.

[0085] S402, perform entity recognition on the text in the filtered text dataset to obtain the key attribute information of the text in the text dataset, and identify the text in the text dataset with key attribute information as long text.

[0086] S403 splits the long text into split text, and then vectorizes the split text to obtain text vectors.

[0087] S404. The text vectors are mapped according to the locality-sensitive hashing algorithm. Text vectors whose hash values ​​meet the preset similarity conditions are mapped to the same hash bucket, resulting in multiple hash buckets containing multiple text vectors.

[0088] S405, calculate the cosine similarity value between every two text vectors in each hash bucket, and determine the similarity value between every two split texts in the hash bucket based on the cosine similarity value.

[0089] S406: Based on the split text in each hash bucket, determine the graph nodes; based on the similarity value between the split texts, determine the graph feature edges; and based on the similarity value between the split texts, filter out the graph feature edges whose similarity value between the split texts does not reach a preset threshold.

[0090] S407: Obtain the remaining feature edges in the hash bucket after filtering out the graph feature edges whose similarity values ​​between the split texts do not reach the preset threshold, and the graph nodes corresponding to the remaining feature edges, and construct a similarity graph.

[0091] S408. Based on the similarity graph, multiple connected subgraphs are generated using the connected component algorithm, and the multiple connected subgraphs are used as multiple communities through the community detection algorithm to construct a text similarity graph.

[0092] S409, Based on the similarity values ​​between the split texts in the text similarity graph, determine the split text with the highest similarity value among the split texts in the connected subgraph of each community corresponding to the text similarity graph, and use it as the center text.

[0093] S410, sort the center text of each connected subgraph according to the similarity value between the corresponding split texts, and obtain the sorted center text.

[0094] S411, in the sorted central text, select the split text with a preset ratio as important text.

[0095] S412, based on the occurrence time of important texts, the current time, the number of times the important texts appear at the regional level, and the extraction region of the important texts, determine the importance score of each important text in the text similarity graph.

[0096] S413. Based on the importance score, determine the text data in the long text corresponding to the important text, and convert the text data in the long text according to the preset format to obtain the target dataset in the preset format for training the government affairs big data model.

[0097] The specific processes from S401 to S413 have been described in detail in the above embodiments and will not be repeated here.

[0098] The technical solution of this application embodiment obtains a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained. These long texts are then split and vectorized to obtain split texts and corresponding text vectors. Using a locality-sensitive hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. Similarity is calculated for the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket. Based on the split texts in the hash bucket and the similarity values ​​between them, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. The importance score of the important texts is calculated, and based on the importance score, a target dataset is determined. This target dataset is used for training the large model. Therefore, splitting and vectorizing the long texts obtained after preprocessing the text dataset for large model training to obtain split texts and text vectors improves the efficiency of long text processing. Text vectors that meet preset similarity conditions are mapped to the same hash bucket. This is used to calculate the similarity value between the split texts in each hash bucket and construct a text similarity graph. This allows for mapping relatively similar split texts that meet preset similarity conditions to the same hash bucket to form a text similarity graph. By selecting a preset proportion of split texts as important texts from the text similarity graph, the split texts with high similarity in the text similarity graph can be centrally deduplicated to obtain important texts. Then, by scoring the important texts, an importance score is obtained, which improves the quality and diversity of the data in the acquired target dataset. Finally, a high-value target dataset is obtained, which can quickly identify high-value data for use in training large models.

[0099] In summary, specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0100] The above is a method for constructing a large model dataset provided by an embodiment of this application. Based on the same idea, an apparatus for constructing a large model dataset is also provided by an embodiment of this application.

[0101] Figure 5 This is a schematic diagram of a large model dataset construction device according to an embodiment of the present invention. Figure 5 As shown, the large model dataset construction device includes: a preprocessing module 51, a similarity module 52, a filtering module 53, and a format conversion module 54. The preprocessing module 51 is used to obtain a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained, and the long texts are split and vectorized to obtain the split texts and corresponding text vectors. The similarity module 52 is used to map text vectors that meet the preset similarity conditions to the same hash bucket using the locality-sensitive hashing algorithm, calculate the similarity of text vectors in each hash bucket, and obtain the similarity value between every two split texts in the hash bucket. The filtering module 53 is used to construct a text similarity graph based on the similarity values ​​between the split texts in the hash bucket. In the text similarity graph, a preset proportion of the split texts are selected as important texts based on the similarity values ​​between the split texts. The format conversion module 54 is used to calculate the importance score of important texts, and determine the target dataset based on the importance score. The target dataset is used for training the large model.

[0102] In one embodiment, the similarity module 52 is used to map text vectors according to the locality-sensitive hashing algorithm, mapping text vectors whose hash values ​​satisfy a preset similarity condition to the same hash bucket, resulting in multiple hash buckets including multiple text vectors; calculating the cosine similarity value of every two text vectors in each hash bucket, and determining the similarity value between every two split texts in the hash bucket based on the cosine similarity value, wherein the preset similarity condition includes the hash value of the text vectors in the hash bucket after hash processing.

[0103] In one embodiment, the filtering module 53 is specifically used to determine graph nodes based on the split text in each hash bucket, determine graph feature edges based on the similarity values ​​between the split texts, filter out graph feature edges whose similarity values ​​between the split texts do not reach a preset threshold based on the similarity values ​​between the split texts, obtain the remaining feature edges in the hash bucket after filtering out the graph feature edges whose similarity values ​​between the split texts do not reach the preset threshold, and the graph nodes corresponding to the remaining feature edges, and construct a similarity graph; based on the similarity graph, generate multiple connected subgraphs using a connected component algorithm, and construct a text similarity graph by treating the multiple connected subgraphs as multiple communities using a community detection algorithm.

[0104] In one embodiment, the filtering module 53 is further configured to determine, based on the similarity values ​​between the split texts in the text similarity graph, the split text with the highest similarity value among the split texts in each connected subgraph corresponding to the text similarity graph, and use it as the center text; sort the center texts of each connected subgraph according to the similarity values ​​between the split texts corresponding to the center texts to obtain the sorted center texts; and select a preset proportion of the split texts as important texts from the sorted center texts.

[0105] In one embodiment, the format conversion module 54 is specifically used to determine the importance score of each important text in the text similarity graph based on the occurrence time of the important text, the current time, the number of times the important text appears at the region level, and the extraction region of the important text; based on the importance score, determine the text data in the long text corresponding to the important text; and convert the text data in the long text according to a preset format to obtain the target dataset in the preset format.

[0106] In one embodiment, the preprocessing module 51 is specifically used to acquire a multi-source text dataset for training a large model, filter the text in the text dataset using a preset filtering method to obtain the filtered text in the text dataset; perform entity recognition on the filtered text in the text dataset to obtain the key attribute information of the text in the text dataset; identify the text in the text dataset with key attribute information as long text, and use the key attribute information to describe the text in the text dataset; split the long text to obtain split text, and perform vectorization processing on the split text to obtain text vectors.

[0107] The technical solution of this application embodiment obtains a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained. These long texts are then split and vectorized to obtain split texts and corresponding text vectors. Using a locality-sensitive hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. Similarity is calculated for the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket. Based on the split texts in the hash bucket and the similarity values ​​between them, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. The importance score of the important texts is calculated, and based on the importance score, a target dataset is determined. This target dataset is used for training the large model. Therefore, splitting and vectorizing the long texts obtained after preprocessing the text dataset for large model training to obtain split texts and text vectors improves the efficiency of long text processing. Text vectors that meet preset similarity conditions are mapped to the same hash bucket. This is used to calculate the similarity value between the split texts in each hash bucket and construct a text similarity graph. This allows for mapping relatively similar split texts that meet preset similarity conditions to the same hash bucket to form a text similarity graph. By selecting a preset proportion of split texts as important texts from the text similarity graph, the split texts with high similarity in the text similarity graph can be centrally deduplicated to obtain important texts. Then, by scoring the important texts, an importance score is obtained, which improves the quality and diversity of the data in the acquired target dataset. Finally, a high-value target dataset is obtained, which can quickly identify high-value data for use in training large models.

[0108] Those skilled in the art will understand that Figure 5 The large model dataset construction device in the document can be used to implement the large model dataset construction method described above. The detailed description should be similar to that in the method section above, and will not be repeated here to avoid being cumbersome.

[0109] Based on the same technical concept, this application also provides an electronic device for executing the above-described method for constructing large model datasets. Figure 6 This is a schematic diagram of the structure of an electronic device to implement various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call a computer program stored in the memory 630 and executable on the processor 610 to perform the following steps: Obtain a text dataset for training a large model. Preprocess the text dataset to obtain multiple long texts. Then, split and vectorize the long texts to obtain the split texts and their corresponding text vectors. By using the Locality Sensitive Hash algorithm, text vectors that meet the preset similarity conditions are mapped to the same hash bucket. The similarity of the text vectors in each hash bucket is calculated to obtain the similarity value between every two split texts in the hash bucket. Based on the similarity values ​​between the split texts in the hash bucket, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. Calculate the importance score of important texts, and determine the target dataset based on the importance score. The target dataset is used to train the large model.

[0110] The technical solution of this application embodiment obtains a text dataset for training a large model. By preprocessing the text dataset, multiple long texts are obtained. These long texts are then split and vectorized to obtain split texts and corresponding text vectors. Using a locality-sensitive hashing algorithm, text vectors that meet preset similarity conditions are mapped to the same hash bucket. Similarity is calculated for the text vectors in each hash bucket to obtain the similarity value between every two split texts in the hash bucket. Based on the split texts in the hash bucket and the similarity values ​​between them, a text similarity graph is constructed. In the text similarity graph, a preset proportion of split texts are selected as important texts based on the similarity values ​​between them. The importance score of the important texts is calculated, and based on the importance score, a target dataset is determined. This target dataset is used for training the large model. Therefore, splitting and vectorizing the long texts obtained after preprocessing the text dataset for large model training to obtain split texts and text vectors improves the efficiency of long text processing. Text vectors that meet preset similarity conditions are mapped to the same hash bucket. This is used to calculate the similarity value between the split texts in each hash bucket and construct a text similarity graph. This allows for mapping relatively similar split texts that meet preset similarity conditions to the same hash bucket to form a text similarity graph. By selecting a preset proportion of split texts as important texts from the text similarity graph, the split texts with high similarity in the text similarity graph can be centrally deduplicated to obtain important texts. Then, by scoring the important texts, an importance score is obtained, which improves the quality and diversity of the data in the acquired target dataset. Finally, a high-value target dataset is obtained, which can quickly identify high-value data for use in training large models.

[0111] The specific execution steps can be found in the various steps of the above-described large model dataset construction method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0112] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0113] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0114] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0115] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0116] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described large model dataset construction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0117] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0118] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described large model dataset construction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0119] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0120] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommended method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0121] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0123] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for constructing a large model data set, characterized in that, The method includes: Obtain a text dataset for training a large model. Preprocess the text dataset to obtain multiple long texts. Then, split and vectorize the long texts to obtain the split texts and their corresponding text vectors. By using the Locality Sensitive Hash algorithm, text vectors that meet the preset similarity conditions are mapped to the same hash bucket. The similarity of the text vectors in each hash bucket is calculated to obtain the similarity value between every two split texts in the hash bucket. Based on the similarity values ​​between the split texts in the hash bucket, a text similarity graph is constructed. In the text similarity graph, a preset proportion of the split texts are selected as important texts based on the similarity values ​​between the split texts. The importance score of the important text is calculated, and the target dataset is determined based on the importance score. The target dataset is used for training the large model.

2. The method of claim 1, wherein, The step involves using a locality-sensitive hashing algorithm to map text vectors that meet preset similarity conditions into the same hash bucket, calculating the similarity of the text vectors in each hash bucket, and obtaining the similarity value between every two split texts in the hash bucket, including: The text vectors are mapped according to the locality-sensitive hashing algorithm, and the text vectors whose hash values ​​satisfy the preset similarity condition are mapped to the same hash bucket, resulting in multiple hash buckets including multiple text vectors. The preset similarity condition includes the hash value of the text vectors in the hash bucket after hash processing. Calculate the cosine similarity value of every two text vectors in each hash bucket, and determine the similarity value between every two split texts in the hash bucket based on the cosine similarity value.

3. The method of claim 1, wherein, The step of constructing a text similarity graph based on the similarity values ​​between the split texts in the hash bucket includes: Based on the split text in each hash bucket, graph nodes are determined, graph feature edges are determined based on the similarity value between the split texts, and graph feature edges whose similarity value between the split texts does not reach a preset threshold are filtered out based on the similarity value between the split texts. Obtain the remaining feature edges in the hash bucket after filtering out the graph feature edges whose similarity values ​​between the split texts do not reach a preset threshold, and the graph nodes corresponding to the remaining feature edges, and construct a similarity graph; Based on the similarity graph, multiple connected subgraphs are generated using the connected component algorithm, and the multiple connected subgraphs are used as multiple communities through the community detection algorithm to construct the text similarity graph.

4. The method of claim 3, wherein, The step of selecting a preset proportion of the split texts as important texts based on the similarity values ​​between the split texts includes: Based on the similarity values ​​between the split texts in the text similarity graph, the split text with the highest similarity value in the connected subgraph of each community corresponding to the text similarity graph is determined and used as the center text; Sort the center texts of each of the connected subgraphs according to the similarity values between the split texts corresponding to the center texts, to obtain the center texts after sorting processing; In the center texts after sorting processing, select a preset proportion of the split texts as the important texts.

5. The method of claim 1, wherein, The method further includes: determining the importance scores of the important texts according to the appearance times of the important texts, the current time, the number of times of the region-level appearance in the important texts, and the extraction region of the important texts; determining the text data in the long text corresponding to the important texts based on the importance scores; converting the text data in the long text into a preset format to obtain the target data set in the preset format.

6. The method of claim 1, wherein, The method further includes: obtaining a plurality of long texts by preprocessing the text data set for large model training, and splitting and vectorizing the long texts to obtain split texts and corresponding text vectors. obtaining a plurality of long texts by preprocessing the text data set for large model training, and splitting and vectorizing the long texts to obtain split texts and corresponding text vectors. The method further includes: 7.A large model dataset construction apparatus, characterized by comprising: obtaining a plurality of long texts by preprocessing the text data set for large model training, and splitting and vectorizing the long texts to obtain split texts and corresponding text vectors. The method further includes: a preprocessing module configured to obtain a text data set for large model training, preprocess the text data set to obtain a plurality of long texts, split and vectorize the long texts to obtain split texts and corresponding text vectors; a similarity module configured to map the text vectors satisfying a preset similarity condition to the same hash bucket by using a local sensitive hashing algorithm, calculate the similarity between the text vectors in each hash bucket to obtain the similarity values between each two split texts in the hash bucket; a screening module configured to construct a text similarity graph according to the split texts in the hash bucket and the similarity values between the split texts, and select a preset proportion of the split texts as important texts in the text similarity graph based on the similarity values between the split texts; a format conversion module configured to calculate the importance scores of the important texts, and determine a target data set for training the large model according to the importance scores.

8. An electronic device, comprising: An apparatus comprising a processor and a memory electrically connected with the processor, the memory storing a computer program, the processor being configured to invoke and execute the computer program from the memory to implement the method of claim 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium is configured to store a computer program, the computer program being executable by a processor to implement the method of claim 1-6.

10. A computer program product, characterised in that, A computer program product, which, when executed by a processor, implements the method of claim 1-6.

Images