Sample construction method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a sample construction method and device, electronic equipment, a storage medium and a program product. The method can be applied to a text processing model of a vehicle-mounted scene, the text content is segmented, and the text sub-content corresponding to each text segment in the text content is obtained; for each text segment, determining current to-be-processed text sub-content from each text sub-content according to the content sequence of each text sub-content in the text segment; adding the currently processed text sub-content obtained every time into a preset text data set, detecting the text length in the text data set after addition every time until the text data set after addition meets a preset sample text length condition, and obtaining a reference text data set corresponding to the text segment; sample text data is constructed based on the reference text data set. In the embodiment of the invention, the sample text data can keep the structure of the original sample text. Therefore, according to the scheme, the high-quality sample text can be obtained.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a sample construction method, device, electronic device, storage medium, and program product. Background Art

[0002] During the training process of a text processing model, a large number of text samples are required for model learning to equip the model with the relevant knowledge. Specifically, by analyzing the input text samples, the model can learn the patterns of language, the usage of vocabulary, and the relationships between contexts. This allows the trained text processing model to generate coherent and reasonable text based on previous training experience, and even answer user questions or provide relevant information. The goal of training a text processing model with samples is to improve the model's language understanding and generation capabilities, making it as close as possible to human expression and thinking. Through continuous iteration and optimization of the training process, we strive to improve the performance of text processing models so that they can be better applied in natural language processing, dialogue systems, and other related fields.

[0003] Currently, when constructing training samples for a model, the original text data is typically cut into fixed-length segments to convert them into batches of data samples. This arbitrary text segmentation destroys the original structure of the text, resulting in poor semantic integrity of the constructed sample text, hindering sample quality and leading to poor performance of the trained model. Furthermore, in some scenarios, to ensure that the constructed samples meet quality requirements, further manual review or annotation is required, making the sample construction process cumbersome and time-consuming. This is not conducive to quickly constructing samples that meet the requirements when processing large-scale, high-dimensional data. Summary of the Invention

[0004] The embodiments of the present application provide a sample construction method, device, electronic device, storage medium and program product, which can enable the constructed sample to maintain the structure of the original text, improve the semantic integrity of the constructed sample, and facilitate the acquisition of high-quality sample text. In addition, the quality of the sample constructed in this way is high, avoiding manual review of the sample, thereby improving the efficiency of sample construction.

[0005] The present invention provides a sample construction method, including:

[0006] Acquire text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence;

[0007] Segmenting the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content;

[0008] For each text segment, according to the content order of each text sub-content in the text segment, determine the current text sub-content to be processed from each text sub-content;

[0009] Add the current to-be-processed text sub-content to a preset text data set to obtain a post-addition text data set corresponding to the text segment, and detect the length of the text in the post-addition text data set;

[0010] When it is detected that the text length in the added text dataset does not meet the preset sample text length condition, the added text dataset is set as a new preset text dataset, and the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment is returned to be executed until the text length in the added text dataset meets the preset sample text length condition, thereby obtaining a reference text dataset corresponding to the text segment;

[0011] Based on the text in the reference text data set, sample text data is constructed.

[0012] The present application also provides a sample construction device, including:

[0013] A text content acquisition unit, configured to acquire text content for constructing a sample, wherein the text content includes at least one text segment, and each text segment includes at least one text sentence;

[0014] A text content segmentation unit, configured to segment the text content based on position information of each text segment and text sentence in the text content, to obtain at least one text sub-content corresponding to each text segment in the text content;

[0015] a sub-content determining unit, configured to determine, for each text segment, a current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment;

[0016] The sub-content adding unit is used to add the current to-be-processed text sub-content to the preset text data set, obtain the added text data set corresponding to the text segment, and detect the text length in the added text data set;

[0017] a set acquisition unit for, when detecting that the text length in the added text data set does not satisfy a preset sample text length condition, taking the added text data set as a new preset text data set, returning to execute the step of determining a current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment, until the text length in the added text data set satisfies the preset sample text length condition, thereby obtaining a reference text data set corresponding to the text segment;

[0018] The sample construction unit is used to construct sample text data based on text in the reference text data set.

[0019] In some embodiments, the apparatus further includes a title acquisition unit, a template determination unit, an instruction generation unit, and a sample pair construction unit:

[0020] A title acquisition unit, used to acquire the content title of the text content;

[0021] a template determining unit, configured to determine a target instruction template matching the content title from preset instruction templates;

[0022] An instruction generation unit, configured to fuse the content title with the target instruction template to obtain text generation instruction information corresponding to the text content;

[0023] The sample pair construction unit is used to construct a sample data pair based on the text generation instruction information and the text in the reference text data set corresponding to the text content. The sample data pair includes the text generation instruction information and the sample text corresponding to the text generation instruction information.

[0024] In some embodiments, the apparatus further includes a text prediction unit, a loss calculation unit, and a model training unit:

[0025] A text prediction unit, configured to perform text prediction processing on the text generation instruction information in the sample data pair through a text processing model to obtain a predicted text;

[0026] A loss calculation unit, configured to perform loss calculation on the predicted text and the sample text in the sample data pair to obtain a predicted loss value;

[0027] The model training unit is used to train the text processing model based on the prediction loss value, so that the text processing model learns the knowledge corresponding to the sample data.

[0028] In some embodiments, the sample construction unit includes a data supplementation subunit and a sample construction subunit:

[0029] The data supplement subunit is used to use preset characters to perform data supplement processing on the reference text data set when the text length in the reference text data set does not reach the preset text length, so as to obtain an updated reference text data set;

[0030] The sample construction subunit is used to construct sample text data based on the updated reference text data set.

[0031] In some embodiments, the text content acquisition unit includes a text acquisition subunit and a text analysis subunit:

[0032] A text acquisition subunit, used to acquire the initial text;

[0033] The text analysis subunit is used to perform text analysis on the initial text to obtain the text content in the initial text that meets the preset main text conditions. The text content is the text content used to construct the sample.

[0034] In some embodiments, the text analysis subunit includes a title information acquisition subunit, a content title determination subunit, and a text content determination subunit:

[0035] A title information acquisition subunit, configured to acquire at least one of a preset title format and a preset title expression;

[0036] a content title determination subunit, configured to determine at least one content title matching the initial text based on at least one of a preset title format and a preset title expression;

[0037] The text content determination subunit is used to determine text content corresponding to each content title in the initial text and used to construct the sample based on the title position of each content title in the initial text.

[0038] In some embodiments, the text analysis subunit includes a semantic segmentation subunit, a semantic similarity determination subunit, and a text content determination subunit:

[0039] A semantic segmentation subunit, used to perform semantic segmentation processing on the initial text to obtain at least one initial text paragraph;

[0040] a semantic similarity determination subunit, configured to determine, for each initial text paragraph, the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph;

[0041] The text content determination subunit is configured to determine the text content for constructing a sample based on the initial text paragraph and its corresponding adjacent initial text paragraph if the paragraph semantic similarity meets a preset semantic similarity condition.

[0042] In some embodiments, the semantic similarity determination subunit is used to perform word prediction processing on each word in the corresponding adjacent initial text paragraph based on the initial text paragraph to obtain the word prediction probability of each word in the corresponding adjacent initial text paragraph; for each word in the corresponding adjacent initial text paragraph, based on the word prediction probability, determine the word prediction relevance index of each word; fuse the word prediction relevance index of each word to obtain the paragraph prediction relevance index of the corresponding adjacent initial text paragraph; and determine the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph based on the paragraph prediction relevance index.

[0043] In some embodiments, the system further includes an associated information acquisition subunit, a word determination subunit, a sensitive information determination subunit, and a desensitization subunit:

[0044] An associated information acquisition subunit is configured to acquire at least one associated word list corresponding to a preset sensitive word type, wherein the associated word list includes at least one associated word, which is a word associated with a sensitive word of the preset sensitive word type;

[0045] a word determination subunit, configured to determine a target word that matches the target word from the text of the sample data pair according to at least one associated word list;

[0046] A sensitive information determination subunit, configured to determine sensitive word information from the text of the sample data pair based on the target word;

[0047] The desensitization sub-unit is used to replace the sensitive word information in the text of the sample data pair with a preset placeholder corresponding to the preset sensitive word type, thereby obtaining a desensitized sample data pair.

[0048] In some embodiments, the association information acquisition subunit is further configured to acquire a preset association word expression corresponding to a preset sensitive word type, where the preset association word expression is an expression composed of words associated with the sensitive word of the preset sensitive word type;

[0049] The word determination subunit is further configured to determine a target word that matches the sample data pair from the text of the sample data pair according to a preset associated word expression.

[0050] In some embodiments, the apparatus further includes a sample similarity calculation unit and a sample deduplication unit:

[0051] A sample similarity calculation unit, configured to calculate the similarity between any two constructed sample data pairs;

[0052] The sample deduplication unit is used to perform deduplication processing on multiple sample data pairs based on similarity to obtain deduplicated sample data pairs.

[0053] In some embodiments, the sample similarity calculation unit is used to perform word segmentation processing on each sample data pair to obtain sample words corresponding to each sample data pair; for any two sample data pairs, repeated sample words are determined from the sample words corresponding to them, and repeated sample words are sample words that appear repeatedly in any two sample data pairs; based on the number of repeated sample words and the total number of sample words in any two sample data pairs, the similarity between any two sample data pairs is obtained.

[0054] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the sample construction methods provided in the embodiments of the present application.

[0055] An embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps in any one of the sample construction methods provided in the embodiment of the present application.

[0056] An embodiment of the present application further provides a computer program product, including a computer program / instruction, which implements the steps of any sample construction method provided in the embodiment of the present application when the computer program / instruction is executed by a processor.

[0057] The embodiment of the present application can obtain text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence; based on the position information of each text segment and text sentence in the text content, the text content is segmented to obtain at least one text sub-content corresponding to each text segment in the text content; for each text segment, according to the content order of each text sub-content in the text segment, the current text sub-content to be processed is determined from each text sub-content; the current text sub-content to be processed is added to a preset text data set to obtain an added text data set corresponding to the text segment, and the text length in the added text data set is detected; when it is detected that the text length in the added text data set does not meet the preset sample text length condition, the added text data set is used as a new preset text data set, and the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment is returned to execute until the text length in the added text data set meets the preset sample text length condition, and a reference text data set corresponding to the text segment is obtained; and sample text data is constructed based on the text in the reference text data set.

[0058] In the present application, by utilizing the position information of each text fragment and text sentence in the text content, it can help to segment the text content and generate text sub-content with complete semantics. When processing each text fragment subsequently, the current text sub-content to be processed with semantic integrity and continuity can be determined according to the content order of each text sub-content in the text fragment, and the current processed text sub-content obtained each time is added to the preset text data set. After each addition, the length of the text in the added text data set is detected until the added text data set meets the preset sample text length condition. In this way, a reference text data set corresponding to the text fragment that meets the input requirements of the text processing model can be obtained, and the reference text data set contains text with semantic integrity and continuity. Therefore, the sample text data constructed based on the text in the reference text data set can maintain the structure of the original text, improve the semantic integrity of the constructed sample, and is conducive to obtaining high-quality sample text. In addition, the quality of the sample constructed in this way is high, avoiding manual review of the sample, thereby improving the efficiency of sample construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1a This is a schematic diagram of a scenario of a sample construction method provided in an embodiment of the present application;

[0061] Figure 1b is a flow chart of a sample construction method provided in an embodiment of the present application;

[0062] Figure 2a This is a flow chart of the sample construction method provided in the embodiment of the present application applied to a large language model;

[0063] Figure 2b This is a schematic diagram of sensitive words corresponding to the preset sensitive word types provided in an embodiment of the present application;

[0064] Figure 2c It is a schematic diagram of the framework of the sample construction method provided in the embodiment of the present application;

[0065] Figure 3 is a schematic structural diagram of a sample construction device provided in an embodiment of the present application;

[0066] Figure 4 It is a structural diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0068] Embodiments of the present application provide a sample construction method, apparatus, electronic device, storage medium, and program product.

[0069] The sample construction device can be integrated into an electronic device, such as a terminal or a server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0070] In some embodiments, the sample construction device may also be integrated into multiple electronic devices. For example, the sample construction device may be integrated into multiple servers, and the sample construction method of the present application may be implemented by the multiple servers.

[0071] In some embodiments, the server may also be implemented in the form of a terminal.

[0072] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0073] It is understandable that in the specific implementation of this application, related data such as text content used to construct samples is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0074] For example, reference Figure 1aThe electronic device V can obtain text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence; based on the position information of each text segment and text sentence in the text content, the text content is segmented to obtain at least one text sub-content corresponding to each text segment in the text content; for each text segment, according to the content order of each text sub-content in the text segment, the current text sub-content to be processed is determined from each text sub-content; the current text sub-content to be processed is added to a preset text data set to obtain an added text data set corresponding to the text segment, and the text length in the added text data set is detected; when it is detected that the text length in the added text data set does not meet the preset sample text length condition, the added text data set is used as a new preset text data set, and the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment is returned to execute until the text length in the added text data set meets the preset sample text length condition, and a reference text data set corresponding to the text segment is obtained; and sample text data is constructed based on the text in the reference text data set.

[0075] By utilizing the position information of each text fragment and text sentence in the text content, it can help segment the text content and generate text sub-content with complete semantics. When processing each text fragment subsequently, the current text sub-content to be processed with semantic integrity and continuity can be determined according to the content order of each text sub-content in the text fragment, and the current processed text sub-content obtained each time is added to the preset text data set. After each addition, the length of the text in the added text data set is detected until the added text data set meets the preset sample text length condition. In this way, a reference text data set corresponding to the text fragment that meets the input requirements of the text processing model can be obtained, and the reference text data set contains text with semantic integrity and continuity. Therefore, the sample text data constructed based on the text in the reference text data set can maintain the structure of the original text, improve the semantic integrity of the constructed sample, and is conducive to obtaining high-quality sample text. In addition, the quality of the sample constructed in this way is high, avoiding manual review of the sample, thereby improving the efficiency of sample construction.

[0076] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0077] Artificial intelligence (AI) is a technology that uses digital computers to simulate human-like perception of the environment, acquisition, and application of knowledge. This technology enables machines to possess human-like perception, reasoning, and decision-making capabilities. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0078] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0079] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0080] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0081] In this embodiment, a sample construction method based on machine learning involving artificial intelligence is provided, such as Figure 1b As shown, the specific process of the sample construction method can be as follows:

[0082] 101. Obtain text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence.

[0083] Text content refers to the content consisting of texts with the same theme. For example, text content can be the content of a chapter in a book, the content of a notification or report, or the main body of a webpage article. The semantics of these text contents are consistent and correspond to the same theme.

[0084] The text content can be obtained locally or remotely. Specifically, web crawlers can be used to obtain articles, news reports, blogs and other text content on related topics from the Internet. It can also be obtained from local or remote course materials, social media, public corpora, etc. The method of obtaining text content is not limited here.

[0085] A text fragment is a continuous section or paragraph within a text. It is a small semantic unit of text content, usually containing certain information or expressing a complete idea. For example, a text fragment can be at least one sentence or at least one paragraph within a text content.

[0086] A text sentence refers to a unit sentence that constitutes a text segment, wherein a text sentence usually ends with a punctuation mark (such as a period, question mark, exclamation mark, etc.).

[0087] In some embodiments, considering that there are multiple types of text content in the initial text, in order to obtain text content for constructing a sample from the initial text, obtaining the text content for constructing the sample includes:

[0088] Get the initial text;

[0089] The initial text is subjected to text analysis processing to obtain text content in the initial text that meets the preset main text conditions. The text content is the text content used to construct the sample.

[0090] The original text refers to the original text data that has not been processed or modified. For example, the original text can be a book, a report, a webpage article, etc.

[0091] The preset body text condition is a pre-set condition used to determine semantic consistency from the initial text.

[0092] For example, by presetting the main text conditions, the main text content corresponding to any chapter in a book, the main text content from a report, the main text content from a web article, etc. can be determined as the text content for constructing a sample, and so on.

[0093] In some embodiments, in order to obtain text content used to construct a sample from the initial text, text analysis processing is performed on the initial text to obtain text content in the initial text that meets preset main text conditions. The text content is the text content used to construct the sample, including:

[0094] Obtain at least one of a preset title format and a preset title expression;

[0095] Determining at least one content title matching the initial text according to at least one of a preset title format and a preset title expression;

[0096] Based on the title position of each content title in the initial text, text content corresponding to each content title and used to construct the sample is determined from the initial text.

[0097] The preset title format is a format that may be used for a preset title. For example, the preset title format may include title level, title number, font style, alignment, first letter capitalization, etc.

[0098] Title levels: Use different levels to indicate the importance or hierarchical relationship of titles, such as using different font sizes, fonts, or indents to distinguish main titles, subtitles, and sub-subtitles.

[0099] Title numbering: Use numbers, letters, Roman numerals, etc. to number titles to indicate the order and hierarchical relationship between titles. For example, use "1", "1.1", "1.1.1" or "I", "II", "III", etc.

[0100] Font style: Use bold, italics, underlining, color change, etc. to highlight the importance or visual impact of the title.

[0101] Alignment: Titles can be aligned left, center, or right to align with text content or other elements.

[0102] Capitalize the first letter: By convention, the first letter of each word in a title should be capitalized unless otherwise specified.

[0103] The preset title expression is a language expression representing the preset title format, which can be used to describe the hierarchical structure, numbering method, font style and other features of the preset title format.

[0104] The content title is the text in the initial text that complies with the preset title format and / or preset title expression.

[0105] Title position is the position of the content title in the initial text.

[0106] For example, if the initial text contains only one content title, the text below the content title is used as the text content for constructing the sample. If the initial text contains multiple content titles, the text between two adjacent content titles is used as the text content for constructing the sample.

[0107] In some embodiments, considering that the initial text may not contain a title, in order to obtain text content used to construct a sample from the initial text, the initial text is subjected to text analysis processing to obtain text content in the initial text that meets preset body text conditions. The text content used to construct the sample includes:

[0108] Performing semantic segmentation on the initial text to obtain at least one initial text paragraph;

[0109] For each initial text paragraph, determining the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph;

[0110] If the paragraph semantic similarity meets the preset semantic similarity condition, the text content used to construct the sample is determined based on the initial text paragraph and its corresponding adjacent initial text paragraph.

[0111] The initial text paragraph is a paragraph in the initial text, and each initial text paragraph is a semantically complete unit.

[0112] Perform semantic segmentation on the initial text, including but not limited to obtaining initial text segments using the following methods:

[0113] 1. Split by blank lines: The most common method is to use blank lines (that is, two carriage returns) as paragraph separators to split the initial text into multiple initial text paragraphs.

[0114] 2. Punctuation-based segmentation: The initial text is segmented into multiple initial text paragraphs based on punctuation marks (such as period, question mark, exclamation mark, etc.).

[0115] 3. Sentence segmentation based on natural language processing technology: Using natural language processing technologies such as word segmentation, part-of-speech tagging, and syntactic analysis, the initial text is segmented into multiple initial text paragraphs.

[0116] The adjacent initial text paragraph is an initial text paragraph adjacent to any initial text paragraph.

[0117] For example, the adjacent initial text paragraph of the first initial text paragraph in the initial text is the second initial text paragraph, the adjacent initial text paragraphs of the second initial text paragraph are the first initial text paragraph and the third initial text paragraph, and so on.

[0118] The paragraph semantic similarity is used to indicate the degree of semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph. The paragraph similarity specifically refers to the perplexity, which is used to measure the semantic consistency between the initial text paragraph and its corresponding adjacent initial text paragraph.

[0119] The preset semantic similarity condition is a pre-set condition for measuring the semantic similarity of paragraphs.

[0120] For example, if the initial text does not contain a clear title, the initial text is segmented according to semantics to obtain at least one initial text paragraph. The paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph is calculated to determine the semantic consistency between them. When the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph meets a preset semantic similarity condition, it is determined that the initial text paragraph and its corresponding adjacent initial text paragraph have semantic consistency, and it can be considered that the initial text paragraph and its corresponding adjacent initial text paragraph belong to continuous text content and have similar semantic expressions.

[0121] In some embodiments, considering that the initial text may not contain a title and may contain multiple text contents, and the semantics of different text contents are different, in order to obtain the semantic similarity between adjacent paragraphs from the initial text and to determine whether the adjacent paragraphs belong to the same text content, for each initial text paragraph, the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph is determined, including:

[0122] For each initial text paragraph, based on the initial text paragraph, word prediction processing is performed on each word in the corresponding adjacent initial text paragraph to obtain the word prediction probability of each word in the corresponding adjacent initial text paragraph;

[0123] For each word in the corresponding adjacent initial text paragraph, determine a word prediction relevance index for each word based on the word prediction probability;

[0124] The word prediction relevance indexes of each word are integrated to obtain the paragraph prediction relevance indexes of the corresponding adjacent initial text paragraphs;

[0125] Based on the paragraph prediction relevance index, the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph is determined.

[0126] The word prediction probability refers to the probability of occurrence of each word in the corresponding adjacent initial text paragraph when a given initial text paragraph is given.

[0127] The word prediction relevance index is used to indicate the relevance index of the word in the corresponding adjacent initial text paragraph with the word in the current context environment during prediction.

[0128] The paragraph prediction relevance index is an index obtained by integrating the word prediction relevance index of each word in the adjacent initial text paragraphs corresponding to the initial text paragraph, and can measure the semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph.

[0129] The paragraph semantic similarity is the paragraph semantic similarity obtained based on the paragraph prediction relevance index.

[0130] The formula related to paragraph semantic similarity (perplexity) is:

[0131] Where x is the position of the word in the adjacent initial text paragraph corresponding to the initial text paragraph, p(x) refers to the word prediction probability of the word at position x in the adjacent initial text paragraph, and p(x)log2p(x) is the word prediction relevance index. is the paragraph prediction relevance index, and H(p) is the paragraph semantic similarity (perplexity).

[0132] In some embodiments, in order to calculate the word prediction relevance index, for each word in its corresponding adjacent initial text paragraph, the word prediction relevance index of each word is determined based on the word prediction probability, including:

[0133] Performing logarithmic calculation on the word prediction probability of each word to obtain the word logarithmic probability of each word;

[0134] For each word in its corresponding adjacent initial text paragraph, a word prediction relevance index of each word is obtained based on the word prediction probability and the word log probability.

[0135] The word log probability is the value obtained by performing logarithmic calculation on the word prediction probability, which is equal to logp(x).

[0136] Word prediction relevance index = p(x)log2p(x).

[0137] In some embodiments, in order to calculate paragraph semantic similarity based on the paragraph prediction quality index, determining the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph based on the paragraph prediction relevance index includes:

[0138] The negative number of the paragraph prediction quality index is exponentially calculated to obtain the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph.

[0139] in,

[0140] 102. Segment the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content.

[0141] Among them, the position information is used to indicate the position of each text fragment and text sentence in the text content. The position information of the text fragment can indicate the fragment order of the text fragment in the text content, and the position information of the text sentence can indicate the text sentence order of the text sentence in the text content.

[0142] The text sub-content is the content obtained by segmenting the text content according to the preset segmentation strategy.

[0143] The preset segmentation strategy is used to limit the segmentation method of text content.

[0144] For example, the preset segmentation strategy can limit the segmentation of text content to be based on the position information of each text fragment in the text content, segment the text content to obtain at least one text fragment, and then segment the text fragment according to the position information of the text sentence in the text fragment, so as to obtain at least one text sub-content corresponding to the text fragment, and the text sub-content includes a fixed number of text sentences, that is, the text sub-content includes at least one text sentence.

[0145] 103. For each text segment, determine the current text sub-content to be processed from the text sub-contents according to the content order of the text sub-contents in the text segment.

[0146] The content order refers to the order obtained by the contextual relationship between the sub-contents of the text in the text segment. For example, the content order may include forward order, reverse order, etc.

[0147] For example, each text sub-content in a text fragment has a corresponding label, and the labels corresponding to the text sub-content are continuous labels (such as continuous numbers, continuous letters, continuous symbols, etc.), which can represent the contextual relationship between each text sub-content in the text fragment, and can represent the content order of the text sub-content in the text fragment.

[0148] Each text sub-content in the text segment can also be stored in a list. The position of each text sub-content in the list is the content order of each text sub-content in the text segment, which can represent the contextual relationship of each text sub-content in the text segment.

[0149] The content order of each text sub-content in the text segment can also be indicated by an arrow, and the direction of the arrow can indicate a forward order or a reverse order.

[0150] The current text sub-content to be processed is the text sub-content in the text segment that currently needs to be processed.

[0151] For example, the current text sub-content to be processed may be determined by processing each text sub-content one by one starting from the beginning or the end of the text segment according to the content order of the text sub-content in the text segment (such as forward order or reverse order).

[0152] The text segment includes multiple text sub-contents. The first text sub-content can be used as the current text sub-content to be processed, according to the order of the text sub-contents. If it is necessary to further determine the current text sub-content to be processed from the text segment, the second text sub-content can also be used as the current text sub-content to be processed, and so on, to determine the current text sub-content to be processed from each text sub-content of the text segment. 104. The current text sub-content to be processed is added to a preset text data set to obtain a post-addition text data set corresponding to the text segment, and the text length in the post-addition text data set is detected.

[0153] The preset text data set is a preset set for summarizing the current to-be-processed text sub-contents corresponding to the text segments, so as to perform subsequent processing on the current to-be-processed text sub-contents summarized in the preset text data set.

[0154] For example, if the preset text data set does not include the current text sub-content to be processed determined from the various text sub-contents of the text segment, the preset text data set is an empty set. If the preset text data set already includes the current text sub-content to be processed and the newly obtained current text sub-content to be processed needs to be added, the preset text data set is not an empty set.

[0155] The added text data set is a set obtained by adding the current text sub-content to be processed to the preset text data set.

[0156] The text length refers to the number of characters or words in the text sub-content to be processed in the added text data set.

[0157] Methods for detecting the length of text in the added text data set include but are not limited to:

[0158] a) Character count detection: Calculate the number of all characters, including spaces and punctuation, in the current text sub-content to be processed in the added text data set;

[0159] b) Word count detection: When processing English or other languages ​​that separate words with spaces, the length of the text is determined by counting the number of words in the current text sub-content to be processed in the added text data set.

[0160] c) Word count detection: For languages ​​such as Chinese that are not separated by spaces, the length of the text is often detected by counting the number of words in the text.

[0161] 105. When it is detected that the text length in the added text data set does not meet the preset sample text length condition, the added text data set is used as a new preset text data set, and the process returns to the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text fragment, until the text length in the added text data set meets the preset sample text length condition, thereby obtaining a reference text data set corresponding to the text fragment.

[0162] The preset sample text length condition is a preset condition for measuring the length of text in the added text data set. For example, the preset sample text length condition may limit the range of a preset number of characters, a preset number of words, a preset number of characters, etc.

[0163] The new preset text data set is a preset text data set formed by the added text data set when the text length in the added text data set does not meet the preset sample text length condition.

[0164] When the length of text in the reference text dataset after the addition satisfies a preset sample text length condition, the added text dataset is taken as the text dataset.

[0165] For example, a text segment includes multiple text sub-contents. The first text sub-content can be used as the current text sub-content to be processed according to the content order of the text sub-contents, and then the first text sub-content can be added to the preset text data set P to obtain the added text data set P corresponding to the text segment. 1 , detect the text length of the first text sub-content in the added text data set, if it does not meet the preset sample text length condition, add the text data set P 1 As a new preset text data set P 1 , continue to determine the current text sub-content to be processed from the text fragment, that is, the second text sub-content can be used as the current text sub-content to be processed, and the second text sub-content is added to the preset text data set P 1 In the example, we get the added text data set P corresponding to the text fragment 2 .

[0166] After adding the text data set P 2 If the total text length of the first text sub-content and the second text sub-content in the sample text meets the preset sample text length condition, a reference text data set including the first text sub-content and the second text sub-content is obtained.

[0167] After adding the text data set P 2 If the total text length of the first and second text sub-contents does not meet the preset sample text length condition, the third text sub-content is used as the current text sub-content to be processed, and so on, until the text length of the final added text data set meets the preset sample text length condition, and the reference text data set corresponding to the text segment is obtained. If the text segment does not have the third text sub-content, the added text data set P is directly used as the current text sub-content to be processed. 2 As a reference text data set corresponding to the text fragment.

[0168] In some embodiments, for the text sub-content in the text segment that has not been added to the text dataset, the above steps 103 to 105 may be continued to obtain a new reference text dataset.

[0169] 106. Construct sample text data based on the text in the reference text data set.

[0170] The text in the reference text data set refers to the sub-content of the current text to be processed included when the length of the text in the added text data set meets the preset sample text length condition.

[0171] Sample text data refers to a set of specific text data selected or constructed from a text data set and used to train a text processing model.

[0172] For example, the sample text data may be text data obtained by constructing the texts in the text data set according to the order of their contents in the text fragment, or may be text data obtained by constructing the texts in the desensitized text data set according to the order of their contents in the text fragment, and so on.

[0173] In some embodiments, in order to construct unsupervised pre-training data and supervised quality fine-tuning data, the method further includes:

[0174] Get the content title of the text content;

[0175] Determine a target instruction template that matches the content title from preset instruction templates;

[0176] Merge the content title with the target instruction template to obtain the text generation instruction information corresponding to the text content;

[0177] Based on the text generation instruction information and the text in the reference text data set corresponding to the text content, a sample data pair is constructed. The sample data pair includes the text generation instruction information and the sample text corresponding to the text generation instruction information.

[0178] The content title refers to a short text of the core theme of the overview text, such as the title of a book, the title of a news report, or the title of an article.

[0179] Preset instruction templates are preset instruction templates used to guide the text generation model to generate specific text content. These preset instruction templates define the parameters, format, content range and goals of the generation task, which can make the text generation process more efficient and targeted.

[0180] Pre-set instruction templates include, but are not limited to, content topic and scope, text structure and format, style and tone, keywords and phrases, and restrictions and constraints. Content topic and scope can be used to explicitly indicate the subject and content scope of the text processing model, helping it focus on specific domains or topics for text generation. Text structure and format can be used to specify the desired structure (e.g., headings, paragraphs, lists) and format (e.g., formal reports, blog posts, conversations, etc.) of the text to suit different application scenarios and reader needs. Style and tone set the writing style (e.g., professional, humorous, friendly, etc.) and tone (e.g., positive, neutral, critical, etc.) of the text to match a specific audience or convey a particular sentiment. Keywords and phrases include specific keywords or phrases to ensure that the generated text is closely related to a specific topic or concept. Restrictions and constraints set any necessary restrictions and constraints, such as text length limits, avoiding sensitive terms, and ensuring factual accuracy, to ensure that the generated text meets specific quality and ethical standards.

[0181] The target instruction template is a preset instruction template that matches the content title.

[0182] For example, the preset instruction model may include text representing its application scenario (such as business hours adjustment notice). If the content title of the text content includes business hours adjustment notice, a target instruction template matching the content title may be obtained.

[0183] For example, different preset instruction model tags have different application scenarios. Since the content title can summarize the short text of the core theme of the text content, the application scenario involved in the content title can also be obtained. Based on the application scenario marked by the preset instruction template and the application scenario involved in the content title, the target instruction template is obtained, and so on.

[0184] The text generation instruction information is used to instruct the text generation model to generate specific text related to the content title.

[0185] For example, the content title is "Notice of Adjustment of Business Hours of Bank A during the Spring Festival." The target instruction template includes "Notice of Adjustment of Business Hours of Bank xx during the Spring Festival, which is aimed at the general bank customers, and the specific content includes: xxxx." By integrating the content title with the target instruction template, the resulting text generation instruction information is "Notice of Adjustment of Business Hours of Bank A during the Spring Festival, which is aimed at the general bank customers, and the specific content includes: xxxx," so that the text generation model can generate specific text related to the content title.

[0186] The text generation instruction information in the sample data pair is the input data of the text processing model, and the sample text corresponding to the text generation instruction information is the text that the expected text processing model can generate based on the input text generation instruction information, so that after the text processing model is trained with the sample data pair, it can generate a text processing model that meets the text generation task.

[0187] The sample text includes text in a text data set corresponding to the text content.

[0188] In some embodiments, considering that after obtaining text content from an initial text without a title, in order to obtain a target instruction template based on the content title of the text content, before obtaining the content title of the text content, the following is further included:

[0189] According to the preset title generation task, the text content is semantically analyzed and processed to obtain the content title corresponding to the text content.

[0190] The pre-set title generation task is used to generate a relevant title or summary based on a given text content. This task typically requires the generated title to accurately and concisely summarize the theme or content of the text content. For example, the pre-set title generation task can be a pre-trained title generation model that can input text content into the title generation model and output the content title corresponding to the text content.

[0191] Content titles can also be generated through preset title generation tasks, extracting phrases or sentences from text content that can accurately and concisely summarize the theme of the text content.

[0192] In some embodiments, in order to train the text processing model, the method further includes:

[0193] Through the text processing model, text prediction processing is performed on the text generation instruction information in the sample data pair to obtain the predicted text;

[0194] Calculate the loss between the predicted text and the sample text in the sample data to obtain the predicted loss value;

[0195] Train a text processing model based on the prediction loss value.

[0196] Text processing models are computational models implemented through algorithms to understand, interpret, generate, or otherwise process natural language text. These models can be applied to a variety of natural language processing (NLP) tasks, such as text classification, sentiment analysis, machine translation, named entity recognition, text summarization, question-answering systems, and language models. The goal of text processing models is to enable computers to understand and manipulate human language, thereby automatically completing specific language processing tasks without human intervention.

[0197] Predictive text refers to the text result generated after using a text processing model to predict the text generation instruction information in the input sample data pair.

[0198] The prediction loss value is a measure of the difference between the predicted text and the sample text in the sample data pair.

[0199] Based on the predicted loss value, the steps of training the text processing model can specifically include: calculating the gradient of the predicted loss value with respect to the text processing model, where the gradient represents the sensitivity of the predicted loss value to changes in the model parameters in the text processing model; based on the calculated gradient, updating the model parameters of the text processing model to reduce the predicted loss value. This process can be repeated multiple times to gradually improve the performance of the text processing model.

[0200] In some embodiments, the text processing model is used to provide intelligent services for user terminals, including but not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0201] In some embodiments, considering that the reference text dataset includes all text sentences of the text fragment, but the length of the text in the reference text dataset does not meet the processing requirements of the text processing model, in order to make the text length meet the model requirements of the text processing model and not introduce text sentences other than the text fragment, sample text data is constructed based on the text in the reference text dataset, including:

[0202] When the text length in the reference text data set does not reach the preset text length, using preset characters to perform data supplementation processing on the reference text data set to obtain an updated reference text data set;

[0203] Based on the updated reference text data set, sample text data is constructed.

[0204] The text length in the reference text dataset is the text length of the current text sub-content to be processed in the text dataset. For example, the text length in the text dataset can be specifically represented by the number of characters, words, or characters of the current text sub-content to be processed in the text dataset.

[0205] The preset text length is a pre-set text length, which is used to measure whether the text length in the text data set reaches the preset text length.

[0206] The preset characters refer to specific characters used to fill in short texts in the text dataset when the length of the text in the text dataset is less than the preset text length.

[0207] The updated reference text data set is a text data set obtained after the preset characters are supplemented and the text length in the supplemented reference text data set reaches the preset text length. The text length in the updated reference text data set meets the input requirements of the text processing model.

[0208] For example, after all text sub-contents of a text fragment are included in the reference text data set, if the text length in the reference text data set still does not reach the preset text length, in order to make the text length in the reference text data set meet the input requirements of the text processing model, preset characters are used to supplement the reference text data set so that the text length in the reference text data set can reach the preset text length, thereby obtaining an updated reference text data set, and using the text in the updated reference text data set (the current text sub-content to be processed and the preset characters) to construct sample text data.

[0209] In some embodiments, in order to use the desensitized sample text data to train the text processing model, after constructing the sample data pair based on the text generation instruction information and the text in the text data set corresponding to the text content, the method further includes:

[0210] Obtaining at least one associated word list corresponding to a preset sensitive word type, the associated word list including at least one associated word, where the associated word is a word associated with a sensitive word of the preset sensitive word type;

[0211] Determining a target word that matches the target word from the text of the sample data pair according to at least one associated word list;

[0212] Based on the target word, determine the sensitive word information from the text of the sample data pair;

[0213] The sensitive word information in the text of the sample data pair is replaced with the preset placeholder corresponding to the preset sensitive word type to obtain the desensitized sample data pair.

[0214] Among them, the preset sensitive word type is a predefined sensitive word category, for example, the preset sensitive word type may include name, account, address, website, Internet Protocol Address (IP address), contact information, etc.

[0215] The associated word list is a list of words associated with sensitive words of a preset sensitive word type.

[0216] Associated words are words that appear along with sensitive words of the preset sensitive word type.

[0217] For example, if the preset sensitive word type is name, the associated word list may include at least one of a list composed of the name's pre-information, a list composed of the name's post-information, a list composed of the name's prompt verbs, etc., wherein the name's pre-information refers to the name's position or title attached to the front of the name, and the name's post-information refers to the name's position or title attached to the back of the name. The name's prompt verbs are mostly followed by the name and personal pronouns, such as ["say", "point out", "high speed", "notify"] and so on.

[0218] If the preset sensitive word type is account, the associated word list may include at least one of an account prefix list and a linking word list. The account prefix list may include ["account", "account number"], etc., and the linking word list may include ["is", "for", ":"], etc.

[0219] If the preset sensitive word type is address, the associated word list may include at least one of the address prefix list, the aforementioned linking word list and the address suffix list. The address prefix list may include ["residence", "address", "location information", "work unit"], etc., and the address suffix list may include ["room", "room", "department", "institute", "school", "field", "library", "institute", "department", "bureau", "ministry", "bureau", "building", "number", "road", "street", "road", "village", "town", "county", "district", "city", "province", "county"], etc.

[0220] If the preset sensitive word type is a URL, the associated word list may include a URL prefix list and the aforementioned linking word list. The URL prefix list may include ["URL", "website"], etc.

[0221] If the preset sensitive word type is IP address, the associated word list may include an IP address prefix list and the aforementioned linking word list. The IP address prefix list may include ["IP", "IP address", "ip", "ip address"], etc., where "ip" is the lowercase version of "IP".

[0222] If the preset sensitive word type is contact information, when the contact information is a phone number, the associated word list may include a phone number prefix list, such as ["phone number", "mobile phone number"], etc. When the contact information is an instant messaging application, the associated word list may include an instant messaging application prefix list, such as ["name of the instant messaging application", "account of the instant messaging application"], etc. When the contact information is an email address, the associated word list may include an email address prefix list, such as ["mailbox", "mail", ["telecom box"], etc.

[0223] Sensitive words in a preset sensitive word type refer to specific sensitive words associated with the preset sensitive word type. For example, sensitive words in a preset sensitive word type may specifically include personal identity information, account information, address information, website information, IP address information, etc.

[0224] The target word refers to a word in a text data set that matches an associated word in the associated word list, that is, the target word is specifically a word in the text of the sample data pair that is the same or similar to the associated word, and the similarity between the word and the associated word is within a preset similarity range. The preset similarity range is used to measure the similarity between the word in the text of the sample data pair and the associated word.

[0225] Sensitive word information is information that accompanies the target word in the text of the sample data pair. The relationship between the associated words and the sensitive words in the associated word list can be used to obtain the front and back positions between the target word and the sensitive word information.

[0226] For example, if the target word is matched by at least one of the prefix-related word list and the linking word list corresponding to the preset sensitive word type, the sensitive word information appears at the end of the target word.

[0227] If the target word is matched using the suffix or post-associated word list corresponding to the preset sensitive word type, the sensitive word information will appear at the front position of the target word.

[0228] The preset placeholders are preset placeholders used to replace sensitive word information in the text of the sample data pair to prevent the exposure of sensitive information and comply with the principles of privacy protection and data security.

[0229] In some embodiments, in order to locate sensitive word information in sample text data, at least one associated word list corresponding to a preset sensitive word type is obtained. The associated word list includes at least one associated word, which is a word associated with a sensitive word of the preset sensitive word type, including:

[0230] Obtaining a preset associated word expression corresponding to a preset sensitive word type, where the preset associated word expression is an expression composed of words associated with the sensitive word of the preset sensitive word type;

[0231] Determine the target words that match them from the text of the sample data pairs according to at least one list of related words, including:

[0232] Determine the target words that match it from the text of the sample data pairs according to the preset related word expression.

[0233] Among them, the preset related word expression can represent the words associated with the sensitive words of the preset sensitive word type.

[0234] For example, if the preset sensitive word category is address, the preset related word expression corresponding to the address can be [\u4e00-\u9fa5 + "room" / "house" / "company"... / "country"], "\u4e00" can indicate the address prefix list, and "\u9fa5" can indicate the connection word list.

[0235] If the preset sensitive word category is website URL, the preset related word expression corresponding to the website URL can be "(?:(?:http:\ / \ / )|(?:https:\ / \ / ))?(?:[\w](?:[\w\-]{0,61}[\w])?\.)+[a-zA-Z]{2,6}(?:\ / )".

[0236] If the preset sensitive word category is IP address, the preset related word expression corresponding to the IP address can be "((?:(?:25[0-5]|2[0-4]\d|

[01] ?\d?\d)\.){3}(?:25[0-5]|2[0-4]\d|

[01] ?\d?\d))".

[0237] If the preset sensitive word category is contact information, the preset related word expression and prefix list corresponding to the contact information can be obtained through Table 1.

[0238]

[0239] Table 1

[0240] In some embodiments, in order to avoid using overlapping sample data pairs to train the text processing model, the method further includes:

[0241] After constructing the sample data pairs based on the text generation instruction information and the text in the reference text data set corresponding to the text content, the method further includes:

[0242] For any two constructed sample data pairs, calculate the similarity between any two sample data pairs;

[0243] Based on the similarity, perform deduplication processing on multiple sample data pairs to obtain the deduplicated sample data pairs.

[0244] Among them, similarity is used to indicate the degree of similarity between any two sample data pairs.

[0245] Similarity can be calculated using methods such as feature cosine similarity, edit distance, hash distance, machine translation evaluation indicators (Bilingual Evaluation Understudy, BLEU score), indicators used for automatic summarization and summary evaluation (Recall-Oriented Understudy for Gisting Evaluation, ROUGE score), and statistical indicators (such as the Dice coefficient) to measure the similarity between two sample data pairs.

[0246] The deduplicated sample text data is sample data pairs from which duplicate or similar data have been eliminated.

[0247] For example, for a given plurality of sample data pairs, the similarity between any two of them is calculated. When the similarity is greater than a preset similarity threshold, the two sample data pairs are considered similar. Any one of the two sample data pairs is retained, and the aforementioned steps are repeated to traverse all sample data pairs to obtain the deduplicated sample data pairs.

[0248] In some embodiments, in order to calculate the similarity between any two sample text data,

[0249] For any two constructed sample text data, calculate the similarity between any two sample data pairs, including:

[0250] Perform word segmentation processing on each sample data pair to obtain sample words corresponding to each sample data pair;

[0251] For any two sample data pairs, determine repeated sample words from their corresponding sample words, where the repeated sample words are sample words that appear repeatedly in any two sample data pairs;

[0252] Based on the number of repeated sample words and the total number of sample words in any two sample data pairs, the similarity between any two sample data pairs is obtained.

[0253] The sample words refer to the words or phrases obtained after word segmentation processing on the sample data.

[0254] The word segmentation methods for sample data pairs include but are not limited to:

[0255] ⅰ) Use regular expressions, dictionary matching and other methods to perform word segmentation.

[0256] ii) Segment the sample text data according to the preset word length.

[0257] For example, the sample data pair includes "abcdefg.", and the preset word length is 2. The sample words obtained after segmentation may include "ab", "cd", "ef", and "g.", where the characters in "abcdefg." are used to represent the words in the sample data pair.

[0258] ⅲ) Use the trained word segmentation model to segment the sample text data.

[0259] Repeated sample words are sample words that appear repeatedly in any two sample data pairs.

[0260] Dice coefficient (used to measure the similarity between any two sample data pairs) = 2(A∩B) / (A+B), where the numerator is twice the number of intersections of A and B, and the denominator is the sum of the lengths of A and B, so it ranges between 0 and 1. A and B are used to represent any two sample data pairs.

[0261] In the specific calculation process, any two sample data pairs to be compared are first divided into sample words (n-grams). Depending on the value of n, n-grams of different lengths can be obtained, such as unigram representing a single letter or word, bigram representing two consecutive letters or words, trigram representing three consecutive letters or words, etc.

[0262] Dice coefficient = 2 x number of shared n-grams / total number of n-grams in the two strings.

[0263] Among them, shared n-grams are n-grams that appear in any two sample data pairs at the same time, that is, repeated sample words. The larger the Dice value, the more similar the two strings are:

[0264] 1.0000: means that any two pairs of sample data consist of the same set of characters, although those characters may be in different orders and may use different capitalization.

[0265] 0.7500: means that any two sample data pairs have 75% similarity.

[0266] 0.0000: This means that any two sample data pairs do not have a shared n-gram (see explanation below), or the specified length of the n-gram used in the calculation is longer than the shorter of any two sample data pairs being compared. Filtering or sorting values ​​based on the Dyss coefficient of the domain median value can identify those values ​​that are most similar to the comparison string.

[0267] For example, any two sample text data include sample data pair 1 and sample data pair 2. Sample data pair 1 includes "abcdefg" and sample data pair 2 includes "abcdefhk". "abcdefg" is segmented into "ab", "cd", "ef", and "g". "abcdefhk" is segmented into "ab", "cd", "ef", and "hk". The repeated sample words in sample data pair 1 and sample data pair 2 are "ab", "cd", and "ef", that is, the number of repeated sample words is 3, and the total number is 8. Similarity = 2x3 / 8.

[0268] As can be seen from the above, the embodiment of the present application can obtain text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence; based on the position information of each text segment and text sentence in the text content, the text content is segmented to obtain at least one text sub-content corresponding to each text segment in the text content; for each text segment, according to the content order of each text sub-content in the text segment, the current text sub-content to be processed is determined from each text sub-content; the current text sub-content to be processed is added to a preset text data set to obtain an added text data set corresponding to the text segment, and the text length in the added text data set is detected; when it is detected that the text length in the added text data set does not meet the preset sample text length condition, the added text data set is used as a new preset text data set, and the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment is returned to execute until the text length in the added text data set meets the preset sample text length condition, and a reference text data set corresponding to the text segment is obtained; based on the text in the text data set, sample text data is constructed.

[0269] Therefore, this solution can help segment the text content and generate text sub-content with complete semantics by utilizing the position information of each text fragment and text sentence in the text content. When processing each text fragment subsequently, the current text sub-content to be processed with semantic integrity and continuity can be determined according to the content order of each text sub-content in the text fragment, and the current processed text sub-content obtained each time is added to the preset text data set. After each addition, the length of the text in the added text data set is detected until the added text data set meets the preset sample text length condition. In this way, a reference text data set corresponding to the text fragment that meets the input requirements of the text processing model can be obtained, and the reference text data set contains text with semantic integrity and continuity. Therefore, the sample text data constructed based on the text in the reference text data set can maintain the structure of the original text, improve the semantic integrity of the constructed sample, and is conducive to obtaining high-quality sample text. In addition, the quality of the sample constructed in this way is high, avoiding manual review of the sample, thereby improving the efficiency of sample construction.

[0270] The method described in the above embodiment will be further described below.

[0271] In this embodiment, the method of the embodiment of the present application will be described in detail by taking the text processing model as a large language model as an example.

[0272] like Figure 2a As shown, the specific process of a sample construction method is as follows:

[0273] 201. Obtain an initial text and perform text analysis on the initial text to obtain text content in the initial text that meets preset main text conditions. The text content is used to construct a sample. The text content includes at least one text segment, and each text segment includes at least one text sentence.

[0274] In some embodiments, before performing text analysis on the initial text to obtain text content in the initial text that meets the preset main text conditions, the method further includes:

[0275] Get preset filtering information;

[0276] Based on the preset filtering information, the initial text is filtered to obtain the filtered initial text.

[0277] The preset filtering information is pre-set information that needs to be filtered out, for example, the preset filtering information may include headers, footers, page numbers, directories, document numbers, issuing units, dates, signatures, advertisements, comments, navigation bars, and other information.

[0278] Data filtering of the initial text may include:

[0279] 1. If the initial text is a book, a regular expression is used to match the text in a specific format such as the header, footer, page number, table of contents, etc. in the book and delete it to obtain the filtered book.

[0280] 2. If the initial text is an official document, regular expressions are used to match and delete fixed-format information such as the document number, issuing unit, date, signature, etc. in the official document, and the attachments carried by the official document are ignored to obtain the filtered official document.

[0281] 3. If the initial text is a report, pattern recognition and regular expressions are used to identify and remove charts, footnotes, references and other information in the report to obtain a filtered report.

[0282] 4. If the initial text is a web page article, remove advertisements, comments, navigation bars, web page tags, blank characters and other information in the web page article to obtain a filtered web page article.

[0283] The data filtering method of the present application has the following advantages: it is automated and does not require human intervention, is applicable to a variety of text data sources, can effectively remove irrelevant and unimportant content, and improves the accuracy and efficiency of subsequent text analysis and mining.

[0284] Perform text analysis on the initial text to obtain text content that meets the preset main text conditions in the initial text, which can be:

[0285] a. When performing initial text segmentation, the content titles in the initial text are located by matching the preset title format with the preset title expression. Based on the title position of each content title in the initial text, two adjacent content titles can be obtained. The content between the two content titles is the text content used to construct the sample.

[0286] b. For the initial text that cannot locate the content title, in order to obtain the text content used to construct the sample in the initial text, the semantic consistency of the text content is guaranteed as much as possible, and irrelevant paragraphs are avoided from being added to the same text content. The paragraph semantic similarity (such as perplexity) is used to determine whether the initial text fragments are divided into the same text content. If the overall perplexity of the text content is significantly improved after a new initial text fragment is added, it means that the initial text fragment may not belong to the same text content and is removed. The text content used to construct the sample is obtained, and its text content can be input into the title generation model through the pre-trained title generation model to output the content title corresponding to the text content.

[0287] 202. Segment the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content.

[0288] For example, when segmenting text content, text segments in the text content and text sentences in the text segments may be obtained, and the text sub-content may be the text sentences in the text segments.

[0289] 203 . For each text segment, determine the current text sub-content to be processed from the text sub-contents according to the content order of the text sub-contents in the text segment.

[0290] For example, the sample text data required to train a text processing model is usually limited to a preset sample text length condition, such as [minimum text length (min_batch_num), maximum text length (max_batch_num)]. For each text segment in the text content, each text sub-content in the text segment is read line by line to obtain the current text sub-content to be processed.

[0291] 204 : Add the current to-be-processed text sub-content to a preset text data set to obtain an added text data set corresponding to the text segment, and detect the text length in the added text data set.

[0292] In some embodiments, before the current text sub-content to be processed is added to the preset text data set, the method further includes:

[0293] The current text sub-content to be processed is converted from "half-width -> full-width" format and non-Chinese garbled characters are identified and filtered to prevent the format and content of the text processing model from being incorrect.

[0294] 205. When it is detected that the text length in the added text data set does not meet the preset sample text length condition, the added text data set is used as a new preset text data set, and the process returns to the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text fragment, until the text length in the added text data set meets the preset sample text length condition, thereby obtaining a reference text data set corresponding to the text fragment.

[0295] For example, it is determined whether the text length of the current to-be-processed text sub-content in the added text data set is greater than min_batch_num. If it is greater, it is marked once and is called "state one". If it is less than, the above process is continued until it reaches "state one". After reaching "state one", the processing of the current text segment is continued, and it is determined whether the text length in the added text data set is greater than max_batch_num. If it is greater, the added text data set in "state one" is stored in a storage area, and the text data set corresponding to the text segment is obtained. If it is less than, it is marked once and is called "state two". The added text data set in "state two" is stored in a storage area, and preset characters are used to supplement the data of the added text data set in "state two" so that its text length reaches max_batch_num, and a reference text data set corresponding to the text segment is obtained.

[0296] This application can maintain the structural and semantic integrity of sentences and paragraphs in text data sets as much as possible, avoiding the situation where sentences and paragraphs are truncate, which causes the text processing model to learn incomplete semantic information and mislead it to output incomplete content. Therefore, it has important significance in actual production applications.

[0297] 206. Obtain the content title of the text content obtained during the initial text segmentation, determine the target instruction template that matches the content title from the preset instruction template, merge the content title with the target instruction template, obtain text generation instruction information corresponding to the text content, and construct sample text data based on the text in the reference text data set corresponding to the text content.

[0298] In some embodiments, further comprising

[0299] Based on the text generation instruction information and the sample text data, a sample data pair is constructed, where the sample data pair includes the text generation instruction information and the sample text corresponding to the text generation instruction information.

[0300] In some embodiments, the method further comprises:

[0301] The text generation instruction information corresponding to the text content is determined through the instruction generation model.

[0302] In practical applications, the above a and b can be organically combined to construct higher-quality supervised sample data pairs. Compared with the previous manual / rule-based construction of supervised sample data pairs, this application can fully automate the data construction process, greatly improving the efficiency of supervised sample data pair construction.

[0303] In some embodiments, after constructing the sample data pair based on the text generation instruction information and the text in the text data set corresponding to the text content, the method further includes:

[0304] Obtaining at least one associated word list corresponding to a preset sensitive word type, the associated word list including at least one associated word, where the associated word is a word associated with a sensitive word of the preset sensitive word type;

[0305] Determining a target word that matches the target word from the text of the sample data pair according to at least one associated word list;

[0306] Based on the target word, determine the sensitive word information from the text of the sample data pair;

[0307] The sensitive word information in the text of the sample data pair is replaced with the preset placeholder corresponding to the preset sensitive word type to obtain the desensitized sample data pair.

[0308] In some embodiments, at least one associated word list corresponding to a preset sensitive word type is obtained, where the associated word list includes at least one associated word, where the associated word is a word associated with a sensitive word of the preset sensitive word type, including:

[0309] Obtaining a preset associated word expression corresponding to a preset sensitive word type, where the preset associated word expression is an expression composed of words associated with the sensitive word of the preset sensitive word type;

[0310] According to at least one associated word list, a target word that matches the target word is determined from the text of the sample data pair, including:

[0311] According to the preset associated word expression, the target word that matches it is determined from the text of the sample data pair.

[0312] When constructing training data for text processing models, the importance of data desensitization cannot be ignored. Data desensitization is a privacy protection technology that modifies, replaces, or deletes sensitive information so that the data cannot be identified or associated with a specific entity while maintaining its original purpose. Currently, the preset sensitive word types considered in this application are as follows: Figure 2b As shown, this includes name, account, address, website, IP address, and contact information. Contact information further includes phone numbers, instant messaging app accounts, email addresses, and other subcategories. The desensitization process first locates the location of sensitive word information and then replaces it with placeholders "{XXX}", where XXX refers to the corresponding preset sensitive word type, such as "name" for name and "address" for address.

[0313] In some embodiments, in order to improve the effectiveness of sample data pairs, the constructed sample data pairs are deduplicated before the formal training of the text processing model begins. The text data deduplication process adopted in the application is to calculate the similarity of each sample data pair in the data set, and then eliminate one of the sample data pairs whose similarity is higher than a preset similarity threshold.

[0314] like Figure 2c As shown, this application aims to automatically process massive amounts of multi-source unstructured initial text into desensitized sample data pairs. The sample data pairs include text generation instruction information and sample text corresponding to the text generation instruction information, which are used to train text processing models. Multi-source unstructured initial text may come from multiple sources, such as books, official documents, reports, web articles, etc. Therefore, it is necessary to perform preliminary filtering and cleaning according to different formats to remove irrelevant and unimportant content to obtain a clean initial text. The initial text is then subjected to text analysis processing to obtain text content in the initial text that meets the preset main text conditions. Based on the same text content, unsupervised sample text data and supervised text generation instruction information can be obtained. Then, a series of desensitization operations are performed on the sample data pairs to ensure that information related to user privacy, such as name, address, and contact information, does not appear in the sample text data, preventing the text processing model from leaking user privacy when outputting answers. Before training the model, the sample data pairs need to be deduplicated to further improve the quality and independent distribution of the data itself. After the above operations, the final unsupervised sample text data and supervised text generation instruction information can be obtained.

[0315] Compared to fixed-length unsupervised sample text data constructed using previous methods, the sample text data constructed in this application maintains the integrity of paragraphs and context as much as possible, and its structure and semantic information are clearer, which helps text processing models learn richer knowledge representations. Furthermore, the sample text data constructed in this application also masks and replaces sensitive word information.

[0316] Compared with the supervised text generation instruction information constructed based on rules in previous methods, the sample text data constructed in this application has a higher degree of matching with the text generation instruction information, the text generation instruction information is more detailed and clear, and the sample text data is more relevant and complete, which is conducive to the text processing model to better understand the user's intentions and generate higher quality output content.

[0317] This application can quickly construct effective desensitized sample data pairs from massive multi-source unstructured initial data, so it has broad application prospects in various industries with high confidentiality requirements. In real-world applications, the text processing model trained by this application has been successfully implemented in specific products such as official document writing assistants and industry intelligent question-and-answer assistants in government affairs, finance, education and other fields. The core functions of these products include but are not limited to: automatic report generation, intelligent answers to user questions, and provision of professional advice. In these applications, the construction method of desensitized sample data pairs plays a key role, effectively protecting the privacy of users, while also ensuring the accuracy and generalization of the answers given by the text processing model.

[0318] For example, in the government sector, this solution can be used by official document writing assistants to provide efficient and accurate document writing services while protecting citizens' personal information. In the financial sector, this solution can be used by intelligent question-and-answer assistants to provide users with accurate and professional financial consulting services while ensuring the security of financial privacy data. In the education sector, this solution can be used by educational document writing assistants to provide teachers with efficient and convenient teaching document writing services while protecting student privacy.

[0319] The implementation of this solution not only improves the efficiency of text data processing in various industries, but also greatly enhances the security of text data processing, providing a strong guarantee for protecting user privacy. As shown in Table 2, the generation effect of the text processing model after training with different data is compared:

[0320]

[0321] Table 2

[0322] The above experiments were conducted to further verify that the sample data pairs constructed in this application can improve the performance of text processing models. A chat generation language model (such as ChatGLM2-6B) was selected as the base model, with approximately 6 billion parameters. "Chat generation" refers to the model's use for generating conversational text, enabling tasks such as chatting and answering user questions. "Language model" refers to the model's use of natural language processing technology to understand and generate text. Using 7,000 fixed-length unsupervised data samples and 2,000 supervised data samples constructed from rules, along with the same number of sample data pairs constructed in this application, two versions of the ChatGLM2-6B model were fine-tuned under the same experimental conditions. The model was tested on a proprietary dataset, the Chinese Normative Writing Evaluation Dataset, and results across seven dimensions are presented. The results demonstrate that the sample data pairs constructed in this application can significantly improve the performance of text processing models across all dimensions, indirectly demonstrating the effectiveness of this application.

[0323] To better implement the above method, the present application also provides a sample construction device. The sample construction device can be integrated into an electronic device, such as a terminal or a server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc. The server can be a single server or a server cluster consisting of multiple servers.

[0324] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the specific integration of the sample construction device into an electronic device as an example.

[0325] For example, Figure 3 As shown, the sample construction device may include a text content acquisition unit 301, a text content segmentation unit 302, a sub-content determination unit 303, a sub-content addition unit 304, a set acquisition unit 305, and a sample construction unit 306, as follows:

[0326] (1) Text content acquisition unit 301.

[0327] The text content acquisition unit 301 is configured to acquire text content for constructing a sample. The text content includes at least one text segment, and each text segment includes at least one text sentence.

[0328] In some embodiments, the text content acquisition unit includes a text acquisition subunit and a text analysis subunit:

[0329] A text acquisition subunit, used to acquire the initial text;

[0330] The text analysis subunit is used to perform text analysis on the initial text to obtain the text content in the initial text that meets the preset main text conditions. The text content is the text content used to construct the sample.

[0331] In some embodiments, the text analysis subunit includes a title information acquisition subunit, a content title determination subunit, and a text content determination subunit:

[0332] A title information acquisition subunit, configured to acquire at least one of a preset title format and a preset title expression;

[0333] a content title determination subunit, configured to determine at least one content title matching the initial text based on at least one of a preset title format and a preset title expression;

[0334] The text content determination subunit is used to determine text content corresponding to each content title in the initial text and used to construct the sample based on the title position of each content title in the initial text.

[0335] In some embodiments, the text analysis subunit includes a semantic segmentation subunit, a semantic similarity determination subunit, and a text content determination subunit:

[0336] A semantic segmentation subunit, used to perform semantic segmentation processing on the initial text to obtain at least one initial text paragraph;

[0337] a semantic similarity determination subunit, configured to determine, for each initial text paragraph, the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph;

[0338] The text content determination subunit is configured to determine the text content for constructing a sample based on the initial text paragraph and its corresponding adjacent initial text paragraph if the paragraph semantic similarity meets a preset semantic similarity condition.

[0339] In some embodiments, the semantic similarity determination subunit is used to perform word prediction processing on each word in the corresponding adjacent initial text paragraph based on the initial text paragraph to obtain the word prediction probability of each word in the corresponding adjacent initial text paragraph; for each word in the corresponding adjacent initial text paragraph, based on the word prediction probability, determine the word prediction relevance index of each word; fuse the word prediction relevance index of each word to obtain the paragraph prediction relevance index of the corresponding adjacent initial text paragraph; and determine the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph based on the paragraph prediction relevance index.

[0340] (2) Text content segmentation unit 302.

[0341] The text content segmentation unit 302 is configured to segment the text content based on the position information of each text segment and text sentence in the text content, and obtain at least one text sub-content corresponding to each text segment in the text content.

[0342] (3) Sub-content determination unit 303.

[0343] The sub-content determining unit 303 is configured to determine, for each text segment, a current text sub-content to be processed from the various text sub-contents according to the content order of the various text sub-contents in the text segment.

[0344] (4) Sub-content adding unit 304.

[0345] The sub-content adding unit 304 is configured to add the current text sub-content to be processed to a preset text data set, obtain an added text data set corresponding to the text segment, and detect the length of the text in the added text data set.

[0346] (5) Set acquisition unit 305.

[0347] The set acquisition unit 305 is used to, when it is detected that the text length in the added text data set does not meet the preset sample text length condition, use the added text data set as a new preset text data set, return to execute the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text fragment, until the text length in the added text data set meets the preset sample text length condition, and obtain a reference text data set corresponding to the text fragment.

[0348] (6) Sample construction unit 306.

[0349] The sample construction unit 306 is configured to construct sample text data based on the text in the reference text data set.

[0350] In some embodiments, the apparatus further includes a title acquisition unit, a template determination unit, an instruction generation unit, and a sample pair construction unit:

[0351] A title acquisition unit, used to acquire the content title of the text content;

[0352] a template determining unit, configured to determine a target instruction template matching the content title from preset instruction templates;

[0353] An instruction generation unit, configured to fuse the content title with the target instruction template to obtain text generation instruction information corresponding to the text content;

[0354] The sample pair construction unit is used to construct a sample data pair based on the text generation instruction information and the text in the reference text data set corresponding to the text content. The sample data pair includes the text generation instruction information and the sample text corresponding to the text generation instruction information.

[0355] In some embodiments, the apparatus further includes a text prediction unit, a loss calculation unit, and a model training unit:

[0356] A text prediction unit, configured to perform text prediction processing on the text generation instruction information in the sample data pair through a text processing model to obtain a predicted text;

[0357] A loss calculation unit, configured to perform loss calculation on the predicted text and the sample text in the sample data pair to obtain a predicted loss value;

[0358] The model training unit is used to train the text processing model based on the prediction loss value, so that the text processing model learns the knowledge corresponding to the sample data.

[0359] In some embodiments, the sample construction unit includes a data supplementation subunit and a sample construction subunit:

[0360] The data supplement subunit is used to use preset characters to perform data supplement processing on the reference text data set when the text length in the reference text data set does not reach the preset text length, so as to obtain an updated reference text data set;

[0361] The sample construction subunit is used to construct sample text data based on the updated reference text data set.

[0362] In some embodiments, the system further includes an associated information acquisition subunit, a word determination subunit, a sensitive information determination subunit, and a desensitization subunit:

[0363] An associated information acquisition subunit is configured to acquire at least one associated word list corresponding to a preset sensitive word type, wherein the associated word list includes at least one associated word, which is a word associated with a sensitive word of the preset sensitive word type;

[0364] a word determination subunit, configured to determine a target word that matches the target word from the text of the sample data pair according to at least one associated word list;

[0365] A sensitive information determination subunit, configured to determine sensitive word information from the text of the sample data pair based on the target word;

[0366] The desensitization sub-unit is used to replace the sensitive word information in the text of the sample data pair with a preset placeholder corresponding to the preset sensitive word type, thereby obtaining a desensitized sample data pair.

[0367] In some embodiments, the association information acquisition subunit is further configured to acquire a preset association word expression corresponding to a preset sensitive word type, where the preset association word expression is an expression composed of words associated with the sensitive word of the preset sensitive word type;

[0368] The word determination subunit is further configured to determine a target word that matches the sample data pair from the text of the sample data pair according to a preset associated word expression.

[0369] In some embodiments, the apparatus further includes a sample similarity calculation unit and a sample deduplication unit:

[0370] A sample similarity calculation unit, configured to calculate the similarity between any two constructed sample data pairs;

[0371] The sample deduplication unit is used to perform deduplication processing on multiple sample data pairs based on similarity to obtain deduplicated sample data pairs.

[0372] In some embodiments, the sample similarity calculation unit is used to perform word segmentation processing on each sample data pair to obtain sample words corresponding to each sample data pair; for any two sample data pairs, repeated sample words are determined from the sample words corresponding to them, and repeated sample words are sample words that appear repeatedly in any two sample data pairs; based on the number of repeated sample words and the total number of sample words in any two sample data pairs, the similarity between any two sample data pairs is obtained.

[0373] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.

[0374] As can be seen from the above, the sample construction device of this embodiment obtains the text content for constructing the sample by the text content acquisition unit, the text content includes at least one text segment, and each text segment includes at least one text sentence; the text content segmentation unit segments the text content based on the position information of each text segment and text sentence in the text content, and obtains at least one text sub-content corresponding to each text segment in the text content; for each text segment, the sub-content determination unit determines the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment; the sub-content adding unit adds the current text sub-content to be processed to the preset text data set, and obtains the text sub-content. The added text data set corresponding to the present fragment is obtained, and the text length in the added text data set is detected; when it is detected that the text length in the added text data set does not meet the preset sample text length condition, the set acquisition unit combines the added text data set as a new preset text data set, and returns to execute the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text fragment, until the text length in the added text data set meets the preset sample text length condition, and a reference text data set corresponding to the text fragment is obtained; the sample construction unit constructs sample text data based on the text in the reference text data set.

[0375] Therefore, the embodiment of the present application can enable the constructed sample to maintain the structure of the original text, improve the semantic integrity of the constructed sample, and facilitate obtaining high-quality sample text. In addition, the quality of the sample constructed in this way is high, avoiding manual review of the sample, thereby improving the efficiency of sample construction.

[0376] The present application also provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer, or the like; the server may be a single server or a server cluster consisting of multiple servers, or the like.

[0377] In some embodiments, the sample construction device may also be integrated into multiple electronic devices. For example, the sample construction device may be integrated into multiple servers, and the sample construction method of the present application may be implemented by the multiple servers.

[0378] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 4 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:

[0379] The server may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that Figure 4 The server structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0380] Processor 401 is the server's control center, connecting various components of the server using various interfaces and circuits. It executes software programs and / or modules stored in memory 402 and accesses data stored in memory 402 to perform various server functions and process data. In some embodiments, processor 401 may include one or more processing cores. In some embodiments, processor 401 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0381] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0382] The server also includes a power supply 403 that supplies power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0383] The server may further include an input module 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0384] The server may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The server may use the wireless module of the communication module 405 to perform short-range wireless transmission, thereby providing users with wireless broadband Internet access. For example, the communication module 405 may be used to help users send and receive emails, browse web pages, and access streaming media.

[0385] Although not shown, the server may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the server loads the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 runs the application stored in the memory 402, thereby implementing the steps of the method in each embodiment of the present application.

[0386] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0387] From the above, it can be seen that this application can enable the constructed samples to maintain the structure of the original text, improve the semantic integrity of the constructed samples, and is conducive to obtaining high-quality sample text. In addition, the quality of the samples constructed in this way is high, avoiding manual review of samples, thereby improving the efficiency of sample construction.

[0388] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0389] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the sample construction methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0390] Acquire text content for constructing a sample, where the text content includes at least one text segment, and each text segment includes at least one text sentence;

[0391] Segmenting the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content;

[0392] For each text segment, according to the content order of each text sub-content in the text segment, determine the current text sub-content to be processed from each text sub-content;

[0393] Add the current to-be-processed text sub-content to a preset text data set to obtain a post-addition text data set corresponding to the text segment, and detect the length of the text in the post-addition text data set;

[0394] When it is detected that the text length in the added text dataset does not meet the preset sample text length condition, the added text dataset is set as a new preset text dataset, and the step of determining the current text sub-content to be processed from each text sub-content according to the content order of each text sub-content in the text segment is returned to be executed until the text length in the added text dataset meets the preset sample text length condition, thereby obtaining a reference text dataset corresponding to the text segment;

[0395] Based on the text in the reference text data set, sample text data is constructed, and the sample text data is used to train the text processing model so that the text processing model learns the knowledge corresponding to the sample text data.

[0396] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0397] According to one aspect of the present application, a computer program product or computer program is provided, comprising a computer program / instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in various optional implementations of the sample construction aspects provided in the above-described embodiments.

[0398] Since the instructions stored in the storage medium can execute the steps in any sample construction method provided in the embodiments of the present application, the beneficial effects that can be achieved by any sample construction method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0399] The above is a detailed introduction to a sample construction method, device, server and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A sample construction method, characterized in that: include: Acquire text content for constructing a sample, wherein the text content includes at least one text segment, and each text segment includes at least one text sentence; Segmenting the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content; For each text segment, determining the current text sub-content to be processed from the various text sub-contents according to the content order of the various text sub-contents in the text segment; Adding the current to-be-processed text sub-content to a preset text data set to obtain an added text data set corresponding to the text segment, and detecting the text length in the added text data set; When it is detected that the text length in the added text dataset does not meet the preset sample text length condition, the added text dataset is set as a new preset text dataset, and the step of determining the current text sub-content to be processed from the text sub-contents according to the content order of the text sub-contents in the text segment is returned to execution until the text length in the added text dataset meets the preset sample text length condition, thereby obtaining a reference text dataset corresponding to the text segment; Based on the text in the reference text data set, sample text data is constructed.

2. The sample construction method according to claim 1, wherein: The method further comprises: Get the content title of the text content; Determine a target instruction template that matches the content title from preset instruction templates; Merging the content title with the target instruction template to obtain text generation instruction information corresponding to the text content; Based on the text generation instruction information and the text in the reference text data set corresponding to the text content, a sample data pair is constructed, wherein the sample data pair includes the text generation instruction information and the sample text corresponding to the text generation instruction information.

3. The sample construction method according to claim 2, wherein: The method further comprises: Performing text prediction processing on the text generation instruction information in the sample data pair through a text processing model to obtain a predicted text; Calculate the loss between the predicted text and the sample text in the sample data to obtain the predicted loss value; Based on the prediction loss value, the text processing model is trained so that the text processing model learns the knowledge corresponding to the sample data.

4. The sample construction method according to claim 1, wherein: The step of constructing sample text data based on the text in the reference text data set includes: When the text length in the reference text data set does not reach the preset text length, using preset characters to perform data supplementation processing on the reference text data set to obtain an updated reference text data set; Based on the updated reference text data set, sample text data is constructed.

5. The sample construction method according to claim 1 or 2, wherein: The obtaining of text content for constructing a sample includes: Get the initial text; The initial text is subjected to text analysis processing to obtain text content in the initial text that meets the preset main text conditions, and the text content is the text content used to construct the sample.

6. The sample construction method according to claim 5, wherein: The text analysis processing is performed on the initial text to obtain text content in the initial text that meets the preset main text conditions, and the text content is used to construct the sample, including: Obtain at least one of a preset title format and a preset title expression; Determining at least one content title matching the initial text according to at least one of the preset title format and the preset title expression; Based on the title positions of the respective content titles in the initial text, text contents corresponding to the respective content titles and used to construct samples are determined from the initial text.

7. The sample construction method according to claim 5, wherein: The text analysis processing is performed on the initial text to obtain text content in the initial text that meets the preset main text conditions, and the text content is used to construct the sample, including: Performing semantic segmentation processing on the initial text to obtain at least one initial text paragraph; For each of the initial text paragraphs, determining the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph; If the paragraph semantic similarity satisfies a preset semantic similarity condition, the text content used to construct the sample is determined according to the initial text paragraph and its corresponding adjacent initial text paragraph.

8. The sample construction method according to claim 7, wherein: The step of determining, for each of the initial text paragraphs, the paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph includes: For each of the initial text paragraphs, based on the initial text paragraph, word prediction processing is performed on each word in the corresponding adjacent initial text paragraph to obtain a word prediction probability for each word in the corresponding adjacent initial text paragraph; For each word in the corresponding adjacent initial text paragraph, determining a word prediction relevance index for each word based on the word prediction probability; fusing the word prediction relevance indices of the respective words to obtain paragraph prediction relevance indices of the corresponding adjacent initial text paragraphs; Based on the paragraph prediction relevance index, a paragraph semantic similarity between the initial text paragraph and its corresponding adjacent initial text paragraph is determined.

9. The sample construction method according to claim 2, wherein: After constructing a sample data pair based on the text generation instruction information and the text in the reference text data set corresponding to the text content, the method further includes: Obtaining at least one associated word list corresponding to a preset sensitive word type, the associated word list including at least one associated word, the associated word being a word associated with a sensitive word of the preset sensitive word type; Determining, from the text of the sample data pair, a target word that matches the target word according to the at least one associated word list; Based on the target word, determining sensitive word information from the text of the sample data pair; The sensitive word information in the text of the sample data pair is replaced with a preset placeholder corresponding to the preset sensitive word type to obtain a desensitized sample data pair.

10. The sample construction method according to claim 9, wherein: The step of obtaining at least one associated word list corresponding to a preset sensitive word type, wherein the associated word list includes at least one associated word, and the associated word is a word associated with a sensitive word of the preset sensitive word type, including: Obtaining a preset associated word expression corresponding to a preset sensitive word type, wherein the preset associated word expression is an expression composed of words associated with the sensitive word of the preset sensitive word type; The step of determining a target word that matches the target word from the text of the sample data pair according to the at least one associated word list comprises: According to the preset associated word expression, a target word that matches the preset associated word expression is determined from the text of the sample data pair.

11. The sample construction method according to any one of claims 2, wherein: After constructing a sample data pair based on the text generation instruction information and the text in the reference text data set corresponding to the text content, the method further includes: For any two constructed sample data pairs, calculating the similarity between the two sample data pairs; Based on the similarity, duplicate removal is performed on the plurality of sample data pairs to obtain duplicate-free sample data pairs.

12. The sample construction method according to claim 11, wherein: The step of calculating the similarity between any two constructed sample text data pairs includes: Perform word segmentation processing on each sample data pair to obtain sample words corresponding to each sample data pair; For any two sample data pairs, determining repeated sample words from the sample words corresponding thereto, wherein the repeated sample words are sample words that appear repeatedly in the any two sample data pairs; Based on the number of repeated sample words and the total number of sample words in the any two sample data pairs, the similarity between the any two sample data pairs is obtained.

13. A sample construction device, characterized in that: include: A content acquisition unit, configured to acquire text content for constructing a sample, wherein the text content includes at least one text segment, and each text segment includes at least one text sentence; a content segmentation unit configured to segment the text content based on position information of each text segment and text sentence in the text content to obtain at least one text sub-content corresponding to each text segment in the text content; a content determination unit configured to determine, for each text segment, a current text sub-content to be processed from the various text sub-contents in the text segment according to the content order of the various text sub-contents; A content adding unit, configured to add the current to-be-processed text sub-content to a preset text data set, obtain an added text data set corresponding to the text segment, and detect the length of the text in the added text data set; a text acquisition unit configured to, when detecting that the text length in the added text dataset does not satisfy a preset sample text length condition, use the added text dataset as a new preset text dataset, return to executing the step of determining a current text sub-content to be processed from the text sub-contents according to the content order of the text sub-contents in the text segment, until the text length in the added text dataset satisfies the preset sample text length condition, thereby obtaining a reference text dataset corresponding to the text segment; The sample construction unit is used to construct sample text data based on the text in the reference text data set.

14. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the sample construction method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the sample construction method according to any one of claims 1 to 12.

16. A computer program product comprising a plurality of instructions, characterized in that: When the instructions are executed by a processor, the steps of the sample construction method according to any one of claims 1 to 12 are implemented.

Images