Training data generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention relates to a training data generation method and device, equipment, a storage medium and a program product, and relates to the field of data processing technologies, text generation technologies, large model technologies and large language model technologies. The method comprises the following steps: acquiring a plurality of initial training samples; performing field matching on the initial training sample based on a preset segmentation classification field, and determining at least one first candidate segmentation category and a candidate segmentation position to which the initial training sample belongs; based on the category occurrence probability of each preset segmentation category and the total sample data amount, determining the sample data amount of the preset segmentation category; determining a target segmentation category of the initial training sample from each first candidate segmentation category based on the sample data volume, a preset segmentation category and a matching relationship of the first candidate segmentation categories; and performing text segmentation processing on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample. Therefore, the difference between the generated target training sample and the real situation can be reduced, and the accuracy of the generative model trained by relying on the target training sample is indirectly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a training data generation method, apparatus, device, storage medium and program product. Background Art

[0002] Generative models with text completion capabilities can help users improve work efficiency, reduce input time, and reduce the occurrence of spelling and grammatical errors by providing candidate texts.

[0003] At present, the main method for constructing training data for generative models with text completion functions is to randomly split the collected original data and construct training data from the split data. However, this random split method may cause the constructed training data to be significantly different from the actual online distribution, resulting in poor accuracy of the trained generative model. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide a training data generation method, apparatus, device, storage medium and program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating training data, the method comprising:

[0006] Acquire multiple initial training samples; the initial training samples include text content with complete structure;

[0007] Performing field matching on any initial training sample based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category;

[0008] Determine the amount of sample data for each preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category;

[0009] Based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category, determining the target segmentation category of the initial training sample from the first candidate segmentation categories to which the initial training sample belongs;

[0010] The initial training sample is subjected to text segmentation processing based on the candidate segmentation positions corresponding to the target segmentation category to generate a target training sample; the target training sample contains multiple segmented text contents.

[0011] In a second aspect, the present disclosure also provides a training data generating device, the device comprising:

[0012] An acquisition module is used to acquire a plurality of initial training samples; the initial training samples include text content with complete structure;

[0013] A matching module, configured to perform field matching on any initial training sample based on a preset segmentation category field, and determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category;

[0014] A determination module, used to determine the amount of sample data for a preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category;

[0015] The determination module is further used to determine the target segmentation category of the initial training sample from each first candidate segmentation category to which the initial training sample belongs based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category;

[0016] The processing module is used to perform text segmentation processing on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample; the target training sample contains multiple segmented text contents.

[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0018] processor;

[0019] A memory for storing executable instructions;

[0020] The processor is used to read executable instructions from the memory and execute the executable instructions to implement the training data generation method described in any embodiment of the present disclosure.

[0021] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the training data generation method described in any embodiment of the present disclosure.

[0022] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which is used to execute the training data generation method described in any embodiment of the present disclosure.

[0023] The training data generation method, device, equipment, storage medium and program product of the embodiment of the present disclosure can first determine at least one first candidate segmentation category to which each initial training sample belongs and the candidate segmentation position corresponding to each first candidate segmentation category according to the preset segmentation category field. And determine the sample data volume of each preset segmentation category according to the total sample data and the category occurrence probability of each preset segmentation category. Then, according to each preset segmentation category, the sample data volume of each preset segmentation category, and the first candidate segmentation category, determine the target segmentation category of the initial training sample from the first candidate segmentation category. Finally, according to the candidate segmentation position corresponding to the target segmentation category, the initial training sample is segmented and processed to generate the target training sample. In this way, the segmentation of the initial training sample is guided by the preset segmentation category field and the category occurrence probability of the preset segmentation category of the real statistics, avoiding the target training sample generated by random segmentation, reducing the difference between the generated target training sample and the real situation, and indirectly improving the accuracy of the generative model with text completion function trained by the target training sample.

[0024] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0026] Figure 1 A flowchart of a training data generation method provided in an embodiment of the present disclosure;

[0027] Figure 2 A schematic diagram of a flow chart of determining the probability of occurrence of a category of a preset segmentation category in a training data generation method provided in an embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of a process for determining preset segmentation categories corresponding to target text content in a training data generation method provided in an embodiment of the present disclosure;

[0029] Figure 4 for Figure 1A schematic diagram of a detailed process of S14 in the training data generation method shown;

[0030] Figure 5 for Figure 1 A schematic diagram of a detailed process of S15 in the training data generation method shown;

[0031] Figure 6 A schematic diagram of the structure of a training data generating device provided in an embodiment of the present disclosure;

[0032] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] A generative model with a text completion function is used to help users complete the text content that they may enter next when they are writing text, thereby improving the efficiency of users writing text. For example, in the process of writing code, a generative model with a text completion function can be used to automatically generate candidate codes for users to complete when users pause while writing code; this scenario can be referred to as a code completion scenario; for another example, in the process of editing notes, a generative model with a text completion function can be used to automatically provide users with candidate texts when users pause while recording notes, so as to complete the notes that they may want to enter.

[0040] Taking the code completion scenario as an example, the data for training a generative model with text completion function includes two data forms, namely the next unit prediction (Next Token Prediction, NTP) data form and the fill in the middle (Fill in the Middle, FIM) data form.

[0041] Among them, the NTP data format is a standard sequential code format, that is, the original code, without any changes, and the original code is directly used as training data to train and generate a generative model with text completion function. The FIM data format needs to first divide the standard sequential code into three training data segments, namely the front code segment (the code before the cursor stop position), the middle code segment (the code to be supplemented), and the back code segment (the code after the cursor stop position). Then, the three training data segments are shuffled, and the front code segment and the back code segment are used as input, and the middle code segment is used as output to train and generate a generative model with text completion function. When training and generating a generative model with text completion function through FIM data format, it can be divided into PrefixSuffixMiddle (PSM) method and SuffixPrefixMiddle (SPM) method according to the order of the front code segment and the back code segment. Among them, the PSM method refers to the method of predicting the middle code segment using the front code segment and the back code segment, and the SPM method refers to the method of predicting the middle code segment using the back code segment and the front code segment.

[0042] It can be seen that when determining the training data in the form of NTP data, a more difficult problem is how to correctly divide the front-end code, middle-end code, and back-end code.

[0043] At present, the method of splitting the front-end code, middle-end code and back-end code is usually to split the code by relying on simple rules plus random splitting. For example, while ensuring that the front-end code accounts for no less than 30% of the overall code, split any position in the original code to obtain the front-end code, middle-end code and back-end code.

[0044] There are two main problems with the training data determined by this segmentation method: First, the determined training data is unreasonable. For example, when a user writes to the end of a complete structure, the user is unlikely to trigger any completion content at this time. However, according to the above segmentation method, it is very likely that the complete structure will be segmented as the front-end code, and a combination of the complete structure, the middle-end code, and the back-end code will be obtained, which is obviously unreasonable. Second, the determined training data may be significantly different from the actual online distribution. In real business scenarios, the number of each cursor stop position has a certain regularity. For example, in the completion scenario of Structured Query Language (SQL), the user is very likely to stop the cursor after selecting this syntax to seek completion, but the data trigger position generated by random means does not have this regularity. Each syntax has the same probability of being used as a data trigger position, which leads to poor accuracy of the generative model trained with text completion function.

[0045] Based on the above situation, an embodiment of the present disclosure provides a training data generation method, including obtaining multiple initial training samples, wherein the initial training samples include text content with complete structure. Field matching is performed on any initial training sample based on the preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and the candidate segmentation position corresponding to the first candidate segmentation category. And based on the category occurrence probability and the total amount of sample data of each preset segmentation category, the amount of sample data of the preset segmentation category is determined. Afterwards, based on the matching relationship between the sample data amount, the preset segmentation category and the first candidate segmentation category, the target segmentation category of the initial training sample is determined from each first candidate segmentation category to which the initial training sample belongs. Finally, text segmentation processing is performed on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample. Wherein, the target training sample contains multiple segmented text contents. The embodiments of the present disclosure can establish universal preset segmentation category fields for specific scenarios, and establish an automatic field matching algorithm based on the preset segmentation category fields, thereby improving the rationality of determining target training samples; using the preset segmentation category fields and the category occurrence probabilities of preset segmentation categories to guide the segmentation of initial training samples, avoiding the target training samples generated by random segmentation, and narrowing the difference between the generated target training samples and the actual situation; from the perspective of data utilization, when training a generative model with a text completion function, the embodiments of the present disclosure can learn more features that are useful to the generative model with a text completion function in the same amount of time, thereby improving the efficiency of data utilization.

[0046] The training data generation method provided in the embodiments of the present disclosure can be applied to scenarios such as programming development tools, cloud notes, and online intelligent assistants. The method can be performed by a training data generation device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device. Among them, the electronic device can include but is not limited to smart phones, personal digital assistants (PDAs), tablet computers (Tablet Personal Computers, Tablet PCs), laptops, vehicle terminals (such as vehicle navigation terminals), wearable devices, smart home devices such as digital televisions, desktop computers, and the like.

[0047] Figure 1 FIG. 1 is a flow chart of a method for generating training data provided by an embodiment of the present disclosure. Figure 1 As shown, the training data generation method may include the following steps:

[0048] S11. Obtain multiple initial training samples.

[0049] The initial training sample includes text content with complete structure. The text content can be the corresponding text content in scenarios such as programming development tools, cloud notes, and online intelligent assistants. For example, in the code completion scenario, the initial training sample can be a complete code that can run successfully.

[0050] Specifically, the method for obtaining multiple initial training samples can be to obtain multiple initial training samples input by the user, or to obtain multiple initial training samples from a corresponding database based on sample acquisition information input by the user. For example, in a code completion scenario, multiple initial training samples can be obtained from data corresponding to a programming development tool based on sample acquisition information input by the user.

[0051] S12: Perform field matching on any initial training sample based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category.

[0052] The preset segmentation category field may be a field obtained by pre-statistics or a field set by relevant personnel based on experience.

[0053] In some embodiments, the preset segmentation category field is obtained by pre-stating the frequency of occurrence of the field corresponding to the input stop position in the text editing process. Specifically, the content that the generative model with a text completion function needs to complete is closely related to the user's input stop position. For example, when the generative model with a text completion function is a generative model for code completion in the SQL language, when the user edits to select, the generative model with a text completion function is required to supplement the field name in the database table; and when the user edits to select*from, the generative model with a text completion function is required to supplement the appropriate database table name. Therefore, when determining the preset segmentation category field, the frequency of occurrence of the field corresponding to the input stop position of the user in the text editing process can be counted first, and then the field with a larger frequency of occurrence is determined as the preset segmentation category field.

[0054] In some embodiments, the preset segmentation category field can be divided into one or more categories. For example, when the generative model with text completion function is a generative model for code completion in SQL language, the preset segmentation category field can be divided into a primary category field and a secondary category field. The primary category field uses specific keywords, and the secondary category field, as a supplement to the primary category field, adds keywords such as function and logic to further subdivide the content that the user wants to supplement.Exemplarily, the first-level category fields may include: select, where, join, group, from, insert, on, casual, with, config_code, create, update, order, etc.; the second-level category fields may include: normal, nvl, array, concat, float, median, and, first, any, datediff, slice, decode, or, int, from_unixtime, double, space, str_to_map, if, cos, length, then, to_date, log10, when, avg, bin, unix_timestamp, grouping, encode, in, last, substr, percentile, trunc, to_json, as, pi, timestamp, lower, upper, ceil, limit, get_json_object, std, contains, collectlist, chr, having, cast, sign, collectset, elt, sort_array, over, min, char, substring, addmonths, greatest, view, exp, tan, array_contains, corr, cot, union, coalesce, regexp, date_format, date_add, next_day, union all, map, now, trim, rlike, least, truncate, rand, replace, explode, locate, binary, set, log, round, stddev, negative, random, count, shuffle, year, pow, positive, mask, case, sin, regexp_replace, window, decimal, rint, sum, second, concat_ws, percentile_approx, date_diff, Ipad, max, posexplode, etc.Among them, the first-level category field includes the first default keyword (config_code), which means that if the corresponding category is not found in the code, this situation is likely that the user is writing some configuration parameters, such as "'author: XXX, createtime: xxx"'", then the code is classified as config_code; similarly, the second-level category also includes the second default keyword (normal), which is also the case that the corresponding second-level category field is not found in the code, then the code is classified as the first-level category field-normal.

[0055] First, according to the preset segmentation category field, field matching is performed on each initial training sample to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs.

[0056] Specifically, for any initial training sample, when a certain preset segmentation category field is included in the initial training sample, it is determined that the initial training sample belongs to the first candidate segmentation category corresponding to the certain preset segmentation category field.

[0057] For example, when the initial training sample is:

[0058] SELECT employees.name AS EmployeeName, departments.department_name ASDepartmentName / *Select the employee's name and rename it to EmployeeName, select the department name and rename it to DepartmentName* /

[0059] FROM employees / *Get data from the employee table* /

[0060] JOIN departments ON employees.department_id=departments.department_id / *Join the employee table with the department table and match by department ID* /

[0061] WHERE departments.department_name = 'IT'; / *Filter the results and return only the records where the department name is 'IT'* /

[0062] The generative model with text completion function is a generative model for code completion of SQL language. When the preset segmentation category fields are the primary category fields and secondary category fields in the above example, it can be seen that the primary category fields included in the initial training sample are: SELECT, FROM, JOIN, ON, WHERE, and the secondary category fields are AS. In this way, when determining the first candidate segmentation category to which the initial training sample belongs, the primary category field can be directly determined as the first candidate segmentation category to which the initial training sample belongs, that is, the initial training sample is respectively classified as SELECT, FROM, JOIN, ON, WHERE. The initial training sample can also be classified as SELECT-AS, FROM-normal, JOIN-normal, ON-normal, WHERE-normal according to the primary category field and the secondary category field.

[0063] Secondly, according to the first candidate segmentation category to which each initial training sample belongs, the candidate segmentation position corresponding to the initial training sample under the first candidate segmentation category is determined. Specifically, any position between every two preset segmentation category fields can be determined as the candidate segmentation position corresponding to the initial training sample.

[0064] S13. Determine the amount of sample data for each preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category.

[0065] Among them, the category occurrence probability of each preset segmentation category can be obtained by pre-counting the number of occurrences of each preset segmentation category in the edited text content including the input dwell position within a preset time period. During the online text editing process of the edited text content, the input dwell position should correspond to the predicted text. For example, the generative model with text completion function is a generative model for code completion in the SQL language. The number of occurrences of each preset segmentation category in the edited text content is as follows: select-and appears 80 times, select-sum appears 60 times, select-normal appears 160 times, join-and appears 80 times, join-sum appears 60 times, and join-normal appears 160 times. The total number of occurrences of the preset segmentation category is 80+60+160+80+60+160=600 times, and the category occurrence probability of select-and is 80÷600= 0.13, the category occurrence probability of select-sum is 60÷600=0.1, and the category occurrence probability of select-normal is 160÷600=0.27; the category occurrence probability of join-and is 80÷600=0.13, the category occurrence probability of join-sum is 60÷600=0.1, and the category occurrence probability of join-normal is 160÷600=0.27; the category occurrence probability of select is (80+60+160)÷600=0.5, and the category occurrence probability of join is (80+60+160)÷600=0.5.

[0066] Specifically, based on the category occurrence probability of each preset segmentation category and the total amount of sample data, the method of determining the sample data amount of the preset segmentation category can be to multiply the total amount of sample data by the category occurrence probability of each preset segmentation category as the sample data amount of each preset segmentation category. For example, if the total amount of sample data is 100, the preset segmentation category is select, and the category occurrence probability is 0.25, then the sample data amount of select is 100×0.25=25.

[0067] S14, based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category, determining the target segmentation category of the initial training sample from the first candidate segmentation categories to which the initial training sample belongs.

[0068] Specifically, for each preset segmentation category, from each first candidate segmentation category to which the initial training sample belongs, a first candidate segmentation category with the same category and sample data volume as the preset segmentation category is matched as the target segmentation category of the initial training sample. For example, if the sample data volume of select is 25, then from each first candidate segmentation category to which the initial training sample belongs, 25 first candidate segmentation categories belonging to select are matched as the target segmentation category of the initial training sample.

[0069] In some embodiments, since an initial training sample can be divided into multiple first candidate segmentation categories, such as in the example of step S12, an initial training sample is classified into SELECT-AS, FROM-normal, JOIN-normal, ON-normal, and WHERE-normal. Therefore, the same initial training sample can be counted as a target segmentation category multiple times as different first candidate segmentation categories.

[0070] S15. Perform text segmentation processing on the initial training samples based on the candidate segmentation positions corresponding to the target segmentation categories to generate target training samples.

[0071] The target training sample contains multiple segmented text contents.

[0072] Specifically, when the candidate segmentation position is any position between every two preset segmentation category fields, all segmentable target segmentation categories in the initial training sample can be confirmed by string matching, corresponding to the preset segmentation category field position, and then starting from the preset segmentation category field position, searching backward until encountering the preset segmentation category field corresponding to the next target segmentation category, and then any position between the two preset segmentation category fields is determined as the candidate segmentation position corresponding to the initial training sample, and the initial training sample is segmented to obtain multiple segmented text contents, and the target training sample is generated. For example, "selecta,b,cfromtable_awherea=b" has three preset segmentation category fields, "select", "from", and "where". Any position from select to from is the candidate segmentation position range under the select trigger position. Similarly, the candidate segmentation position range of from is any position in the middle to where. The same is true for where. That is to say, it can be segmented into "selecta,b," "cfromtable_a," "wherea=b" or "selecta,b,cfrom," "table_a," "wherea=b".

[0073] The disclosed embodiment can first determine at least one first candidate segmentation category to which each initial training sample belongs, and the candidate segmentation position corresponding to each first candidate segmentation category, based on the preset segmentation category field. And determine the amount of sample data for each preset segmentation category based on the total amount of sample data and the category occurrence probability of each preset segmentation category. Then, determine the target segmentation category of the initial training sample from the first candidate segmentation category based on each preset segmentation category, the sample data amount of each preset segmentation category, and the first candidate segmentation category. Finally, according to the candidate segmentation position corresponding to the target segmentation category, the initial training sample is segmented and processed to generate the target training sample. In this way, the segmentation of the initial training sample is guided by the preset segmentation category field and the category occurrence probability of the preset segmentation category of the real statistics, avoiding the target training sample generated by random segmentation, reducing the difference between the generated target training sample and the real situation, and indirectly improving the accuracy of the generative model with text completion function trained by the target training sample.

[0074] Figure 2 is a flow chart of a method for determining the probability of occurrence of a preset segmentation category provided by an embodiment of the present disclosure. Figure 2 The method for determining the probability of occurrence of a category of a preset segmentation category includes:

[0075] S1. Obtain edited text content during an online text editing process within a preset time period.

[0076] Among them, the edited text content includes the position where the cursor stops during the text editing process (i.e., the input stop position). The input stop position corresponds to the predicted text content output by the generative model with text completion function. In other words, the edited text content is a section of edited text in the online text editing process where the user's input cursor has stopped and the predicted text content has been given at the input stop position. For example, when the user's cursor stops when editing to "select a, b, sum(", "sum(" is used to specify the specific field to be queried. At this time, the position where the cursor stops will be given by the generative model with text completion function. The position where the cursor stops is the input stop position, which is included in the edited text content corresponding to "select a, b, sum(".

[0077] In some embodiments, the edited text content may be unfinished or completed text content being edited online.

[0078] S2. Perform line processing on the edited text content, and filter the text content obtained by line processing based on the complete text structure in the text editing mode to generate target text content.

[0079] First, the edited text content is processed in lines. Specifically, the code can be processed in lines according to line breaks.

[0080] Secondly, based on the complete text structure in the text editing mode, the text content obtained by line branching is filtered to generate target text content.

[0081] Specifically, taking the generative model with text completion function as a generative model for SQL language code completion as an example, when searching from the input stop position to the front, the search range needs to be considered. According to normal readable habits, the search indentation from the back to the front must be getting closer and closer to the top grid, such as the following code:

[0082]

[0083] It can be seen that the input stop position is after a subquery. The user wants to add the alias of the subquery. The correct category should be from-normal. However, if the indentation is not considered and the input stop position is matched forward, the result will match the where category in the subquery. Therefore, when determining the target text content, filter out the complete text structure with an indentation greater than the indentation of the row where the input stop position is located to avoid the influence of the preset segmentation category field in the complete structure on the overall statistical results.

[0084] S3. Perform field matching on the target text content based on the preset segmentation category field to determine the preset segmentation categories corresponding to the target text content, count the number of occurrences of each preset segmentation category, and determine the category occurrence probability of the preset segmentation category.

[0085] First, the target text content is matched based on the preset segmentation category field to determine the preset segmentation categories corresponding to the target text content. Specifically, the method of matching the target text content based on the preset segmentation category field to determine the preset segmentation categories corresponding to the target text content is the same as the method of matching any initial training sample based on the preset segmentation category field in step S12 to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs, and will not be repeated here.

[0086] Secondly, the number of occurrences of each preset segmentation category is counted to determine the category occurrence probability of the preset segmentation category. Specifically, the method of counting the number of occurrences of each preset segmentation category and determining the category occurrence probability of the preset segmentation category is detailed in step S13 and will not be repeated here.

[0087] Exemplarily, when the generative model with text completion function is a generative model for code completion of SQL language, the category occurrence probability of the preset segmentation category counted in a certain time period is as shown in Table 1 below, including each preset segmentation category and the category occurrence probability corresponding to each preset segmentation category.

[0088] Table 1

[0089]

[0090] The disclosed embodiment counts the occurrence probabilities of categories of preset segmentation categories based on the edited text content distributed online, and can reasonably plan the data composition of the target training samples according to the actual online situation, so that the generated target training samples must be real examples encountered online, and can be aligned with the actual online distribution as much as possible while ensuring the completeness of the target training sample types, thereby improving the rationality of determining the target training samples and further improving the accuracy of the generative model with text completion function trained by the target training samples. In addition, since the complete text structure usually does not need to be completed, the disclosed embodiment also filters the complete text structure when generating the target text content, and reduces the amount of data processing as much as possible while ensuring the rationality of the data, saving computing resources.

[0091] Figure 3 4 is a flowchart of another method for determining the probability of occurrence of a preset segmentation category provided by an embodiment of the present disclosure, in which step S3 is further optimized. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 3 The method for determining the probability of occurrence of a category of a preset segmentation category includes:

[0092] S1. Obtain edited text content during an online text editing process within a preset time period.

[0093] S2. Perform line processing on the edited text content, and filter the text content obtained by line processing based on the complete text structure in the text editing mode to generate target text content.

[0094] S31 . Perform field matching on the target text content based on a preset segmentation category field to determine each second candidate segmentation category corresponding to the target text content.

[0095] Specifically, the method of performing field matching on the target text content based on the preset segmentation category field to determine the preset segmentation categories corresponding to the target text content is the same as the method of performing field matching on any initial training sample based on the preset segmentation category field in step S12 to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs, and will not be repeated here.

[0096] S32: Based on the text editing mode, verify each second candidate segmentation category to determine each preset segmentation category corresponding to the target text content.

[0097] Specifically, certain restrictions need to be considered when matching the second candidate segmentation category, such as the verification example in Table 2 below.

[0098] Table 2

[0099]

[0100] For the first code segment, the input stops after the word sum. The user may want the function sum(boy_num, girl_num) or the field summary. The preset segmentation category should be select-normal. However, if the match is only performed according to the preset segmentation category field, the preset segmentation category will be select-sum. Obviously, the user does not necessarily want a function at this time. The restriction check ensures that the conditions that some preset segmentation category fields must meet are matched. For example, when sum is matched as a secondary category field, there must be ( before encountering sum, that is, when "sum(" appears, it can be confirmed that the user wants to add a function. In addition to the left and right brackets, the restriction check also includes line break check and separation check. The line break check requires that there are at most N line breaks before encountering the corresponding keyword, and the separation check requires that there are at most N commas before encountering the corresponding keyword. Taking the above Table 2 as an example, the trigger position is in the next line of "and b=c". In fact, the and condition is complete, and the correct preset segmentation category is select-normal. Therefore, it is necessary to restrict the matching of the and keyword to at most 0 line breaks before it. Similarly, in the example of the separation check, the input stop position is in the next line of "asid," and the as alias is complete. The correct preset segmentation category is select-normal, so it is necessary to restrict the matching of the as keyword to at most 0 commas before it. In the restriction check, at most / at least, the number are all hyperparameters and can be configured according to actual conditions.

[0101] S33, counting the number of occurrences of each preset segmentation category, and determining the category occurrence probability of the preset segmentation category.

[0102] The disclosed embodiment can verify each second candidate segmentation category to ensure the accuracy of determining the preset segmentation category, avoid the influence of the preset segmentation category with errors on the statistical probability of occurrence of the preset segmentation category, and ensure the accuracy of the probability of occurrence of the preset segmentation category.

[0103] Figure 4 FIG. 1 is a flow chart of another training data generation method provided by an embodiment of the present disclosure, in which step S14 is further optimized. Explanations of terms that are the same or corresponding to the above embodiments are not repeated here. Figure 4 , the training data generation method includes:

[0104] S11. Obtain multiple initial training samples.

[0105] Among them, the initial training samples include text content with complete structure.

[0106] S12: Perform field matching on any initial training sample based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category.

[0107] S13. Determine the amount of sample data for each preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category.

[0108] S141. Sort the preset segmentation categories based on the amount of sample data from small to large.

[0109] Specifically, each preset segmentation category may be sorted from small to large based on the amount of sample data by a sorting algorithm, for example, bubble sort, selection sort, and the like.

[0110] S142. According to the order of the preset segmentation categories in the sorting result, when the current sample data amount of the preset segmentation category is less than the sample data amount, the preset segmentation category is matched with each first candidate segmentation category to which each initial training sample belongs.

[0111] Specifically, first, according to the order of the preset segmentation categories in the sorting results, matching starts from the preset segmentation category with the least sample data. Then, when the current sample data volume of the preset segmentation category is less than the sample data volume, that is, there are still preset segmentation categories that have not been matched, the preset segmentation categories are matched with the first candidate segmentation categories to which the initial training samples belong. For example, the sorting results are: 1 select-avg, 2 select-and, and 76 select-or. According to the sorting results, the preset segmentation category with the least sample data volume is select-avg, which has only 1. Since the matching has not started yet, the current sample data volume is 0. At this time, 0<1, and the preset segmentation category is matched with the first candidate segmentation category to which the initial training samples belong. For another example, according to the category occurrence probabilities of each preset segmentation category in Table 1, select accounts for 0.4328 and where accounts for 0.2326. Assuming that the total sample data is 300, the sample data volumes corresponding to select and where are 171 and 41 respectively. Based on the construction principle of maintaining data diversity as much as possible, data of the where type should be matched first.

[0112] S143, determining the first candidate segmentation category that is successfully matched as the target segmentation category of the corresponding initial training sample, and accumulating the current sample data volume of the preset segmentation category by 1 until the current sample data volume of the preset segmentation category is equal to the sample data volume.

[0113] Specifically, first, any first candidate segmentation category that successfully matches the preset segmentation category is determined as the target segmentation category of the corresponding initial training sample. After that, the current sample data volume of the preset segmentation category is accumulated by 1, and when the accumulated current sample data volume is still less than the sample data volume, return to execute steps S141-S143 until the current sample data volume of all preset segmentation categories is equal to the sample data volume, and the target segmentation category of the initial training sample is obtained.

[0114] In some embodiments, if there are multiple first candidate segmentation categories of initial training samples that successfully match the preset segmentation category, the first candidate segmentation category that successfully matches is determined as the target segmentation category corresponding to the initial training sample with the least number of first candidate segmentation categories. For example, the initial training sample includes "SELECT d.d_name FROM emJOIN dpm ON em.d_id=dpm.d_id WHERE d.d_name='IT'" and "select a,b,c fromtable_a where a=b". Among them, the first candidate segmentation category of the first initial training sample includes select, from, join, on, where, and the first candidate segmentation category of the second initial training sample includes select, from, where. In this way, when the preset segmentation category is where, both initial training samples can be matched successfully. When selecting, the second initial training sample with the least number of first candidate segmentation categories is preferentially selected as the target segmentation category corresponding to the preset segmentation category where.

[0115] S15. Perform text segmentation processing on the initial training samples based on the candidate segmentation positions corresponding to the target segmentation categories to generate target training samples.

[0116] The target training sample contains multiple segmented text contents.

[0117] The disclosed embodiment can match the target segmentation category based on the amount of sample data from small to large, thus avoiding the situation where some preset segmentation categories with small amounts of sample data cannot be matched to the corresponding first candidate segmentation category when directly matching the target segmentation category from a large amount of sample data, thereby ensuring the diversity of the target segmentation category determined at the preset segmentation category level. In addition, the disclosed embodiment can also determine the successfully matched first candidate segmentation category as the target segmentation category corresponding to the initial training samples with the least number of first candidate segmentation categories when the preset segmentation category matches the first candidate segmentation category of multiple initial training samples, thereby ensuring the diversity of the target segmentation category determined at the first candidate segmentation category level.

[0118] Figure 5 FIG. 1 is a flow chart of another training data generation method provided by an embodiment of the present disclosure, in which step S15 is further optimized. Explanations of terms that are the same or corresponding to the above embodiments are not repeated here. Figure 5 , the training data generation method includes:

[0119] S11. Obtain multiple initial training samples.

[0120] Among them, the initial training samples include text content with complete structure.

[0121] S12: Perform field matching on any initial training sample based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category.

[0122] S13. Determine the amount of sample data for each preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category.

[0123] S14, based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category, determining the target segmentation category of the initial training sample from the first candidate segmentation categories to which the initial training sample belongs.

[0124] S151 . Based on the candidate segmentation positions corresponding to the target segmentation category and the preset number of segments, the initial training sample is subjected to text segmentation processing to generate the preset number of segmented contents.

[0125] The preset number of segments is preset, for example, it can be a default value, or a value set by relevant personnel according to actual conditions, for example, the preset number of segments is three.

[0126] Specifically, the initial training sample is segmented into a preset number of segmented contents according to the candidate segmentation positions corresponding to the target segmentation category.

[0127] In some embodiments, the initial training sample is a complete code text content; the preset segmentation category field includes at least a primary category field and a secondary category field, the primary category field includes at least a code keyword and a first default keyword, and the secondary category field includes at least a function name, a logic keyword, and a second default keyword; the preset segmentation category is determined based on the primary category field, or the preset segmentation category is determined based on the primary category field and the secondary category field. The classification of the primary category field and the secondary category field is detailed in step S12 and will not be repeated here.

[0128] If the preset number of segments is three, the initial training sample is subjected to text segmentation processing based on the candidate segmentation positions corresponding to the target segmentation category and the preset number of segments, and the method of generating the preset number of segmented contents can be to perform text segmentation processing on the complete code text content based on the candidate segmentation positions corresponding to the target segmentation category to generate the front code segment, the middle code segment and the back code segment. For example, if the initial training sample is selecta,b,cfromtable_awherea=b, and the preset number of segments is three, the segmented contents after segmentation processing can be "selecta,b," "cfromtable_a," "wherea=b," or "selecta,b,cfrom," "table_a," "wherea=b," etc.

[0129] S152: Arrange the contents of each segment based on a preset reorganization method to generate target training samples.

[0130] Among them, the target training samples are used to train the generative model with text completion function.

[0131] Specifically, the method of organizing the contents of each segment based on a preset reorganization method to generate a target training sample can be to organize the contents of each segment based on a PSM method to generate a target training sample; or it can be to organize the contents of each segment based on an SPM method to generate a target training sample.

[0132] The disclosed embodiment can first perform text segmentation processing on the initial training sample, generate segmented content with a preset number of segments, and then reorganize the segmented content to generate target training samples for training the generative model with text completion function, so as to improve the generalization ability and accuracy of the generative model with text completion function trained by relying on the target training samples.

[0133] In some embodiments, a generative model with a text completion function can be generated directly based on target training sample training; or the target training samples determined in the form of FIM data and the initial training samples in the form of NTP data can be combined in a certain ratio (such as 1:1) to generate training data for a generative model with a text completion function, and a generative model with a text completion function can be generated through a model training process.

[0134] In some embodiments, special marks are added to the target training sample determined by the FIM data format to distinguish the front code segment, the middle code segment and the back code segment. For example, the front code segment is marked with <|fim begin|>, the middle code segment is marked with <|fim hole|>, and the back code segment is marked with <|fim end|>. For another example, the marking format is as follows:

[0135]

[0136] In some embodiments, when training a generative model with a text completion function, the training task is to predict the next word given a sentence. The function of the generative model with a text completion function is to predict the next word given a sentence until an end symbol is encountered. Regardless of the NTP data format or the FIM data format, at the level of the generative model with a text completion function, a sentence is input and the next word is predicted, but according to the input form, two input forms, NTP data form and FIM data form, are distinguished.

[0137] In some embodiments, in order to train the effectiveness of the data generation method, an evaluation set from a real business scenario is established in the embodiment of the present application, totaling 300 questions. This evaluation set covers most of the situations of online samples in the classification system. During the evaluation, the training results are evaluated using the mature automatic evaluation algorithm in the business. The key indicators are Excatly Match (EM) and Score (SCORE). EM represents the number of results predicted by the generative model with text completion function that completely match the answer. SCORE represents the score of the predicted result of the generative model with text completion function. If it is a perfect match, the score is 1. If it is completely irrelevant, the score is 0. The final SCORE is the average score of all samples in the evaluation set. The results of the test are shown in Table 3 below:

[0138] Table 3

[0139] Data volume SCORE EM Base / 0.2345 17 Training data generation method_v0.1 0.05B 0.2838(+0.0493) 29 Random Splitting Strategy_v0.1 0.05B 0.2749(+0.0404) 27 Training data generation method_v0.2 0.19B 0.3101(+0.0756) 36 Random Splitting Strategy_v0.2 0.19B 0.3075(+0.073) 35 Training data generation method_v0.3 1.1B 0.3598(+0.1253) 50 Random Splitting Strategy_v0.3 1.1B 0.3225(+0.088) 37

[0140] Among them, the base model used in the embodiment of the present disclosure is deepseek-coder-1.3b-base, and its SCORE index on the evaluation set is 0.2345, and the number of EMs is 17. In terms of data scale, a total of 0.05B, 0.19B, and 1.1B scales were used for comparison. The benchmark group is the random segmentation strategy, and the control group is the training data generation method of the embodiment of the present disclosure. From the data in Table 3, it can be seen that after continuing pre-training, both the random segmentation strategy and the training data generation method of the embodiment of the present disclosure have better effects than the original base model; the training data generation method of the embodiment of the present disclosure is better than the random segmentation strategy in all magnitudes, and the trend will become obvious as the data set increases.

[0141] The following is an embodiment of a training data generating device provided in an embodiment of the present invention. The device and the training data generating methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the training data generating device, reference can be made to the embodiment of the above-mentioned training data generating method.

[0142] Figure 6 FIG. 1 is a schematic diagram showing a structure of a training data generating device provided by an embodiment of the present disclosure. Figure 6 As shown, the training data generating device 600 may include: an acquisition module 61 , a matching module 62 , a determination module 63 and a processing module 64 .

[0143] An acquisition module 61 is used to acquire a plurality of initial training samples; the initial training samples include text content with complete structure;

[0144] A matching module 62 is used to perform field matching on any initial training sample based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category;

[0145] A determination module 63, configured to determine the amount of sample data for a preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category;

[0146] The determination module 63 is further used to determine the target segmentation category of the initial training sample from each of the first candidate segmentation categories to which the initial training sample belongs based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category;

[0147] The processing module 64 is used to perform text segmentation processing on the initial training sample based on the candidate segmentation positions corresponding to the target segmentation category to generate a target training sample; the target training sample contains a plurality of segmented text contents.

[0148] In some embodiments, the determination module 63 is specifically configured to:

[0149] Sort each preset segmentation category based on the amount of sample data from small to large;

[0150] According to the order of the preset segmentation categories in the sorting result, when the current sample data amount of the preset segmentation category is less than the sample data amount, the preset segmentation category is matched with each first candidate segmentation category to which each initial training sample belongs;

[0151] The first candidate segmentation category that is successfully matched is determined as the target segmentation category of the corresponding initial training sample, and the current sample data volume of the preset segmentation category is accumulated by 1 until the current sample data volume of the preset segmentation category is equal to the sample data volume.

[0152] In some embodiments, the determination module 63 is specifically configured to:

[0153] If there are multiple first candidate segmentation categories of initial training samples that successfully match the preset segmentation category, the first candidate segmentation category that successfully matches is determined as the target segmentation category corresponding to the initial training samples with the least number of first candidate segmentation categories.

[0154] In some embodiments, the preset segmentation category field is obtained by pre-stating the occurrence frequency of the field corresponding to the input stop position during the text editing process.

[0155] In some embodiments, the acquisition module 61 is further used to acquire the edited text content during the online text editing process within a preset time period; the edited text content includes the input stop position; the input stop position corresponds to the predicted text content;

[0156] The processing module 64 is further used to perform line processing on the edited text content, and filter the text content obtained by line processing based on the complete text structure in the text editing mode to generate target text content;

[0157] The determination module 63 is further used to perform field matching on the target text content based on the preset segmentation category field, determine the preset segmentation categories corresponding to the target text content, and count the number of occurrences of each preset segmentation category to determine the category occurrence probability of the preset segmentation category.

[0158] In some embodiments, the determination module 63 is specifically configured to:

[0159] Perform field matching on the target text content based on the preset segmentation category field to determine each second candidate segmentation category corresponding to the target text content;

[0160] Based on the text editing method, each second candidate segmentation category is verified to determine each preset segmentation category corresponding to the target text content.

[0161] In some embodiments, the processing module 64 is specifically configured to:

[0162] Based on the candidate segmentation positions corresponding to the target segmentation category and the preset number of segments, the initial training sample is segmented to generate the preset number of segmented contents;

[0163] The contents of each segment are sorted based on the preset reorganization method to generate target training samples.

[0164] In some embodiments, the initial training sample is a complete code text content;

[0165] The preset segmentation category field includes at least a primary category field and a secondary category field, the primary category field includes at least a code keyword and a first default keyword, and the secondary category field includes at least a function name, a logic keyword, and a second default keyword;

[0166] The preset segmentation category is determined based on the primary category field, or the preset segmentation category is determined based on the primary category field and the secondary category field;

[0167] The processing module 64 is specifically used for:

[0168] If the preset number of segments is three, the complete code text content is segmented based on the candidate segmentation positions corresponding to the target segmentation category to generate the front code segment, the middle code segment and the back code segment.

[0169] The training data generating device 600 provided in the embodiment of the present invention can execute the training data generating method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] It is worth noting that in the embodiment of the above-mentioned training data generating device 600, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure.

[0171] The present disclosure also provides an electronic device, which may include a processor and a memory, wherein the memory may be used to store executable instructions. The processor may be used to read the executable instructions from the memory and execute the executable instructions to implement the training data generation method in the above embodiment.

[0172] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0173] like Figure 7As shown, the electronic device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output interface (I / O interface) 705 is also connected to the bus 704.

[0174] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data.

[0175] It should be noted that Figure 7 The electronic device 700 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0176] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the training data generation method of any embodiment of the present disclosure are executed.

[0177] The embodiments of the present disclosure further provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the training data generating method in any embodiment of the present disclosure.

[0178] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0179] In some embodiments, the client and server may communicate using any currently known or future developed network protocol such as HTTP, and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0180] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0181] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the steps of the training data generating method described in any embodiment of the present disclosure.

[0182] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0185] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0187] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0188] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A training data generation method, characterized in that: include: Obtain multiple initial training samples; The initial training sample includes text content with complete structure; Performing field matching on any of the initial training samples based on a preset segmentation category field to determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category; Determine the amount of sample data for each preset segmentation category based on the category occurrence probability and the total amount of sample data for each preset segmentation category; Based on the matching relationship between the sample data volume, the preset segmentation category and the first candidate segmentation category, determining the target segmentation category of the initial training sample from each of the first candidate segmentation categories to which the initial training sample belongs; Performing text segmentation processing on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample; The target training sample contains a plurality of segmented text contents.

2. The method according to claim 1, characterized in that The determining, based on the matching relationship between the sample data volume, the preset segmentation category and the first candidate segmentation category, of the target segmentation category of the initial training sample from each of the first candidate segmentation categories to which the initial training sample belongs, includes: Sorting the preset segmentation categories based on the amount of sample data from small to large; According to the order of the preset segmentation categories in the sorting result, when the current sample data amount of the preset segmentation category is less than the sample data amount, the preset segmentation category is matched with each of the first candidate segmentation categories to which each of the initial training samples belongs; The first candidate segmentation category that is successfully matched is determined as the target segmentation category of the corresponding initial training sample, and the current sample data volume of the preset segmentation category is increased by 1 until the current sample data volume of the preset segmentation category is equal to the sample data volume.

3. The method according to claim 2, characterized in that The step of determining the first candidate segmentation category that matches successfully as the target segmentation category of the corresponding initial training sample includes: If there are multiple initial training samples whose first candidate segmentation categories successfully match the preset segmentation category, the first candidate segmentation category that successfully matches is determined as the target segmentation category corresponding to the initial training samples with the least number of the first candidate segmentation categories.

4. The method according to claim 1, characterized in that: The preset segmentation category fields are obtained by pre-stating the occurrence frequencies of the fields corresponding to the input stop positions during the text editing process.

5. The method according to claim 1, characterized in that The category occurrence probability of the preset segmentation category is obtained in advance by the following method: Obtaining edited text content during an online text editing process within a preset time period; the edited text content includes an input stop position; the input stop position corresponds to predicted text content; Performing line processing on the edited text content, and filtering the text content obtained by line processing based on the complete text structure in the text editing mode to generate target text content; Based on the preset segmentation category field, field matching is performed on the target text content to determine each preset segmentation category corresponding to the target text content, and the number of occurrences of each preset segmentation category is counted to determine the category occurrence probability of the preset segmentation category.

6. The method according to claim 5, characterized in that The performing field matching on the target text content based on the preset segmentation category field to determine each preset segmentation category corresponding to the target text content includes: Performing field matching on the target text content based on the preset segmentation category field to determine each second candidate segmentation category corresponding to the target text content; Based on the text editing method, each of the second candidate segmentation categories is verified to determine each of the preset segmentation categories corresponding to the target text content.

7. The method according to claim 1, characterized in that The performing text segmentation processing on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample includes: Based on the candidate segmentation positions corresponding to the target segmentation category and the preset number of segments, the initial training sample is subjected to text segmentation processing to generate the preset number of segmented contents; The content of each segment is reorganized based on a preset reorganization method to generate the target training sample.

8. The method according to claim 7, characterized in that The initial training sample is a complete code text content; The preset segmentation category field includes at least a primary category field and a secondary category field, the primary category field includes at least a code keyword and a first default keyword, and the secondary category field includes at least a function name, a logic keyword, and a second default keyword; The preset segmentation category is determined based on the primary category field, or the preset segmentation category is determined based on the primary category field and the secondary category field; If the preset number of segments is three, then based on the candidate segmentation positions corresponding to the target segmentation category and the preset number of segments, the initial training sample is subjected to text segmentation processing to generate the preset number of segmented contents, including: The candidate segmentation positions corresponding to the target segmentation categories are used to perform text segmentation processing on the complete code text content to generate front-end code, middle-end code and back-end code.

9. A training data generating device, characterized in that: include: An acquisition module, used for acquiring multiple initial training samples; The initial training sample includes text content with complete structure; A matching module, configured to perform field matching on any of the initial training samples based on a preset segmentation category field, and determine at least one first candidate segmentation category to which the corresponding initial training sample belongs and a candidate segmentation position corresponding to the first candidate segmentation category; A determination module, configured to determine the amount of sample data of the preset segmentation category based on the category occurrence probability and the total amount of sample data of each preset segmentation category; The determination module is further configured to determine a target segmentation category of the initial training sample from each of the first candidate segmentation categories to which the initial training sample belongs based on the sample data volume, the matching relationship between the preset segmentation category and the first candidate segmentation category; A processing module, configured to perform text segmentation processing on the initial training sample based on the candidate segmentation position corresponding to the target segmentation category to generate a target training sample; The target training sample contains a plurality of segmented text contents.

10. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the training data generation method described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the training data generation method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product is used to implement the training data generation method according to any one of claims 1 to 8.

Images