Semantic segmentation method and device for lines, electronic equipment and readable medium

Abstract

The invention relates to a line semantic segmentation method and device, electronic equipment and a readable medium, and the method comprises the steps: carrying out the semantic segmentation of first line data and second line data through a first semantic segmentation model, and generating a first data set and a second data set according to the segmentation result, the first data set and the second data set both comprise sample data of different difficulty levels; integrating the first data set and the second data set according to the difficulty level to obtain a third data set; generating training sets of different training stages according to the third data set; performing staged tuning training on the second semantic segmentation model by using the training set to obtain a target semantic segmentation model; and performing semantic segmentation on the target line data by using the target semantic segmentation model to obtain a semantic segmentation result. The problem that the accuracy of line semantic segmentation can be improved only by using a large number of training samples is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and readable medium for semantic segmentation of lines. Background Art

[0002] With the rapid development of Internet technology and the popularization of smart terminal devices, the dissemination of videos and short videos is becoming more and more widespread. In the field of video content understanding and short video secondary creation, it is very important to accurately segment the lines with semantic integrity.

[0003] At present, the methods of speech semantic segmentation can be divided into two categories: one is to use small models trained on special data sets for implementation, but due to the limitations of training data and model capacity, small models are not accurate enough in speech semantic segmentation; the other is to use large models trained on general data sets. These large models have stronger understanding and generalization capabilities because they have been trained with a large amount of data. Due to the high complexity of the model, its output often has a certain degree of randomness, and it also requires the collection of a large number of training samples, which also requires greater computing resources. Therefore, the need to use a large number of training samples to improve the accuracy of speech semantic segmentation has become a key problem that needs to be solved in the field of semantic segmentation.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a method, device, electronic device and readable medium for semantic segmentation of lines to solve the above-mentioned technical problem of "using a large number of training samples to improve the accuracy of semantic segmentation of lines".

[0006] According to one aspect of an embodiment of the present application, the present application provides a method for semantic segmentation of lines, including: using a first semantic segmentation model to perform semantic segmentation on first line data and second line data, respectively, and generating a first data set and a second data set based on the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; integrating the first data set and the second data set according to the difficulty level to obtain a third data set; generating training sets of different training stages based on the third data set; using the training set to perform stage-by-stage tuning training on the second semantic segmentation model to obtain a target semantic segmentation model; using the target semantic segmentation model to perform semantic segmentation on the target line data to obtain a semantic segmentation result.

[0007] Optionally, the first semantic segmentation model is used to perform semantic segmentation on the first dialogue data and the second dialogue data, respectively, and a first data set and a second data set are generated based on the segmentation results, including: performing semantic segmentation on the first dialogue data using the first semantic segmentation model to obtain a first segmentation result, and dividing the first segmentation result into first data sets of different difficulty levels; performing semantic segmentation on the second dialogue data using the first semantic segmentation model to obtain a second segmentation result, and dividing the second segmentation result into second data sets of different difficulty levels.

[0008] Optionally, the first segmentation result is divided into a first data set of different difficulty levels, including: verifying the first segmentation result to obtain a first sample that passes the verification and a second sample that fails the verification, and correcting the second sample that fails the verification; extracting the confidence of each sample in the first sample and the second sample, and comparing each confidence with a target confidence; determining a sample whose confidence is greater than or equal to the target confidence as a first simple sample, and determining a sample whose confidence is less than the target confidence and belongs to the first sample as a first medium sample, and determining a sample whose confidence is less than the target confidence and belongs to the second sample as a first difficult sample; determining the first simple sample, the first medium sample, and the first difficult sample as a first data set.

[0009] Optionally, the second segmentation result is divided into second data sets of different difficulty levels, including: extracting the confidence of each sample in the second data set, and comparing each confidence with the target confidence; determining the sample whose confidence is greater than or equal to the target confidence as the second simple sample; verifying the sample whose confidence is less than the target confidence to obtain a second medium sample that passes the verification and a second difficult sample that fails the verification; and determining the second simple sample, the second medium sample and the second difficult sample as the second data set.

[0010] Optionally, the first data set and the second data set are integrated according to the difficulty level to obtain a third data set, including: calculating the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the first data set to obtain a first ratio, and calculating the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the second data set to obtain a second ratio; calculating the ratio difference between the first ratio and the second ratio; if the ratio difference is less than or equal to the target difference, integrating the first data set and the second data set into the third data set.

[0011] Optionally, training sets for different training stages are generated based on the third data set, including: converting the third data set according to the training instruction structure corresponding to the second semantic segmentation model to obtain a training instruction set; dividing the training instruction set into a first instruction set corresponding to an early stage of training, a second instruction set corresponding to a middle stage of training, and a third instruction set corresponding to a late stage of training according to the difficulty level, wherein the first instruction set has the highest proportion of instruction samples of simple difficulty, the second instruction set has the highest proportion of instruction samples of medium difficulty, and the third instruction set has the highest proportion of instruction samples of difficult difficulty.

[0012] Optionally, the second semantic segmentation model is fine-tuned and trained in stages using the training set to obtain a target semantic segmentation model, including: performing LORA training on the second semantic segmentation model using the first instruction set to obtain a third semantic segmentation model; performing LORA training on the third semantic segmentation model using the second instruction set to obtain a fourth semantic segmentation model; and performing LORA training on the fourth semantic segmentation model using the third instruction set to obtain a target semantic segmentation model.

[0013] According to another aspect of an embodiment of the present application, the present application provides a semantic segmentation device for lines, including: a first segmentation module, used to use a first semantic segmentation model to perform semantic segmentation on first line data and second line data respectively, and generate a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; an integration module, used to integrate the first data set and the second data set according to the difficulty level to obtain a third data set; a generation module, used to generate training sets of different training stages according to the third data set; a training module, used to use the training set to perform stage-by-stage tuning training on the second semantic segmentation model to obtain a target semantic segmentation model; a second segmentation module, used to use the target semantic segmentation model to perform semantic segmentation on the target line data to obtain a semantic segmentation result.

[0014] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0015] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application provides a method for semantic segmentation of lines, including: using a first semantic segmentation model to semantically segment the first line data and the second line data respectively, and generating a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; integrating the first data set and the second data set according to the difficulty level to obtain a third data set; generating training sets of different training stages according to the third data set; using the training set to perform stage-by-stage fine-tuning training on the second semantic segmentation model to obtain a target semantic segmentation model; using the target semantic segmentation model to perform semantic segmentation on the target line data to obtain a semantic segmentation result. By first dividing the semantically segmented sample data according to different difficulty levels, and then using the sample data of different difficulty levels to perform stage-by-stage fine-tuning training on the semantic segmentation model to obtain an optimized semantic segmentation model, and finally using the optimized semantic segmentation model to perform semantic segmentation of lines, the problem of using a large number of training samples to improve the accuracy of line semantic segmentation is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of an optional method for semantic segmentation of lines provided according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of generating an optional target semantic segmentation model according to an embodiment of the present application;

[0022] Figure 3 A block diagram of an optional speech semantic segmentation device provided according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0026] With the rapid development of Internet technology and the popularization of smart terminal devices, the dissemination of videos and short videos is becoming more and more widespread. In the field of video content understanding and short video secondary creation, it is very important to accurately segment the lines with semantic integrity.

[0027] At present, the methods of speech semantic segmentation can be divided into two categories: one is to use small models trained on special data sets for implementation, but due to the limitations of training data and model capacity, small models are not accurate enough in speech semantic segmentation; the other is to use large models trained on general data sets. These large models have stronger understanding and generalization capabilities because they have been trained with a large amount of data. Due to the high complexity of the model, its output often has a certain degree of randomness, and it also requires the collection of a large number of training samples, which also requires greater computing resources. Therefore, the need to use a large number of training samples to improve the accuracy of speech semantic segmentation has become a key problem that needs to be solved in the field of semantic segmentation.

[0028] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiment of the present application, an embodiment of a method for semantic segmentation of lines is provided, such as Figure 1 As shown, including:

[0029] Step 101, using a first semantic segmentation model to perform semantic segmentation on the first speech data and the second speech data respectively, and generating a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels;

[0030] Step 103, integrating the first data set and the second data set according to the difficulty level to obtain a third data set;

[0031] Step 105, generating training sets of different training stages according to the third data set;

[0032] Step 107, using the training set to perform stage-by-stage fine-tuning training on the second semantic segmentation model to obtain a target semantic segmentation model;

[0033] Step 109, using the target semantic segmentation model to perform semantic segmentation on the target dialogue data to obtain a semantic segmentation result.

[0034] The present application divides the semantically segmented sample data into different difficulty levels, then uses the sample data of different difficulty levels to perform phased fine-tuning training on the semantic segmentation model to obtain an optimized semantic segmentation model, and finally uses the optimized semantic segmentation model to perform semantic segmentation of the lines, thereby solving the problem of using a large number of training samples to improve the accuracy of semantic segmentation of the lines.

[0035] The first semantic segmentation model is a small model trained using an existing dedicated dataset, and the second semantic segmentation model is a large lightweight language model trained on a general dataset.

[0036] Compared with the lightweight large model before training, the target semantic segmentation model has significantly improved the format standardization and accuracy of the segmentation results. Compared with the small model segmentation method, the target semantic segmentation model has significantly reduced the amount of data and training cycle required for model optimization, and the segmentation effect is improved.

[0037] As an optional embodiment, the first semantic segmentation model is used to perform semantic segmentation on the first dialogue data and the second dialogue data respectively, and a first data set and a second data set are generated according to the segmentation results, including: using the first semantic segmentation model to perform semantic segmentation on the first dialogue data to obtain a first segmentation result, and dividing the first segmentation result into first data sets of different difficulty levels; using the first semantic segmentation model to perform semantic segmentation on the second dialogue data to obtain a second segmentation result, and dividing the second segmentation result into second data sets of different difficulty levels.

[0038] The first speech data and the second speech data are different. The second speech data is obtained by expanding the video speech data collection range based on the first speech data.

[0039] An existing semantic segmentation model (i.e., a first semantic segmentation model) trained using a general or dedicated dataset is used to perform semantic segmentation processing on a small amount of video dialogue data to generate corresponding dialogue segmentation results (i.e., segmentation results).

[0040] The methods for obtaining the first data set and the second data set are roughly the same, except that the first speech data and the second speech data are different, and when obtaining the first data set, the samples are first verified and then the difficulty levels are divided, and when obtaining the second data set, simple samples are first selected and then verified, and samples of other difficulty levels are obtained. Next, the methods for obtaining the first data set and the second data set are described in detail.

[0041] As an optional embodiment, the first segmentation result is divided into a first data set of different difficulty levels, including: verifying the first segmentation result to obtain a first sample that passes the verification and a second sample that fails the verification, and correcting the second sample that fails the verification; extracting the confidence of each sample in the first sample and the second sample, and comparing each confidence with a target confidence; determining a sample whose confidence is greater than or equal to the target confidence as a first simple sample, and determining a sample whose confidence is less than the target confidence and belongs to the first sample as a first medium sample, and determining a sample whose confidence is less than the target confidence and belongs to the second sample as a first difficult sample; determining the first simple sample, the first medium sample, and the first difficult sample as a first data set.

[0042] Use a verification mechanism (such as manual inspection, rule matching, or other pre-trained models, etc.) to verify the segmentation results generated by the first semantic segmentation model to identify which segmentation results are accurate (i.e., "first samples that passed the verification") and which are inaccurate (i.e., "second samples that failed the verification").

[0043] For example, the first segmentation result can be manually verified to repair the erroneous samples in the first segmentation result. For samples that fail the verification, corrections are made. The corrections may include re-segmentation, adjustment of segmentation boundaries, and change of annotations, etc., so that the accuracy of the samples can be improved and made closer to the real semantic segmentation results.

[0044] The segmentation results will carry confidence levels, and the confidence level of each sample will be compared with the target confidence level to perform hierarchical screening of difficult examples in the dialogue data.

[0045] Confidence is the degree of confidence the model has in its segmentation results, usually expressed as a probability value or score.

[0046] The target confidence level can be set or adjusted according to actual conditions, for example, the accuracy of samples with a confidence level greater than the target confidence level needs to reach a set value, etc. This application does not limit the specific value of the target confidence level.

[0047] The first simple samples can be understood as samples with a relatively simple difficulty level that can be accurately segmented by the semantic segmentation model, and the confidence of these samples is greater than or equal to the target confidence.

[0048] The first medium sample can be understood as a sample with a medium difficulty level that can be accurately segmented by the semantic segmentation model. The confidence of the first medium sample is less than the target confidence, but the sample belongs to the first sample that has passed the verification.

[0049] The first difficult sample can be understood as a sample with a difficulty level that is difficult for the semantic segmentation model to accurately segment. The confidence of the first difficult sample is less than the target confidence, and it belongs to the second sample that has not passed the verification (although it has been corrected).

[0050] As an optional embodiment, the second segmentation result is divided into second data sets of different difficulty levels, including: extracting the confidence of each sample in the second data set, and comparing each confidence with the target confidence; determining the samples with a confidence greater than or equal to the target confidence as second simple samples; verifying the samples with a confidence less than the target confidence to obtain a second medium sample that passes the verification and a second difficult sample that fails the verification; and determining the second simple sample, the second medium sample, and the second difficult sample as the second data set.

[0051] First, samples with confidence greater than or equal to the target confidence are determined as the second simple samples, and then samples with confidence less than the target confidence are verified, which can reduce the verification workload to a certain extent.

[0052] The samples with confidence greater than or equal to the target confidence are directly determined as the second simple samples, and then the samples with confidence less than the target confidence are verified. The second medium samples and the second difficult samples are determined according to the verification results, among which the samples that pass the verification are the second medium samples, and the samples that fail the verification are the second difficult samples.

[0053] The second easy sample, the second medium sample, and the second difficult sample are combined together to form a second data set containing different difficulty levels.

[0054] By constructing a dataset with different difficulty levels, it helps to improve the generalization ability and robustness of the model in training and optimizing the semantic segmentation model.

[0055] As an optional embodiment, the first data set and the second data set are integrated according to the difficulty level to obtain a third data set, including: calculating the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the first data set to obtain a first ratio, and calculating the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the second data set to obtain a second ratio; calculating the ratio difference between the first ratio and the second ratio; if the ratio difference is less than or equal to the target difference, integrating the first data set and the second data set into the third data set.

[0056] The difference between the first ratio and the second ratio is calculated to obtain the ratio difference. The ratio difference can reflect the difference in the distribution of medium-difficulty and difficult-difficulty samples in the two data sets. Therefore, it is necessary to ensure that the difference in the distribution of medium-difficulty and difficult-difficulty samples in the two data sets is small, that is, the ratio difference is less than or equal to the target difference.

[0057] If the ratio difference is greater than the target difference, it means that the first ratio and the second ratio cannot be maintained at the same level, and the difficulty level classification of each sample in the second data set needs to be adjusted. For example, set more stringent filtering conditions (for example, increase the target confidence), classify the difficulty of the samples in the second data set according to the new target confidence, and calculate the ratio of the number of samples with medium difficulty and difficult difficulty in the second data set (i.e., the new second ratio), and then calculate the new ratio difference between the first ratio and the new second ratio until the new ratio difference is less than or equal to the target difference.

[0058] The target difference can be set according to actual conditions, and this application does not limit the specific value of the target difference.

[0059] The first data set and the second data set are integrated according to the difficulty level to obtain a third data set covering a wider range of difficulty levels, which provides a data basis for subsequent model training.

[0060] As an optional embodiment, training sets for different training stages are generated based on the third data set, including: converting the third data set according to the training instruction structure corresponding to the second semantic segmentation model to obtain a training instruction set; dividing the training instruction set into a first instruction set corresponding to the initial training stage, a second instruction set corresponding to the middle training stage, and a third instruction set corresponding to the late training stage according to the difficulty level, wherein the first instruction set has the highest proportion of instruction samples of simple difficulty, the second instruction set has the highest proportion of instruction samples of medium difficulty, and the third instruction set has the highest proportion of instruction samples of difficult difficulty.

[0061] The third data set is converted according to the training instruction structure corresponding to the second semantic segmentation model. For example, the samples in the data set are converted into the format required for model training, including input data, labels, and possible additional information (such as weights, masks, etc.). The converted data set is called a training instruction set, which contains all the instructions required for model training.

[0062] The training instruction set is divided into three subsets according to the difficulty level: the first instruction set, the second instruction set and the third instruction set. Each subset corresponds to a different stage of training: the first instruction set corresponds to the initial stage of training, the second instruction set corresponds to the middle stage of training, and the third instruction set corresponds to the late stage of training.

[0063] When dividing, it is necessary to ensure that the difficulty level ratio of instruction samples in each subset meets the set requirements, including: the first instruction set has the highest proportion of instruction samples with simple difficulty, the second instruction set has the highest proportion of instruction samples with medium difficulty, and the third instruction set has the highest proportion of instruction samples with difficult difficulty. Reducing the proportion of simple samples in the later stage of training can improve the recognition effect of the model on difficult data.

[0064] A training set suitable for different training stages is generated according to the third data set, so as to more effectively train the second semantic segmentation model.

[0065] This application adopts a batch multi-round training method to adjust the proportion of difficult samples in the line segmentation instruction set, reduce the difficulty of model convergence, and ensure the effect of supervised training.

[0066] This application uses the existing small semantic segmentation model (i.e., the first semantic segmentation model) to generate a preliminary version of the dialogue semantic segmentation results, and then filters the simple samples based on the confidence level, and manually verifies and modifies the erroneous samples with low confidence levels. Finally, samples of different difficulty levels are integrated to construct a training set for dialogue segmentation training.

[0067] As an optional embodiment, the second semantic segmentation model is fine-tuned and trained in stages using the training set to obtain a target semantic segmentation model, including: using the first instruction set to perform LORA training on the second semantic segmentation model to obtain a third semantic segmentation model; using the second instruction set to perform LORA training on the third semantic segmentation model to obtain a fourth semantic segmentation model; using the third instruction set to perform LORA training on the fourth semantic segmentation model to obtain the target semantic segmentation model.

[0068] The tuning training provided in this application is divided into three stages, using the instruction sets corresponding to different stages as training data in turn, and using the training results of the previous stage as the pre-training model. The tuning training of each stage is implemented using the LoRA (Low-Rank Adaptation of Large Language Models) training method, that is, only some parameter weights are updated, which can effectively reduce the resource requirements during the training process.

[0069] It should be noted that the number of training stages can be adjusted according to actual needs, and the three stages provided in this application are only for reference as the optimal embodiment.

[0070] This application adopts the LoRA training method, combined with training sets of different difficulty levels, to perform supervised tuning training on lightweight language large models. The core idea of ​​LoRA is to replace the full weight update with low-rank approximation. Specifically, it freezes the original weights of the pre-trained model and introduces a low-rank matrix ΔW in the weight matrix of certain specific layers, which can be expressed as the product of two smaller trainable matrices A and B (ie ΔW = A·B). During the training process, only the two smaller matrices A and B need to be optimized without updating the weights of the entire model.

[0071] Figure 2 A schematic diagram of the generation of the target semantic segmentation model provided for the present application, as shown in the figure, uses a small semantic segmentation model to recognize video dialogue data, and then undergoes semantic paragraph reasoning, manual verification, and automatic difficult example screening to obtain semantic segmentation instruction sets corresponding to different training stages (including the dialogue semantic segmentation instruction set for stage 1, the dialogue semantic segmentation instruction set for stage 2, and the dialogue semantic segmentation instruction set for stage 3). The semantic segmentation instruction set is then used to perform phased tuning on the large lightweight language model to obtain a dialogue semantic segmentation model (i.e., the target semantic segmentation model of the present application).

[0072] After obtaining the target semantic segmentation model, the video dialogue sequence in the target dialogue data can be processed, and the segmentation type corresponding to the dialogue sequence can be output through the target semantic segmentation model. The dialogue sequences that do not need to be segmented are combined to form a dialogue semantic paragraph; the long video dialogue data is divided into several dialogue semantic paragraphs, which achieves the effect of automatic dialogue semantic segmentation.

[0073] The present application provides a method for semantic segmentation of lines, including: using a first semantic segmentation model to semantically segment the first line data and the second line data respectively, and generating a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; integrating the first data set and the second data set according to the difficulty level to obtain a third data set; generating training sets of different training stages according to the third data set; using the training set to perform stage-by-stage fine-tuning training on the second semantic segmentation model to obtain a target semantic segmentation model; using the target semantic segmentation model to perform semantic segmentation on the target line data to obtain a semantic segmentation result. By first dividing the semantically segmented sample data according to different difficulty levels, and then using the sample data of different difficulty levels to perform stage-by-stage fine-tuning training on the semantic segmentation model to obtain an optimized semantic segmentation model, and finally using the optimized semantic segmentation model to perform semantic segmentation of lines, the problem of using a large number of training samples to improve the accuracy of line semantic segmentation is solved.

[0074] According to another aspect of the embodiment of the present application, the present application provides a semantic segmentation device for lines, such as Figure 3 As shown, including:

[0075] A first segmentation module 302 is used to perform semantic segmentation on the first speech data and the second speech data respectively using a first semantic segmentation model, and generate a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels;

[0076] An integration module 304, configured to integrate the first data set and the second data set according to the difficulty level to obtain a third data set;

[0077] A generating module 306, configured to generate training sets of different training stages according to the third data set;

[0078] A training module 308 is used to perform stage-by-stage fine-tuning training on the second semantic segmentation model using the training set to obtain a target semantic segmentation model;

[0079] The second segmentation module 310 is used to perform semantic segmentation on the target dialogue data using the target semantic segmentation model to obtain a semantic segmentation result.

[0080] It should be noted that the first segmentation module 302 in this embodiment can be used to execute step 101 in the embodiment of the present application, the integration module 304 in this embodiment can be used to execute step 103 in the embodiment of the present application, the generation module 306 in this embodiment can be used to execute step 105 in the embodiment of the present application, the training module 308 in this embodiment can be used to execute step 107 in the embodiment of the present application, and the second segmentation module 310 in this embodiment can be used to execute step 109 in the embodiment of the present application.

[0081] Optionally, the first segmentation module 302 includes:

[0082] A first segmentation submodule, configured to perform semantic segmentation on the first dialogue data using a first semantic segmentation model to obtain a first segmentation result, and divide the first segmentation result into first data sets of different difficulty levels;

[0083] The second segmentation submodule is used to perform semantic segmentation on the second dialogue data using the first semantic segmentation model to obtain a second segmentation result, and divide the second segmentation result into second data sets of different difficulty levels.

[0084] Optionally, the first segmentation submodule is also used to verify the first segmentation result, obtain a first sample that passes the verification and a second sample that fails the verification, and correct the second sample that fails the verification; extract the confidence of each sample in the first sample and the second sample, and compare each confidence with the target confidence; determine the sample whose confidence is greater than or equal to the target confidence as a first simple sample, and determine the sample whose confidence is less than the target confidence and belongs to the first sample as a first medium sample, and determine the sample whose confidence is less than the target confidence and belongs to the second sample as a first difficult sample; determine the first simple sample, the first medium sample, and the first difficult sample as a first data set.

[0085] Optionally, the second segmentation submodule is also used to extract the confidence of each sample in the second data set, and compare each confidence with the target confidence; determine the sample whose confidence is greater than or equal to the target confidence as the second simple sample; verify the sample whose confidence is less than the target confidence to obtain a second medium sample that passes the verification and a second difficult sample that fails the verification; determine the second simple sample, the second medium sample and the second difficult sample as the second data set.

[0086] Optionally, the integration module 304 is also used to calculate the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the first data set to obtain a first ratio, and calculate the ratio of the number of samples with a difficulty level of medium difficulty to that of difficult difficulty in the second data set to obtain a second ratio; calculate the ratio difference between the first ratio and the second ratio; if the ratio difference is less than or equal to the target difference, integrate the first data set and the second data set into a third data set.

[0087] Optionally, the generation module 306 is also used to convert the third data set according to the training instruction structure corresponding to the second semantic segmentation model to obtain a training instruction set; divide the training instruction set into a first instruction set corresponding to the initial stage of training, a second instruction set corresponding to the middle stage of training, and a third instruction set corresponding to the late stage of training according to the difficulty level, wherein the first instruction set has the highest proportion of instruction samples of simple difficulty, the second instruction set has the highest proportion of instruction samples of medium difficulty, and the third instruction set has the highest proportion of instruction samples of difficult difficulty.

[0088] Optionally, the training module 308 is also used to perform LORA training on the second semantic segmentation model using the first instruction set to obtain a third semantic segmentation model; to perform LORA training on the third semantic segmentation model using the second instruction set to obtain a fourth semantic segmentation model; and to perform LORA training on the fourth semantic segmentation model using the third instruction set to obtain a target semantic segmentation model.

[0089] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0090] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 4 As shown, it includes a memory 401, a processor 403, a communication interface 405 and a communication bus 407. The memory 401 stores a computer program that can be run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, the steps of the above method are implemented.

[0091] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0092] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0093] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0094] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0095] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0096] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0097] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0098] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0105] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for semantic segmentation of lines, characterized in that: include: Using a first semantic segmentation model to perform semantic segmentation on the first speech data and the second speech data respectively, and generating a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; Integrate the first data set and the second data set according to the difficulty level to obtain a third data set; Generating training sets of different training stages according to the third data set; Using the training set to perform stage-by-stage fine-tuning training on the second semantic segmentation model to obtain a target semantic segmentation model; The target semantic segmentation model is used to perform semantic segmentation on the target dialogue data to obtain a semantic segmentation result.

2. The method according to claim 1, characterized in that The method of using the first semantic segmentation model to semantically segment the first speech data and the second speech data, and generating the first data set and the second data set according to the segmentation results, includes: Using the first semantic segmentation model to perform semantic segmentation on the first dialogue data to obtain a first segmentation result, and dividing the first segmentation result into the first data sets of different difficulty levels; The second dialogue data is semantically segmented using the first semantic segmentation model to obtain a second segmentation result, and the second segmentation result is divided into second data sets of different difficulty levels.

3. The method according to claim 2, characterized in that The step of dividing the first segmentation result into the first data sets of different difficulty levels includes: Verifying the first segmentation result to obtain a first sample that passes the verification and a second sample that fails the verification, and correcting the second sample that fails the verification; Extracting the confidence of each sample in the first sample and the second sample, and comparing each of the confidences with a target confidence; Determine a sample whose confidence is greater than or equal to the target confidence as a first simple sample, determine a sample whose confidence is less than the target confidence and belongs to the first sample as a first medium sample, and determine a sample whose confidence is less than the target confidence and belongs to the second sample as a first difficult sample; The first simple sample, the first medium sample, and the first difficult sample are determined as the first data set.

4. The method according to claim 2, characterized in that: The step of dividing the second segmentation result into second data sets of different difficulty levels includes: Extracting the confidence of each sample in the second data set, and comparing each of the confidences with a target confidence; Determine the sample whose confidence is greater than or equal to the target confidence as a second simple sample; Verifying the samples whose confidence is less than the target confidence, to obtain a second medium sample that passes the verification and a second difficult sample that fails the verification; The second easy sample, the second medium sample, and the second difficult sample are determined as the second data set.

5. The method according to claim 1, characterized in that The step of integrating the first data set and the second data set according to the difficulty level to obtain a third data set includes: Calculating the ratio of the number of samples with medium difficulty to that with difficult difficulty in the first data set to obtain a first ratio, and calculating the ratio of the number of samples with medium difficulty to that with difficult difficulty in the second data set to obtain a second ratio; Calculating a ratio difference between the first ratio and the second ratio; If the ratio difference is less than or equal to the target difference, the first data set and the second data set are integrated into a third data set.

6. The method according to claim 1, characterized in that Generating training sets of different training stages according to the third data set includes: Converting the third data set according to the training instruction structure corresponding to the second semantic segmentation model to obtain a training instruction set; According to the difficulty level, the training instruction set is divided into a first instruction set corresponding to the initial stage of training, a second instruction set corresponding to the middle stage of training, and a third instruction set corresponding to the late stage of training, wherein the first instruction set has the highest proportion of instruction samples of simple difficulty, the second instruction set has the highest proportion of instruction samples of medium difficulty, and the third instruction set has the highest proportion of instruction samples of difficult difficulty.

7. The method according to claim 6, characterized in that The step of using the training set to perform stage-by-stage fine-tuning training on the second semantic segmentation model to obtain a target semantic segmentation model includes: Using the first instruction set to perform LORA training on the second semantic segmentation model to obtain a third semantic segmentation model; Performing LORA training on the third semantic segmentation model using the second instruction set to obtain a fourth semantic segmentation model; The fourth semantic segmentation model is trained with LORA using the third instruction set to obtain the target semantic segmentation model.

8. A semantic segmentation device for lines, characterized in that: include: A first segmentation module, configured to perform semantic segmentation on the first speech data and the second speech data respectively using a first semantic segmentation model, and generate a first data set and a second data set according to the segmentation results, wherein the first data set and the second data set both include sample data of different difficulty levels; an integration module, configured to integrate the first data set and the second data set according to the difficulty level to obtain a third data set; A generating module, used for generating training sets of different training stages according to the third data set; A training module, used to perform stage-by-stage fine-tuning training on the second semantic segmentation model using the training set to obtain a target semantic segmentation model; The second segmentation module is used to perform semantic segmentation on the target dialogue data using the target semantic segmentation model to obtain a semantic segmentation result.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 7.

Images