Model training method and device for task processing model, equipment and storage medium

Abstract

The invention provides a model training method and device for a task processing model, equipment and a storage medium. The method comprises the steps of extracting various types of shared feature information in a training corpus through a shared feature extraction model; when task types of to-be-executed sub-tasks of different sub-task models are not distinguished, a first-layer model input node of each sub-task model is input into a second-layer model input node of each sub-task model; and synchronously inputting the various types of shared feature information and training text information based on training corpus labeling into each sub-task model in a second input mode of first-layer input, and training the multiple sub-task models in parallel until the overall loss function of the multiple sub-task models meets a training cut-off condition. Thus, it is guaranteed that each sub-task model can be independently trained, meanwhile, various kinds of shared feature information related to the sub-tasks executed by the sub-task models can be provided for the different sub-task models, and then the overall model training effect of the task processing model is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a model training method, apparatus, device and storage medium for a task processing model. Background Art

[0002] In the field of natural language processing, when using text information as model training data, the model can be trained to perform various types of model training tasks for text information, such as text recognition tasks for identifying named entities in specific business scenarios, sentiment classification tasks for identifying sentence sentiments expressed by different sentences, etc. Specifically, taking the above-mentioned text recognition tasks and sentiment classification tasks as examples, although the model training difficulty and model training purpose of these two types of tasks are different, considering that the text recognition of named entities (equivalent to the semantic recognition of word segmentation) belongs to the recognition basis of sentence sentiment recognition, therefore, in the relevant technology, for models with correlations between such model training tasks, the model training steps are usually performed in an overall training manner.

[0003] At present, in the relevant technology, the "assembly-line learning mode" is often used to conduct overall training for the above-mentioned types of task models. Here, still taking the text recognition task and the sentiment classification task as an example, when the assembly-line learning mode is used to conduct overall training for the task model, the first-level sub-task model is first trained to perform the named entity recognition task, and then, with the help of the first-level sub-task model's recognition results for the named entities, the second-level sub-task model is further trained to learn to recognize and classify the semantic emotions, sentiment expression themes, etc. of different named entities in the text information. In this way, based on the assembly-line learning mode, the task model is often prone to cascading accumulation of recognition errors during the overall training process, resulting in the recognition errors generated by each level of the sub-task model being postponed to the next level, which in turn causes the overall model training effect under the assembly-line learning mode to be unsatisfactory. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a model training method, device, equipment and storage medium for a task processing model, so as to construct a multi-task learning model framework, while ensuring that each subtask model can be trained independently, and provide different subtask models with a variety of shared feature information related to the subtasks they perform, which is conducive to improving the overall model training effect of the task processing model.

[0005] In a first aspect, an embodiment of the present application provides a model training method for a task processing model, which is applied to a multi-task learning model framework, wherein the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, wherein the task processing model includes multiple subtask models; the model training method includes:

[0006] Acquire training corpus, and input the training corpus into the shared feature extraction model, and extract multiple categories of shared feature information in the training corpus through the shared feature extraction model;

[0007] According to a preset input method, the multiple categories of shared feature information and the training text information annotated based on the training corpus are input into the multiple subtask models, and the multiple subtask models are trained in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition;

[0008] During the independent training of the multiple subtask models, the task training loss of each subtask model is obtained, and the weight coefficient of the subtask model is adjusted according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as trained task processing models.

[0009] In an optional embodiment, the multiple categories of shared feature information include: character feature vectors after the training corpus is segmented into character sequences; word feature vectors representing syntactic dependencies between words in the training corpus; and sentence feature vectors in the training corpus.

[0010] In an optional implementation, multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method:

[0011] According to the target task dependency between the multiple subtasks to be executed by the multiple subtask models, multiple information categories corresponding to the target task dependency are determined from a preset task dependency table as the multiple categories of shared feature information to be extracted; wherein the task dependency table pre-stores multiple information categories corresponding to the multiple task dependencies.

[0012] In an optional implementation, multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method:

[0013] According to the target task to be executed by the task processing model, taking the multiple subtask models included in the task processing model as the first search space, taking the ability to execute the target task as the first search strategy, performing a neural network structure search on the subtask model combination mode between different subtask models in the first search space, and obtaining the optimal subtask model combination mode that meets the first search strategy;

[0014] Taking each subtask model included in the optimal subtask model combination as the first subtask model;

[0015] According to the first task dependency relationship between the subtasks to be executed in each of the first subtask models, multiple information categories corresponding to the first task dependency relationship are determined from a preset task dependency relationship table as the multiple categories of the shared feature information to be extracted.

[0016] In an optional implementation, multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method:

[0017] According to the target task to be performed by the task processing model, obtaining a variety of text feature information related to completing the target task;

[0018] Taking the multiple types of text feature information as the second search space, taking the ability of the multiple subtask models to complete the target task based on the information combination of different text feature information as the second search strategy, performing a neural network structure search on the information combination mode between different text feature information in the second search space, and obtaining the optimal information combination mode that meets the second search strategy;

[0019] The information category to which each type of text feature information included in the optimal information combination mode belongs is used as the multiple categories of the shared feature information to be extracted.

[0020] In an optional implementation, the inputting of the multiple categories of shared feature information and the training text information annotated based on the training corpus into the multiple subtask models according to a preset input method includes:

[0021] At the first-layer model input node of each of the subtask models, input the training text information into each of the subtask models;

[0022] The multiple categories of shared feature information are hierarchically input to different training nodes in each of the subtask models according to the correspondence between the information categories and the training nodes in a first input method of hierarchical input; wherein the different training nodes in each of the subtask models are sorted from shallow to deep levels of the neural network in the subtask model.

[0023] In an optional implementation, the step of inputting the multiple categories of shared feature information and the training text information annotated based on the training corpus into the multiple subtask models in a preset input manner further includes:

[0024] At the first-layer model input node of each of the subtask models, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models in the second input mode of the first-layer input.

[0025] In an optional implementation, the second input method of the first-level input synchronously inputs the multiple categories of shared feature information and the training text information into each of the subtask models, including:

[0026] When the task types of the subtasks to be executed by different subtask models are not differentiated, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models in the second input mode;

[0027] or,

[0028] When distinguishing the task types to which the subtasks to be executed of different subtask models belong, for each of the subtask models, according to the subtask to be executed by the subtask model, determining the target shared feature information that matches the subtask to be executed by the subtask model among the multiple categories of shared feature information;

[0029] In the second input mode, the multiple categories of shared feature information, the training text information and the target shared feature information are synchronously input into the subtask model.

[0030] In an optional implementation, the overall loss function of the multiple subtask models is determined based on the product of the gradient of the task training loss of each subtask model and the weight coefficient of the subtask model in the multi-task learning model framework; the weight coefficient of the subtask model is adjusted according to the gradient change of the task training loss of each subtask model, including:

[0031] For each of the subtask models, taking the gradient of the task training loss of the subtask model as the target gradient, within a gradient detection period, obtaining the periodic variation amplitude of the target gradient within the detection period;

[0032] When it is detected that the periodic variation amplitude of the target gradient is greater than or equal to the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in a descending manner according to the gradient reduction adjustment coefficient;

[0033] When it is detected that the periodic variation amplitude of the target gradient is smaller than the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in an increasing manner according to the gradient increasing adjustment coefficient.

[0034] In an optional embodiment, each subtask model in the task processing model is used to execute a corresponding subtask, and different subtask models cooperate with each other to process the target task to be executed by the task processing model; when the target task is related to identifying the semantic emotions expressed by text information, the task processing model includes at least one named entity recognition model and one sentiment classification model; wherein the named entity recognition model is used to execute the text recognition task for the named entities included in the training text information; and the sentiment classification model is used to execute the sentiment classification task for the sentence emotions represented by each sentence in the training text information.

[0035] In a second aspect, an embodiment of the present application provides a model training device for a task processing model, which is applied to a multi-task learning model framework, wherein the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, wherein the task processing model includes multiple subtask models; the model training device includes:

[0036] An extraction module, used for acquiring training corpus, and inputting the training corpus into the shared feature extraction model, and extracting multiple categories of shared feature information in the training corpus through the shared feature extraction model;

[0037] An input module, used to input the multiple categories of shared feature information and the training text information annotated based on the training corpus into the multiple subtask models according to a preset input method, and train the multiple subtask models in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition;

[0038] A training module is used to obtain the task training loss of each subtask model during the independent training of the multiple subtask models, and adjust the weight coefficient of the subtask model according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as trained task processing models.

[0039] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned model training method when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned model training method are executed.

[0041] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0042] The embodiment of the present application provides a model training method, device, equipment and storage medium for a task processing model, which first obtains training corpus and inputs the training corpus into a shared feature extraction model, and extracts multiple categories of shared feature information in the training corpus through the shared feature extraction model; then, according to a preset input method, the multiple categories of shared feature information and training text information based on the annotation of the training corpus are input into multiple subtask models, and the multiple subtask models are trained in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition; in the process of independent training of multiple subtask models, the task training loss of each subtask model is obtained, and the weight coefficient of the subtask model is adjusted according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as the trained task processing models.

[0043] In this way, the present application can construct a multi-task learning model framework to ensure that each subtask model can be trained independently while providing different subtask models with a variety of shared feature information related to the subtasks they perform, which is conducive to improving the overall model training effect of the task processing model.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a flow chart of a model training method for a task processing model provided in an embodiment of the present application is shown;

[0047] Figure 2 A flow chart of a method for dynamically adjusting the weight coefficient of each subtask model in a multi-task learning model framework provided by an embodiment of the present application is shown;

[0048] Figure 3A schematic diagram showing a flow chart of a first neural network structure search method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram showing a flow chart of a second neural network structure search method provided in an embodiment of the present application;

[0050] Figure 5 A schematic flow chart of a method for inputting shared feature information in a first input mode provided in an embodiment of the present application is shown;

[0051] Figure 6 A schematic diagram of the structure of a model training device for a task processing model provided in an embodiment of the present application is shown;

[0052] Figure 7 A schematic diagram of the structure of a computer device 700 provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0054] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0055] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0056] At present, in the relevant technology, the "assembly-line learning mode" is often used to conduct overall training for the above-mentioned types of task models. Here, still taking the text recognition task and the sentiment classification task as an example, when the assembly-line learning mode is used to conduct overall training for the task model, the first-level sub-task model is first trained to perform the named entity recognition task, and then, with the help of the first-level sub-task model's recognition results for the named entities, the second-level sub-task model is further trained to learn to recognize and classify the semantic emotions, sentiment expression themes, etc. of different named entities in the text information. In this way, based on the assembly-line learning mode, the task model is often prone to cascading accumulation of recognition errors during the overall training process, resulting in the recognition errors generated by each level of the sub-task model being postponed to the next level, which in turn causes the overall model training effect under the assembly-line learning mode to be unsatisfactory.

[0057] Based on this, the embodiments of the present application provide a model training method, apparatus, device and storage medium for a task processing model, first obtaining a training corpus, and inputting the training corpus into a shared feature extraction model, and extracting multiple categories of shared feature information in the training corpus through the shared feature extraction model; then, according to a preset input method, multiple categories of shared feature information and training text information based on the annotations of the training corpus are input into multiple subtask models, and the multiple subtask models are trained in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition; in the process of independent training of multiple subtask models, the task training loss of each subtask model is obtained, and the weight coefficient of the subtask model is adjusted according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as the trained task processing models.

[0058] In this way, the present application can construct a multi-task learning model framework to ensure that each subtask model can be trained independently while providing different subtask models with a variety of shared feature information related to the subtasks they perform, which is conducive to improving the overall model training effect of the task processing model.

[0059] The following is a detailed introduction to the model training method, device, equipment and storage medium of a task processing model provided in an embodiment of the present application.

[0060] Reference Figure 1 As shown, Figure 1A flow chart of a model training method for a task processing model provided in an embodiment of the present application is shown, wherein the model training method is applied to a multi-task learning model framework, wherein the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, wherein the task processing model includes a plurality of subtask models; the model training method includes steps S101-S103; specifically:

[0061] S101, obtaining training corpus, and inputting the training corpus into the shared feature extraction model, and extracting multiple categories of shared feature information in the training corpus through the shared feature extraction model.

[0062] Here, the training processes of different subtask models in the multi-task learning model framework are independent of each other, that is, the subtasks to be executed by different subtask models can be the same or different, and the embodiments of the present application do not impose any limitations on this.

[0063] In an embodiment of the present application, the training corpus obtained in step S101 represents: corpus information of the same text type used by the above-mentioned multiple subtask models in the training process; the information category of the above-mentioned shared feature information is determined according to the task dependency relationship between the subtasks to be executed by different subtask models.

[0064] Specifically, in an embodiment of the present application, the above-mentioned multiple categories of shared feature information may include: character feature vectors after the training corpus is segmented into character sequences; word feature vectors in the training corpus that represent the syntactic dependency relationship between words; and sentence feature vectors in the training corpus.

[0065] Regarding the above-mentioned multiple subtask models, it should be noted that, in the embodiments of the present application, the subtask models in the multi-task learning model framework are not a collection of arbitrary models without any restrictions; that is, the specific technical application scenarios applicable to the embodiments of the present application (i.e., the model range of the above-mentioned multiple subtask models) are: multiple subtask models under a multi-task learning model framework that "perform tasks related to processing text information" and "use corpus information of the same text type for training" (equivalent to the above-mentioned "multiple subtask models are used to perform different subtasks based on corpus information of the same text type during training").

[0066] Here, for the character feature vector, word feature vector and sentence feature vector in the above shared feature information, it should be noted that:

[0067] The character feature vector may be obtained by segmenting the training corpus according to any fixed step length. For example, the training corpus may be segmented according to a fixed step length of 2 characters to obtain the character feature vector, or the training corpus may be segmented according to a fixed step length of 3 characters to obtain the character feature vector. The specific segmentation method of the above character feature vector is not limited in any way in the embodiment of the present application.

[0068] The word feature vector can be a high-order word feature vector obtained by performing dependency syntactic analysis on the training corpus, or a simple word feature vector obtained by a common word segmentation method. The specific vector form of the word feature vector is not limited in any way in the embodiment of the present application.

[0069] The sentence feature vector can represent a high-dimensional feature vector that maps different sentences in the training corpus under multiple dimensional features. The embodiment of the present application does not impose any limitation on the specific dimensional values ​​of the vector in the sentence feature vector.

[0070] Specifically, with respect to the specific information categories of the above-mentioned shared feature information, it should be noted that, in special cases, when the subtasks to be executed by two different subtask models are the same:

[0071] In a first optional implementation, the text feature information required to execute the subtask may be determined, that is, the shared feature information corresponding to the two current different subtask models;

[0072] In a second optional implementation, it may also be determined that there is no task dependency between the subtasks to be executed by the current two different subtask models, that is, it is determined that there is no shared feature information to be extracted between the current two different subtask models.

[0073] S102, according to a preset input method, input the multiple categories of shared feature information and the training text information annotated based on the training corpus into the multiple subtask models, and train the multiple subtask models in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition.

[0074] Here, the specific annotation method of the above-mentioned training corpus is determined according to the subtask executed by the subtask model currently to be input. For example, if subtask model A is used to perform a text recognition task for named entities included in the training corpus, the named entities included in the training corpus are annotated, and the training corpus after entity annotation is input into subtask model A as training text information.

[0075] Here, the above-mentioned preset input method at least includes: a first input method of layered input and a second input method of first-level input.

[0076] Specifically, in the embodiment of the present application, when the preset input mode is the first input mode, the step S102 may be performed in the following manner 1:

[0077] Method 1: at the first-layer model input node of each subtask model, input the training text information into each subtask model;

[0078] The multiple categories of shared feature information are input hierarchically to different training nodes in each of the subtask models in a first input mode of hierarchical input according to the correspondence between the information categories and the training nodes.

[0079] Here, the different training nodes in each of the subtask models are sorted according to the layers of the neural network in the model from shallow to deep; as an optional embodiment, for shared feature information of different information categories, it can be set that the lower the information amount of the shared feature information, the shallower the neural network layer of the training node corresponding to the information category of the shared feature information is.

[0080] As an exemplary explanation, taking the shared feature information corresponding to subtask model A and subtask model B as character feature vectors and word feature vectors as an example, if subtask model A is a three-layer neural network model composed of neural network a, neural network b and neural network c, and the depth order of the neural networks in subtask model A is: neural network a < neural network b < neural network c, then when neural network a is the first-layer neural network of subtask model A, the character feature vectors with lower information content and training text information in the shared feature information are input into subtask model A from the input node where neural network a is located; the word feature vectors with higher information content in the shared feature information are input into subtask model A from the input node where neural network b is located.

[0081] Specifically, in the embodiment of the present application, when the preset input method is the second input method, the step S102 may be performed according to the following method 2:

[0082] Method 2: At the first-layer model input node of each of the subtask models, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models using the second input method of the first-layer input.

[0083] For example, taking the shared feature information corresponding to subtask model A and subtask model B as character feature vectors and word feature vectors as an example, according to the second input method mentioned above, the character feature vectors, word feature vectors and training text information can be directly input into subtask model A from the first-level model input node of subtask model A (such as the input node of the lowest-level neural network) without determining the specific model structure of subtask model A (i.e., the number of neural network layers in the model).

[0084] S103, during the independent training of the multiple subtask models, obtain the task training loss of each of the subtask models, and adjust the weight coefficient of the subtask model according to the gradient change of the task training loss of each of the subtask models, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and use the trained multiple subtask models as trained task processing models.

[0085] It should be noted that the model training process of each subtask model in the multi-task learning model framework is independent of each other. Therefore, the task training loss functions used by different subtask models can be the same or different. This embodiment of the present application does not impose any limitations on this.

[0086] In an embodiment of the present application, the overall loss function of the multiple subtask models is determined based on the product of the gradient of the task training loss of each subtask model and the weight coefficient of the subtask model in the multi-task learning model framework.

[0087] Here, considering that the subtasks to be executed by different subtask models may be different, and the subtask model with higher difficulty in completing the task has a more complex corresponding specific model structure, therefore, between subtask models with different model structures, the gradient change rate of the task training loss of the subtask model (equivalent to the gradient magnitude of the back propagation of the model training task) is difficult to balance (it is difficult to ensure model convergence at a similar or identical training speed), that is, it is difficult to keep the training rates of multiple subtask models synchronized (that is, the training rates are the same or the training rates are within the same numerical range) under the same multi-task learning model framework.

[0088] Based on this, in order to ensure that subtask models with different model structure complexity can be trained at a similar training speed (that is, the training rate is within the same numerical range) under the multi-task learning model framework, and improve the model learning adequacy of each subtask model, in an optional implementation, it is also possible to perform the following Figure 2 The dynamic adjustment method shown is used to dynamically adjust the weight coefficient of each subtask model in the multi-task learning model framework. Specifically:

[0089] Reference Figure 2 As shown, Figure 2 A flow chart of a method for dynamically adjusting the weight coefficient of each subtask model in a multi-task learning model framework provided by an embodiment of the present application is shown. The method includes steps S201-S203; specifically:

[0090] S201, for each of the subtask models, taking the gradient of the task training loss of the subtask model as the target gradient, within a gradient detection period, obtaining the periodic variation amplitude of the target gradient within the detection period.

[0091] Here, under the multi-task learning model framework, the gradient detection cycles corresponding to different subtask models are the same, that is, within the same gradient detection cycle, the gradient of the task training loss of each subtask model is gradient detected; the specific time length of the gradient detection cycle is not limited in any way in the embodiments of the present application.

[0092] S202, when it is detected that the periodic variation amplitude of the target gradient is greater than or equal to the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in a descending manner according to the gradient reduction adjustment coefficient.

[0093] Here, when it is detected that the periodic change amplitude of the target gradient is greater than or equal to the reference gradient change, it can be determined that within the current gradient detection cycle, the gradient of the task training loss of the subtask model has changed significantly (i.e., increased significantly). At this time, by dynamically adjusting the weight coefficient of the subtask model in a descending manner, the proportion of the task training loss of the subtask model in the overall loss of the multi-task learning model framework (or task processing model) (equivalent to the product of the gradient of the task training loss of the subtask model and the weight coefficient of the subtask model) can be maintained in dynamic balance, thereby ensuring that subtask models with different levels of model structure complexity can be trained at a similar training speed under the multi-task learning model framework, thereby improving the model learning adequacy of each subtask model.

[0094] S203, when it is detected that the periodic variation amplitude of the target gradient is smaller than the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in an increasing manner according to the gradient increasing adjustment coefficient.

[0095] Here, corresponding to the above-mentioned step S202, when it is detected that the periodic change amplitude of the target gradient is less than the reference gradient change amount, it can be determined that within the current gradient detection cycle, the gradient of the task training loss of the subtask model also changes significantly (i.e., decreases significantly). At this time, by dynamically adjusting the weight coefficient of the subtask model in an increasing manner, the proportion of the task training loss of the subtask model in the overall loss of the multi-task learning model framework (equivalent to the product of the gradient of the task training loss of the subtask model and the weight coefficient of the subtask model) can be maintained to achieve a dynamic balance, thereby ensuring that subtask models with different levels of model structure complexity can be trained at a similar training speed under the multi-task learning model framework, thereby improving the model learning adequacy of each subtask model.

[0096] In order to more clearly reflect the implementation details of the above steps S101-S103 in the embodiment of the present application, the following takes the "semantic sentiment analysis task" that appears more frequently in the text information processing process as an example of the overall learning task of the above multi-task learning model framework, and introduces the implementation details of the above steps S101-S103 in detail:

[0097] First of all, it should be explained that in the embodiment of the present application, each subtask model in the task processing model is used to execute the corresponding subtask, and different subtask models cooperate with each other to process the target task to be executed by the task processing model; the target task to be executed by the task processing model can be any learning task related to "text information processing", and is not limited to the above-mentioned "semantic sentiment analysis task". Here, only a relatively complex text information processing task (i.e., "semantic sentiment analysis" task) is selected as an example to more clearly highlight the specific implementation details of the embodiment of the present application in the process of "training multiple subtask models". The embodiment of the present application does not impose any restrictions on which type of "text information processing tasks" the target task to be executed by the task processing model belongs to.

[0098] With respect to the implementation process of the above-mentioned step S101, when the target task to be executed by the task processing model is related to identifying the semantic emotions expressed by the text information, the task processing model (i.e., the multiple subtask models) includes at least one named entity recognition model and one sentiment classification model; wherein the named entity recognition model is used to perform the text recognition task for the named entities included in the training text information; and the sentiment classification model is used to perform the sentiment classification task for the sentence emotions represented by each sentence in the training text information.

[0099] Based on this, in the embodiment of the present application, when the pre-training methods for the shared feature extraction model are different, according to different model training methods, in step S101, at least the following three different optional methods can be used to determine the specific information category of the shared feature information to be extracted by the shared feature extraction model, specifically:

[0100] Optional method (1) is to determine, based on the target task dependency relationship between the multiple subtasks to be executed by the multiple subtask models, multiple information categories corresponding to the target task dependency relationship from a preset task dependency table as the multiple categories of shared feature information to be extracted.

[0101] Here, the task dependency table pre-stores a plurality of information categories corresponding to a plurality of task dependencies. For example, the task dependency table pre-stores a plurality of information categories corresponding to task dependency a: information categories x1, x2 and x3.

[0102] Regarding the implementation of the above optional method (1), here, taking the multiple subtask models in the task processing model as: named entity recognition model M and sentiment classification model Q as an example, during the implementation of the above optional method (1), the named entity recognition model M needs to perform a text recognition task for the named entities included in the training text information; the sentiment classification model Q needs to perform a sentiment classification task for the sentence sentiment represented by each sentence in the training text information; at this time, the sentiment classification task performed by the sentiment classification model Q is to perform text processing with "sentence" as the minimum analysis unit, and the text recognition task performed by the named entity recognition model M is to perform text processing with the text recognition order of "character-word". Based on this, it can be determined that the target task dependency relationship between the named entity recognition model M and the sentiment classification model Q is: the subtask to be performed by the sentiment classification model Q (i.e., the text analysis task at the sentence level) depends on the text recognition result of the character / word of the named entity recognition model M (i.e., the text recognition task at the character / word level); from the task dependency table, determine the information category corresponding to the target task dependency relationship: character feature vector and word feature vector as the information category of the shared feature information to be extracted.

[0103] Here, in combination with the above analysis content, in the embodiment of the present application, as an optional embodiment, when the target task of the task processing model is related to identifying the semantic emotions expressed by text information, the multiple categories of shared feature information may at least include:

[0104] 1. Character-level shared feature information, that is, the word feature vector in the above step S101.

[0105] Here, the character-level shared feature information is used to represent the character arrangement feature information that can constitute the target word segmentation.

[0106] As an exemplary explanation, taking the use of adjacent word bigram (secondary grammar) vectors as an example, the training corpus can be cut into bigram word sequences. For example, the sentence "Beijing today the north wind blows hard and the blue sky dominates the screen" in the training text information will be cut into the sequence: "Beijing / Beijing today / Today / North Sky / North Wind / Blowing Hard / Blowing Blue / Blue Sky / Sky Dominates / Dominates the Screen" and then, using the word2vec (a group of related models used to generate word vectors) method for training, a 50-dimensional adjacent word bigram vector can be obtained.

[0107] It should be noted that, for character-level shared feature information, in addition to the adjacent word bigram vectors in the above example, tri-gram (i.e., every 3 characters are grouped together to segment the sentence) feature vector, 4-gram (i.e., every 4 characters are grouped together to segment the sentence) feature vector... n-Gram (i.e., every n characters are grouped together to segment the sentence) feature vector can also be used. The specific method of obtaining the above-mentioned character-level shared feature information is not limited in any way in the embodiments of the present application.

[0108] 2. Shared feature information at the word level, that is, the word feature vector in the above step S101.

[0109] Here, the word segmentation level shared feature information is used to represent the part-of-speech feature information of the subordinate relationship between different word segments in the same context.

[0110] Here, as an optional embodiment, StandFordNLP (a natural language processing tool) can be used to perform dependency syntactic analysis on the input training text information to obtain the syntactic dependency relationship between words in the training text information, thereby obtaining the above-mentioned word segmentation level shared feature information.

[0111] Here, as another optional embodiment, an adjacency matrix of a directed graph may also be used to store and obtain the syntactic dependency relationship between words.

[0112] For example, first, ignore the specific type of dependency between words (for example, whether it is a subject-predicate relationship or a verb-object relationship, it is considered to have a "subordinate relationship"). Then, according to the preset dependency indication direction (for example, i pointing to j in the matrix can indicate that i is a subordinate word of j), use the word segmentation results of each sentence as the rows and columns of the matrix to create an adjacency matrix. At this time, if the above-mentioned "subordinate relationship" exists between word i and word j, the corresponding adjacency matrix element aij has a value of 1; if the above-mentioned "subordinate relationship" does not exist between word i and word j, the corresponding adjacency matrix element aij has a value of 0. Finally, use each row in the adjacency matrix as the word feature vector corresponding to each column of words to obtain the above-mentioned word segmentation level shared feature information.

[0113] 3. Sentence-level high-dimensional shared feature information, that is, the sentence feature vector in the above step S101.

[0114] Here, the sentence-level high-dimensional shared feature information is used to represent high-dimensional feature vectors mapped under multiple dimensional features of different sentences in text information; the target word segmentation is used to represent word segmentations with semantic meanings.

[0115] As an exemplary explanation, as an optional embodiment, the BERT pre-trained language model can be used to perform text embedding processing on the training text information, and the sentence vector of each sentence is obtained as the above-mentioned sentence-level high-dimensional shared feature information.

[0116] Optional method (2) is to train the shared feature extraction model through NAS (neural network structure search). When the target task to be executed by the task processing model can be split, the information category of the shared feature information is determined and extracted by searching for the optimal sub-task model combination required to complete the target task.

[0117] Reference Figure 3 As shown, Figure 3 A schematic flow chart of a first neural network structure search method provided in an embodiment of the present application is shown, the method comprising steps S301-S303; specifically:

[0118] S301, according to the target task to be executed by the task processing model, taking the multiple subtask models included in the task processing model as the first search space, taking the ability to execute the target task as the first search strategy, performing a neural network structure search on the subtask model combination method between different subtask models in the first search space, and obtaining the optimal subtask model combination method that meets the first search strategy.

[0119] Here, the target task in the first search strategy can be used to characterize the highest-level learning task that the task processing model can perform, and can also be used to characterize the secondary learning task under the above-mentioned highest-level learning task.

[0120] For example, the highest-level learning task that the task processing model can perform is the above-mentioned "semantic sentiment analysis" task (named entity recognition task + sentence sentiment classification task). In step S301, the target task in the first search strategy can be the "semantic sentiment analysis" task, or it can be a secondary learning task under the above-mentioned highest-level learning task: named entity recognition task or sentence sentiment classification task.

[0121] Here, the definition of "optimal" in the optimal subtask model combination method can be determined according to the user's actual model training needs. For example, when it is detected that the user's actual model training needs tend to increase the model training speed, it can be determined that the optimal subtask model combination method is a model combination method that complies with the above-mentioned first search strategy and has the fastest overall training speed for multiple subtask models; when it is detected that the user's actual model training needs tend to increase the model training accuracy, it can be determined that the optimal subtask model combination method is a model combination method that complies with the above-mentioned first search strategy and has the most accurate overall training results for multiple subtask models.

[0122] S302: taking each subtask model included in the optimal subtask model combination as a first subtask model.

[0123] S303: According to the first task dependency relationship between the subtasks to be executed in each of the first subtask models, determine from a preset task dependency relationship table a plurality of information categories corresponding to the first task dependency relationship as the plurality of categories of the shared feature information to be extracted.

[0124] Here, after determining the above-mentioned optimal subtask model combination method, the specific implementation process of steps S302-S303 is the same as the above-mentioned optional method (1), and the repeated parts are not repeated here.

[0125] Here, with regard to the implementation of the above steps S301-S303, it is also necessary to explain that, in the specific model structure of each subtask model, each subtask model can be regarded as a combination of multiple neural networks of different types / levels. For a subtask model, the structural combination of different neural networks can complete the subtask to be executed by the subtask model (i.e., the model training task of the subtask model) as the above-mentioned first search strategy, and the optimal model structure of each subtask model can also be determined. The repetitions will not be repeated here.

[0126] Optional method (3) is to train the shared feature extraction model through NAS (neural network structure search) to determine and extract the information category of the shared feature information when the framework structure of the multi-task learning model framework is fixed (that is, the target task to be executed by the task processing model cannot be split and executed).

[0127] Reference Figure 4 As shown, Figure 4 A schematic flow chart of a second neural network structure search method provided in an embodiment of the present application is shown, the method comprising steps S401-S403; specifically:

[0128] S401, according to the target task to be executed by the task processing model, obtaining a variety of text feature information related to completing the target task.

[0129] As an exemplary explanation, still taking the target task to be executed by the task processing model as the above-mentioned "semantic sentiment analysis" task (named entity recognition task + sentence sentiment classification task) as an example, without splitting the target task, "character-level shared feature information a, b, c" (equivalent to the text feature information whose information category is word feature vector), "word-level shared feature information d, e, f" (equivalent to the text feature information whose information category is word feature vector), and "sentence-level shared feature information g, h" (equivalent to the text feature information whose information category is sentence feature vector) can be obtained as a variety of text feature information related to completing the target task.

[0130] S402, taking the multiple text feature information as the second search space, taking the ability of the multiple subtask models to complete the target task based on the information combination of different text feature information as the second search strategy, performing a neural network structure search on the information combination method between different text feature information in the second search space, and obtaining the optimal information combination method that meets the second search strategy.

[0131] Here, similar to the above-mentioned step S301, the definition of "optimal" in the optimal information combination method can also be determined based on the user's actual model training needs. For example, when it is detected that the user's actual model training needs tend to increase the model training speed, it can be determined that the optimal information combination method is an information combination method that complies with the above-mentioned second search strategy and has the least information categories of shared feature information that need to be extracted; when it is detected that the user's actual model training needs tend to improve the model training accuracy, it can be determined that the optimal information combination method is an information combination method that complies with the above-mentioned second search strategy and has the most accurate overall training results for multiple subtask models.

[0132] S403: taking the information category to which each type of text feature information included in the optimal information combination mode belongs as the multiple categories of the shared feature information to be extracted.

[0133] For example, still taking the example in S401 above as an example, if it is determined that the text feature information included in the optimal information combination method is: character-level shared feature information a and word-level shared feature information d, then it can be determined that the multiple categories of shared feature information to be extracted are: character feature vectors and word-level feature vectors.

[0134] In the implementation process of the above step S102, when the shared feature information part is input according to the first input method of hierarchical input, in addition to the method 1 in the above step S102, refer to Figure 5As shown, Figure 5 A flow chart of a method for inputting shared feature information in a first input mode provided in an embodiment of the present application is shown. The method includes steps S501-S503; specifically:

[0135] S501 , for each of the subtask models, at a first training node of the subtask model, inputting first shared feature information whose information category is the word feature vector into the subtask model.

[0136] Here, the first training node is used to represent the input node of the shallow neural network in the subtask model.

[0137] S502: At a second training node of the subtask model, second shared feature information whose information category is the word feature vector is input into the subtask model.

[0138] Here, the second training node is used to represent the input node of the intermediate layer neural network in the subtask model.

[0139] S503: At the third training node of the subtask model, third shared feature information whose information category is the sentence feature vector is input into the subtask model.

[0140] Here, the third training node is used to represent the input node of the deep neural network in the subtask model.

[0141] Regarding the implementation of the above steps S501-S503, it should be noted that when the number of layers of the neural network in the subtask model is less than 3 layers, taking the model structure of a 2-layer neural network as an example, the "character-level shared feature information" (i.e., the first shared feature information whose information category is the character feature vector) and the "word-level shared feature information" (i.e., the second shared feature information whose information category is the word feature vector) can be input into the first layer (i.e., the shallowest layer) of the neural network, and the "sentence-level shared feature information" (i.e., the third shared feature information whose information category is the sentence feature vector) can be input into the second layer (i.e., the deepest layer) of the neural network;

[0142] In addition, the "character-level shared feature information" can be input into the first layer of the neural network, and the "word segmentation-level shared feature information" and the "sentence-level shared feature information" can be input into the second layer of the neural network. The embodiment of the present application does not impose any limitation on the specific number of layers of the neural network in the subtask model.

[0143] For the implementation process of the above step S102, when the shared feature information part is input according to the second input method of the first-level input, for the specific implementation of the method 2 in the above step S102, it can also be divided into the following two optional implementation plans according to whether the task type differences of the subtasks to be executed by different subtask models are distinguished, specifically:

[0144] Optional implementation scheme 1: when the task types of the subtasks to be executed by different subtask models are not differentiated, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models using the second input method.

[0145] Optional implementation scheme 2 is divided into step a and step b, specifically:

[0146] Step a: when distinguishing the task types of subtasks to be executed by different subtask models, for each subtask model, according to the subtask to be executed by the subtask model, determine the target shared feature information that matches the subtask to be executed by the subtask model in the multiple categories of shared feature information.

[0147] Step b: using the second input method, synchronously inputting the multiple categories of shared feature information, the training text information and the target shared feature information into the subtask model.

[0148] The model training method of the above-mentioned task processing model provided in the embodiment of the present application,

[0149] First, a training corpus is obtained, and the training corpus is input into a shared feature extraction model, and multiple categories of shared feature information in the training corpus are extracted through the shared feature extraction model; then, the multiple categories of shared feature information and training text information based on the annotations of the training corpus are input into multiple subtask models according to a preset input method, and the multiple subtask models are trained in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition; in the process of independent training of multiple subtask models, the task training loss of each subtask model is obtained, and the weight coefficient of the subtask model is adjusted according to the gradient change of the task training loss of each subtask model so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as trained task processing models.

[0150] In this way, the present application can construct a multi-task learning model framework to ensure that each subtask model can be trained independently while providing different subtask models with a variety of shared feature information related to the subtasks they perform, which is conducive to improving the overall model training effect of the task processing model.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a model training device corresponding to the model training method of the task processing model in the above embodiment. Since the principle of solving the problem by the model training device in the embodiment of the present application is similar to the model training method in the above embodiment of the present application, the implementation of the model training device can refer to the implementation of the aforementioned model training method, and the repeated parts will not be repeated.

[0152] Reference Figure 6 As shown, Figure 6 A schematic diagram of the structure of a model training device for a task processing model provided in an embodiment of the present application is shown; wherein the model training device is applied to a multi-task learning model framework, the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, the task processing model includes a plurality of subtask models; the model training device includes:

[0153] An extraction module 601 is used to obtain training corpus, and input the training corpus into the shared feature extraction model, and extract multiple categories of shared feature information in the training corpus through the shared feature extraction model;

[0154] An input module 602 is used to input the multiple categories of shared feature information and the training text information annotated based on the training corpus into the multiple subtask models in accordance with a preset input method, and train the multiple subtask models in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition;

[0155] The training module 603 is used to obtain the task training loss of each subtask model during the independent training of the multiple subtask models, and adjust the weight coefficient of the subtask model according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as trained task processing models.

[0156] In an optional embodiment, the multiple categories of shared feature information include: character feature vectors after the training corpus is segmented into character sequences; word feature vectors representing syntactic dependencies between words in the training corpus; and sentence feature vectors in the training corpus.

[0157] In an optional implementation, the extraction module 601 is used to determine multiple categories of shared feature information to be extracted by the shared feature extraction model by the following method:

[0158] According to the target task dependency between the multiple subtasks to be executed by the multiple subtask models, multiple information categories corresponding to the target task dependency are determined from a preset task dependency table as the multiple categories of shared feature information to be extracted; wherein the task dependency table pre-stores multiple information categories corresponding to the multiple task dependencies.

[0159] In an optional implementation, the extraction module 601 is used to determine multiple categories of shared feature information to be extracted by the shared feature extraction model by the following method:

[0160] According to the target task to be executed by the task processing model, taking the multiple subtask models included in the task processing model as the first search space, taking the ability to execute the target task as the first search strategy, performing a neural network structure search on the subtask model combination mode between different subtask models in the first search space, and obtaining the optimal subtask model combination mode that meets the first search strategy;

[0161] Taking each subtask model included in the optimal subtask model combination as the first subtask model;

[0162] According to the first task dependency relationship between the subtasks to be executed in each of the first subtask models, multiple information categories corresponding to the first task dependency relationship are determined from a preset task dependency relationship table as the multiple categories of the shared feature information to be extracted.

[0163] In an optional implementation, the extraction module 601 is used to determine multiple categories of shared feature information to be extracted by the shared feature extraction model by the following method:

[0164] According to the target task to be performed by the task processing model, obtaining a variety of text feature information related to completing the target task;

[0165] Taking the multiple types of text feature information as the second search space, taking the ability of the multiple subtask models to complete the target task based on the information combination of different text feature information as the second search strategy, performing a neural network structure search on the information combination mode between different text feature information in the second search space, and obtaining the optimal information combination mode that meets the second search strategy;

[0166] The information category to which each type of text feature information included in the optimal information combination mode belongs is used as the multiple categories of the shared feature information to be extracted.

[0167] In an optional implementation, when the multiple categories of shared feature information and the training text information annotated based on the training corpus are input into the multiple subtask models according to the preset input method, the input module 602 is specifically used to:

[0168] At the first-layer model input node of each of the subtask models, input the training text information into each of the subtask models;

[0169] The multiple categories of shared feature information are hierarchically input to different training nodes in each of the subtask models according to the correspondence between the information categories and the training nodes in a first input method of hierarchical input; wherein the different training nodes in each of the subtask models are sorted from shallow to deep levels of the neural network in the subtask model.

[0170] In an optional implementation, when the multiple categories of shared feature information and the training text information annotated based on the training corpus are input into the multiple subtask models according to the preset input method, the input module 602 is further used to:

[0171] At the first-layer model input node of each of the subtask models, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models in the second input mode of the first-layer input.

[0172] In an optional implementation, when the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models in the second input mode of the first-level input, the input module 602 is specifically used to:

[0173] When the task types of the subtasks to be executed by different subtask models are not differentiated, the multiple categories of shared feature information and the training text information are synchronously input into each of the subtask models in the second input mode;

[0174] or,

[0175] When distinguishing the task types to which the subtasks to be executed of different subtask models belong, for each of the subtask models, according to the subtask to be executed by the subtask model, determining the target shared feature information that matches the subtask to be executed by the subtask model among the multiple categories of shared feature information;

[0176] In the second input mode, the multiple categories of shared feature information, the training text information and the target shared feature information are synchronously input into the subtask model.

[0177] In an optional implementation, the overall loss function of the multiple subtask models is determined according to the product of the gradient of the task training loss of each subtask model and the weight coefficient of the subtask model in the multi-task learning model framework; when adjusting the weight coefficient of the subtask model according to the gradient change of the task training loss of each subtask model, the training module 603 is specifically used to:

[0178] For each of the subtask models, taking the gradient of the task training loss of the subtask model as the target gradient, within a gradient detection period, obtaining the periodic variation amplitude of the target gradient within the detection period;

[0179] When it is detected that the periodic variation amplitude of the target gradient is greater than or equal to the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in a descending manner according to the gradient reduction adjustment coefficient;

[0180] When it is detected that the periodic variation amplitude of the target gradient is smaller than the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in an increasing manner according to the gradient increasing adjustment coefficient.

[0181] In an optional embodiment, each subtask model in the task processing model is used to execute a corresponding subtask, and different subtask models cooperate with each other to process the target task to be executed by the task processing model; when the target task is related to identifying the semantic emotions expressed by text information, the task processing model includes at least one named entity recognition model and one sentiment classification model; wherein the named entity recognition model is used to execute the text recognition task for the named entities included in the training text information; and the sentiment classification model is used to execute the sentiment classification task for the sentence emotions represented by each sentence in the training text information.

[0182] like Figure 7 As shown, an embodiment of the present application provides a computer device 700 for executing the model training method of the task processing model in the present application. The device includes a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702, wherein the processor 702 implements the steps of the model training method of the task processing model when executing the computer program.

[0183] Specifically, the above-mentioned memory 701 and processor 702 can be general-purpose memory and processor, which are not specifically limited here. When the processor 702 runs the computer program stored in the memory 701, it can execute the model training method of the above-mentioned task processing model.

[0184] Corresponding to the model training method of the task processing model in the present application, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method of the above-mentioned task processing model are executed.

[0185] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the model training method of the above-mentioned task processing model.

[0186] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0189] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0190] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0191] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A model training method for a task processing model, It is characterized in that Applied to a multi-task learning model framework, the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, the task processing model includes multiple sub-task models; the model training method includes: Inputting the training corpus into the shared feature extraction model, and extracting multiple categories of shared feature information in the training corpus through the shared feature extraction model; When the task types of the subtasks to be executed by different subtask models are not distinguished, at the first-level model input node of each of the subtask models, the shared feature information of the multiple categories and the training text information annotated based on the training corpus are synchronously input into each of the subtask models using the second input method of the first-level input, and the multiple subtask models are trained in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition.

2. The model training method according to claim 1, It is characterized in that The model training method also includes: During the independent training of the multiple subtask models, the task training loss of each subtask model is obtained, and the multi-task learning model framework adjusts the weight coefficient of the subtask model of the multi-task learning model framework according to the gradient change of the task training loss of each subtask model, so that the training rates of the multiple subtask models are within the same numerical range, until the overall loss function of the multiple subtask models meets the training cutoff condition, and the trained multiple subtask models are used as trained task processing models.

3. The model training method according to claim 1, It is characterized in that The multiple categories of shared feature information include: character feature vectors after the training corpus is segmented into character sequences; word feature vectors representing syntactic dependency relationships between words in the training corpus; and sentence feature vectors in the training corpus.

4. The model training method according to claim 1, It is characterized in that The multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method: According to the target task dependency between the multiple subtasks to be executed by the multiple subtask models, multiple information categories corresponding to the target task dependency are determined from a preset task dependency table as the multiple categories of shared feature information to be extracted; wherein the task dependency table pre-stores multiple information categories corresponding to the multiple task dependencies.

5. The model training method according to claim 1, It is characterized in that The multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method: According to the target task to be executed by the task processing model, taking the multiple subtask models included in the task processing model as the first search space, taking the ability to execute the target task as the first search strategy, performing a neural network structure search on the subtask model combination mode between different subtask models in the first search space, and obtaining the optimal subtask model combination mode that meets the first search strategy; Taking each subtask model included in the optimal subtask model combination as the first subtask model; According to the first task dependency relationship between the subtasks to be executed in each of the first subtask models, multiple information categories corresponding to the first task dependency relationship are determined from a preset task dependency relationship table as the multiple categories of the shared feature information to be extracted.

6. The model training method according to claim 1, It is characterized in that The multiple categories of shared feature information to be extracted by the shared feature extraction model are determined by the following method: According to the target task to be performed by the task processing model, obtaining a variety of text feature information related to completing the target task; Taking the multiple types of text feature information as the second search space, taking the ability of the multiple subtask models to complete the target task based on the information combination of different text feature information as the second search strategy, performing a neural network structure search on the information combination mode between different text feature information in the second search space, and obtaining the optimal information combination mode that meets the second search strategy; The information category to which each type of text feature information included in the optimal information combination mode belongs is used as the multiple categories of the shared feature information to be extracted.

7. The model training method according to claim 1, It is characterized in that The method of inputting the multiple categories of shared feature information and the training text information into the multiple subtask models further includes: At the first-layer model input node of each of the subtask models, input the training text information into each of the subtask models; The multiple categories of shared feature information are hierarchically input to different training nodes in each of the subtask models according to the correspondence between the information categories and the training nodes in a first input method of hierarchical input; wherein the different training nodes in each of the subtask models are sorted from shallow to deep levels of the neural network in the subtask model.

8. The model training method according to claim 1, It is characterized in that The method of inputting the multiple categories of shared feature information and the training text information into the multiple subtask models further includes: When distinguishing the task types to which the subtasks to be executed of different subtask models belong, for each of the subtask models, according to the subtask to be executed by the subtask model, determining the target shared feature information that matches the subtask to be executed by the subtask model among the multiple categories of shared feature information; In the second input mode, the multiple categories of shared feature information, the training text information and the target shared feature information are synchronously input into the subtask model.

9. The model training method according to claim 2, It is characterized in that The overall loss function of the multiple subtask models is determined according to the product of the gradient of the task training loss of each subtask model and the weight coefficient of the subtask model in the multi-task learning model framework; The step of adjusting the weight coefficient of each subtask model according to the gradient change of the task training loss of each subtask model includes: For each of the subtask models, taking the gradient of the task training loss of the subtask model as the target gradient, within a gradient detection period, obtaining the periodic variation amplitude of the target gradient within the detection period; When it is detected that the periodic variation amplitude of the target gradient is greater than or equal to the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in a descending manner according to the gradient reduction adjustment coefficient; When it is detected that the periodic variation amplitude of the target gradient is smaller than the reference gradient variation, the weight coefficient of the subtask model is dynamically adjusted in an increasing manner according to the gradient increasing adjustment coefficient.

10. The model training method according to claim 1, It is characterized in that Each subtask model in the task processing model is used to execute a corresponding subtask, and different subtask models cooperate with each other to process the target task to be executed by the task processing model; when the target task is related to identifying the semantic emotions expressed by text information, the task processing model includes at least one named entity recognition model and one sentiment classification model; wherein the named entity recognition model is used to execute the text recognition task for the named entities included in the training text information; and the sentiment classification model is used to execute the sentiment classification task for the sentence emotions represented by each sentence in the training text information.

11. A model training device for a task processing model, It is characterized in that Applied to a multi-task learning model framework, the multi-task learning model framework includes a task processing model and a pre-trained shared feature extraction model, the task processing model includes multiple sub-task models; the model training device includes: An extraction module, used for inputting the training corpus into the shared feature extraction model, and extracting multiple categories of shared feature information in the training corpus through the shared feature extraction model; An input module is used to synchronously input the shared feature information of multiple categories and the training text information annotated based on the training corpus into each of the subtask models at the first-level model input node of each of the subtask models without distinguishing the task types of the subtasks to be executed by different subtask models, and train the multiple subtask models in parallel so that the overall loss function of the multiple subtask models meets the training cutoff condition.

12. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the model training method as described in any one of claims 1 to 10 are performed.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the model training method as described in any one of claims 1 to 10.

Images