Task processing method, natural language processing method, device and program product

Abstract

The invention provides a task processing method, a natural language processing method, equipment and a program product. According to the method provided by the invention, the key layer which has higher importance on domain knowledge injection (namely, plays a key role on domain knowledge injection) in the pre-training model is determined, the extension layer corresponding to at least one key layer is inserted into the pre-training model, and the domain knowledge is injected into the extension layers which are more critical for domain knowledge injection; the task processing capability of the model in the target domain is improved, the general capability of the pre-training model is well reserved, the general capability of the model and the task processing capability in the target domain can be effectively balanced, and more effective domain knowledge injection is realized, so that the performance of the model in processing the tasks in the target domain can be improved, and the user experience is improved. And the quality of the task processing result of the target field is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a task processing method, a natural language processing method, equipment and a program product. BACKGROUND

[0002] In recent years, a large language model (LLM) and other deep learning models have shown strong general capabilities in various task scenarios in the field of natural language processing (NLP). However, the LLM performs poorly in various tasks in specific fields such as mathematics, programming and law.

[0003] Through the method of knowledge injection, the performance of the model in the tasks of the specific field can be enhanced. The traditional comprehensive fine-tuning method of training all parameters of the model using field data requires a large amount of computing resources and a large amount of labeled data, and is prone to cause the model to overfit, causing the problem of catastrophic forgetting of the general capability of the model.

[0004] At present, the mainstream knowledge injection adopts the method of parameter-efficient fine-tuning (PEFT), which trains part of the parameters of the model using field data. Although the strong general capability of the model is retained, the effectiveness of the knowledge injection is poor, resulting in poor performance of the model in processing specific field tasks. SUMMARY

[0005] The present application provides a task processing method, a natural language processing method, equipment and a program product to solve the problem of poor effectiveness of parameter-efficient fine-tuning method knowledge injection, resulting in poor performance of the model in processing specific field tasks.

[0006] In a first aspect, the present application provides a task processing method, comprising:

[0007] In response to a task processing request of a target field, input data of the target field is obtained;

[0008] The input data of the target field is input into a field model for task processing to obtain a task processing result, wherein the field model is obtained by inserting at least one extension layer corresponding to a key layer in a pre-trained model and injecting field knowledge of the target field into the extension layer, and the key layer is determined according to the importance of each layer in the pre-trained model to the injection of the field knowledge;

[0009] The task processing result is output.

[0010] In a second aspect, the present application provides a natural language processing method, comprising:

[0011] receiving a calling request for a domain model of a target domain, obtaining input data of the target domain;

[0012] inputting the input data of the target domain into the domain model for natural language processing, and obtaining a natural language processing result, wherein the domain model is obtained by inserting at least one extension layer corresponding to a key layer into a pre-training model, and injecting domain knowledge of the target domain into the extension layer, and the key layer is determined according to importance of each layer in the pre-training model to the injection of the domain knowledge;

[0013] returning the natural language processing result.

[0014] In a third aspect, the present application provides a server, comprising:

[0015] at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the server to perform the method provided in any one of the preceding aspects.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method provided in any one of the preceding aspects is implemented.

[0017] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the method provided in any one of the preceding aspects is implemented.

[0018] The task processing method, the natural language processing method, the device and the program product provided by the present application determine a key layer in a pre-training model which is more important (i.e., plays a key role) to the injection of domain knowledge, insert at least one extension layer corresponding to the key layer into the pre-training model, inject domain knowledge into the extension layers which are more important to the injection of domain knowledge, not only improve the task processing capability of the model in the target domain, but also better retain the general capability of the pre-training model, can effectively balance the general capability of the model and the task processing capability in the target domain, realize more effective injection of domain knowledge, and thus can improve the performance of the model in processing tasks in the target domain, and further improve the quality of the task processing result in the target domain. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0020] Figure 1 A schematic diagram of an example system architecture for use with the present application;

[0021] Figure 2 A flowchart of a task processing method for an example embodiment of the present application;

[0022] Figure 3 A flowchart of constructing a domain model of a target domain for an example embodiment of the present application;

[0023] Figure 4 An example diagram of an extended model for an example embodiment of the present application;

[0024] Figure 5 A flowchart of a natural language processing method for an example embodiment of the present application;

[0025] Figure 6 A structural schematic diagram of a server according to an embodiment of the present application.

[0026] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0027] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers refer to the same or similar elements unless otherwise represented. The embodiments described in the following example embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0028] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] First, the terms involved in the present application are explained:

[0030] Large language model: a deep neural network model with a large number of parameters trained using a large amount of text, which can understand and generate natural language text, also known as large-scale language model, large language model.

[0031] Pre-trained large model: a pre-trained model obtained by pre-training a large model such as a large language model (LLM).

[0032] Continued pre-training: continuing pre-training of a pre-trained model in a general field on unlabeled corpus in a specific field using the same or similar pre-training task.

[0033] Vertical field: an Internet industry term referring to a divided field providing specific services for a limited group, including mathematics, programming, technology, law, e-commerce, entertainment, medicine, environmental protection, and other fields. A vertical field is a small field subdivided from a large field.

[0034] General field: in this application, the general field is a concept opposite to the specific field. The general field contains multiple fields and can cover multiple vertical fields. It can be understood as a relatively general or common field with a large amount of existing corpus data, and can generally refer to a field covered by a large amount of corpus data that can be collected.

[0035] Multi-task instruction fine-tuning: fine-tuning training based on multi-instruction data sets, a method for fine-tuning pre-trained models, which can correctly respond to human task instructions and better solve various task instruction-driven tasks.

[0036] Visual question answering task: determining the answer to a question from the visual information of an input image based on the input image and the question.

[0037] Image description task: generating a description text of an input image.

[0038] Visual implication task: predicting the semantic relevance of input images and text, i.e., implication, neutrality, or contradiction.

[0039] Point representation and understanding task: locating the image area corresponding to the input text in the input image based on the input text.

[0040] Image generation task: generating an image based on an input description text.

[0041] Text-based sentiment classification task: predicting sentiment classification information of input text.

[0042] Text summarization task: generating summary information of input text.

[0043] Multi-modal task: a downstream task involving multiple modal data such as images and text in input and output data, such as visual question answering task, image description task, visual implication task, point representation and understanding task, image generation task, etc.

[0044] Multi-modal pre-training model: refers to a pre-training model involving multiple modal data such as images and texts. After fine-tuning, it can be applied to multi-modal task processing.

[0045] Large model refers to a deep learning model with large-scale model parameters, usually containing hundreds of millions, billions, or even tens of billions of model parameters. Large model can also be called foundation model (FM). Through large-scale unlabeled corpus pre-training, a pre-training model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large-scale language models, multi-modal pre-training models, etc.

[0046] In practical applications, large models only need a small amount of samples to fine-tune the pre-training model and can be applied to different tasks. Large models can be widely used in NLP, computer vision, etc. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0047] Existing pre-training models (such as pre-trained language models, LLMs, and machine learning models) have strong general-purpose capabilities and can adapt to a wide range of downstream tasks. However, pre-training models perform poorly on tasks in specific domains such as mathematics, programming, and law. By injecting knowledge, the performance of the model on specific domain tasks can be enhanced. Traditional comprehensive fine-tuning methods require a large amount of computing resources and a large amount of labeled data, and are prone to overfitting, causing the model to suffer from catastrophic forgetting of general-purpose capabilities. While the PEFT method retains the model's strong general-purpose capabilities, the effectiveness of knowledge injection is poor, resulting in poor performance of the model in processing specific domain tasks.

[0048] The application provides a domain model of a target field, which is obtained by inserting at least one extension layer corresponding to a key layer in a pre-trained model and injecting domain knowledge of the target field into the extension layer. By locating the key layer that is more important for injecting domain knowledge (i.e., plays a key role in injecting domain knowledge) in the pre-trained model, inserting at least one extension layer corresponding to the key layer, and injecting domain knowledge into the extension layer that is more important for injecting domain knowledge, the model's task processing capability in the target field is improved, and the powerful general capability of the pre-trained model is well preserved, which can effectively balance the general capability of the model and the task processing capability in the target field, achieve more effective knowledge injection, and thus improve the performance of the model in processing tasks in a specific field and improve the quality of task processing in the specific field. The pre-trained model can be a pre-trained language model, LLM, large model, machine learning model, etc., which is not limited here. The key layer refers to a layer in the pre-trained model that plays a key role in injecting domain knowledge, which can be determined according to the importance of each layer in the pre-trained model for injecting domain knowledge.

[0049] Based on the above-mentioned domain model of the target field, the application provides a task processing method, which responds to a task processing request of a target field, obtains input target field input data, inputs the target field input data into the domain model for task processing, and obtains a task processing result. The domain model used for processing tasks in the target field is obtained by inserting at least one extension layer corresponding to a key layer in a pre-trained model and injecting domain knowledge of the target field into the extension layer. This model effectively injects domain knowledge of the target field and retains strong general capability, which can improve the quality of task processing in the feature field.

[0050] The scheme of the application can obtain a domain model of any field and process tasks in the corresponding field based on the domain model to improve the performance of the domain model in processing tasks in the corresponding field.

[0051] Figure 1 A schematic diagram of an example system architecture suitable for the application. As shown in Figure 1 The system architecture includes a server and an end-side device. The server and the end-side device have a communicable communication link between them, enabling communication connection between the server and the end-side device.

[0052] The server is a device with computing capability deployed in the cloud or locally, such as a cloud cluster. The server stores a domain model of a target field. The domain model is obtained by inserting at least one extension layer corresponding to a key layer in a pre-trained model and injecting domain knowledge of the target field into the extension layer.

[0053] The end-side device can be an electronic device running a downstream task system, and specifically can be a hardware device with network communication function, operation function and information display function, including but not limited to a smart phone, a tablet computer, a desktop computer, a local server, a cloud server, etc. In the process of running the downstream task system, the end-side device needs to use the domain model to process the input data of the target domain to obtain the task processing result. For example, the downstream task system running on the end-side device can realize the functions of intelligent question answering, information extraction, text classification, text summarization, etc. in the target domain, and at least one function of the downstream task system needs to use the domain model of the target domain to perform task processing.

[0054] Specifically, in the process of running the downstream task system, when the task of the target domain needs to be executed, the end-side device acquires the input data of the target domain, sends a task processing request of the target domain to the server, and the task processing request contains the input data of the target domain.

[0055] In response to receiving the task processing request of the target domain sent by the end-side device, the server acquires the input data of the target domain; inputs the input data of the target domain into the domain model for task processing to obtain a task processing result. The server returns the task processing result to the end-side device.

[0056] The end-side device receives the task processing result returned by the server, and continues to execute the processing logic of the downstream task system according to the task processing result to realize the function of the downstream task system.

[0057] In addition, the server can also be responsible for inserting at least one extension layer corresponding to the key layer in the pre-trained model, and injecting the domain knowledge of the target domain into the extension layer to obtain the domain model of the target domain.

[0058] In an example scenario, in order to obtain a domain model with understanding of the domain knowledge of a specific target domain, the end-side device collects the domain data of the target domain applied by the task system, which can be the public knowledge data of the target domain collected by the end-side device, or the internal knowledge data of the enterprise to which the end-side device belongs. The end-side device provides the domain data of the target domain to the server. In addition, the user can also specify the pre-trained model used through the end-side device. The server inserts at least one extension layer corresponding to the key layer in the pre-trained model, and injects the domain knowledge of the target domain into the extension layer to obtain the domain model of the target domain.

[0059] Further, the server can provide an application programming interface (API) of the domain model of the target domain to the outside. The end-side device calls the domain model of the target domain through the API of the domain model of the target domain, so that the domain model of the target domain implements the task processing of the target domain and obtains the task processing result.

[0060] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0061] Figure 2 A task processing method flowchart is provided for an exemplary embodiment of the present application. The execution subject of the present embodiment is the server in the system architecture described above. As shown in the figure, the specific steps of the method are as follows: Figure 2

[0062] Step S201, in response to a task processing request of a target domain, obtaining input data of the target domain.

[0063] The target domain refers to a specific domain / vertical domain to which the task to be processed belongs according to the actual application scenario requirements. For example, the domains of mathematics, programming, law, e-commerce, entertainment, and medicine. The general domain is a concept opposite to the target domain. The general domain can be understood as a relatively general or common domain with a large amount of existing corpus data. The general domain usually includes multiple domains and covers multiple specific / vertical domains. In addition, the target domain can cover one or more specific / vertical domains. For example, the target domain can be the legal domain, or the target domain can include the two domains of mathematics and programming.

[0064] In actual applications, the end-side device running the downstream task system will generate various tasks, such as intelligent question answering, legal document processing, information extraction, text classification, text generation, and other natural language processing tasks. When a task of a target domain needs to be executed, the end-side device obtains input data of the target domain, which is data of the target domain; the end-side device sends a task processing request of the target domain to the server, and the task processing request includes the input data of the target domain.

[0065] The server receives the task processing request of the target domain sent by the end-side device, and in response to receiving the task processing request of the target domain, extracts the input data of the target domain from the task processing request.

[0066] ​In step S202, input data of the target field is input into the field model for task processing to obtain a task processing result. The field model is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model and injecting field knowledge of the target field into the extension layer, where the key layer is determined according to the importance of each layer of the pre-trained model to the injection of field knowledge.

[0067] In this step, after obtaining the input data of the target field, the server inputs the input data of the target field into the field model of the target field for task processing to obtain a task processing result.

[0068] In this embodiment, the field model of the target field is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model and injecting field knowledge of the target field into the extension layer. The pre-trained model can be a pre-trained language model, LLM, large model, machine learning model, etc., which is not limited here.

[0069] Specifically, by locating the key layer with high importance (i.e., playing a key role in the injection of field knowledge) in the pre-trained model, at least one extension layer corresponding to the key layer is inserted, and the field knowledge is injected into the extension layer which is more critical for the injection of field knowledge. Not only improves the model's task processing ability in the target field, but also better preserves the general ability of the pre-trained model, effectively balances the general ability of the model and the task processing ability in the target field, and realizes more effective knowledge injection, thereby improving the performance of the model in processing specific field tasks.

[0070] In step S203, the task processing result is output.

[0071] After obtaining the task processing result, the server returns the task processing result to the end-side device.

[0072] For example, according to the requirements of the downstream task system, the server can return the task processing result to the downstream task system running on the end-side device and execute the subsequent processing logic of the downstream task system.

[0073] For example, the server can also output the task processing result through the front-end visual interface.

[0074] The method of the present application, the domain model for performing task processing in the target domain, is obtained by inserting at least one extension layer corresponding to a key layer in the pre-trained model and injecting domain knowledge of the target domain into the extension layer; based on the key layer in the pre-trained model which is more important for injecting domain knowledge (i.e., plays a key role in injecting domain knowledge), at least one extension layer corresponding to the key layer is inserted in the pre-trained model, and domain knowledge is injected into these extension layers which are more critical for injecting domain knowledge, not only improving the model's task processing ability in the target domain, but also better preserving the general ability of the pre-trained model, which can effectively balance the general ability of the model and the task processing ability in the target domain, achieve more effective domain knowledge injection, and thus improve the performance of the model in processing tasks in the target domain and improve the quality of the task processing result in the target domain.

[0075] Figure 3 A flowchart for constructing a domain model of a target domain is provided for an exemplary embodiment of the present application. In this embodiment, the process of obtaining the domain model of the target domain by inserting at least one extension layer corresponding to a key layer in the pre-trained model and injecting domain knowledge of the target domain into the extension layer in the foregoing step S202 is described in detail.

[0076] As shown in Figure 3 , the process of constructing a domain model of a target domain includes:

[0077] Step S301, determining the key layer in the pre-trained model.

[0078] Among them, the key layer refers to the layer in the pre-trained model which plays a key role in injecting domain knowledge, which can be determined according to the importance of each layer in the pre-trained model for injecting domain knowledge.

[0079] In this step, based on the domain data of the target domain, the importance indicators of each layer in the pre-trained model for injecting domain knowledge are evaluated; according to the importance indicators of each layer in the pre-trained model, the layer (Block) with higher importance for injecting domain knowledge is selected as the key layer of the pre-trained model.

[0080] In the first optional implementation, based on the domain data of the target domain, the importance indicators of each layer in the pre-trained model for injecting domain knowledge can be realized in the following way:

[0081] The domain data is input into the pre-trained model, and the first importance indicator of each layer is calculated according to the distance between the input features and the output features of each layer in the pre-trained model, wherein the larger the first importance indicator, the higher the importance of the corresponding layer for injecting domain knowledge.

[0082] Specifically, the field data of the target field is input into the pre-training model for processing, and the input features and output features of each layer (Block) in the pre-training model are obtained. Each layer (Block) in the pre-training model refers to each layer (Block) in the backbone network of the pre-training network. The pre-training model can also include an embedding module (i.e., Embedding) and a task head. After the field data is input into the pre-training model, the input field data is mapped into embedding features through the embedding module (Embedding), and the embedding features are input into a plurality of layers (Blocks) stacked in sequence, and the input features are processed in sequence by each layer (Block) to generate output features, and the output features are input as input features of the next layer and continue to be processed in the next layer. The input features of the last layer are input into the task head for processing to obtain the task processing result.

[0083] Taking the i-th layer (Block) in the pre-training model as an example, let x i represent the input features of the i-th layer (Block), and let x i+1 represent the output features of the i-th layer (Block). The output features of the i-th layer (Block) are the input features of the (i+1)-th layer (Block). The distance between the input features x i and the output features x i+1 of the i-th layer (Block) is calculated. The distance between the input features x i and the output features x i+1 of the i-th layer (Block) can measure the importance of the i-th layer (Block) for injecting field knowledge. The higher the distance between the input features x i and the output features x i+1 , the greater the difference between the input features and the output features of the i-th layer (Block), indicating that the i-th layer (Block) plays a more critical role in adapting to new field knowledge, that is, the i-th layer (Block) is more important for injecting field knowledge. Therefore, the distance between the input features x i and the output features x i+1 of the i-th layer (Block) is taken as the first importance indicator of the i-th layer (Block).

[0084] Optionally, the distance between the input features and the output features of any layer (such as the i-th layer) can be an angular distance (AD) or a cosine distance, or other indicators for measuring the distance between two features, which are not limited in this embodiment.

[0085] For example, taking the distance between the input features and the output features as an angular distance as an example, the first importance indicator of any layer (such as the i-th layer) can be calculated using the following formula (1):

[0086]

[0087] wherein, A i,i+1 denotes the angular distance between the input feature x i and the output feature x i+1 of the i-th layer. denotes the transpose of the input feature x i .‖·‖ denotes the L2 norm, and arccos(·) is the inverse cosine function.

[0088] In the second optional implementation, based on the domain data of the target domain, the importance indicators of each layer in the pre-training model for injecting domain knowledge can be realized in the following manner:

[0089] Each layer in the pre-training model is taken as a target layer, the target layer is removed from the pre-training model to obtain a first intermediate model; the performance degradation of the first intermediate model compared with the pre-training model is evaluated using the domain data of the target domain as a second importance indicator of the target layer, wherein the greater the second importance indicator of the target layer is, the higher the importance of the target layer for injecting domain knowledge is.

[0090] Exemplarily, taking the i-th layer (Block) in the pre-training model as an example, the i-th layer is taken as a target layer, the i-th layer is removed from the pre-training model, and the output feature of the original i-1-th layer is directly input into the original i+1-th layer to obtain a first intermediate model. The performance of the first intermediate model and the pre-training model is evaluated based on the domain data of the target domain, and the performance degradation of the first intermediate model compared with the pre-training model is calculated. The higher the performance degradation of the first intermediate model obtained after removing the target layer compared with the pre-training model is, the more critical the role of the removed target layer in adapting to new domain knowledge is, and the higher the importance of the target layer for injecting domain knowledge is. Therefore, the performance degradation of the first intermediate model after removing the target layer compared with the pre-training model is taken as the second importance indicator of the target layer.

[0091] In the implementation of evaluating the performance of the first intermediate model and the pre-training model based on the domain data of the target domain, the domain data of the target domain is used to construct test data, and the first intermediate model and the pre-training model are respectively tested by the test data to obtain the performance indicators of the first intermediate model and the pre-training model. The model testing of the first intermediate model and the pre-training model can be realized by using any existing performance testing scheme of the pre-training model, which will not be described here.

[0092] Similar processing is performed on each layer in the pre-training model to obtain the second importance indicators of each layer. The layer with a greater second importance indicator has a higher importance for injecting domain knowledge.

[0093] Optionally, when evaluating the performance of the first intermediate model and the pre-training model after removing the target layer, the performance of the first intermediate model and the pre-training model in processing the question and answer task of the target field can be evaluated. For example, the performance indicators of the first intermediate model and the pre-training model can be evaluated by using the question and answer benchmark test methods such as Massive Multitask Language Understanding (MMLU), AI2Reasoning Challenge (ARC), and the like. Further, the decline amplitude (such as the decline value or the decline ratio, etc.) of the performance indicators of the first intermediate model after removing the target layer compared with the pre-training model is calculated as the second importance indicator of the target layer.

[0094] In a third optional implementation, based on the field data of the target field, the importance indicators of each layer of the pre-training model for injecting field knowledge can be evaluated in the following manner:

[0095] Each layer of the pre-training model is taken as a target layer, a new layer is inserted after the target layer, the parameters of the new layer are determined according to the parameters of the target layer, and a second intermediate model is obtained. The field data of the target field is used to evaluate the performance decline degree of the second intermediate model compared with the pre-training model as the third importance indicator of the target layer, wherein the greater the third importance indicator of the target layer is, the higher the importance of the target layer for injecting field knowledge is.

[0096] Exemplarily, taking the i-th layer (Block) in the pre-training model as an example, the i-th layer is taken as a target layer, a new layer is inserted after the i-th layer, the output features of the original (i-1)-th layer are input into the inserted layer, and the output features of the inserted layer are input into the original (i+1)-th layer to obtain a second intermediate model. The parameters of the layer inserted after the i-th layer are determined according to the parameters of the i-th layer.

[0097] Optionally, the parameters of the layer inserted after the i-th layer are the parameters of the i-th layer, that is, the layer inserted after the i-th layer is a copy of the i-th layer. Optionally, the parameters of the layer inserted after the i-th layer can be the mean value of the parameters of the i-th layer and the parameters of the (i+1)-th layer. By using the mean value of the parameters of the adjacent two layers (i.e., the i-th layer and the (i+1)-th layer) to initialize the parameters of the newly inserted layer, a more coherent initialization effect between layers can be achieved.

[0098] Further, based on the domain data of the target domain, the performance of the second intermediate model and the pre-training model is evaluated, and the performance degradation of the second intermediate model compared with the pre-training model is calculated. The higher the performance degradation of the second intermediate model compared with the pre-training model obtained by inserting a new layer after the target layer, the more important the target layer is in adapting to new domain knowledge, and the more important the domain knowledge injection is. Therefore, the performance degradation of the first intermediate model obtained by inserting a new layer after the target layer compared with the pre-training model is used as the third importance index of the target layer.

[0099] In the implementation of evaluating the performance of the second intermediate model and the pre-training model based on the domain data of the target domain, the test data is constructed using the domain data of the target domain, and the model test is performed on the second intermediate model and the pre-training model respectively through the test data to obtain the performance indicators of the second intermediate model and the pre-training model. The model test on the second intermediate model and the pre-training model can be implemented by using any existing performance test scheme of the pre-training model, which will not be described here.

[0100] Similarly, the third importance index of each layer in the pre-training model can be obtained by processing each layer in the pre-training model. The layer with a larger third importance index is more important for domain knowledge injection.

[0101] Optionally, when evaluating the performance of the second intermediate model and the pre-training model, the performance of the second intermediate model and the pre-training model in handling the question and answer task of the target domain can be evaluated. For example, the performance indicators of the second intermediate model and the pre-training model can be evaluated by using the methods of large-scale multi-task language understanding (MMLU), AI2 reasoning challenge (ARC) and other question and answer benchmark tests. Further, the performance indicator degradation (such as the degradation value or the degradation ratio, etc.) of the second intermediate model compared with the pre-training model is calculated as the third importance index of the corresponding target layer.

[0102] It should be noted that when determining the key layer of the pre-training model in this step, at least one importance index of each layer can be calculated by selecting at least one of the three optional implementation manners of calculating the importance index of each layer. According to the at least one importance index, the layer (Block) with higher importance for domain knowledge injection is selected as the key layer of the pre-training model.

[0103] In an optional embodiment, after at least one importance index of each layer in the pre-training model is calculated, the server can output at least one importance index of each layer through a front-end visual interface, so that a technical personnel can refer to at least one importance index of each layer to configure the key layer in the pre-training model in the visual interface. The server determines the key layer configured in the visual interface as the key layer of the pre-training model.

[0104] In an optional embodiment, the key layers in the pre-trained model can be configured by a relevant technical person through a front-end interface. Illustratively, the server can output relevant information of the pre-trained model (such as the name of the pre-trained model, the hierarchical information, etc.) through the front-end interface for the relevant technical person to refer to and configure the hierarchical information of the key layers in the pre-trained model. The server receives the hierarchical information of the key layers configured through the front-end interface, and determines the layers in the pre-trained model corresponding to the hierarchical information of the key layers as the key layers.

[0105] In step S302, a corresponding expansion layer is inserted after at least one key layer in the pre-trained model to obtain an expansion model, wherein the parameters of the expansion layer are determined according to the parameters of the corresponding key layer.

[0106] After determining the key layers of the pre-trained model, all the key layers can be expanded, or one or more key layers can be selected from the key layers of the pre-trained model for expansion.

[0107] Illustratively, at least one key layer is selected from the key layers of the pre-trained model as an expansion object layer; an expansion layer adjacent to the expansion object layer is inserted after the expansion object layer; the parameters of the expansion layer corresponding to the expansion object layer are updated according to the parameters of the expansion object layer to obtain an expansion model. By selecting part of the key layers from the key layers of the pre-trained model for expansion, the expansion model can be made more lightweight, the trainable parameters of the model can be reduced, and the efficiency of model training can be improved.

[0108] In an example scenario, the key layers of the pre-trained model include a plurality of layers that are continuously adjacent in the pre-trained model. The plurality of layers that are continuously adjacent are divided into key layer groups, any key layer group includes at least one key layer that are continuously adjacent; the last layer in each key layer group is taken as an expansion object layer. Further, an expansion layer adjacent to the expansion object layer is inserted after the expansion object layer.

[0109] Specifically, the pre-trained model is a model including L layers, wherein the first half (i.e. the shallow layer) of L / 2 layers are key layers. Then the first half (i.e. the shallow layer) of L / 2 key layers are divided into N key layer groups, each key layer group contains L N consecutive key layers. For each key layer group, the last layer (i.e. the L N th layer of the deepest layer) in the key layer group is taken as an expansion object layer, and an expansion layer is inserted after the last layer of each key layer group. Wherein, L is usually even, N and L N are positive integers.

[0110] For example, assume that the pre-trained model is a model including 32 layers (Blocks), and the first half (i.e., the shallow layers) of the 16 layers are key layers. Then, the first half (i.e., the shallow layers) of the 16 key layers are divided into 8 key layer groups, each of which includes 2 consecutive key layers. For each key layer group, the last layer (i.e., the 2nd deepest layer) of the key layer group is taken as the expansion target layer, and an expansion layer is inserted after the 2nd layer of each key layer group.

[0111] In another example scenario, the key layers of the pre-trained model include at least one layer that is discrete in the pre-trained model, and each of the layers that are key layers can be taken as an expansion target layer.

[0112] Specifically, the pre-trained model is a model including L layers (Blocks), and M layers that are discrete are key layers, and the M layers are not adjacent to each other. Then, the M key layers are taken as expansion target layers, and an expansion layer is inserted after each key layer.

[0113] It should be noted that the key layers of the pre-trained model can include both consecutive key layers and discrete key layers. For the consecutive key layers in the pre-trained model, the consecutive key layers are divided into key layer groups, and each key layer group includes at least one consecutive key layer; the last layer of each key layer group is taken as an expansion target layer. For the discrete key layers in the pre-trained model, each of the discrete key layers is taken as an expansion target layer.

[0114] Optionally, when at least one key layer in the key layers of the pre-trained model is selected as an expansion target layer, the key layer with higher importance of injecting domain knowledge can also be selected as an expansion target layer according to the importance indicators of the key layers.

[0115] Optionally, when at least one key layer in the key layers of the pre-trained model is selected as an expansion target layer, at least one key layer of the pre-trained model can also be selected and specified as an expansion target layer by a related technical person according to actual application requirements and experience.

[0116] In this embodiment, the parameters of the expansion layer corresponding to the expansion target layer can be determined according to the parameters of the expansion target layer, or can be determined according to the parameters of the two layers adjacent to the expansion layer.

[0117] Optionally, when the parameters of the expansion layer corresponding to the expansion target layer are updated according to the parameters of the expansion target layer, the parameters of the expansion layer corresponding to the expansion target layer can be directly updated to the parameters of the expansion target layer. In this case, the expansion layer is a copy of the expansion target layer.

[0118] Optionally, according to the parameters of the extended object layer, when updating the parameters of the extended layer corresponding to the extended object layer, for any extended layer, the parameters of the extended layer are updated using the mean value of the parameters of the two adjacent layers of the extended layer. In this case, the parameters of the extended layer are the mean value of the parameters of the two adjacent layers. By using the mean value of the parameters of the two adjacent layers to initialize the parameters of the extended layer, a more coherent initialization effect between layers can be achieved.

[0119] In step S303, the parameters of the extended layer in the extended model are adjusted using the domain data of the target domain to inject the domain knowledge contained in the domain data into the extended layer, and a domain model is obtained.

[0120] In this embodiment, after inserting the corresponding extended layer after at least one key layer in the pre-trained model to obtain the extended model, the extended model is further pre-trained using the domain data of the target domain to obtain the pre-trained extended model. During the further pre-training, the parameters of the original layers belonging to the pre-trained model in the extended model are fixed, and the parameters of the extended layer in the extended model are adjusted, which can significantly reduce the amount of training parameters and improve the efficiency of further pre-training. Further, the pre-trained extended model is fine-tuned using the instruction data set of the target domain to obtain a domain model of the target domain.

[0121] The target domain refers to a specific domain / vertical domain to which the task to be processed belongs according to the requirements of the actual application scenario. For example, the domains of mathematics, programming, law, e-commerce, entertainment, and medicine. The target domain can cover one or more specific / vertical domains. For example, the target domain can be the legal domain, or the target domain can include the domains of mathematics and programming.

[0122] The domain data of the target domain refers to the training data used for model pre-training based on the data of the target domain. The domain data of the target domain contains rich domain knowledge of the target domain. The domain data of the target domain can be provided by an end-side device or constructed by a server.

[0123] For example, the server can use the data in the open source data set of the target domain as the domain data, or the server can obtain the domain data of the target domain by collecting publicly available knowledge data of the target domain.

[0124] For example, the end-side device can collect publicly available knowledge data of the target domain, or the end-side device can obtain knowledge data within the enterprise to which it belongs as the domain data of the target domain and send it to the server. The server receives the domain data of the target domain sent by the end-side device.

[0125] The instruction data set of the target domain refers to the instruction data set used for fine-tuning the model for a specific task in the target domain.

[0126] Exemplarily, the instruction dataset of the target domain is constructed by converting the domain data of the target domain into instruction data, or is obtained by collecting the instruction dataset used for fine-tuning the pre-trained model for each type of task disclosed in the target domain.

[0127] In this embodiment, the continued pre-training of the extended model is consistent with the implementation principle of pre-training the pre-trained model, and can be implemented by using any pre-training strategy for the pre-trained model, which is not limited here.

[0128] The fine-tuning training of the pre-trained extended model using the instruction dataset of the target domain obtains the domain model of the target domain, which is consistent with the implementation principle of fine-tuning the pre-trained model using the instruction dataset, and can be implemented by using any fine-tuning training strategy for the pre-trained model, which is not limited here.

[0129] The method of this embodiment inserts the corresponding extended layer after at least one key layer in the pre-trained model to obtain an extended model, and adjusts the parameters of the extended layer in the extended model using the domain data of the target domain, so as to inject the domain knowledge contained in the domain data into the extended layer which is more critical for domain knowledge injection. This not only improves the task processing capability of the model in the target domain, but also better preserves the powerful general capability of the pre-trained model, can effectively balance the general capability of the model and the task processing capability in the target domain, realizes more effective knowledge injection, and thus can improve the performance of the model in processing specific domain tasks.

[0130] In an optional embodiment, after inserting the corresponding extended layer after at least one key layer in the pre-trained model to obtain an extended model, the layers which are not important for domain knowledge injection (i.e., have lower influence on domain knowledge injection) in the extended model can be pruned to further lighten the extended model.

[0131] Specifically, the pruning layer in the pre-trained model is determined, the pruning layer is removed from the extended model, and the parameters of the previous layer of the pruning layer are updated according to the parameters of the pruning layer. The pruning layer refers to a layer which is not important for domain knowledge injection, and the pruning layer is determined according to the importance of each layer in the pre-trained model for domain knowledge injection.

[0132] In determining the pruning layers in the pre-training model, according to the importance indicators of the layers in the pre-training model, a layer (Block) with low importance of injecting domain knowledge is selected as the pruning layer of the pre-training model. The calculation principle of the importance indicators of the layers in the pre-training model is described in the foregoing embodiments, which will not be repeated here.

[0133] It should be noted that in the embodiment, in determining the pruning layers of the pre-training model, at least one importance indicator of each layer can be calculated based on the three optional implementation manners of calculating the importance indicators of the layers. According to the at least one importance indicator, a layer (Block) with low importance of injecting domain knowledge is selected as the pruning layer of the pre-training model.

[0134] In an optional embodiment, after at least one importance indicator of each layer in the pre-training model is calculated, the server can output at least one importance indicator of each layer through a front-end visual interface, so that a technical personnel can refer to at least one importance indicator of each layer to configure the pruning layers in the pre-training model in the visual interface. The server determines the pruning layers configured in the visual interface as the pruning layers of the pre-training model.

[0135] In an optional embodiment, the key layers in the pre-training model can be configured by a technical personnel through a front-end interface. For example, the server can output the relevant information of the pre-training model (such as the name of the pre-training model, the hierarchical information, etc.) through the front-end interface, so that a technical personnel can refer to the hierarchical information of the pruning layers in the pre-training model. The server receives the hierarchical information of the pruning layers configured through the front-end interface, and determines the layers in the pre-training model corresponding to the hierarchical information of the pruning layers as the pruning layers.

[0136] In the embodiment, the determined pruning layers of the pre-training model can include a plurality of pruning layers that are continuously adjacent in the pre-training model, and / or at least one pruning layer that is discrete. For the plurality of pruning layers that are continuously adjacent, the plurality of pruning layers that are continuously adjacent are deleted from the pre-training model, and the parameters of the previous layer are updated according to the parameters of the deleted plurality of pruning layers that are continuously adjacent.

[0137] Optionally, the parameters of the previous layer are updated according to the mean of the parameters of the deleted plurality of pruning layers that are continuously adjacent; or the parameters of the previous layer are updated according to the mean of the parameters of the deleted plurality of pruning layers that are continuously adjacent and the previous layer.

[0138] Exemplarily, assuming that the pruning layers of the pre-training model include the last D layers of the pre-training model, the parameters of the last D layers of the pre-training model are used to update the parameters of the previous layer (i.e., the (D+1)th layer from the bottom) of the last D layers; or, the parameters of the (D+1)th layer from the bottom of the pre-training model are used to update the parameters of the previous layer (i.e., the (D+1)th layer from the bottom) of the last D layers. Further, the last D layers of the pre-training model are deleted.

[0139] For the discrete pruning layers in the pre-training model, the parameters of the previous layer are updated according to the parameters of the deleted pruning layer, and the pruning layer is deleted from the pre-training model. Alternatively, the parameters of the previous layer can be updated as the parameters of the deleted pruning layer, or the parameters of the previous layer are updated as the parameter mean of the deleted pruning layer and the previous layer.

[0140] Exemplarily, assuming that the pruning layers of the pre-training model include the dth layer therein, the parameters of the dth layer of the pre-training model are used to update the parameters of the previous layer (i.e., the (d-1)th layer); or, the parameter mean of the dth layer and the (d-1)th layer of the pre-training model is used to update the parameters of the (d-1)th layer. Further, the dth layer is deleted.

[0141] Further, after the corresponding expansion layers are inserted behind the at least one key layer in the pre-training model and the at least one pruning layer is deleted to obtain the expansion model, the expansion model is further pre-trained using the domain data of the target domain to obtain the pre-trained expansion model. During the further pre-training, the parameters of the original layers belonging to the pre-training model in the expansion model are fixed, and the parameters of the previous layers of the expansion layers and the pruning layers in the expansion model are adjusted. Further, the pre-trained expansion model is fine-tuned using the instruction data set of the target domain to obtain the domain model of the target domain.

[0142] The method of the embodiment can inject the domain knowledge into the expansion layers which are more critical for the domain knowledge injection in the expansion model, improve the task processing capability of the model in the target domain, and better retain the powerful general capability of the pre-training model, so as to effectively balance the general capability of the model and the task processing capability of the model in the target domain, achieve more effective knowledge injection, and thus improve the performance of the model in processing the task in the specific domain. Meanwhile, by deleting the pruning layers in the model, the expansion model obtained can be more lightweight, and the training efficiency of the expansion model can be improved.

[0143] Figure 4 An example diagram of the expansion model provided by an example embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the pre-training model includes a plurality of layers, and the at least one key layer is inserted behind the at least one key layer in the pre-training model to obtain the expansion model. Figure 4As shown, taking a 32-layer (Block) large language model (such as the large language model of the LIama series) as an example, it includes 31 layers (Block) as shown in Figure 4 Original Layer 0, Original Layer 1, …, Original Layer 16, …, Original Layer 31. In addition, as shown in Figure 4 The large language model usually also includes an embedding layer (such as Figure 4 Embedding) and a task head (such as Figure 4 LM Head).

[0144] By calculating the importance indicators of the domain knowledge injection of each layer in the large language model, it is determined that the first half (i.e. the shallow layer) of the 16 layers (such as Figure 4 Original Layer 0, Original Layer 1, …, Original Layer 16) in the large language model are key layers. The first half (i.e. the shallow layer) of the 16 key layers is divided into 8 key layer groups, and each key layer group contains 2 consecutive adjacent key layers. The 8 key layer groups include: Original Layer 0, Original Layer 1; Original Layer 2, Original Layer 3; Original Layer 4, Original Layer 5; …; Original Layer 14, Original Layer 15.

[0145] For each key layer group, the last (i.e. the second deepest) layer in the key layer group is taken as the expansion object layer, and an expansion layer is inserted after the second layer of each key layer group. As shown in Figure 4 Taking the first key layer group as an example, the key layer group includes two key layers Original Layer 0 and Original Layer 1, and an expansion layer is inserted after the deepest Original Layer 1, as shown in Figure 4 Expanded Layer 1. The parameters of Expanded Layer 1 are initialized using the parameters of Original Layer 1, or the parameters of Expanded Layer 1 are initialized using the average of the parameters of Original Layer 1 and Original Layer 2.

[0146] Similarly, in the second key layer group, an expanded layer is inserted after Original Layer 3, as shown in Expanded Layer 2 in Figure 4 . The parameters of Expanded Layer 2 are initialized using the average values of the parameters of Original Layer 3 and Original Layer 4, so as to achieve more coherent initialization between layers.

[0147] By analogy, in the eighth key layer group, an expanded layer is inserted after Original Layer 15, as shown in Expanded Layer 8 in Figure 4 . The parameters of Expanded Layer 8 are initialized using the average values of the parameters of Original Layer 15 and Original Layer 16, so as to achieve more coherent initialization between layers.

[0148] In addition, the importance index of each layer in the large language model for injecting domain knowledge can determine that the last two layers in the large language model (e.g., Original Layer 30 and Original Layer 31 shown in Figure 4 ) are the pruning layers with low importance for injecting domain knowledge. The parameters of the pruning layers (i.e., Original Layer 30 and Original Layer 31) are updated using the average values of the parameters of the previous layer of the pruning layers (i.e., Original Layer 29), to obtain Updated Layer 29 shown in Figure 4 , and the Original Layer 30 and the Original Layer 31 are pruned to obtain an expanded model, so that the expanded model is more lightweight.

[0149] In the process of continuing pre-training the expanded model using the domain data of the target domain, the parameters of the inserted expanded layers (e.g., the expanded layers with flame marks shown in Figure 4 , including Expanded Layer 1, Expanded Layer 2, …, and Expanded Layer 8) and the parameters of the previous layer of the pruning layer (e.g., Updated Layer 29 shown in Figure 5 ) are adjusted, and the original layers (e.g., Original Layer 0, Original Layer 1, …, and Original Layer 28 shown in Figure 5 ) in the pre-trained model are fixed, which can greatly reduce the amount of parameters to be trained and improve the efficiency of the continuing pre-training.

[0150] Figure 6 A flowchart of a natural language processing method is provided for an exemplary embodiment of the present application. As shown in the figure, the method has the following specific steps: Figure 6

[0151] Step S501: receiving a calling request for a domain model of a target domain, and obtaining input data of the target domain.

[0152] In this embodiment, the server can provide an application programming interface (API) of the domain model of the target domain to the outside. The end-side device can send a calling request to the server through the API of the domain model of the target domain. The calling request contains input data of the target domain required to input the domain model.

[0153] The server receives the calling request for the domain model of the target domain sent by the end-side device, and extracts the input data of the target domain from the calling request.

[0154] It should be noted that the server can pre-obtain one or more domain models of different target domains, and the different domain models have different APIs. The server can provide the APIs of the domain models of different target domains to the end-side device, and the end-side device can select to call different APIs according to the target domain to which the current natural language processing task belongs, to use the domain model of the target domain corresponding to the API. In addition, in actual application, the server can receive calling requests from multiple different end-side devices, and provide natural language processing capabilities based on the domain model of the target domain to multiple different end-side devices.

[0155] Step S502: inputting the input data of the target domain into the domain model for natural language processing, and obtaining a natural language processing result, wherein the domain model is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model, and injecting domain knowledge of the target domain into the extension layer, and the key layer is determined according to the importance of each layer in the pre-trained model to the injection of domain knowledge.

[0156] After obtaining the domain data of the target domain, the server inputs the input data of the target domain into the domain model for natural language processing, and obtains a natural language processing result.

[0157] In this embodiment, the domain model of the target domain is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model, and injecting domain knowledge of the target domain into the extension layer. The key layer is determined according to the importance of each layer in the pre-trained model to the injection of domain knowledge.

[0158] Specifically, the process of inserting at least one extension layer corresponding to a key layer into a pre-trained model, and injecting domain knowledge of the target domain into the extension layer to obtain the domain model of the target domain includes:​

[0159] determining a key layer in the pre-trained model; inserting a corresponding expansion layer after the at least one key layer in the pre-trained model to obtain an expansion model, wherein parameters of the expansion layer are determined according to parameters of the corresponding key layer; and adjusting the parameters of the expansion layer in the expansion model using domain data of the target domain to inject domain knowledge contained in the domain data of the target domain into the expansion layer, to obtain a domain model.

[0160] In this embodiment, the specific implementation principle and technical effects of obtaining the domain model of the target domain are the same as those of the related content of the foregoing embodiments, which will not be described here again.

[0161] In step S503, the natural language processing result is returned.

[0162] After obtaining the natural language processing result, the server returns the natural language processing result to the end-side device.

[0163] In this embodiment, the server provides an API of the domain model of the target domain to the end-side device. Since the pre-trained model is trained to maintain the original strong semantic understanding and representation ability as much as possible, and more knowledge of the target domain is injected into the domain model to enhance the semantic understanding and generation ability of the domain model in the target domain, the quality of the natural language processing result can be greatly improved when the downstream task system uses the domain model to perform the natural language processing task of the target domain.

[0164] Figure 6 A structural schematic diagram of a server provided in an embodiment of the present application is shown in FIG. 6. Figure 6 As shown in FIG. 6, the server includes a memory 601 and a processor 602. The memory 601 is configured to store computer execution instructions and can be configured to store other various data to support operations on the server. The processor 602 is communicatively connected with the memory 601 and is configured to execute the computer execution instructions stored in the memory 601 to implement the technical solutions provided by any of the method embodiments described above, and the specific functions and technical effects that can be achieved are similar, which will not be described here again.

[0165] Optionally, as shown in FIG. 6, the server further includes a firewall 603, a load balancer 604, a communication component 605, a power supply component 606 and other components. Figure 6 Only part of the components are shown in the figure, which does not mean that the server only includes the components shown in the figure. Figure 6 Only the server is exemplarily illustrated as a cloud server deployed in the cloud in the figure, and the server can also be deployed locally, which will not be limited here. ​ ​

[0166] ​​The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the method of any of the foregoing embodiments is implemented, and the specific functions and the technical effects that can be achieved are not repeated here.

[0167] The embodiment of the present application further provides a computer program product, including a computer program. The computer program is executed by a processor to implement the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of a server can read the computer program from the readable storage medium. The at least one processor executes the computer program to enable the server to execute the technical solutions provided in any of the method embodiments, and the specific functions and the technical effects that can be achieved are not repeated here.

[0168] The embodiment of the present application provides a chip, including a processing module and a communication interface. The processing module can execute the technical solutions of the server in the foregoing method embodiments. Optionally, the chip further includes a storage module (for example, a memory). The storage module is used to store instructions. The processing module is used to execute the instructions stored in the storage module. The execution of the instructions stored in the storage module enables the processing module to execute the technical solutions provided in any of the method embodiments.

[0169] The integrated modules in the form of software function modules described above can be stored in a computer readable storage medium. The software function modules described above are stored in a storage medium, and include a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method of each embodiment of the present application.

[0170] It should be understood that the processor described above can be a central processing unit (CPU), a graphics processing unit (GPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor.

[0171] The memory can include a high-speed random access memory (RAM) and can also include a non-volatile storage, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0172] The memory described above can be an object storage (OSS).

[0173] The memory described above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0174] The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second generation mobile communication system (2G), a third generation mobile communication system (3G), a fourth generation mobile communication system (4G) / long term evolution (LTE), a fifth generation mobile communication system (5G), or other mobile communication networks, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared technology, ultra wide band (UWB) technology, Bluetooth technology, and other technologies.

[0175] The power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0176] The storage medium can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0177] An exemplary storage medium is coupled to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor and the storage medium can be located in a special purpose computing device. The processor and the storage medium can also be located in a general purpose computing device.

[0178] It is to be appreciated that the term "include," "comprise," or variations thereof, as used in this document, is intended to cover the presence of one or more elements, components, steps, actions, etc. but does not exclude the presence of one or more other elements, components, steps, actions, etc. Unless otherwise indicated, the use of the term "or" in this document is used to mean "and / or," that is, the term "or" is not exclusive, unless otherwise indicated. In addition, it is to be appreciated that the use of any form of "present" or "comprise" herein, such as "comprises", "comprising", "includes", "including", "contains", "containing" or variants thereof, is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, includes, contains elements or steps do not include only those elements or steps but can also include other elements or steps not expressly listed or inherent to such process, method, article, or apparatus.

[0179] The sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed in the order they appear in this document or in parallel, and only to distinguish different operations. The sequence itself does not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. described herein are used to distinguish different messages, devices, modules, etc. and do not represent the order. "First" and "second" are different types. The meaning of "multiple" is more than two, unless otherwise explicitly specified.

[0180] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art to make contributions can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method of each embodiment of the present application.

[0181] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the present application along with its general principles and features. This application is intended to cover any adaptations or variations of the application.

[0182] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A task processing method, characterized in that, include: In response to a task processing request from the target domain, obtain the input data from the target domain; Input data from the target domain is input into a domain model for task processing to obtain task processing results. The domain model is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model and injecting domain knowledge of the target domain into the extension layer. The key layer is determined according to the importance of each layer of the pre-trained model to the injection of the domain knowledge. Output the task processing results.

2. The method according to claim 1, characterized in that, The step of inserting at least one extended layer corresponding to a key layer into the pre-trained model and injecting domain knowledge of the target domain into the extended layer includes: Identify the key layers in the pre-trained model; An extended layer is inserted after at least one key layer in the pre-trained model to obtain an extended model, wherein the parameters of the extended layer are determined according to the parameters of the corresponding key layer. Using the domain data of the target domain, the parameters of the extension layer in the extension model are adjusted to inject the domain knowledge contained in the domain data into the extension layer, thereby obtaining the domain model.

3. The method according to claim 2, characterized in that, The process of determining the key layers in the pre-trained model includes: Based on the domain data of the target domain, evaluate the importance metrics of each layer in the pre-trained model for injecting domain knowledge. The key layers of the pre-trained model are determined based on the importance index of each layer in the pre-trained model.

4. The method according to claim 3, characterized in that, The evaluation of the importance metrics for the injection of domain knowledge into each layer of the pre-trained model based on domain data of the target domain includes: The domain data is input into the pre-trained model for processing. Based on the distance between the input features and output features of each layer in the pre-trained model, the first importance index of each layer is calculated. The larger the first importance index, the higher the importance of the corresponding layer for the injection of domain knowledge.

5. The method according to claim 3, characterized in that, The evaluation of the importance metrics for the injection of domain knowledge into each layer of the pre-trained model based on domain data of the target domain includes: Each layer in the pre-trained model is taken as a target layer, and the target layer is removed from the pre-trained model to obtain the first intermediate model; Using domain data from the target domain, the performance degradation of the first intermediate model compared to the pre-trained model is evaluated as a second importance indicator for the target layer. The larger the second importance indicator of the target layer, the higher the importance of the target layer in injecting domain knowledge.

6. The method according to claim 3, characterized in that, The evaluation of the importance metrics for the injection of domain knowledge into each layer of the pre-trained model based on domain data of the target domain includes: Each layer in the pre-trained model is taken as a target layer, and a new layer is inserted after the target layer to obtain a second intermediate model. The parameters of the new layer are determined according to the parameters of the target layer. Using domain data from the target domain, the performance degradation of the second intermediate model compared to the pre-trained model is evaluated as the third importance index of the target layer. The larger the third importance index of the target layer, the higher the importance of the target layer in injecting domain knowledge.

7. The method according to claim 2, characterized in that, The step of inserting a corresponding extension layer after at least one key layer in the pre-trained model to obtain an extended model includes: Select at least one key layer from the key layers of the pre-trained model as the extended object layer; Insert an extension layer adjacent to the extension object layer after the extension object layer; Based on the parameters of the extended object layer, the parameters of the adjacent extended layers are updated to obtain the extended model.

8. The method according to claim 7, characterized in that, The step of selecting at least one key layer from the key layers of the pre-trained model as the extended object layer includes: If the key layer of the pre-trained model includes multiple consecutive adjacent layers in the pre-trained model, then the multiple consecutive adjacent layers are divided into key layer groups, and any key layer group includes at least one consecutive adjacent key layer. The last layer in each of the aforementioned key layer groups is taken as the extended object layer.

9. The method according to claim 7, characterized in that, The step of selecting at least one key layer from the key layers of the pre-trained model as the extended object layer includes: If the key layer of the pre-trained model includes at least one discrete layer in the pre-trained model, then the at least one discrete key layer is used as the extended object layer.

10. The method according to claim 7, characterized in that, The step of updating the parameters of the extension layer corresponding to the extension object layer based on the parameters of the extension object layer includes: Update the parameters of the extension layer corresponding to the extension object layer to the parameters of the extension object layer; or, For any of the aforementioned extended layers, the parameters of the extended layer are updated using the average parameter value of the two adjacent layers.

11. The method according to any one of claims 2-10, characterized in that, The step of using domain data from the target domain to adjust the parameters of the extension layer in the extended model, thereby injecting the domain knowledge contained in the domain data into the extension layer to obtain the domain model, includes: Using the domain data of the target domain, the extended model is pre-trained. The parameters of the original layer belonging to the pre-trained model in the extended model are fixed, and the parameters of the extended layer in the extended model are adjusted to obtain the pre-trained extended model. The pre-trained extended model is fine-tuned using the instruction dataset of the target domain to obtain the domain model.

12. The method according to claim 2, characterized in that, After inserting a corresponding extension layer after at least one key layer in the pre-trained model to obtain the extended model, the method further includes: The pruning layer in the pre-trained model is determined based on the importance of each layer in the pre-trained model for the injection of domain knowledge; Remove the clipping layer from the extended model, and update the parameters of the layer preceding the clipping layer based on the parameters of the clipping layer.

13. The method according to claim 12, characterized in that, The step of using domain data from the target domain to adjust the parameters of the extension layer in the extended model, thereby injecting the domain knowledge contained in the domain data into the extension layer to obtain the domain model, includes: Using the domain data of the target domain, the extended model is pre-trained. The parameters of the original layer belonging to the pre-trained model in the extended model are fixed, and the parameters of the layer before the extended layer and the pruning layer in the extended model are adjusted to obtain the pre-trained extended model. The pre-trained extended model is fine-tuned using the instruction dataset of the target domain to obtain the domain model.

14. A natural language processing method, characterized in that, include: Receive a request to invoke the domain model of the target domain, and obtain the input data of the target domain; Input data from the target domain is input into the domain model for natural language processing to obtain natural language processing results. The domain model is obtained by inserting at least one extension layer corresponding to a key layer into a pre-trained model and injecting domain knowledge of the target domain into the extension layer. The key layer is determined according to the importance of each layer of the pre-trained model to the injection of the domain knowledge. Return the natural language processing result.

15. The method according to claim 14, characterized in that, The step of inserting at least one extended layer corresponding to a key layer into the pre-trained model and injecting domain knowledge of the target domain into the extended layer includes: Identify the key layers in the pre-trained model; An extended layer is inserted after at least one key layer in the pre-trained model to obtain an extended model, wherein the parameters of the extended layer are determined according to the parameters of the corresponding key layer. Using the domain data of the target domain, the parameters of the extension layer in the extension model are adjusted to inject the domain knowledge contained in the domain data into the extension layer, thereby obtaining the domain model.

16. A server, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the server to perform the method according to any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-15.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-15.

Images