Task processing method and server

Abstract

The application provides a task processing method and a server. The method of the application uses mixed pre-training data of vertical fields and general fields to continue pre-training of a pre-training model, obtains a pre-trained field-based model, and makes the model familiar with the characteristics of the text format and data distribution of the vertical field and injects more knowledge of the vertical field into the model, thereby enhancing the text understanding and generation capability of the model in the vertical field on the basis of maintaining the original strong semantic understanding and representation capability of the pre-training model as much as possible; then, a field-based model suitable for the vertical field is obtained by using a data set of the vertical field to fine-tune and train the field-based model, and the field-based model common to various natural language processing tasks in the vertical field can be obtained, and the text understanding and generation capability of the model in the vertical field is improved, thereby improving the generation quality of the task processing result when the model is applied to various NLP tasks in the vertical field.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a task processing method and server. Background Technology

[0002] In recent years, Natural Language Processing (NLP) technology has developed rapidly. Deep learning models for NLP, such as Large Language Models (LLMs), unify different NLP tasks into a text generation paradigm, achieving good results on a large number of NLP tasks.

[0003] However, mainstream deep learning models for NLP (such as LLM) perform poorly on various tasks in vertical domains (such as e-commerce, healthcare, education, technology, finance, etc.), and the quality of the generated processing results is low. Summary of the Invention

[0004] This application provides a task processing method and server to solve the problem that large language models perform poorly on various tasks in vertical domains and generate low-quality processing results.

[0005] In a first aspect, this application provides a task processing method, comprising: using mixed pre-training data from vertical and general domains to further pre-train a pre-trained model to obtain a pre-trained domain-based model; using the dataset from the vertical domain to fine-tune the domain-based model to obtain a domain model suitable for the vertical domain, wherein the domain model is used to perform a natural language processing task in the vertical domain and generate a corresponding task processing result.

[0006] Secondly, this application provides a task processing method, comprising: using mixed pre-training data from the e-commerce domain and the general domain to further pre-train a pre-trained large model to obtain a pre-trained basic large model in the e-commerce domain; using a dataset from the e-commerce domain to fine-tune the basic large model in the e-commerce domain to obtain a large model in the e-commerce domain, wherein the large model in the e-commerce domain is used to perform natural language processing tasks in the e-commerce domain and generate corresponding task processing results.

[0007] Thirdly, this application provides a task processing method applied to a server, comprising: receiving a call request for a domain model sent by an end-side device, the call request containing task instructions, the task instructions being generated based on task prompt format information and input data of a natural language processing task to be executed; inputting the task instructions into the domain model, executing task processing based on the task instructions through the domain model, generating a task processing result, wherein the domain model is obtained by further pre-training a pre-trained model using mixed pre-training data from vertical and general domains to obtain a pre-trained domain basic model, and then fine-tuning the domain basic model using the dataset from the vertical domain; and returning the task processing result to the end-side device.

[0008] Fourthly, this application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the server to perform the method described in any of the above aspects.

[0009] The task processing method and server provided in this application further pre-train the pre-trained model using a mixture of vertical and general domain pre-trained data to obtain a pre-trained domain-based model. While maintaining the original powerful semantic understanding and representation capabilities of the pre-trained model as much as possible, the model becomes familiar with the characteristics of text formats and data distributions in the vertical domain, and more vertical domain knowledge is injected into the model, thereby enhancing the model's text understanding and generation capabilities in the vertical domain. Then, using the vertical domain dataset, the domain-based model is fine-tuned to obtain a domain model suitable for the vertical domain. This allows for obtaining a domain model applicable to various natural language processing tasks in the vertical domain, improving the model's text understanding and generation capabilities in the vertical domain, and thus improving the generation quality of task processing results when the model is applied to various NLP tasks in the vertical domain. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 This is a schematic diagram of an example system architecture to which this application applies;

[0012] Figure 2 This is a schematic diagram of another example system architecture to which this application applies;

[0013] Figure 3 A flowchart of a task processing method provided for an exemplary embodiment of this application;

[0014] Figure 4 A flowchart for constructing pre-trained data for a vertical domain is provided as an exemplary embodiment of this application;

[0015] Figure 5 A flowchart for constructing a vertical domain multi-task instruction dataset is provided as an exemplary embodiment of this application;

[0016] Figure 6 A schematic diagram of a fine-tuning training framework for multi-task instruction data provided for an exemplary embodiment of this application;

[0017] Figure 7 A framework diagram for obtaining a domain model of a vertical domain provided for an exemplary embodiment of this application;

[0018] Figure 8 A flowchart of a task processing method in the e-commerce field provided as an exemplary embodiment of this application;

[0019] Figure 9 A flowchart illustrating task processing based on a domain model, provided as an exemplary embodiment of this application;

[0020] Figure 10 A flowchart for implementing task processing based on a domain model is provided as another exemplary embodiment of this application;

[0021] Figure 11 A schematic diagram of the process framework for a product comparison task provided in an exemplary embodiment of this application;

[0022] Figure 12 A flowchart illustrating a product comparison method provided in an exemplary embodiment of this application;

[0023] Figure 13 An example diagram of an interface for displaying product comparison results provided for an exemplary embodiment of this application;

[0024] Figure 14 This is a schematic diagram of the structure of a server provided for an exemplary embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0028] First, let me explain the terms used in this application:

[0029] Large Language Model (LLM): A deep neural network model with a huge number of parameters trained on a large amount of text. It can understand and generate natural language text. It is also called a large-scale language model.

[0030] Pre-trained large models: Pre-trained models obtained by pre-training large models such as large-scale language models (LLM).

[0031] Domain-based pre-training: Based on a pre-trained model in a general domain, continue pre-training on unlabeled corpora in a specific domain using the same or similar pre-training tasks.

[0032] Vertical domain: An internet industry term referring to a segmented field that provides specific services to a limited group of people, including fields such as e-commerce, entertainment, healthcare, environmental protection, education, and sports. A vertical domain is a smaller sub-field that is vertically subdivided from a larger domain.

[0033] General domain: In this application, the general domain is a concept relative to the vertical domain. The general domain includes multiple domains and can cover multiple vertical domains. It can be understood as a relatively general or common domain with a large amount of existing corpus data. The data in the general domain can cover data from one or more vertical domains.

[0034] Multi-task instruction fine-tuning: Fine-tuning training based on multi-task instruction data is a method for fine-tuning a pre-trained model so that the trained model can correctly respond to task instructions given by humans, thereby better solving various task-driven tasks.

[0035] Named Entity Recognition (NER): Identifying named entities with specific meanings from text.

[0036] Relation Extraction (RE): Identifying the relationships between text segments.

[0037] Chain of Thought (CoT) is a method that provides natural language descriptions of the reasoning steps involved in complex reasoning, progressively deriving the final result. For example, in a product comparison task, given information about two products, the task is to compare them and provide a recommendation that meets the user's needs. One possible chain of thought for solving this task would be: based on the information about the two products, describe their characteristics, similarities, and differences; then describe the advantages of each product; and finally, provide a recommended product based on the user's needs. By using the chain of thought method, the model performs task reasoning step-by-step, providing the crucial reasoning process alongside the final result.

[0038] Visual question answering task: Based on the input image and the question, determine the answer to the question from the visual information of the input image.

[0039] Image description task: Generate descriptive text for the input image.

[0040] Visual entailment task: Predict the semantic relevance between input images and text, i.e., entailment, neutrality, or contradiction.

[0041] The task of expression and understanding is to locate the image region in the input image that corresponds to the input information based on the input information.

[0042] Image generation task: Generate an image based on the input descriptive text.

[0043] Text-based sentiment classification task: Predict the sentiment classification information of input information.

[0044] Text summarization task: Generate a summary of the input information.

[0045] Multimodal tasks refer to downstream tasks that involve multiple modalities of data, such as images and text, in their input and output. Examples include visual question answering, image description, visual entailment, representation and understanding, and image generation.

[0046] Multimodal pre-trained models refer to pre-trained models whose input and output data involve multiple modalities such as images and text. After fine-tuning and training, they can be applied to multimodal task processing.

[0047] Large models refer to deep learning models with a massive number of parameters, typically containing hundreds of millions, tens of billions, or even trillions of parameters. Large models are also known as foundation models (FM). They are pre-trained on large-scale unlabeled corpora, producing pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLM) and Multi-modal Pre-training Models.

[0048] Large models can be widely applied in fields such as natural language processing and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0049] Pre-trained models commonly used in the field of Natural Language Processing (NLP) (such as pre-trained language models and pre-trained large language models) possess powerful natural language understanding capabilities. Based on the generative capabilities of pre-trained models, they can perform various NLP tasks such as information extraction (e.g., Named Entity Recognition (NER), Relation Extraction (RE), etc.), text classification, and text generation. They have wide applications in various scenarios such as e-commerce, healthcare, intelligent transportation, online education, and digital assistants. For example, in the e-commerce field, they enable intelligent question answering, product comparison, and product search for product-related information; in the online education field, they enable intelligent customer service for education-related knowledge and text topic content extraction.

[0050] However, the pre-trained models that are applicable to multiple tasks in the field of natural language processing are pre-trained based on general domain pre-training data. Since the scale of labeled data used for fine-tuning training is very small, the models obtained by fine-tuning training with a small amount of fine-tuning data still perform unsatisfactorily on various tasks in vertical domains (such as e-commerce, medical care, education, technology, etc.), and the quality of the generated processing results is low.

[0051] Taking e-commerce as an example, due to its enormous commercial value, many engineers are researching how to use natural language processing (NLP) to solve numerous problems in e-commerce, such as product information extraction, user query understanding, product content generation, and intelligent customer service dialogue. However, general-purpose large language models (LLMs) cannot effectively solve various NLP tasks in e-commerce because e-commerce text has the following characteristics compared to general domain text:

[0052] 1. Much text in the e-commerce field is not a coherent, fluent sentence, but rather has its own unique structure. For example, product titles are often a continuous string of important entities or concepts, while product attribute lists are typically semi-structured key-value pairs rather than complete sentences. This poses a significant challenge to general LLM (Local Management Model) understanding of e-commerce text.

[0053] 2. The e-commerce sector contains a large number of novel entities and concepts. The distribution of these entities and concepts differs significantly from general text and is rapidly updated in response to changes in products and trends. For example, the e-commerce sector has many unique brand and merchant names, and the meanings behind these entities may differ from their literal meanings.

[0054] Based on the unique characteristics of text in the e-commerce field, even after fine-tuning a pre-trained model trained using pre-trained data from a general domain and then fine-tuning it with a small amount of fine-tuning data, the quality of the generated task processing results is still relatively low when applied to natural language processing tasks in the e-commerce field.

[0055] To address the aforementioned technical problems, this application provides a task processing method. By using hybrid pre-trained data from vertical and general domains, a pre-trained model is further pre-trained to obtain a pre-trained domain-based model. While preserving the original powerful semantic understanding and representation capabilities of the pre-trained model as much as possible, the model becomes familiar with the text formats and data distribution characteristics of the vertical domain, and more vertical domain knowledge is injected into the model, thereby enhancing its text understanding and generation capabilities in the vertical domain. Then, using a vertical domain dataset, the domain-based model is fine-tuned to obtain a domain model suitable for the vertical domain. This allows for the acquisition of a domain model applicable to various natural language processing tasks in the vertical domain, improving the model's text understanding and generation capabilities in the vertical domain, and thus enhancing the quality of the generated results when the model is applied to various NLP tasks in the vertical domain. The pre-trained model can be a model applied to natural language processing, such as various pre-trained language models or pre-trained large language models.

[0056] In this application, to distinguish it from pre-training data, the dataset used for fine-tuning training is referred to as the "fine-tuning dataset." Typically, pre-training datasets do not contain labeled information, while fine-tuning training datasets do contain labeled information. In this application, a fine-tuning dataset from the vertical domain is used when fine-tuning the domain-based model.

[0057] Figure 1 This is a schematic diagram of an example system architecture to which this application applies. Figure 1As shown, the system architecture includes a server and endpoint devices. The server and endpoint devices have a communication link, enabling communication between them.

[0058] The server is a computing device deployed in the cloud or locally, such as a cloud cluster. The server stores pre-trained models, pre-built mixed pre-trained data from both vertical and general domains, and fine-tuned datasets from the vertical domain. The server is responsible for first using the mixed pre-trained data to further pre-train the pre-trained model, obtaining a pre-trained domain-based model; then, using the fine-tuned dataset from the vertical domain, it fine-tunes the domain-based model to obtain a domain model suitable for the vertical domain. This domain model can be applied to various natural language processing tasks within the vertical domain, including but not limited to information extraction (such as Named Entity Recognition (NER), Relation Extraction (RE), etc.), text classification, and text generation. Different types of tasks use different task prompt formats, which indicate the task requirements, input data, and output results to be performed by the domain model.

[0059] Edge devices can be electronic devices that run downstream application systems. Specifically, they can be hardware devices with network communication, computing, and information display capabilities, including but not limited to smartphones, tablets, desktop computers, local servers, and cloud servers. Edge devices need to utilize the natural language processing capabilities of the domain model when running downstream application systems. For example, a downstream application system running on an edge device could implement functions such as intelligent question answering, information extraction, text classification, and text summarization within a vertical domain. Implementing at least one function of the downstream application system requires utilizing the natural language processing capabilities of the domain model within that vertical domain.

[0060] based on Figure 1 The system architecture shown illustrates that when a vertical domain natural language processing task needs to be executed, the edge device acquires the input data and the task prompt format information. Based on the task prompt format information and the input data, it generates a task instruction and sends a request to the server to invoke the domain model. This request contains the task instruction. The server receives the request, obtains the task instruction to be executed, inputs the task instruction into the vertical domain model, executes the task processing based on the task instruction through the domain model, generates a task processing result, and returns the result to the edge device. The edge device receives the task processing result returned by the server and continues to execute the processing logic of the downstream application system based on the result, thus implementing the functionality of the downstream application system.

[0061] Figure 2 This is a schematic diagram of another example system architecture to which this application applies. Figure 2As shown, the system architecture includes a server and a model service. The server and the model service have a communication link that enables communication between them.

[0062] The model service is a service that can provide pre-trained models. The model service can provide the server with download information for the pre-trained model. The server then downloads the pre-trained model from the model service to its local machine based on this information. Alternatively, the server can send a pre-trained model retrieval request to the model service, and the model service, in response, sends the pre-trained model to the server. Furthermore, the server can also obtain the pre-trained model from the model service through other interactive methods, which are not specifically limited here.

[0063] The server can be a computing device deployed in the cloud or locally by various organizations or system platforms, such as a cloud cluster or local server. The server model service acquires pre-trained models and stores pre-built hybrid pre-trained data from both vertical and general domains, as well as fine-tuned datasets from the vertical domain. The server is responsible for first using the hybrid pre-trained data from both vertical and general domains to further pre-train the pre-trained model, obtaining a pre-trained domain-based model; then, using the fine-tuned dataset from the vertical domain, it fine-tunes the domain-based model to obtain a domain model suitable for the vertical domain. This domain model can be applied to various natural language processing tasks in the vertical domain, including but not limited to information extraction (such as Named Entity Recognition (NER), Relation Extraction (RE), etc.), text classification, and text generation. Different types of tasks use different task prompt formats, which indicate the task requirements, input data, and output results to be performed by the domain model.

[0064] based on Figure 2 In the architecture shown, the server also runs downstream application systems that utilize the domain model of the vertical domain. During the execution of these downstream application systems, when a natural language processing task in the vertical domain needs to be performed, the server obtains the input data and the task prompt format information. Based on the task prompt format information and the input data, it generates task instructions, inputs these instructions into the domain model, and the domain model executes the task processing based on the instructions, generating task processing results and returning them to the downstream application systems. The server then continues to execute the subsequent processing logic of the downstream application systems based on the task processing results, thus implementing the functionality of the downstream application systems.

[0065] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0066] Figure 3 A flowchart of a task processing method provided for an exemplary embodiment of this application. The execution subject of this embodiment is... Figure 1 or Figure 2 The system architecture shown depicts a server. The method in this embodiment aims to further pre-train and fine-tune a pre-trained model based on the user's specific vertical domain, thereby obtaining a domain model suitable for that particular vertical domain. For example... Figure 3 As shown, the specific steps of this method are as follows:

[0067] Step S31: Use the mixed pre-training data of vertical and general domains to continue pre-training the pre-trained model to obtain the pre-trained domain basic model.

[0068] Among them, the pre-trained model can be a model applied to natural language processing, such as various pre-trained language models, pre-trained large language models, etc.

[0069] In this embodiment, the vertical domain refers to the specific domain in which the pre-trained model needs to be applied, based on the actual application scenario requirements. Examples include e-commerce, healthcare, transportation, and sports. The general domain is a concept relative to the vertical domain. A general domain can be understood as a relatively common or widely used domain with a large amount of existing corpus data. A general domain typically includes multiple domains and can cover various vertical domains.

[0070] Vertical domain pre-training data refers to data used for model pre-training, constructed based on data from a specific vertical domain. General domain pre-training data refers to data used for model pre-training, constructed based on data from a general domain.

[0071] The construction method for pre-training data in the general domain is the same as that for the training data used in the pre-training phase of the pre-trained model, and will not be repeated here. Furthermore, the pre-training data for the general domain can directly adopt the training data used in the pre-training phase of the pre-trained model.

[0072] The construction method of pre-training data in vertical domains is similar to that in general domains. The difference lies in the source of the data used. Pre-training data in vertical domains is built solely from data from the vertical domain, while pre-training data in general domains is built from data from the general domain. However, the construction process is the same, and can be consistent with the construction method of training data used in the pre-training stage of the pre-training model. It will not be elaborated here.

[0073] Hybrid pre-training data from vertical and general domains is obtained by combining pre-training data from both domains. Specifically, pre-training data from both domains can be merged into a single training set, which serves as the training set for further pre-training of the pre-trained model. This training set contains hybrid pre-training data from both domains.

[0074] Optionally, for the training set obtained by merging pre-training data from vertical and general domains, the order of the training data in the training set can be further shuffled to fully mix the pre-training data from vertical and general domains, which can improve the training effect of further pre-training of the pre-trained model in the domain.

[0075] It's important to note that when constructing mixed pre-training data from vertical and general domains, the proportion of vertical domain pre-training data should be neither too small nor too large. If the proportion is too small, the model won't learn enough knowledge from the vertical domain, affecting its text understanding and generation capabilities. Conversely, if the proportion is too large, the model won't retain its original strong semantic understanding and representation capabilities, also impacting its text understanding and generation capabilities. A good ratio is approximately 1 / 2 for vertical domain pre-training data, with a roughly 1:1 mix of vertical and general domain data. Using this mixed pre-training data for further pre-training allows the model to become familiar with the text formats and data distribution characteristics of the vertical domain while preserving its original strong semantic understanding and representation capabilities as much as possible. It also infuses the model with more vertical domain knowledge, thereby enhancing its text understanding and generation capabilities. In some embodiments, the training samples in the mixed pre-training data are unsupervised text, meaning the training samples are unlabeled sample text.

[0076] In this step, a hybrid pre-training data from both the vertical and general domains is used to further pre-train the pre-trained model, resulting in a pre-trained domain-based model. This further pre-training process can employ the same training methods as the initial pre-training process. Alternatively, this further pre-training step can utilize different training methods than the initial pre-training process.

[0077] In this step, mixed pre-training data from vertical and general domains can be used to perform any of the following pre-training tasks on the pre-trained model: Causal Language Modeling (CLM) task or masked language modeling task, so as to continue pre-training the pre-trained model and obtain a pre-trained domain-based model.

[0078] The causal language modeling task includes forward causal language modeling and backward causal language modeling. The mixed pre-training data contains multiple training samples, each containing multiple words (also called tokens). The forward causal language modeling task predicts the subsequent words in a training sample from the preceding words. The backward causal language modeling task predicts the preceding words in a training sample from the subsequent words. The masked language modeling task predicts the masked words in a training sample from the non-masked words in the training sample.

[0079] For example, in this step, mixed pre-training data from vertical and general domains can be used to perform causal language modeling pre-training tasks (such as forward causal language modeling pre-training tasks or reverse causal language modeling pre-training tasks) on the pre-trained model to further pre-train the pre-trained model and obtain a pre-trained domain-based model.

[0080] Taking the forward causal language modeling pre-training task as an example, the server samples training samples from the mixed pre-training data, performs word segmentation on the training samples to obtain each word in the training samples, determines the word vector corresponding to each word in the training samples, and encodes each word vector through the pre-trained model to obtain the encoding vector of each word vector. Further, for any word in the training samples, based on the encoding vector of the given word and the encoding vector of the historical words, the forward prediction result of that given word is predicted. That is, the forward prediction result of that given word is the word in the training samples that follows and is adjacent to that given word. The historical words are the words in the training samples that precede that given word. For example, the pre-trained model predicts the output of the first word X1 based on the Begin-Of-Sentence (BOS) marker in the training samples. Then, it predicts the output of the second word X2 based on the BOS and the first word X1, then the third word X3 based on the BOS, the first word X1, and the second word X2, and so on, until the last word or end-Of-Sentence (EOS marker) is output based on the BOS and all preceding words. Further, a forward pre-training loss function is determined based on the forward prediction results; this forward pre-training loss function characterizes the loss function for the forward causal language modeling pre-training task.

[0081] In this step, the pre-trained model is further pre-trained by using a mixture of pre-trained data from vertical and general domains. While maintaining the original strong semantic understanding and representation capabilities of the pre-trained model as much as possible, the model becomes familiar with the characteristics of text formats and data distribution in the vertical domain, and more knowledge from the vertical domain is injected into the model, thereby enhancing the model's ability to understand and generate text in the vertical domain.

[0082] Step S32: Using the dataset of the vertical domain, fine-tune the domain base model to obtain a domain model suitable for the vertical domain. The domain model is used to perform natural language processing tasks in the vertical domain and generate corresponding task processing results.

[0083] In this embodiment, to distinguish it from the pre-training data, the dataset used for fine-tuning training is referred to as the "fine-tuning dataset". Typically, the dataset used for pre-training does not contain labeled information, while the data used for fine-tuning training contains labeled information.

[0084] By using a mixture of pre-trained data from vertical and general domains to further pre-train the pre-trained model and obtain a pre-trained domain-based model, the domain-based model can be fine-tuned using a fine-tuning dataset from the vertical domain to obtain a domain model suitable for the vertical domain. Through fine-tuning the domain-based model, the final domain model can be better applied to natural language processing tasks in the vertical domain, thereby further improving the model's performance.

[0085] The method in this embodiment uses mixed pre-training data from vertical and general domains to further pre-train the pre-trained model, obtaining a pre-trained domain-based model. While maintaining the original powerful semantic understanding and representation capabilities of the pre-trained model as much as possible, it familiarizes the model with the characteristics of text formats and data distribution in the vertical domain and injects more knowledge from the vertical domain into the model, thereby enhancing the model's text understanding and generation capabilities in the vertical domain. Furthermore, the domain-based model is fine-tuned using a fine-tuning dataset from the vertical domain to obtain a domain model suitable for the vertical domain. This allows the final domain model to be better applied to natural language processing tasks in the vertical domain, further improving the model's performance and thus greatly enhancing the quality of task processing results generated when the model is applied to various NLP tasks in the vertical domain.

[0086] In an optional embodiment, before using the hybrid pre-training data from the vertical and general domains in step S31 to further pre-train the pre-trained model and obtain the pre-trained domain base model, the server pre-constructs hybrid pre-training data from the vertical and general domains. Specifically, this can be achieved as follows: constructing pre-training data from the vertical domain, and then mixing the pre-training data from the vertical domain with the pre-training data from the general domain according to a preset mixing method to obtain hybrid pre-training data. The preset mixing method is a pre-configured method used to mix the pre-training data from the vertical and general domains. In this embodiment, multiple available mixing methods and their corresponding specific mixing rules are pre-configured. In practical applications, users can select one of the mixing methods as the preset mixing method according to their application needs.

[0087] For example, when a user triggers the selection of a blending method, the server displays the configured blending methods and their corresponding blending rules through the blending method configuration interface. Users can browse the configured blending methods and their corresponding rules through the interface and select a blending method suitable for their application needs as the preset blending method used in the model training process to obtain the domain model for their vertical field. Users can trigger a selection operation by selecting a blending method. In response to the selection operation of any blending method, the server uses the selected blending method as the currently used preset blending method. The selection operation can be triggered by the user clicking the "Select" control on the interface, or by the user clicking on a blending method displayed on the interface. Alternatively, the selection operation can be implemented in other ways that have a similar function to selecting an option; no specific limitations are made here.

[0088] In this embodiment, users are supported in editing and updating the mixing rules corresponding to the configured mixing methods. Specifically, the server displays the configured mixing methods and the corresponding mixing rules through the mixing method configuration interface. Users can edit the mixing rules corresponding to one or more of the displayed mixing methods in the mixing method configuration interface to modify the specific mixing rules for each mixing method.

[0089] In response to an edit operation on the blending rule corresponding to any blending method, the server obtains the new blending rule for the edited blending method and updates the blending rule corresponding to the blending method. Furthermore, the user can select the blending method after the update of the blending rule as the preset blending method used in the model training process to obtain the domain model of the vertical domain.

[0090] Optionally, a typical mixing method is as follows: Pre-trained data from the vertical domain is mixed with pre-trained data from the general neighborhood according to a preset mixing ratio to obtain hybrid pre-trained data. The mixing rules for this method configure a preset mixing ratio. This preset mixing ratio is a pre-configured ratio for mixing the pre-trained data from the vertical and general domains. The specific ratio value can be configured and adjusted according to the needs of the actual application scenario, and is not specifically limited here. A preset mixing ratio of 1:1 or close to 1:1 can achieve better training results. The specific value can be configured and adjusted according to the needs of the actual application scenario, and is not specifically limited here. For example, the preset mixing ratio can be configured as 1:1, 2:3, 2:1, etc. In addition, the server sets a default value for the preset mixing ratio, which can be adjusted and updated by technical personnel according to the needs of the actual application scenario. The default value for the preset mixing ratio can be 1:1 or other values. Furthermore, based on different mixing ratios, multiple mixing methods can be set according to the ratio, allowing users to choose a mixing ratio suitable for their application needs.

[0091] In practical applications, a vertical domain can encompass multiple sub-domains. For example, a vertical domain could be the e-commerce domain, including sub-domains such as product search, product comparison, image search, and short video display. The pre-training data for a vertical domain includes pre-training data from multiple sub-domains. Another typical hybrid approach is to combine the pre-training data from the corresponding sub-domain with pre-training data from the general domain, based on the application domain of the domain model, to obtain hybrid pre-training data. This hybrid approach can enhance the data features of the sub-domains, enabling the trained domain model to better learn these features and better suit its application in specific domains.

[0092] For example, if a user can determine that the specific application domain of the domain model in the e-commerce field is the sub-domain of product comparison, then in the pre-training stage of the domain model, a mixture of pre-training data from the product comparison sub-domain and pre-training data from the general domain can be used to enable the model to learn the data characteristics of the product comparison sub-domain, thereby improving the performance and generation quality of the obtained domain model when applied to the product comparison task.

[0093] In one alternative embodiment, pre-trained data for a vertical domain can be constructed in the following manner:

[0094] Step S1: Collect various types of sample data in the vertical field.

[0095] In this step, a large amount of sample data is collected based on the vertical domain in which the model is applied. Depending on the task performed by the domain model, the modality of the sample data input to the model can vary. For multimodal tasks, the model input data typically includes samples of multiple modalities, including but not limited to text, images, videos, and code. Specifically, this step can collect sample data of one or more modalities from the vertical domain, based on the modality of the input data for the task performed by the model. Different types of sample data refer to the source type of the sample data, such as sample data from different data platforms, with different data platforms corresponding to different data types.

[0096] Step S2: Perform data cleaning on various types of sample data to obtain various types of input samples in the vertical field.

[0097] In this step, data cleaning is performed on various types of sample data collected from the vertical domain to obtain high-quality sample data as input samples, thereby ensuring the quality of the input samples. Specifically, data cleaning of various types of sample data includes at least filtering and deduplication of the sample data. In addition, other commonly used data cleaning methods can also be used to improve the quality of sample data and obtain higher-quality input samples; this embodiment does not specify any particular method. Furthermore, different filtering rules can be used for sample data of different modalities. The specific rules can be set and adjusted according to the needs of the actual application scenario; this is not specifically limited here. For example, for images, images containing text content that does not meet the requirements can be filtered out, or images whose colors do not meet the preset requirements can be filtered out, etc. The text content that does not meet the requirements and the images that do not meet the preset requirements can be configured according to the actual application scenario; this is not specifically limited here.

[0098] In one optional embodiment, sampling weights for different types of input samples can be configured. The sampling weights for different types of input samples can be determined based on the quality and size of each type of input sample. Specifically, these weights can be configured and adjusted by relevant technical personnel based on the quality and / or size of each type of input sample, combined with empirical values; no specific limitations are imposed here.

[0099] For example, if the quality of a certain type of input sample is high, a larger sampling weight is assigned to that type of input sample; if the quality of a certain type of input sample is low, a smaller sampling weight is assigned to that type of input sample. This can improve the quality of input samples participating in further pre-training. If the size of a certain type of input sample is large, a smaller sampling weight is assigned to that type of input sample; if the size of a certain type of input sample is small, a larger sampling weight is assigned to that type of input sample. This can improve the diversity of input samples participating in further pre-training.

[0100] When using various types of input samples from a vertical domain to further pre-train the pre-trained model, batch sampling is performed based on the sampling weights of different types of input samples. The sampled input samples are then used to perform an iterative training of the pre-trained model. Input samples with larger sampling weights are more likely to be included in further pre-training, while input samples with smaller sampling weights are less likely to be included. This improves the quality and diversity of input samples used for further pre-training, thereby enhancing the model's training performance.

[0101] In one optional embodiment, using a pre-trained model as a pre-trained language model, during the continued pre-training phase, a causal language modeling pre-training task (such as a forward causal language modeling pre-training task or a backward causal language modeling pre-training task) is performed on the pre-trained model using mixed pre-training data from vertical and general domains. This allows for further pre-training of the pre-trained model to obtain a pre-trained domain-based model. For example, in this scenario... Figure 4 As shown, the specific steps for constructing pre-trained data for a vertical domain are as follows:

[0102] Step S41: Collect various types of text data in the vertical field.

[0103] In this step, based on the vertical domain of the model application, a large amount of text data is collected within that vertical domain. Specifically, one or more types of unlabeled text data can be collected, including but not limited to: news, books, product information, user interaction behavior data, promotional copy, and user-generated content (UGC) within the vertical domain. Product information includes relevant knowledge about various products within the vertical domain, such as product attributes, usage methods, uses, and functional descriptions. User interaction behavior data refers to data generated based on user interactions, such as search behavior data and conversation data. User-generated content is user-created content, including text posted by users, such as comments, updates, articles, press releases, etc.

[0104] In addition, when collecting various types of text data in a vertical field, there is no limitation on the language type of the text data. Text data in multiple languages ​​can be collected, including but not limited to Chinese, English and other languages.

[0105] Step S42: Perform data cleaning on various types of text data to obtain various types of text samples in the vertical field.

[0106] In this step, the collected text data of various types in the vertical field is cleaned to obtain high-quality text data as text samples, thereby ensuring the quality of the text samples.

[0107] Specifically, data cleaning is performed on various types of text data, including at least filtering and deduplication. Additionally, other commonly used data cleaning methods can be used to improve the quality of text data and obtain higher-quality text samples; this embodiment does not impose specific limitations on these methods.

[0108] For example, the server can filter various types of text data to remove low-quality text data that does not meet the requirements. Specifically, this includes, but is not limited to, filtering out text that is too short or too long, text with an excessive proportion of numbers, text containing a large number of special symbols, and text with abnormal sentence breaks and punctuation.

[0109] Optionally, the server can filter various types of text data according to preset filtering rules to remove low-quality text data that does not meet the requirements. The filtering rules can be configured and adjusted according to the actual application scenario; no specific limitations are specified here.

[0110] For example, a filtering rule could be: length greater than the upper length limit or length less than the lower length limit. By filtering out text data that meets this rule, text data that is too short or too long can be filtered out. The specific values ​​of the upper and lower length limits can be configured and adjusted according to the needs of the actual application scenario and empirical values, and are not specifically limited here.

[0111] For example, another filtering rule could be: the proportion of numbers exceeds a preset threshold for the proportion of numbers. By filtering out text data that meets this rule, text data with an excessively high proportion of numbers is filtered out. The preset threshold for the proportion of numbers can be configured and adjusted according to the needs of the actual application scenario and empirical values; no specific limitations are imposed here.

[0112] For example, defining a set of special symbols, another example of a filtering rule could be: the number of special symbols appearing in the set is greater than or equal to a preset threshold. By filtering out text data that meets this filtering rule, text containing a large number of special symbols can be filtered out. Alternatively, a filtering rule could be set as: the proportion of special symbols is greater than or equal to a preset symbol proportion threshold. This filtering rule can also effectively filter out text containing a large number of special symbols. The set of special symbols, the preset threshold, and the preset symbol proportion threshold can be configured and adjusted according to the needs of the actual application scenario and empirical values; no specific limitations are imposed here.

[0113] Optionally, the server can train a text classification model based on preset filtering rules. This model is used to identify the category corresponding to the text data. Categories include: categories that do not satisfy the filtering rules, and categories corresponding to each filtering rule. For example, if m (m is a positive integer) filtering rules are set, then m+1 different categories can be set, including the category that does not satisfy the filtering rules, and the categories corresponding to each of the m filtering rules. The server trains a text classification model to predict the category (one of the m+1 categories) corresponding to the input text data. By inputting each piece of text data into the text classification model, the text classification module can identify the category corresponding to the text data. Based on the category corresponding to the text data, if the text data corresponds to the category of any filtering rule, that is, if the text data satisfies any filtering rule, then the text data is filtered out.

[0114] For example, given any text data, if a text classification model is used to classify and identify the text data, and based on the classification results, if it is determined that the text data meets the filtering rule of containing abnormal sentence breaks and punctuation, then the text is filtered out. This allows filtering out text data containing abnormal sentence breaks and punctuation.

[0115] Optionally, data cleaning for various types of text data also includes removing invisible characters and invalid text from each type of text data. Invisible characters, also known as control characters or non-printable characters, are characters that cannot be displayed in the text, such as spaces, tabs, carriage returns, and line feeds. Invalid text refers to pre-defined content that is not part of the original text content and may affect the natural language understanding of the text. For example, repeating prefixes added at the beginning of the text, such as the filename of the source file. For instance, an invisible character set and an invalid text set can be predefined, and content appearing in these sets can be removed from the text data. The specific content included in the invisible character set and invalid text set can be configured and adjusted according to the needs and experience of the actual application scenario; no specific limitations are made here.

[0116] Since duplicate data can affect the model training process, this embodiment performs deduplication on each type of text data during data cleaning. Specifically, this can be achieved as follows: generate text segments of a preset length for each type of text data, where the length of a text segment refers to the number of words (also called tokens) contained in the text segment; calculate the summary information of the text segments for each type of text data; and determine duplicate text data based on the degree of overlap in the summary information of the text segments from different types of text data, and then perform deduplication on the duplicate text data.

[0117] In this context, any text data may contain text segments of a preset length, which can be text segments of length n generated by an n-gram algorithm (i.e., n-grams). Each text segment consists of a sequence of n consecutive words from the text data, where n is the preset length of the text segment. This length can be configured according to the needs of the actual application scenario; for example, n can be 1, 2, 3, etc., without specific limitations. A set of summary information for multiple text segments of the text data is obtained by calculating the summary information of each text segment using a digest algorithm. The degree of overlap between the sets of text segment summary information of different text data is used as the similarity between different text data, and text data with a high degree of overlap are identified as duplicate text data. The digest algorithm can be a Secure Hash Algorithm (SHA), an MD5 Message-Digest Algorithm (MD5), etc.

[0118] For example, for sets A and B of text segment summary information for any two text data T1 and T2, assuming text data T1 contains s1 text segments and text data T2 contains s2 text segments, then set A contains summary information for s1 text segments, and set B contains summary information for s2 text segments. Assume the number of identical summary information in sets A and B is s, where min{s1,s2}≤s≤max{s1,s2}, where min{s1,s2} is the minimum of s1 and s2, and max{s1,s2} is the maximum of s1 and s2. The degree of overlap between sets A and B can be determined as s / (s1+s2-s). When the degree of overlap between sets A and B is greater than or equal to the overlap threshold, text data T1 and T2 are determined to be duplicate text data. The overlap threshold can be configured and adjusted according to the needs of the actual application scenario and empirical values; no specific limitations are made here.

[0119] Optionally, the degree of overlap between set A and set B can also be calculated as follows: s / (s1+s2). Different overlap calculation methods can be used to set different overlap thresholds. Optionally, deduplication of text data of various types can be performed, specifically by comparing whether two pieces of text data are completely identical to determine duplicate text data. In addition, other existing methods for determining whether two different texts belong to duplicate text can also be used to determine duplicate text data in various types of text data collected in the vertical field. This embodiment does not specifically limit this method.

[0120] Optionally, for text data identified as duplicates, one text data point can be randomly selected and retained, while other text data that duplicates the retained text data can be deleted. Alternatively, when removing duplicate text data, a duplicate relationship graph can be constructed based on the duplicate relationships between the text data. Each text data point corresponds to a node in the graph, and an edge between two nodes indicates that the text data corresponding to the two nodes are duplicates. By identifying connected components (nodes containing mutually duplicate text data) in the duplicate relationship graph, for each connected component, only one randomly selected node is retained, and the other nodes within the connected component are deleted. Simultaneously with deleting a node, the text data corresponding to that node is also deleted. This allows for the retention of only one text data point from several mutually duplicated text data points while deleting all others.

[0121] Step S43: Concatenate text samples of the same type, and divide the concatenation result into multiple input samples based on the preset maximum input length to construct input sample sets of various types. The pre-training data of the vertical domain includes input sample sets of various types in the vertical domain.

[0122] After cleaning the text data of various types in the vertical domain, each cleaned text data is used as a text sample for further pre-training.

[0123] In this step, for any type of text sample, all text samples of that type are concatenated. The concatenated result is then segmented into multiple segments according to the model's preset maximum input length. Each segment serves as an input sample, constructing the input sample set for that type. The length of each segmented input sample is equal to or less than the preset maximum input length. The preset maximum input length can be set based on the maximum input length the model can accept. By concatenating multiple text samples of the same type and segmenting the input samples according to the model's preset maximum input length, the efficiency of further pre-training the pre-trained model can be improved by increasing the length of the input samples in the pre-training data.

[0124] Optionally, for any type of text sample, when concatenating all text samples of that type, an end-of-words symbol (EOS) can be added to the end of each text sample before concatenating the text samples with the end-of-words symbol, thus using the end-of-words symbol to separate different text samples. When performing the causal language modeling pre-training task, the end-of-words symbol (EOS) in the input samples is treated as a word segmentation. The pre-trained model predicts the output of the first word X1 based on the beginning-of-words symbol (BOS) in the training samples, then predicts the output of the second word X2 based on the BOS and the first word X1, then predicts the output of the third word X3 based on the BOS, the first word X1, and the second word X2, and so on, until the last word is output based on the BOS and all the preceding words.

[0125] In this embodiment, through steps 41-S43, various types of text data in the vertical domain are collected, and the various types of text data are cleaned to obtain high-quality text samples of various types in the vertical domain. Furthermore, multiple text samples of the same type are concatenated, and the input samples are segmented according to the preset maximum input length of the model. This can increase the length of the input samples in the pre-training data, thereby improving the efficiency of continuing to pre-train the pre-trained model.

[0126] Step S44: Obtain pre-trained data for the general neighborhood.

[0127] Optionally, the pre-training data for the general domain can directly use the training data used in the pre-training stage of the pre-trained model. Optionally, the server can also collect various types of text data in the general domain, and based on these various types of text data, construct pre-training data for the general neighborhood using a method similar to steps S42-S43 above. For specific implementation details, please refer to the content of steps S42-S43 above, which will not be repeated here.

[0128] This involves collecting various types of text data from a general domain. Specifically, this can be done by collecting unlabeled text data of various types from a general domain, including but not limited to: encyclopedic knowledge, books, blogs, news, Q&A data, and other texts of different types / sources. When collecting various types of text data from a general domain, there is no limitation on the language used; text data in multiple languages ​​from a general domain can be collected, including but not limited to Chinese, English, and other languages.

[0129] Step S45: According to the preset mixing method, mix the pre-trained data of the vertical domain with the pre-trained data of the general neighborhood to obtain mixed pre-trained data.

[0130] For example, in this embodiment, the preset mixing method is to mix the pre-trained data of the vertical domain with the pre-trained data of the general neighborhood according to a preset mixing ratio. The preset mixing ratio is a pre-configured mixing ratio of the pre-trained data of the vertical domain and the pre-trained data of the general neighborhood.

[0131] When constructing hybrid pre-training data from vertical and general domains, the proportion of pre-training data from the vertical domain should not be too small or too large. If the proportion of pre-training data from the vertical domain is too small, the model will not learn enough knowledge from the vertical domain, affecting the model's text understanding and generation capabilities in the vertical domain. If the proportion of pre-training data from the vertical domain is too large, the model will not be able to maintain the original strong semantic understanding and representation capabilities of the pre-trained model, which will also affect the model's text understanding and generation capabilities in the vertical domain. The proportion of pre-training data from the vertical domain is about 1 / 2, that is, the vertical domain and general domain are mixed in a ratio of about 1:1. The resulting hybrid pre-training data is used to further pre-train the pre-trained model. This can, while maintaining the original strong semantic understanding and representation capabilities of the pre-trained model as much as possible, allow the model to become familiar with the characteristics of text formats and data distribution in the vertical domain, and inject more knowledge from the vertical domain into the model, thereby enhancing the model's text understanding and generation capabilities in the vertical domain. In this embodiment, a preset mixing ratio of 1:1 or close to 1:1 can achieve better training results. The specific value can be configured and adjusted according to the needs of the actual application scenario, and no specific limitation is made here. For example, the preset mixing ratio can be configured as 1:1, 2:3, 2:1, etc.

[0132] In an optional embodiment, when using the vertical domain dataset to fine-tune the domain base model in step S32, any existing method of using labeled data from the vertical domain to fine-tune the pre-trained model can be used, which will not be elaborated here.

[0133] In another optional embodiment, when fine-tuning the domain base model using the vertical domain dataset in step S32, the server can use the constructed vertical domain multi-task instruction dataset to fine-tune the domain base model, obtaining a domain model suitable for the vertical domain. The multi-task instruction dataset contains instruction data for various tasks. Instruction data refers to the data used in the fine-tuning training phase to generate task instructions for the input model, including task prompt format information, input information, and output results. Different tasks use different task prompt format information. The task prompt format information is used to instruct the model to execute the corresponding task processing procedure to obtain the task processing result. Specifically, such as... Figure 5 As shown, the server can construct a multi-task instruction dataset for a vertical domain through the following steps S51-S53:

[0134] Step S51: Obtain instruction data for various existing tasks in the vertical field. The instruction data includes task prompt format information, input information, and output results. Different tasks use different task prompt format information.

[0135] In this step, labeled training data for existing tasks in the vertical domain can be obtained by collecting publicly available fine-tuning datasets from the vertical domain used for fine-tuning pre-trained models for various tasks (the aim is to obtain models suitable for specific tasks). In this embodiment, tasks with pre-built fine-tuning datasets are referred to as existing tasks.

[0136] Furthermore, for labeled training data of various existing tasks in vertical domains, the format of instruction data for various tasks is standardized, including: task prompt format information, input information, and output results. The task prompt format information is used to instruct the model to execute the corresponding task processing procedure to obtain the task processing result. The input information includes one or more inputs to the task, and the output result is the labeled task processing result.

[0137] For example, the task prompt format information for various tasks includes at least the following items: Prompt, Input, and Output. The Prompt indicates the task requirements / purpose. The Input indicates which information items are included in the input. The Output indicates the information items that the model needs to generate, i.e., the information items included in the task processing result. Additionally, the task prompt format information for some tasks may also include pre-set example information, such as example information for the input items, example information for the target items, etc., providing the model with reference examples.

[0138] For example, common natural language processing tasks can be divided into three main categories: information extraction tasks, text classification tasks, and text generation tasks. Information extraction tasks include named entity recognition (detecting named entities in a given text and assigning their classification), entity detection (detecting named entities in a given text), attribute detection and extraction tasks (detecting information describing entity attributes in a given text, such as the origin of a product), and question-answering tasks related to a given entity (such as a product). Text classification tasks include entity classification tasks (identifying the entity's classification category given text related to an entity, such as its name, attributes, and description), sentiment classification tasks (classifying the sentiment of a given text, such as the sentiment classification of user comments), and dialogue intent classification tasks (identifying the user's intent based on the dialogue content). Text generation tasks include title generation (generating titles for products based on their attributes and descriptions), copywriting generation (generating promotional copy for products based on their attributes and descriptions), dialogue response generation (generating answers to input questions), and user query rewriting (rewriting user queries in a search system).

[0139] In this embodiment, labeled training data of various natural language processing tasks in the vertical domain can be collected, and instruction data of existing tasks can be constructed based on a unified instruction data format.

[0140] In one alternative implementation, the task prompt format information can be configured based on the chain of thought (CoT), combining input items, output items, and intermediate reasoning steps. This not only incorporates the final output item into the prompt but also incorporates the intermediate reasoning steps, which can effectively improve the quality of the task processing results and help users understand the task processing results.

[0141] For example, taking the product comparison task in the e-commerce field as an example, the mind chain-based method can configure the following task prompt format information:

[0142] Hint: We need to perform a product comparison function. Based on the titles and attributes of two products, we need to summarize the characteristics, similarities and differences between the two products, describe the advantages of the two products in terms that consumers can understand, and finally provide recommendations based on different applicable scenarios.

[0143] The input items are as follows:

[0144] Product 1: {Title, attributes, and price of Product 1}

[0145] Product 2: {Title, attributes, and price of Product 1}

[0146] The output items and output format are as follows:

[0147] Characteristics of Product 1: {Characteristics of Product 1}.

[0148] Characteristics of Product 2: {Characteristics of Product 2}.

[0149] Similarities: {Similarities between the two products}.

[0150] Differences: {Differences between the two products}.

[0151] The advantages of product 1 are {advantages of product 1}.

[0152] The advantages of product 2 are {advantages of product 2}.

[0153] Recommendations: {Purchase suggestions based on different applicable scenarios}.

[0154] Based on this task prompt format, the prompt includes the intermediate inference process of the model performing product comparison. The input items indicate the information required for comparing product 1 and product 2; this example illustrates this by including the product information as title, attributes, and price. The output format specifies that the output items include the characteristics of product 1, the characteristics of product 2, the similarities between the two products, the differences between the two products, the advantages of product 1, the advantages of product 2, and a recommendation. Thus, the task processing result provided by the model not only includes the final recommendation but also the intermediate comparison process between the two products. Furthermore, based on the model's output task processing result, the corresponding parts are extracted and displayed to the user using methods such as regular expressions.

[0155] Step S52: For instruction data of at least one existing task, generate instruction data for a new task according to the corresponding task transformation rules.

[0156] After constructing the instruction data for existing tasks, considering that the number of existing tasks with labeled training data in the vertical domain may be relatively small, this step involves transforming at least one existing task to obtain new tasks, and transforming the instruction data of at least one existing task to obtain instruction data for new tasks. In this way, through data augmentation, the task types are increased, more task instruction data is obtained, and the diversity and volume of multi-task instruction datasets in the vertical domain are improved.

[0157] Optionally, at least one task simplification rule corresponding to an existing task can be pre-configured. Based on the task simplification rule, new tasks of varying difficulty can be constructed by adjusting the input and / or output information of the existing task. In this step, the server can simplify the input information and / or output results in the instruction data of the existing task according to the task simplification rule corresponding to the existing task, based on the instruction data of at least one existing task, to obtain the input information and output results of the new task, and configure the task prompt format information of the new task.

[0158] For example, the named entity recognition task identifies named entities contained in input text and assigns the corresponding classification category (i.e., entity type) to each named entity. By adjusting the output of the named entity recognition task to include only the named entities contained in the input text, the task is simplified to an entity detection task. Furthermore, by adjusting the input to the names of the named entities and the output to the corresponding classification categories, the task is simplified to an entity classification task.

[0159] The task simplification rules for existing tasks and the task prompt format information for new tasks can be configured by relevant technical personnel according to the existing and new tasks in the actual application scenario, and no specific restrictions are imposed here.

[0160] Optionally, at least one task reversal rule corresponding to an existing task can be pre-configured. Based on the task reversal rule, a new task is constructed by reversing the input and output of the existing task. In this step, the server can, based on the instruction data of at least one existing task and according to the task reversal rule corresponding to the existing task, use the output of the existing task as the input information of the new task and the input information of the existing task as the output of the new task to construct the input information and output of the new task, and configure the task prompt format information of the new task.

[0161] For example, given existing question-and-answer tasks, we can reverse the question and answer structure of a new task by using them as the answer and question for that task, respectively. For instance, consider an existing task that generates product description text from product titles. The output of this task is the product description text. By reversing the input and output of the existing task, we can create a new task that generates product description text from product titles. The input of this new task is the product title, and the output is the product description text.

[0162] The task reversal rules for existing tasks and the task prompt format information for new tasks can be configured by relevant technical personnel according to the existing and new tasks in the actual application scenario, and no specific restrictions are imposed here.

[0163] Optionally, at least one task reorganization rule corresponding to an existing task can be pre-configured. Based on the task reorganization rule, a new task is constructed by splitting and reorganizing the input and output of the existing task. In this step, the server can, based on the instruction data of at least one existing task, split and reorganize the input information and output results of the existing task according to the task reorganization rule corresponding to the existing task to construct the input information and output results of the new task, and configure the task prompt format information of the new task.

[0164] For example, an existing task could be a product matching task, where the input is the titles and attributes of two products, and the output is the matching result of whether the two products are the same product. By splitting and recombining the input and output of an existing task, a new product title-attribute matching task can be constructed: the input is the title of a given product, and two attributes; the output is the attribute that matches the given product title.

[0165] The task reorganization rules for existing tasks and the task prompt format information for new tasks can be configured by relevant technical personnel according to the existing and new tasks in the actual application scenario, and no specific restrictions are imposed here.

[0166] Optionally, for instruction data of at least one existing task, a large model can be used to perform corresponding task processing based on the task prompt format information and input information in the instruction data, obtain the task processing result, and use the task processing result as the labeled output to construct new instruction data, which can increase the amount of instruction data.

[0167] Step S53: Based on the instruction data of existing tasks and new tasks, construct a multi-task instruction dataset for the vertical domain.

[0168] Based on the instruction data of the existing tasks constructed in step S51 and the instruction data of the new tasks constructed in step S52, a multi-task instruction dataset for the vertical domain is constructed. The multi-task instruction dataset for the vertical domain includes both the instruction data of the existing tasks constructed in step S51 and the instruction data of the new tasks constructed in step S52.

[0169] The method in this embodiment acquires instruction data for multiple existing tasks in a vertical domain, and generates instruction data for new tasks based on the corresponding task transformation rules for instruction data of at least one existing task. This increases the scale of instruction data used for fine-tuning training and the diversity of task types covered, thereby improving the effect of fine-tuning training, the performance of the trained domain model in the vertical domain, and the quality of the generated task processing results.

[0170] Furthermore, in step S32, the domain base model is fine-tuned and trained using the constructed multi-task instruction dataset of the vertical domain to obtain a domain model suitable for the vertical domain.

[0171] Specifically, instruction data is sampled from a multi-task instruction dataset within a vertical domain. Based on the task prompt format information and input information in the instruction data, task instructions containing input information and conforming to the task prompt format are generated. The domain-based model then executes the task processing according to the task instructions to obtain the task processing result. Based on the task processing result and the labeled output results in the instruction data, the loss function is calculated, and the parameters of the domain-based model are updated. This trains the model's ability to generate output results based on task prompt format information and input information, enabling iterative fine-tuning of the domain-based model. Fine-tuning the domain-based model using a multi-task instruction dataset within a vertical domain can improve the model's performance and generalization ability across various tasks within that vertical domain.

[0172] The task prompt format information can be understood as a template for task instructions. By substituting the input information into this template, task instructions containing the input information can be generated according to the format requirements of the task prompt format information. The method for generating task instructions can refer to existing methods for generating model input instructions based on a given prompt format / template and input information, which will not be elaborated here.

[0173] For example, Figure 6 This is a schematic diagram illustrating the fine-tuning training of multi-task instructions for the domain foundation model provided in this application embodiment. For example... Figure 6 As shown, fine-tuning training is performed using instruction datasets based on two tasks: named entity recognition and dialogue intent detection. Figure 6 Examples of task instructions for these two tasks are provided. For the Named Entity Recognition (NER) task, the instructions include: Prompt (a prompt in the task prompt format information) instructing the model to extract all named entities related to attributes, brands, components, and product types from the input text; Input is the input text; and Output is the part to be generated by the model. By inputting the task instructions into the domain-based model, the model generates and outputs the NER recognition results. For the Dialogue Intent Detection (DIC) task, the instructions include: Prompt (a prompt in the task prompt format information) instructing the model to classify based on the intent of the last sentence in the input dialogue text, and to select the output from candidate intent categories; Input is the input dialogue text; Candidate Labels are the candidate intent categories; and Output is the part to be generated by the model. By inputting the task instructions into the domain-based model, the model generates and outputs the DIC detection results. By fine-tuning the domain-based model using instruction datasets for various tasks, the trained domain model can be applied to various tasks in a vertical domain, making it a multi-task general-purpose model.

[0174] Figure 7 This is a framework diagram for obtaining a domain model of a vertical domain based on a pre-trained model, provided as an exemplary embodiment of this application. In this embodiment, as shown... Figure 7As shown, taking the e-commerce domain as a vertical neighborhood, this paper illustrates the process of obtaining a domain-specific large language model for e-commerce. Based on a general pre-trained large language model, unlabeled pre-training data from the e-commerce domain and unlabeled pre-training data from the general domain are used together to further pre-train the pre-trained large language model, obtaining a basic large language model for e-commerce. While maintaining the original powerful semantic understanding and representation capabilities of the pre-trained large language model as much as possible, the model is made familiar with the characteristics of text formats and data distribution in the e-commerce domain, and more e-commerce domain knowledge is injected into the model, thereby enhancing the model's text understanding and generation capabilities in the e-commerce domain. Furthermore, a fine-tuning training set using instruction data from various tasks in the e-commerce domain (such as information extraction, text classification, and text generation) is used to fine-tune the basic large language model for e-commerce, obtaining a large language model for e-commerce. This final e-commerce domain large language model is better suited for natural language processing tasks in the e-commerce domain, further improving the model's performance and significantly enhancing the generation quality of the large language model when performing various NLP tasks in the e-commerce domain.

[0175] Figure 8 A flowchart illustrating a task processing method in the e-commerce field provided as an exemplary embodiment of this application. The execution entity in this embodiment is... Figure 1 or Figure 2 The system architecture shown depicts a server. The method in this embodiment aims to further pre-train and fine-tune a pre-trained large model to obtain a large e-commerce model suitable for the e-commerce domain. For example... Figure 8 As shown, the specific steps of this method are as follows:

[0176] Step S81: Use the mixed pre-trained data from the e-commerce domain and the general domain to continue pre-training the pre-trained large model to obtain the pre-trained basic large model for the e-commerce domain.

[0177] The pre-trained large model can be a large model applied to natural language processing, such as various pre-trained large language models. In this step, the e-commerce domain is used as the vertical domain. Mixed pre-training data from the e-commerce domain and the general domain are used to further pre-train the large model. This allows the large model to become familiar with the text format and data distribution characteristics of the e-commerce domain, and to inject more knowledge from the vertical domain into the large model, thereby enhancing the large model's text understanding and generation capabilities in the e-commerce domain.

[0178] Specifically, by constructing pre-trained data in the e-commerce domain, and then mixing the pre-trained data in the e-commerce domain with pre-trained data in the general neighborhood according to a preset mixing method, hybrid pre-trained data is obtained.

[0179] The specific implementation of this step is similar to that of the aforementioned step S31. Please refer to the relevant content in the aforementioned embodiments for details, which will not be repeated here.

[0180] Step S82: Using a dataset from the e-commerce domain, fine-tune and train the basic large model in the e-commerce domain to obtain a large model in the e-commerce domain. The large model in the e-commerce domain is used to perform natural language processing tasks in the e-commerce domain and generate corresponding task processing results.

[0181] After further pre-training the pre-trained large model using a mixture of e-commerce and general domain pre-trained data to obtain a basic pre-trained e-commerce domain model, this basic model can be fine-tuned using an e-commerce domain dataset to obtain a large e-commerce domain model suitable for e-commerce. Through this fine-tuning, the final e-commerce domain model is better suited for natural language processing tasks in the e-commerce domain, thereby further improving the model's performance. The specific implementation of this step is similar to that of step S32 described above; please refer to the relevant content in the aforementioned embodiments for details, which will not be repeated here.

[0182] The method in this embodiment uses mixed pre-trained data from the e-commerce domain and the general domain to further pre-train a large pre-trained model, obtaining a basic large pre-trained model for the e-commerce domain. While maintaining the original powerful semantic understanding and representation capabilities of the large pre-trained model as much as possible, it familiarizes the large model with the characteristics of text formats and data distribution in the e-commerce domain, and injects more knowledge from the e-commerce domain into the large model, thereby enhancing the large model's text understanding and generation capabilities in the e-commerce domain. Furthermore, the basic large model is fine-tuned using a fine-tuning dataset from the e-commerce domain to obtain a large e-commerce domain model suitable for the e-commerce domain. This allows the final e-commerce domain model to be better applied to natural language processing tasks in the e-commerce domain, further improving the performance of the large model and thus greatly improving the generation quality of task processing results when the large model is applied to various NLP tasks in the e-commerce domain.

[0183] Figure 9 This is a flowchart illustrating task processing based on a domain model of a vertical domain, provided as an exemplary embodiment of this application. After obtaining the domain model of the vertical domain using any of the foregoing method embodiments, in this embodiment, a natural language processing task in the vertical domain is executed based on the domain model to obtain the task processing result.

[0184] It should be noted that the domain model of the vertical domain obtained based on the foregoing embodiments can be deployed locally on a server or on a server of another organization or system platform. The execution entity in this embodiment is a server running the domain model of the vertical domain, which also runs downstream application systems capable of using the domain model.

[0185] like Figure 9 As shown, the specific steps for implementing task processing based on a domain model of a vertical domain are as follows:

[0186] Step S91: In response to the task processing request of the natural language processing task in the vertical domain, obtain the input data and the task prompt format information of the natural language processing task.

[0187] In this embodiment, when the application system running on the server needs to perform a natural language processing task in a vertical domain, it submits a task processing request to the server. This task processing request includes input data and task category information for the natural language processing task. The task category information differs for different natural language processing tasks.

[0188] In practical applications, downstream application systems may generate various natural language processing (NLP) tasks during operation, including but not limited to information extraction tasks, text classification tasks, and text generation tasks. Different NLP tasks have different task categories. Based on the task category, the corresponding NLP task can be determined, and then the task prompt format information for the corresponding NLP task can be obtained.

[0189] For example, taking an e-commerce system as an example, when a user inputs a query text, the e-commerce system might generate a product recognition task: Based on the user's query text, it extracts entities of various categories, such as products, product attributes, and brands, and determines their corresponding entity categories. After obtaining the product recognition results, the e-commerce system searches for knowledge documents matching the current product and generates a question-and-answer task: based on the searched knowledge documents, it generates a response to the user's query text. After obtaining the response, the e-commerce system outputs the response information.

[0190] In addition, in other scenarios and fields within the e-commerce sector, there are many application systems that generate various natural language processing tasks. The solution in this embodiment uses a general domain model to perform various natural language processing tasks in that vertical domain, which will not be listed here.

[0191] Step S92: Generate task instructions based on the task prompt format information and input data.

[0192] After obtaining the input data and task prompt format information for the natural language processing task to be executed, the server generates task instructions containing the input data based on the task prompt format information. For example, for a product recognition task, the input data is the user-inputted consultation text. Substituting the input data into the input fields of the task prompt format information generates the task instructions for the product recognition task, such as... Figure 6 The task instructions for the named entity recognition task shown can extract entities of categories such as products, attributes, brands, and components from the input text and provide the corresponding entity categories.

[0193] Step S93: Input the task instructions into the domain model, and execute the task processing based on the task instructions through the domain model to generate the task processing results.

[0194] After generating the task instructions for the natural language processing task to be executed, the task instructions are input into the domain model. Through the reasoning of the domain model, natural language processing based on the task instructions is realized, and the task processing results are generated.

[0195] Step S94: Return the task processing result.

[0196] After generating the task processing results for a natural language processing task, the server can return the results. For example, based on the needs of downstream application systems, the server can return the results to those systems and execute their subsequent processing logic. For instance, after performing a product recognition task using a domain model and obtaining the product recognition results, the server returns these results to an e-commerce system. The e-commerce system then searches for knowledge documents matching the products based on the recognition results.

[0197] In this embodiment, the domain model of the vertical domain trained by any of the aforementioned method embodiments is used as a general model for various natural language processing tasks in that vertical domain. Since the original powerful semantic understanding and representation capabilities of the pre-trained model are maintained as much as possible during the training process, more knowledge of the vertical domain is injected into the domain model, enhancing the domain model's text understanding and generation capabilities in the vertical domain. Using the domain model to perform various natural language processing tasks in that vertical domain, including but not limited to named entity recognition, text classification, text generation, and other tasks, can greatly improve the quality of natural language processing results.

[0198] In an optional embodiment, after obtaining the domain model of the vertical domain based on the method of any of the foregoing method embodiments, the server can provide an Application Programming Interface (API) for the domain model to the outside world. External devices can use the natural language processing capabilities of the domain model through the API of the domain model.

[0199] Figure 10 This is a flowchart illustrating task processing based on a domain model within a vertical domain, provided as another exemplary embodiment of this application. In this embodiment, the domain model and the downstream application system utilizing the natural language processing capabilities of the domain model run on different electronic devices. The electronic device running the domain model acts as a server, and the electronic device running the downstream application system acts as an edge device. Figure 10 As shown, the specific steps for implementing natural language processing tasks based on a domain model of a vertical domain are as follows:

[0200] Step S1001: In response to the task processing request of the natural language processing task in the vertical domain, the edge device obtains the input data and the task prompt format information of the natural language processing task.

[0201] In this embodiment, when the application system running on the edge device needs to perform a natural language processing task in a vertical domain, it submits a task processing request to the edge device. This task processing request includes input data and task category information for the natural language processing task. The task category information differs for different natural language processing tasks.

[0202] In practical applications, downstream application systems may generate various natural language processing (NLP) tasks during operation, including but not limited to information extraction tasks, text classification tasks, and text generation tasks. Different NLP tasks have different task categories. Based on the task category, the corresponding NLP task can be determined, and then the task prompt format information for the corresponding NLP task can be obtained.

[0203] For example, taking an e-commerce system as an example, when a user inputs a query text, the e-commerce system might generate a product recognition task: Based on the user's query text, it extracts entities of various categories, such as products, product attributes, and brands, and determines their corresponding entity categories. After obtaining the product recognition results, the e-commerce system searches for knowledge documents matching the current product and generates a question-and-answer task: based on the searched knowledge documents, it generates a response to the user's query text. After obtaining the response, the e-commerce system outputs the response information.

[0204] In addition, in other scenarios and fields within the e-commerce sector, there are many application systems that generate various natural language processing tasks. The solution in this embodiment uses a general domain model to perform various natural language processing tasks in that vertical domain, which will not be listed here.

[0205] Step S1002: The end device generates a task instruction based on the task prompt format information and input data.

[0206] The specific implementation of step S1002 is the same as that of step S92 mentioned above. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0207] Step S1003: The terminal device sends a request to the server to invoke the domain model. The request includes task instructions.

[0208] In this embodiment, the server can provide an application programming interface (API) for the domain model of the vertical domain. The edge device can send a call request to the server through the API of the domain model of the vertical domain. This call request contains task instructions that require input from the domain model. These task instructions are generated based on the task prompt format information and input data of the natural language processing task to be executed.

[0209] It's important to note that the server can pre-acquire one or more domain models from different vertical domains, each with its own API. The server can expose these APIs to client devices, allowing them to select and invoke different APIs based on the application system they are running or the vertical domain of their current natural language processing task. Furthermore, in practical applications, the server can receive requests from multiple client devices, providing them with natural language processing capabilities based on the domain models of different vertical domains.

[0210] Step S1004: The server inputs the task instructions into the domain model, and the domain model executes the task processing based on the task instructions to generate the task processing results.

[0211] Upon receiving a request to invoke the domain model, the server obtains the task instructions and inputs them into the domain model of the vertical domain. The domain model then processes the task based on the instructions and generates the task processing result. In this embodiment, the domain model running on the server is obtained by further pre-training the pre-trained model using a mixture of vertical and general domain pre-trained data to obtain a pre-trained domain base model. This pre-trained model is then fine-tuned using a fine-tuning dataset from the vertical domain. The specific process of obtaining the domain model is detailed in the aforementioned embodiments and will not be repeated here.

[0212] Step S1005: The server returns the task processing result to the end device.

[0213] In this embodiment, the domain model trained by any of the aforementioned method embodiments is used as a general model for various natural language processing tasks in that vertical domain, providing APIs for external edge devices to use the domain model. Because the training process preserves the original powerful semantic understanding and representation capabilities of the pre-trained model as much as possible, and injects more vertical domain knowledge into the domain model, it enhances the domain model's text understanding and generation capabilities in the vertical domain. Downstream application systems can use this domain model to perform various natural language processing tasks in that vertical domain, including but not limited to named entity recognition, text classification, text generation, and other tasks, which can greatly improve the quality of natural language processing results.

[0214] Based on any of the aforementioned method embodiments, the task prompt format information for any type of task can be configured in conjunction with the Thinking Chain (CoT) method. The task prompt format information contains a natural language description of the reasoning steps, instructing the model to perform task reasoning step by step to obtain the final result. It can not only output the final result, but also output the intermediate reasoning process, which can effectively prompt the quality of the output result and reduce the user's understanding and decision-making costs.

[0215] For example, consider a product comparison task in e-commerce: given information about two products, compare them and provide a recommendation that meets the user's needs. One possible approach to solving this task is: based on the information about the two products, describe their characteristics, similarities, and differences; then describe the advantages of each product; and finally, provide a recommended product based on the user's needs. Based on this approach, a task prompt format for the product comparison task can be configured as follows:

[0216] Hint: We need to perform a product comparison function. Based on the titles and attributes of two products, we need to summarize the characteristics, similarities and differences between the two products, describe the advantages of the two products in terms that consumers can understand, and finally provide recommendations based on different applicable scenarios.

[0217] The input items are as follows:

[0218] Product 1: {Title, attributes, and price of Product 1}

[0219] Product 2: {Title, attributes, and price of Product 1}

[0220] The output items and output format are as follows:

[0221] Characteristics of Product 1: {Characteristics of Product 1}.

[0222] Characteristics of Product 2: {Characteristics of Product 2}.

[0223] Similarities: {Similarities between the two products}.

[0224] Differences: {Differences between the two products}.

[0225] The advantages of product 1 are {advantages of product 1}.

[0226] The advantages of product 2 are {advantages of product 2}.

[0227] Recommendations: {Purchase suggestions based on different applicable scenarios}.

[0228] Based on this task prompt format, the prompt includes the intermediate inference process of the model performing product comparison. The input items indicate the information required for comparing product 1 and product 2; this example illustrates this by including the product information as title, attributes, and price. The output format specifies that the output results include the characteristics of product 1, the characteristics of product 2, the similarities between the two products, the differences between the two products, the advantages of product 1, the advantages of product 2, and a recommendation. Thus, the task processing result provided by the model not only includes the final recommendation but also the intermediate comparison process between the two products. Furthermore, based on the model's output task processing result, the output items to be displayed can be extracted from the task processing result using pre-configured rules and presented to the user through the front-end interface. For example, key information from the task processing result can be extracted and displayed to the user using pre-configured regular expressions.

[0229] like Figure 11 As shown, based on the task prompt format information of the aforementioned product comparison task, the process framework for implementing the product comparison task using a domain model in the e-commerce field is as follows: First, based on the given information of Product 1 and Product 2, a task instruction based on the Thinking Chain (CoT) method is generated according to the task prompt format information configured. The task instruction is then input into the e-commerce domain model, which executes a step-by-step reasoning process based on the Thinking Chain (CoT) task instruction to generate the comparison result. The generated comparison result includes the characteristics of Product 1, the characteristics of Product 2, the similarities between the two products, the differences between the two products, the advantages of Product 1, the advantages of Product 2, and recommendations. Additionally, in other examples, the disadvantages of the two products can also be given separately. Which intermediate reasoning steps in the thinking chain are included in the task's thinking chain and task prompt format information can be configured by relevant technical personnel according to the needs of the actual application scenario; no specific limitations are made here.

[0230] Figure 12 The flowchart of the product comparison method provided in this embodiment is as follows: Figure 12 As shown, the specific steps for implementing product comparison based on a large language model in the e-commerce field are as follows:

[0231] Step S121: Obtain information and task prompt format information for the multiple products to be compared.

[0232] The information obtained about the products includes key information such as the product title, attributes, price, usage instructions, applicable scenarios, and product description. The specific key information to be included can be configured in the input fields of the task prompt format information according to the actual application scenario, and no specific limitation is made here.

[0233] The product information obtained in this step is the input information required to perform the product comparison task. The task prompt format for the product comparison task can be found in the example of the aforementioned embodiment, and will not be repeated here.

[0234] Step S122: Generate task instructions based on the task prompt format information and information about multiple products.

[0235] Step S123: Input the task instructions into the large language model of the e-commerce domain, and perform product comparison processing according to the task instructions through the large language model to obtain the product comparison output results.

[0236] Step S124: Output the product comparison results.

[0237] For example, Figure 13 An example interface for displaying product comparison results, such as Figure 13 As shown, the interface can display the characteristics of two products, summarize their advantages and disadvantages, and provide recommendations based on user scenarios. Additionally, the interface can display various product parameters, highlighting key parameters (such as advantages) based on the product's characteristics, advantages, and disadvantages.

[0238] In this embodiment, task prompt format information for various tasks is configured based on the Thinking Chain (CoT) method. This allows the task prompt format information to include natural language descriptions of intermediate reasoning steps, instructing the domain model to perform task reasoning step by step to obtain the final result. It can output not only the final result but also the intermediate reasoning process, effectively improving the quality of the output result and reducing the user's understanding and decision-making costs.

[0239] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 14 As shown, the server includes a memory 1401 and a processor 1402. The memory 1401 stores computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1402 is communicatively connected to the memory 1401 and executes the computer-executable instructions stored in the memory 1401 to implement the technical solutions provided in any of the above-described method embodiments. Their specific functions and the technical effects they achieve are similar and will not be repeated here.

[0240] Figure 14 The example used here is a cloud server deployed in the cloud; however, the server can also be a local server. Optional, such as... Figure 14 As shown, the server also includes other components such as a firewall 1403, a load balancer 1404, a communication component 1405, and a power supply component 1406. Figure 14 The diagram only shows a portion of the components and does not imply that the server only includes... Figure 14 The components shown.

[0241] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0242] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solution provided in any of the above method embodiments. The specific functions and the technical effects that can be achieved are not described here.

[0243] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided in any of the aforementioned method embodiments.

[0244] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0245] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0246] The aforementioned memory can be object storage service (OSS). This memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies. The aforementioned power supply component provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides. The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0247] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0248] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0249] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0250] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0251] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A task processing method, characterized in that, include: Using a mixture of pre-trained data from vertical and general domains, the pre-trained model is further pre-trained to obtain a pre-trained domain-based model. The pre-trained model is a pre-trained large language model. The further pre-training includes using the mixed pre-trained data to perform a causal language modeling pre-training task on the pre-trained large language model. Using the constructed multi-task instruction dataset of the vertical domain, the domain base model is fine-tuned and trained to obtain a domain model suitable for the vertical domain. The instruction dataset contains multiple instruction data, which includes task prompt format information, input information, and output results. Different tasks use different task prompt format information. The domain model is used to execute natural language processing tasks in the vertical domain and generate corresponding task processing results. The method of using hybrid pre-training data from vertical and general domains to further pre-train the pre-trained model to obtain a pre-trained domain-based model includes: The hybrid pre-training data contains multiple types of text samples. Text samples of the same type are concatenated, and the concatenation result is divided into multiple input samples based on a preset maximum input length to construct input sample sets of each type. Based on the input sample sets of each type, a causal language modeling pre-training task is performed on the pre-trained model to further pre-train the pre-trained model and obtain a pre-trained domain basic model. Constructing a multi-task instruction dataset for the aforementioned vertical domain includes: Obtain instruction data for various existing tasks in the vertical field; Given instruction data for at least one existing task, generate instruction data for a new task according to the corresponding task transformation rules; Based on the instruction data of the existing tasks and the new tasks, construct a multi-task instruction dataset for the vertical domain; The task transformation rule includes at least one of a task simplification rule, a task reversal rule, and a task reorganization rule. The task simplification rule is used to simplify the input information and / or output results in the instruction data of the existing task. The task reversal rule is used to use the output results of the existing task as the input information of the new task and the input information of the existing task as the output results of the new task. The task reorganization rule is used to split and reorganize the input information and output results of the existing task.

2. The method according to claim 1, characterized in that, Before using a mixture of vertical and general domain pre-training data to further pre-train the pre-trained model and obtain a pre-trained domain-based model, the process also includes: Construct pre-trained data for the aforementioned vertical domain; According to a preset mixing method, the pre-trained data of the vertical domain is mixed with the pre-trained data of the general domain to obtain mixed pre-trained data.

3. The method according to claim 2, characterized in that, The step of mixing the pre-trained data from the vertical domain with the pre-trained data from the general domain according to a preset mixing method to obtain mixed pre-trained data includes: According to a preset mixing ratio, the pre-trained data of the vertical domain is mixed with the pre-trained data of the general domain to obtain mixed pre-trained data; or, The pre-training data for the vertical domain includes pre-training data for multiple sub-domains. Based on the application domain of the domain model, the pre-training data for the sub-domains corresponding to the application domain are mixed with the pre-training data for the general domain to obtain hybrid pre-training data.

4. The method according to claim 2, characterized in that, Also includes: The configuration interface displays the configured mixing methods and the corresponding mixing rules. In response to an edit operation on the blending rule corresponding to any of the blending methods, the new blending rule of the blending method after the edit is obtained, and the blending rule corresponding to the blending method is updated; In response to the selection of any of the aforementioned mixing modes, the selected mixing mode is set as the currently used preset mixing mode.

5. The method according to claim 1, characterized in that, The process of generating instruction data for a new task based on instruction data for at least one existing task, according to the corresponding task transformation rules, includes at least one of the following: Based on instruction data of at least one existing task, and according to the task simplification rules corresponding to the existing task, the input information and / or output results in the instruction data of the existing task are simplified to obtain the input information and output results of the new task, and the task prompt format information of the new task is configured. Based on instruction data of at least one existing task, according to the task reversal rule corresponding to the existing task, the output result of the existing task is used as the input information of the new task, and the input information of the existing task is used as the output result of the new task to construct the input information and output result of the new task, and the task prompt format information of the new task is configured. Based on instruction data of at least one existing task, and according to the task reorganization rules corresponding to the existing task, the input information and output results of the existing task are split and reorganized to construct the input information and output results of the new task, and the task prompt format information of the new task is configured.

6. The method according to any one of claims 1-5, characterized in that, After using the dataset from the vertical domain to fine-tune and train the domain-based model to obtain a domain model suitable for the vertical domain, the process further includes: In response to the task processing request of the natural language processing task in the vertical domain, the input data and the task prompt format information of the natural language processing task are obtained; Based on the task prompt format information and input data, generate task instructions; The task instructions are input into the domain model, and the domain model executes task processing based on the task instructions to generate task processing results. Output the task processing results.

7. A task processing method, characterized in that, include: Using mixed pre-training data from the e-commerce domain and the general domain, the pre-trained model is further pre-trained to obtain a pre-trained basic large model for the e-commerce domain. The pre-trained model is a pre-trained large language model. The further pre-training includes using the mixed pre-training data to perform a causal language modeling pre-training task on the pre-trained large language model. Using the constructed multi-task instruction dataset in the e-commerce domain, the basic large model in the e-commerce domain is fine-tuned and trained to obtain a large model in the e-commerce domain. The instruction dataset contains multiple instruction data, which includes task prompt format information, input information, and output results. Different tasks use different task prompt format information. The large model in the e-commerce domain is used to perform natural language processing tasks in the e-commerce domain and generate corresponding task processing results. The method of using mixed pre-trained data from the e-commerce domain and the general domain to further pre-train the pre-trained model to obtain a pre-trained domain-based model includes: The hybrid pre-training data contains multiple types of text samples. Text samples of the same type are concatenated, and the concatenation result is divided into multiple input samples based on a preset maximum input length to construct input sample sets of each type. Based on the input sample sets of each type, a causal language modeling pre-training task is performed on the pre-trained model to further pre-train the pre-trained model and obtain a pre-trained domain basic model. Constructing the multi-task instruction dataset for the e-commerce domain includes: Obtain instruction data for various existing tasks in the e-commerce field. The instruction data includes task prompt format information, input information, and output results. Different tasks use different task prompt format information. Given instruction data for at least one existing task, generate instruction data for a new task according to the corresponding task transformation rules; Based on the instruction data of the existing tasks and the new tasks, construct a multi-task instruction dataset for the e-commerce domain; The task transformation rule includes at least one of a task simplification rule, a task reversal rule, and a task reorganization rule. The task simplification rule is used to simplify the input information and / or output results in the instruction data of the existing task. The task reversal rule is used to use the output results of the existing task as the input information of the new task and the input information of the existing task as the output results of the new task. The task reorganization rule is used to split and reorganize the input information and output results of the existing task.

8. The method according to claim 7, characterized in that, Before using mixed pre-trained data from the e-commerce domain and the general domain to further pre-train the pre-trained model and obtain a pre-trained basic large model for the e-commerce domain, the process also includes: Building pre-trained data for the e-commerce field; According to the preset mixing method, the pre-trained data in the e-commerce field is mixed with the pre-trained data in the general field to obtain the mixed pre-trained data.

9. A task processing method, characterized in that, Applied to servers, including: The receiving end device sends a request to invoke the domain model, the request containing task instructions, which are generated based on the task prompt format information and input data of the natural language processing task to be executed; The task instructions are input into the domain model, which then performs task processing based on the instructions to generate task processing results. The domain model is obtained by further pre-training a pre-trained model using hybrid pre-training data from both vertical and general domains to obtain a pre-trained domain base model. This pre-trained model is then fine-tuned using a constructed multi-task instruction dataset from the vertical domain. The instruction dataset contains multiple instruction data entries, including task prompt format information, input information, and output results. Different tasks use different task prompt formats. The pre-trained model is a pre-trained large language model. Further pre-training involves using the hybrid pre-training data to perform a causal language modeling pre-training task on the pre-trained large language model. The hybrid pre-training data is obtained by processing various types of text samples. The processing methods include concatenating text samples of the same type and segmenting the concatenated result into multiple input samples based on a preset maximum input length to construct input sample sets for each type. The instruction dataset is constructed based on instruction data from various existing tasks in the vertical domain and instruction data for new tasks generated from the existing task instruction data according to corresponding task transformation rules. Return the task processing result to the terminal device; The task transformation rule includes at least one of a task simplification rule, a task reversal rule, and a task reorganization rule. The task simplification rule is used to simplify the input information and / or output results in the instruction data of the existing task. The task reversal rule is used to use the output results of the existing task as the input information of the new task and the input information of the existing task as the output results of the new task. The task reorganization rule is used to split and reorganize the input information and output results of the existing task.

10. A server, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to cause the server to perform the method of any one of claims 1-9.

Images