Method, device and storage medium for training vertical domain large model

Abstract

The present application relates to the field of artificial intelligence technology, and in particular, to a method, device and storage medium for training a vertical domain big model. It includes: determining multiple participants corresponding to a target vertical domain, and multiple participants own local text data of the target vertical domain; distributing the initial big model of the target vertical domain to multiple participants, so that multiple participants iteratively train the initial big model based on the local text data of the target vertical domain they own; obtaining the temporary big models returned by the iterative training obtained by each of the multiple participants, and aggregating the temporary big models that have been verified to be compliant to obtain the current latest aggregated big model; when the termination conditions of the iterative training are met, the current latest aggregated big model is determined as the vertical domain big model corresponding to the target vertical domain so that the data coverage of the vertical domain big model is wider, the diversity of the data is improved, and the subsequent user query retrieval is more accurate and comprehensive, and the data confidentiality of each participant is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method, device and storage medium for training a large vertical domain model. Background Art

[0002] RAG (Retrieval-Augmented Generation) is widely used in intelligent question-answering systems. Existing RAGs are based on sparse or dense retrieval to obtain reference contexts related to user query questions. Among them, sparse retrieval usually indexes and retrieves data through keywords or phrases. Dense retrieval embeds each document and query into a dense vector to capture the semantic features of the document through the dimensions of the vector. Both of the above methods need to be executed based on a fixed and visible database. This means that when there are errors in the data in the database, the retrieval process is also wrong, resulting in the inability to give the user the correct answer in the future. In addition, for the sake of data privacy and data security, data between different local databases generally cannot be shared, resulting in existing retrieval or data acquisition can only be in a single database. Some complex queries often require data from multiple local databases at the same time, resulting in the retrieved data being very one-sided and single, making the retrieval results of user queries inaccurate and incomplete. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device and storage medium for training a large vertical domain model, so as to solve the problems of one-sided retrieval and inaccurate question and answer caused by data privacy and data security in the prior art.

[0004] In order to achieve the above-mentioned object, the first aspect of the present application provides a method for training a vertical domain large model, the method comprising:

[0005] Determine multiple participants corresponding to the target vertical domain, and multiple participants have local text data of the target vertical domain;

[0006] Distribute the initial large model of the target vertical domain to multiple participants, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party;

[0007] Obtain the temporary large models returned by multiple participants through iterative training, and aggregate the temporary large models that have been verified to be compliant to obtain the latest aggregated large model;

[0008] When the termination conditions of the iterative training are met, the current latest aggregated large model is determined as the vertical domain large model corresponding to the target vertical domain.

[0009] In an embodiment of the present application, determining multiple participants corresponding to the target vertical domain includes: obtaining verification text data sent by multiple registrants; for the verification text data sent by any registrant, obtaining a semantic embedding vector corresponding to the verification text data; when the semantic embedding vector is in the vertical domain text data space corresponding to the target vertical domain, determining any registrant as a participant of the target vertical domain.

[0010] In an embodiment of the present application, the method also includes: generating test text data based on a temporary large model returned by any participant; and determining that the temporary large model is verification compliant when the semantic embedding vector corresponding to the test text data is in the vertical domain text data space corresponding to the target vertical domain.

[0011] In an embodiment of the present application, the method also includes: respectively obtaining the semantic embedding vectors corresponding to the sample text data owned by each registrant; performing clustering processing on all the obtained semantic embedding vectors to obtain N vertical domain clustering centers, where N is a positive integer; based on the N vertical domain clustering centers, dividing the global semantic space into corresponding N vertical domain text data spaces, wherein each vertical domain text data space respectively contains a corresponding unique vertical domain clustering center, and the distance between any semantic embedding vector in any vertical domain text data space and the vertical domain clustering center in any vertical domain text data space is not greater than the distance between the arbitrary semantic embedding vector and the other vertical domain clustering centers.

[0012] In an embodiment of the present application, the method also includes: performing averaging processing on the semantic embedding vectors corresponding to all the acquired sample text data to obtain the global semantic embedding center corresponding to all the sample text data; determining the edge semantic embedding vector that is farthest from the global semantic embedding center from the semantic embedding vectors corresponding to all the sample text data; determining the standardized high-dimensional spherical space in which all the sample text data are located based on the global semantic embedding center and the edge semantic embedding vector, wherein the center of the standardized high-dimensional spherical space is the global semantic embedding center, and the radius is a preset multiple of the distance between the global semantic embedding center and the edge semantic embedding vector, and the preset multiple is not less than 1; and determining the standardized high-dimensional spherical space as the global semantic space.

[0013] In an embodiment of the present application, aggregating the temporary large models that have been verified to be compliant to obtain the current latest aggregated large model includes: weighted summing up the model parameters corresponding to all the temporary large models that have been verified to be compliant according to preset weights to obtain the aggregated model parameters, and constructing the current latest aggregated large model based on the aggregated model parameters, wherein the preset weight of any temporary large model is determined by the rating of at least one attribute of the participant providing any temporary large model.

[0014] In an embodiment of the present application, the method also includes: when the termination condition of the iterative training is not met, the initial large model is updated to the current latest aggregated large model and distributed to multiple participants again, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party, and return the temporary large model obtained by the iterative training.

[0015] In an embodiment of the present application, the termination conditions include: the number of iterative training times of multiple participants for the initial large model reaches a preset number, or the model performance of the current latest aggregated large model reaches the expected indicators.

[0016] In an embodiment of the present application, the method also includes: obtaining a user's query question; when the query question belongs to the target vertical domain, inputting the query question into the vertical domain big model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general big model to obtain a response to the query question.

[0017] In an embodiment of the present application, the method also includes: when the termination condition of the iterative training is met, saving the temporary big model obtained by the last round of iterative training returned by multiple participants; obtaining the user's query question; when the query question belongs to the target vertical domain, determining the participation weights of the multiple participants based on the query question; aggregating the temporary big models obtained by the last round of iterative training returned by multiple participants based on the participation weights of the multiple participants to obtain a segmented vertical domain big model for the query question; inputting the query question into the segmented vertical domain big model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general big model to obtain a reply to the query question.

[0018] In an embodiment of the present application, determining the participation weights of multiple participants includes: obtaining data descriptions of local text data of target vertical domains owned by multiple participants, inputting query questions and the data descriptions provided by each of the multiple participants into an association recognition large model to obtain the participation weights of the multiple participants, the association recognition large model is used to identify the correlation between the query question and the data description provided by each participant, and output it as the participation weights of the multiple participants; or, obtaining local semantic embedding centers corresponding to local text data of target vertical domains owned by each of the multiple participants, and determining the participation weights of the multiple participants based on the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant, wherein the participation weight of any participant is positively correlated to the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by the participant.

[0019] In an embodiment of the present application, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the query question belongs from each target vertical domain based on all current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0020] In an embodiment of the present application, the method also includes: obtaining the user's current query intent, the current query intent is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, the current user portrait is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0021] A second aspect of the present application provides a method for training a vertical domain large model, the method comprising:

[0022] Obtain an initial large model of the target vertical domain distributed by the central server;

[0023] The initial big model is iteratively trained based on the local text data of the target vertical domain, and the temporary big model obtained by the iterative training is returned to the central server, so that the central server can aggregate the verified compliant temporary big models to obtain the current latest aggregated big model, and determine the current latest aggregated big model as the vertical domain big model corresponding to the target vertical domain when the termination conditions of the iterative training are met.

[0024] In an embodiment of the present application, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the user's query question belongs from each target vertical domain based on all current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0025] In an embodiment of the present application, the method also includes: obtaining the user's current query intent, the current query intent is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, the current user portrait is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0026] A third aspect of the present application provides a device for training a large vertical domain model, comprising:

[0027] a memory configured to store instructions;

[0028] The processor is configured to call instructions from the memory and implement the above-mentioned method for training a vertical domain large model when executing the instructions.

[0029] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for training a vertical domain large model.

[0030] Through the above technical solution, the initial big model of the target vertical domain is distributed to multiple participants, so that multiple participants can iteratively train the initial big model based on the local text data of the target vertical domain they own, and obtain the temporary big models returned by the iterative training, and aggregate the temporary big models that have been verified to be compliant to obtain the current latest aggregated big model. When the termination conditions of the iterative training are met, the current latest aggregated big model is determined as the vertical domain big model corresponding to the target vertical domain, so that the data coverage of the vertical domain big model is wider, the diversity of the data is improved, and the subsequent user query retrieval is more accurate and comprehensive, and at the same time the data of each participant is kept confidential.

[0031] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0033] Figure 1 A schematic diagram of a process of a method for training a vertical domain large model according to an embodiment of the present application is schematically shown;

[0034] Figure 2 Another flowchart of a method for training a vertical domain large model according to an embodiment of the present application is schematically shown;

[0035] Figure 3 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0038] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0039] Figure 1 A schematic diagram of a process of training a vertical domain large model according to an embodiment of the present application is shown. Figure 1 As shown, an embodiment of the present application provides a method for training a large vertical domain model, which may include the following steps.

[0040] Step 101: Determine multiple participants corresponding to the target vertical domain, and the multiple participants have local text data of the target vertical domain.

[0041] A vertical domain refers to a small field that is vertically subdivided under a large field. The processor can determine multiple participants corresponding to the target vertical domain, and multiple participants have local text data of the target vertical domain. For example, the target vertical domain can refer to any one of the engineering field, the financial field, and the legal field. Assuming that the target vertical domain is the engineering field, the local text data of multiple participants include but are not limited to text data in the engineering field.

[0042] In an embodiment of the present application, determining multiple participants corresponding to the target vertical domain includes: obtaining verification text data sent by multiple registrants; for the verification text data sent by any registrant, obtaining a semantic embedding vector corresponding to the verification text data; when the semantic embedding vector is in the vertical domain text data space corresponding to the target vertical domain, determining any registrant as a participant of the target vertical domain.

[0043] The processor can determine that the target is in the corresponding multiple participants. Specifically, the processor can obtain the verification text data sent by multiple registrants. For the verification text data sent by any registrant, the processor can obtain the semantic embedding vector corresponding to the verification text data. After obtaining the semantic embedding vector, the processor can determine whether the semantic embedding vector is in the vertical domain text data space corresponding to the vertical domain. In the case where the semantic embedding vector is in the vertical domain text data space corresponding to the target vertical domain, the processor can determine any registrant as a participant in the target vertical domain. In a specific embodiment, the processor can input the verification text data into a semantic feature extraction model to obtain a semantic embedding vector corresponding to the verification text data.

[0044] Step 102: Distribute the initial large model of the target vertical domain to multiple participants, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party.

[0045] Step 103: Obtain the temporary large models returned by multiple participants through iterative training, and aggregate the temporary large models that have been verified to be compliant to obtain the latest aggregated large model.

[0046] After determining the multiple participants of the target vertical domain, the processor can distribute the initial large model of the target vertical domain to the multiple participants, so that the multiple participants can iteratively train the initial large model based on the local text data of the target vertical domain they own. After distributing the initial large model of the target vertical domain to the multiple participants, the processor can obtain the temporary large models returned by the multiple participants and obtained through iterative training, and aggregate the temporary large models that have been verified to be compliant to obtain the latest aggregated large model. In a specific embodiment, the processor can send the model parameters of the initial large model of the target vertical domain to the multiple participants, so that the multiple participants can iteratively train the initial large model based on the local text data of the target vertical domain they own. The processor can also obtain the model parameters returned by the multiple participants and obtained through iterative training, and then verify the model parameters returned by the multiple participants to aggregate the verified compliant model parameters, and build the latest aggregated large model based on the aggregated model parameters. Specifically, when aggregating model parameters, the processor may average the model parameters that have been verified to be compliant, or set matching weights so that the model parameters of different participants have different levels of importance. For example, the weight of each participant may be dynamically matched based on the data quality, credit rating, industry representativeness, etc. of each participant, and finally aggregated based on the weights of each participation degree.

[0047] In an embodiment of the present application, aggregating the temporary large models that have been verified to be compliant to obtain the current latest aggregated large model includes: weighted summing up the model parameters corresponding to all the temporary large models that have been verified to be compliant according to preset weights to obtain the aggregated model parameters, and constructing the current latest aggregated large model based on the aggregated model parameters, wherein the preset weight of any temporary large model is determined by the rating of at least one attribute of the participant providing any temporary large model.

[0048] The processor may aggregate the temporary large models that have been verified to be compliant to obtain the latest aggregated large model. Specifically, the processor may perform weighted summation of the model parameters corresponding to all the temporary large models that have been verified to be compliant according to preset weights to obtain the aggregated model parameters. After obtaining the aggregated model parameters, the processor may construct the latest aggregated large model based on the aggregated model parameters. The preset weight of any temporary large model is determined by the rating of the participants of any temporary large model on at least one attribute.

[0049] In an embodiment of the present application, the method also includes: generating test text data based on a temporary large model returned by any participant; and determining that the temporary large model is verification compliant when the semantic embedding vector corresponding to the test text data is in the vertical domain text data space corresponding to the target vertical domain.

[0050] The processor can obtain a temporary large model obtained by iterative training returned by each of the multiple participants, wherein the temporary large model obtained by iterative training returned by each of the multiple participants refers to the model obtained after the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each of them. For the temporary large model returned by any participant, the processor can generate test text data based on the temporary large model. After obtaining the test text data, the processor can determine whether the semantic embedding vector corresponding to the test text data is in the vertical domain text space corresponding to the target vertical domain. In the case that the semantic embedding vector corresponding to the test text data is in the vertical domain text space corresponding to the target vertical domain, the processor can determine that the temporary large model verification is compliant to improve the accuracy of the model.

[0051] In an embodiment of the present application, the method also includes: respectively obtaining the semantic embedding vectors corresponding to the sample text data owned by each registrant; performing clustering processing on all the obtained semantic embedding vectors to obtain N vertical domain clustering centers, where N is a positive integer; based on the N vertical domain clustering centers, dividing the global semantic space into corresponding N vertical domain text data spaces, wherein each vertical domain text data space respectively contains a corresponding unique vertical domain clustering center, and the distance between any semantic embedding vector in any vertical domain text data space and the vertical domain clustering center in any vertical domain text data space is not greater than the distance between the arbitrary semantic embedding vector and the other vertical domain clustering centers.

[0052] The processor can obtain the semantic embedding vector corresponding to the sample text data owned by each registrant. In an optional embodiment, each registrant can perform semantic feature extraction based on the local text data owned by itself to obtain a semantic embedding vector, and send it to the processor, thereby avoiding privacy leakage. Alternatively, each registrant can select some representative sample text data and send it to the processor, and the processor calculates the received sample text data to obtain the corresponding semantic embedding vector, so that the risk of privacy leakage is within a controllable range. After obtaining the semantic embedding vector corresponding to each registrant, the processor can cluster all the acquired semantic embedding vectors to obtain N vertical domain clustering centers, where N is a positive integer. After obtaining the N vertical domain clustering centers, the processor can divide the global semantic space into N vertical domain text data spaces based on the N vertical domain clustering centers, wherein each vertical domain text data space contains a unique vertical domain clustering center, and the distance between any semantic embedding vector in any vertical domain text data space and the vertical domain clustering center of any vertical domain text data space is not greater than the distance between the arbitrary semantic embedding vector and other vertical domain clustering centers.

[0053] In an embodiment of the present application, the method also includes: performing averaging processing on the semantic embedding vectors corresponding to all the acquired sample text data to obtain the global semantic embedding center corresponding to all the sample text data; determining the edge semantic embedding vector that is farthest from the global semantic embedding center from the semantic embedding vectors corresponding to all the sample text data; determining the standardized high-dimensional spherical space in which all the sample text data are located based on the global semantic embedding center and the edge semantic embedding vector, wherein the center of the standardized high-dimensional spherical space is the global semantic embedding center, and the radius is a preset multiple of the distance between the global semantic embedding center and the edge semantic embedding vector, and the preset multiple is not less than 1; and determining the standardized high-dimensional spherical space as the global semantic space.

[0054] The processor can respectively obtain the semantic embedding vectors corresponding to the sample text data owned by each registrant. After obtaining the semantic embedding vectors corresponding to the sample text data owned by each registrant, the processor can perform mean processing on the semantic embedding vectors corresponding to all the obtained sample text data to obtain the global semantic embedding center corresponding to all the sample text data. After obtaining the global semantic embedding center, the processor can determine the edge semantic embedding vector farthest from the center of the global semantic embedding vector from the semantic embedding vectors corresponding to all the sample text data. After obtaining the global semantic embedding center and the edge semantic embedding vector, the processor can determine the standardized high-dimensional spherical space where all the sample text data are located based on the global semantic embedding center and the edge semantic embedding vector, wherein the center of the standardized high-dimensional spherical space is the global semantic embedding center, and the radius is a preset multiple of the distance between the global semantic embedding center and the edge semantic embedding vector, and the preset multiple is not less than 1. Preferably, the preset multiple can be set to 1.5. The processor can determine the standardized high-dimensional spherical space as the global semantic space.

[0055] Step 104: When the termination condition of the iterative training is met, the current latest aggregated large model is determined as the vertical domain large model corresponding to the target vertical domain.

[0056] After obtaining the current latest aggregated large model, the processor can determine whether the termination condition of the iterative training is met. In an embodiment of the present application, the termination condition may include that the number of iterative trainings of multiple participants for the initial large model reaches a preset number, or that the model performance of the current latest aggregated large model reaches the expected index. Among them, the preset number of times and the expected index can be defined based on actual conditions. After meeting the termination condition of the iterative training, the processor can determine the current latest aggregated large model as the vertical domain large model corresponding to the target vertical domain.

[0057] In an embodiment of the present application, the method also includes: when the termination condition of the iterative training is not met, the initial large model is updated to the current latest aggregated large model and distributed to multiple participants again, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party, and return the temporary large model obtained by the iterative training.

[0058] If the termination condition of the iterative training is not met, the processor can update the initial large model to the current latest aggregated large model, and distribute it back to multiple participants again, so that multiple participants can iteratively train the initial large model based on the local text data of their respective target vertical domains, and return the temporary large model obtained by the iterative training.

[0059] In an embodiment of the present application, the method also includes: obtaining a user's query question; when the query question belongs to the target vertical domain, inputting the query question into the vertical domain big model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general big model to obtain a response to the query question.

[0060] The processor may obtain the user's query question and determine whether the query question belongs to the target vertical domain. If it is determined that the query question belongs to the target vertical domain, the processor may input the query question into the vertical domain macro model to obtain a reference context corresponding to the query question. After obtaining the reference context, the processor may input the query question and the reference context into the general macro model to obtain a response to the query question.

[0061] In an embodiment of the present application, the method also includes: when the termination condition of the iterative training is met, saving the temporary big model obtained by the last round of iterative training returned by multiple participants; obtaining the user's query question; when the query question belongs to the target vertical domain, determining the participation weights of the multiple participants based on the query question; aggregating the temporary big models obtained by the last round of iterative training returned by multiple participants based on the participation weights of the multiple participants to obtain a segmented vertical domain big model for the query question; inputting the query question into the segmented vertical domain big model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general big model to obtain a reply to the query question.

[0062] When the termination conditions of the iterative training are met, the processor can save the temporary large model obtained by the last round of iterative training returned by each of the multiple participants. The processor can obtain the user's query question. And determine whether the query question belongs to the target vertical domain. When the query question belongs to the target vertical domain, the processor can determine the participation weights of multiple participants based on the query question. And based on the participation weights of multiple participants, the temporary large models obtained by the last round of iterative training returned by each of the multiple participants are aggregated to obtain a segmented vertical domain large model for the query question. After obtaining the segmented vertical domain large model, the processor can input the query question into the segmented vertical domain large model to obtain a reference context corresponding to the query question. After obtaining the reference context, the processor can input the query question and the reference context into the general large model to obtain a response to the query question.

[0063] In a specific embodiment, the processor may determine the participation weights of multiple participants as the fusion weights of the model parameters obtained from the last round of iterative training returned by each of the multiple participants, so that the model parameters of different participants have different importance. Then, all model parameters are aggregated based on the fusion weights of the model parameters of each participant to obtain the aggregated model parameters, and then a segmented vertical domain large model is constructed. The query question is input into the segmented vertical domain large model to obtain a reference context for the query question. The processor may input the query question and the reference context into the general large model to obtain a response to the query question.

[0064] In an embodiment of the present application, determining the participation weights of multiple participants includes: obtaining data descriptions of local text data of target vertical domains owned by multiple participants, inputting query questions and the data descriptions provided by each of the multiple participants into an association recognition large model to obtain the participation weights of the multiple participants, the association recognition large model is used to identify the correlation between the query question and the data description provided by each participant, and output it as the participation weights of the multiple participants; or, obtaining local semantic embedding centers corresponding to local text data of target vertical domains owned by each of the multiple participants, and determining the participation weights of the multiple participants based on the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant, wherein the participation weight of any participant is positively correlated to the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by the participant.

[0065] The processor can determine the participation weights of multiple participants. Specifically, the processor can obtain data descriptions of the local text data of the target vertical domain owned by multiple participants. Among them, the data description can be obtained by multiple participants inputting the reference text of the target vertical domain into their respective local general large models to semantically understand and summarize the reference text through the local general large model. And the query question and the data descriptions provided by each of the multiple participants are input into the association recognition large model to obtain the participation weights of multiple participants. The association recognition large model is used to identify the correlation between the query question and the data description provided by each participant, and output it as the participation weights of multiple participants. Alternatively, the processor can obtain the local semantic embedding center corresponding to the local text data of the target vertical domain owned by multiple participants. Among them, the local semantic embedding center of each participant can be obtained after the semantic embedding vector corresponding to the local text data of the target vertical domain owned by each participant is averaged. And the participation weights of multiple participants are determined based on the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant. The participation weight of any participant is positively correlated to the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by the participant. In an optional embodiment, the processor can determine the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant based on a preset algorithm such as a softmax function.

[0066] In an embodiment of the present application, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the query question belongs from each target vertical domain based on all current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0067] The processor can obtain the user's query question. After obtaining the query question, the processor can input the query question into the embedding model to output the embedding vector of the query question through the embedding model. After obtaining the embedding vector of the query question, the processor can input the embedding vector into the vertical domain selection model to obtain the selection probability vector output by the vertical domain selection model. After obtaining the selection probability vector, the processor can determine the current selection vector that characterizes whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector, and determine the final vertical domain to which the query question belongs from each target vertical domain based on all the current selection vectors. After obtaining the final vertical domain, the processor can determine the response to the query question based on the vertical domain macro model corresponding to the query question and the final vertical domain, so as to improve the efficiency of vertical domain selection and improve the accuracy of the response.

[0068] In an embodiment of the present application, the method also includes: obtaining the user's current query intent, the current query intent is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, the current user portrait is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0069] The processor can obtain the user's current query intention, which is obtained by parsing the query question input by the user and its historical context. The processor can also obtain the user's current user portrait, which is a set of indicator parameters constructed based on the user's personal information and historical behavior data. The processor can also obtain a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intention and historical user portrait and its corresponding output. After obtaining the positive sample data set, the processor can generate task scenario prompt words based on the positive sample data set for the vertical domain selection large model, and any positive sample data in the positive sample data set is a mapping of a set of historical data of the input structure to the historical data of the output structure of the historical call vertical domain selection large model, wherein the input structure includes query intention and user portrait, and the output structure includes vertical domain selection scores for multiple target vertical domains. After obtaining the task scenario prompt words, the processor can construct a vertical domain selection large model based on the task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection large model to obtain the output data of the output structure based on the input data of the input structure. After constructing the vertical domain selection big model, the processor can input the current query intent and the current user portrait into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain. After obtaining the vertical domain selection score of each target vertical domain, the processor can select the final vertical domain for the user's current query intent from each currently available target vertical domain according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain. The preset selection rules can be the output format of the vertical domain selection big model defined based on the actual situation. After obtaining the final vertical domain, the processor can determine the response to the query question according to the vertical domain big model corresponding to the current query intent and the final vertical domain, so as to improve the efficiency of vertical domain selection and improve the accuracy of the response.

[0070] Figure 2 Another schematic diagram of a method for training a vertical domain large model according to an embodiment of the present application is shown schematically. Figure 2 As shown, an embodiment of the present application provides a method for training a large vertical domain model, which may include the following steps.

[0071] Step 201: Obtain an initial large model of the target vertical domain distributed by a central server.

[0072] Step 202: Iteratively train the initial big model based on the local text data of the target vertical domain, and return the temporary big model obtained by the iterative training to the central server, so that the central server will aggregate the verified compliant temporary big models to obtain the current latest aggregated big model, and determine the current latest aggregated big model as the vertical domain big model corresponding to the target vertical domain when the termination condition of the iterative training is met.

[0073] The processor can obtain the initial big model of the target vertical domain distributed by the central service. After obtaining the initial big model, the processor can iteratively train the initial big model based on the local text data of the target vertical domain, and return the temporary big model obtained by the iterative training to the central server, so that the central server can aggregate the verified compliant temporary big models to obtain the current latest aggregated big model, and when the termination conditions of the iterative training are met, the current latest aggregated big model is determined as the vertical domain big model corresponding to the target vertical domain.

[0074] In an embodiment of the present application, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the query question belongs from each target vertical domain based on all current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0075] The processor can obtain the user's query question. After obtaining the query question, the processor can input the query question into the embedding model to output the embedding vector of the query question through the embedding model. After obtaining the embedding vector of the query question, the processor can input the embedding vector into the vertical domain selection model to obtain the selection probability vector output by the vertical domain selection model. After obtaining the selection probability vector, the processor can determine the current selection vector that characterizes whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector, and determine the final vertical domain to which the query question belongs from each target vertical domain based on all the current selection vectors. After obtaining the final vertical domain, the processor can determine the response to the query question based on the vertical domain macro model corresponding to the query question and the final vertical domain, so as to improve the efficiency of vertical domain selection and improve the accuracy of the response.

[0076] In an embodiment of the present application, the method also includes: obtaining the user's current query intent, the current query intent is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, the current user portrait is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0077] The processor can obtain the user's current query intention, which is obtained by parsing the query question input by the user and its historical context. The processor can also obtain the user's current user portrait, which is a set of indicator parameters constructed based on the user's personal information and historical behavior data. The processor can also obtain a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intention and historical user portrait and its corresponding output. After obtaining the positive sample data set, the processor can generate task scenario prompt words based on the positive sample data set for the vertical domain selection large model, and any positive sample data in the positive sample data set is a mapping of a set of historical data of the input structure to the historical data of the output structure of the historical call vertical domain selection large model, wherein the input structure includes query intention and user portrait, and the output structure includes vertical domain selection scores for multiple target vertical domains. After obtaining the task scenario prompt words, the processor can construct a vertical domain selection large model based on the task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection large model to obtain the output data of the output structure based on the input data of the input structure. After constructing the vertical domain selection big model, the processor can input the current query intent and the current user portrait into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain. After obtaining the vertical domain selection score of each target vertical domain, the processor can select the final vertical domain for the user's current query intent from each currently available target vertical domain according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain. The preset selection rules can be the output format of the vertical domain selection big model defined based on the actual situation. After obtaining the final vertical domain, the processor can determine the response to the query question according to the vertical domain big model corresponding to the current query intent and the final vertical domain, so as to improve the efficiency of vertical domain selection and improve the accuracy of the response.

[0078] Through the above technical solution, the data coverage of the vertical domain large model is made wider, the data diversity is improved, and the subsequent user query retrieval is made more accurate and comprehensive, while at the same time the data of each participant is kept confidential.

[0079] Figure 1 and 2 FIG. 1 is a flow chart of a method for training a vertical domain large model in one embodiment. It should be understood that although Figure 1 and 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0080] The embodiment of the present application also provides a device for training a large vertical domain model, including:

[0081] a memory configured to store instructions;

[0082] The processor is configured to call instructions from the memory and implement the above-mentioned method for training a vertical domain large model when executing the instructions.

[0083] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned method of training a vertical domain large model.

[0084] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected via a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data such as local text data, initial large models, temporary large models, the latest current aggregated large models, and vertical domain large models. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a method for training a vertical domain large model is implemented.

[0085] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0086] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: determining multiple participants corresponding to a target vertical domain, where the multiple participants have local text data of the target vertical domain; distributing an initial large model of the target vertical domain to the multiple participants, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain they respectively own; obtaining temporary large models returned by the multiple participants obtained through iterative training, and aggregating the temporary large models verified to be compliant to obtain the current latest aggregated large model; and when the termination conditions of the iterative training are met, determining the current latest aggregated large model as the vertical domain large model corresponding to the target vertical domain.

[0087] In one embodiment, determining multiple participants corresponding to the target vertical domain includes: obtaining verification text data sent by multiple registrants; for the verification text data sent by any registrant, obtaining a semantic embedding vector corresponding to the verification text data; and determining any registrant as a participant of the target vertical domain when the semantic embedding vector is in the vertical domain text data space corresponding to the target vertical domain.

[0088] In one embodiment, the method also includes: generating test text data based on a temporary large model returned by any participant; and determining that the temporary large model is verification compliant when the semantic embedding vector corresponding to the test text data is in the vertical domain text data space corresponding to the target vertical domain.

[0089] In one embodiment, the method also includes: respectively obtaining semantic embedding vectors corresponding to the sample text data owned by each registrant; performing clustering processing on all the obtained semantic embedding vectors to obtain N vertical domain clustering centers, where N is a positive integer; dividing the global semantic space into corresponding N vertical domain text data spaces based on the N vertical domain clustering centers, wherein each vertical domain text data space respectively contains a corresponding unique vertical domain clustering center, and the distance between any semantic embedding vector in any vertical domain text data space and the vertical domain clustering center in any vertical domain text data space is not greater than the distance between the arbitrary semantic embedding vector and other vertical domain clustering centers.

[0090] In one embodiment, the method also includes: performing averaging processing on the semantic embedding vectors corresponding to all the acquired sample text data to obtain the global semantic embedding center corresponding to all the sample text data; determining the edge semantic embedding vector farthest from the global semantic embedding center from the semantic embedding vectors corresponding to all the sample text data; determining the standardized high-dimensional spherical space in which all the sample text data are located based on the global semantic embedding center and the edge semantic embedding vector, wherein the center of the standardized high-dimensional spherical space is the global semantic embedding center, and the radius is a preset multiple of the distance between the global semantic embedding center and the edge semantic embedding vector, and the preset multiple is not less than 1; determining the standardized high-dimensional spherical space as the global semantic space.

[0091] In one embodiment, aggregating the temporary large models that have been verified to be compliant to obtain the current latest aggregated large model includes: weighted summing up the model parameters corresponding to all the temporary large models that have been verified to be compliant according to preset weights to obtain the aggregated model parameters, and constructing the current latest aggregated large model based on the aggregated model parameters, wherein the preset weight of any temporary large model is determined by the rating of at least one attribute of the participant providing any temporary large model.

[0092] In one embodiment, the method also includes: when the termination condition of the iterative training is not met, the initial large model is updated to the current latest aggregated large model and distributed to multiple participants again, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party, and return the temporary large model obtained by the iterative training.

[0093] In one embodiment, the termination conditions include: the number of iterative trainings of the initial large model by multiple participants reaches a preset number, or the model performance of the latest aggregated large model reaches the expected index.

[0094] In one embodiment, the method also includes: obtaining a user's query question; when the query question belongs to the target vertical domain, inputting the query question into the vertical domain macro model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general macro model to obtain a response to the query question.

[0095] In one embodiment, the method also includes: when the termination condition of the iterative training is met, saving the temporary big model obtained by the last round of iterative training returned by multiple participants; obtaining the user's query question; when the query question belongs to the target vertical domain, determining the participation weights of multiple participants based on the query question; aggregating the temporary big models obtained by the last round of iterative training returned by multiple participants based on the participation weights of multiple participants to obtain a segmented vertical domain big model for the query question; inputting the query question into the segmented vertical domain big model to obtain a reference context corresponding to the query question; and inputting the query question and the reference context into the general big model to obtain a reply to the query question.

[0096] In one embodiment, determining the participation weights of multiple participants includes: obtaining data descriptions of local text data of target vertical domains owned by multiple participants, inputting query questions and the data descriptions provided by the multiple participants into an association recognition large model to obtain the participation weights of the multiple participants, the association recognition large model is used to identify the correlation between the query question and the data description provided by each participant, and output it as the participation weights of the multiple participants; or, obtaining local semantic embedding centers corresponding to local text data of target vertical domains owned by multiple participants, and determining the participation weights of the multiple participants based on the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant, wherein the participation weight of any participant is positively correlated to the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by the participant.

[0097] In one embodiment, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the query question belongs from each target vertical domain based on all the current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0098] In one embodiment, the method further includes: obtaining the user's current query intent, which is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, which is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0099] An embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining an initial large model of the target vertical domain distributed by a central server; iteratively training the initial large model based on local text data of the target vertical domain, and returning a temporary large model obtained through the iterative training to the central server, so that the central server aggregates the verified compliant temporary large models to obtain the current latest aggregated large model, and determines the current latest aggregated large model as the vertical domain large model corresponding to the target vertical domain when the termination condition of the iterative training is met.

[0100] In one embodiment, the method also includes: obtaining a user's query question; inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; determining a current selection vector representing whether each target vertical domain is the final vertical domain to which the query question belongs based on the selection probability vector; determining the final vertical domain to which the user's query question belongs from each target vertical domain based on all current selection vectors; and determining a response to the query question based on a vertical domain macro model corresponding to the query question and the final vertical domain.

[0101] In one embodiment, the method further includes: obtaining the user's current query intent, which is obtained by parsing the query question input by the user and its historical context; obtaining the user's current user portrait, which is a set of indicator parameters constructed based on the user's personal information and historical behavior data; obtaining a positive sample data set in the sample data set, wherein the sample data set is constructed based on each user's historical query intent and historical user portrait and their corresponding outputs; generating task scenario prompt words based on the positive sample data set for the vertical domain selection big model, wherein any positive sample data in the positive sample data set is a mapping of a set of historical data of an input structure to historical data of an output structure for historically calling the vertical domain selection big model, Among them, the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; a vertical domain selection big model is constructed based on task scenario prompt words, and the task scenario prompt words are used to enable the vertical domain selection big model to obtain output data of the output structure based on the input data of the input structure; the current query intent and the current user portrait are input into the vertical domain selection big model to obtain the vertical domain selection score of each currently available target vertical domain; according to the preset selection rules and the vertical domain selection score of each currently available target vertical domain, the final vertical domain for the user's current query intent is selected from each currently available target vertical domain; the answer to the query question is determined according to the vertical domain big model corresponding to the current query intent and the final vertical domain.

[0102] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the method steps of initializing training a vertical domain large model.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0109] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for training a large vertical domain model, characterized in that: The method comprises: Determine multiple participants corresponding to the target vertical domain, wherein the multiple participants have local text data of the target vertical domain; Distributing the initial large model of the target vertical domain to the multiple participants, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each of them; Obtaining the temporary large models returned by the multiple participants through iterative training, and aggregating the temporary large models that have been verified to be compliant to obtain the latest aggregated large model; When the termination condition of the iterative training is met, the current latest aggregated large model is determined as the vertical domain large model corresponding to the target vertical domain; The method further comprises: When the termination condition of the iterative training is met, saving the temporary large model obtained by the last round of iterative training returned by each of the multiple participants; Get the user's query question; In the case where the query question belongs to the target vertical domain, determining the participation weights of the multiple participants based on the query question; Aggregating the temporary large models returned by the multiple participants from the last round of iterative training based on the participation weights of the multiple participants to obtain a segmented vertical domain large model for the query problem; Inputting the query question into the segmented vertical domain macro model to obtain a reference context corresponding to the query question; Inputting the query question and the reference context into a universal macro model to obtain a response to the query question; Wherein, determining the participation weights of the multiple participants includes: Obtain data descriptions of the local text data of the target vertical domain owned by each of the multiple participants, input the query question and the data descriptions provided by each of the multiple participants into the association recognition big model to obtain the participation weights of the multiple participants, the association recognition big model is used to identify the degree of association between the query question and the data description provided by each participant, and output it as the participation weights of the multiple participants; Alternatively, obtain the local semantic embedding centers corresponding to the local text data of the target vertical domain owned by each of the multiple participants, and determine the participation weights of the multiple participants based on the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by each participant, wherein the participation weight of any participant is positively correlated to the similarity between the semantic embedding vector corresponding to the query question and the local semantic embedding center provided by the participant.

2. The method for training a vertical domain large model according to claim 1, characterized in that: The multiple participants corresponding to the target vertical domain are determined to include: Get verification text data sent by multiple registrants; For the verification text data sent by any registrant, obtain the semantic embedding vector corresponding to the verification text data; When the semantic embedding vector is in the vertical domain text data space corresponding to the target vertical domain, any one of the registered parties is determined as a participant of the target vertical domain.

3. The method for training a vertical domain large model according to claim 1, characterized in that: The method further comprises: For a temporary large model returned by any participant, generate test text data based on the temporary large model; When the semantic embedding vector corresponding to the test text data is in the vertical domain text data space corresponding to the target vertical domain, it is determined that the temporary large model is verified to be compliant.

4. The method for training a vertical domain large model according to claim 2 or 3, characterized in that: The method further comprises: Obtain the semantic embedding vectors corresponding to the sample text data owned by each registrant respectively; Cluster all the acquired semantic embedding vectors to obtain N vertical cluster centers, where N is a positive integer; Based on the N vertical domain clustering centers, the global semantic space is divided into corresponding N vertical domain text data spaces, wherein each vertical domain text data space contains a corresponding unique vertical domain clustering center, and the distance between any semantic embedding vector in any vertical domain text data space and the vertical domain clustering center in any vertical domain text data space is not greater than the distance between the any semantic embedding vector and other vertical domain clustering centers.

5. The method for training a vertical domain large model according to claim 4, characterized in that: The method further comprises: Performing mean processing on the semantic embedding vectors corresponding to all the acquired sample text data to obtain the global semantic embedding center corresponding to all the sample text data; Determine, from the semantic embedding vectors corresponding to all the sample text data, an edge semantic embedding vector that is farthest from the global semantic embedding center; Determine a standardized high-dimensional spherical space in which all sample text data are located based on the global semantic embedding center and the edge semantic embedding vector, wherein the center of the standardized high-dimensional spherical space is the global semantic embedding center, and the radius is a preset multiple of the distance between the global semantic embedding center and the edge semantic embedding vector, and the preset multiple is not less than 1; The standardized high-dimensional spherical space is determined as the global semantic space.

6. The method for training a vertical domain large model according to claim 1, characterized in that: The step of aggregating the verified compliance temporary large models to obtain the latest aggregated large model includes: The model parameters corresponding to all verified compliant temporary large models are weightedly summed according to preset weights to obtain aggregate model parameters, and the current latest aggregate large model is constructed based on the aggregate model parameters, wherein the preset weight of any temporary large model is determined by the rating of at least one attribute of the participant providing any temporary large model.

7. The method for training a vertical domain large model according to claim 1, characterized in that: The method further comprises: If the termination condition of the iterative training is not met, the initial large model is updated to the current latest aggregated large model and distributed to the multiple participants again, so that the multiple participants iteratively train the initial large model based on the local text data of the target vertical domain owned by each party, and return the temporary large model obtained by the iterative training.

8. The method for training a vertical domain large model according to claim 1, characterized in that: The termination conditions include: The number of iterative trainings of the multiple participants on the initial large model reaches a preset number; Alternatively, the model performance of the current latest aggregated large model reaches the expected indicators.

9. The method for training a vertical domain large model according to claim 1, characterized in that: The method further comprises: Get the user's query question; In the case where the query question belongs to the target vertical domain, inputting the query question into the vertical domain macro model to obtain a reference context corresponding to the query question; The query question and the reference context are input into a universal macro model to obtain a response to the query question.

10. The method for training a vertical domain large model according to claim 1, characterized in that: The method further comprises: Get the user's query question; Inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; Inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; Determine, according to the selection probability vector, a current selection vector representing whether each target vertical domain is a final vertical domain to which the query question belongs; Determine the final vertical domain to which the query question belongs from each target vertical domain according to all current selection vectors; A response to the query question is determined based on the query question and the vertical domain macro model corresponding to the final vertical domain.

11. The method for training a vertical domain large model according to claim 1, characterized in that: The method further comprises: Obtaining the user's current query intent, where the current query intent is obtained by parsing the query question input by the user and its historical context; Obtaining a current user profile of the user, where the current user profile is a set of indicator parameters constructed based on the user's personal information and historical behavior data; Obtaining a positive sample data set from the sample data set, wherein the sample data set is constructed based on each user's historical query intention and historical user profile and their corresponding outputs; Generate task scenario prompt words based on the positive sample data set for the vertical domain selection big model, where any positive sample data in the positive sample data set is a mapping of historical data of a set of input structures of historically calling the vertical domain selection big model to historical data of output structures, wherein the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; Building a vertical domain selection macromodel based on the task scenario prompt words, wherein the task scenario prompt words are used to enable the vertical domain selection macromodel to obtain output data of the output structure based on the input data of the input structure; Inputting the current query intent and the current user portrait into the vertical domain selection macro model to obtain a vertical domain selection score for each currently available target vertical domain; According to a preset selection rule and a vertical domain selection score of each currently available target vertical domain, a final vertical domain for the current query intention of the user is selected from each currently available target vertical domain; The answer to the query question is determined according to the current query intention and the vertical domain big model corresponding to the final vertical domain.

12. A method for training a large vertical domain model, characterized in that: The method comprises: Obtain an initial large model of the target vertical domain distributed by the central server; Iteratively train the initial big model based on the local text data of the target vertical domain, and return the temporary big model obtained by the iterative training to the central server, so that the central server aggregates the verified compliant temporary big models to obtain the current latest aggregated big model, and determines the current latest aggregated big model as the vertical domain big model corresponding to the target vertical domain when the termination condition of the iterative training is met; The method further comprises: Obtaining the user's current query intent, where the current query intent is obtained by parsing the query question input by the user and its historical context; Obtaining a current user profile of the user, where the current user profile is a set of indicator parameters constructed based on the user's personal information and historical behavior data; Obtaining a positive sample data set from the sample data set, wherein the sample data set is constructed based on each user's historical query intention and historical user profile and their corresponding outputs; Generate task scenario prompt words based on the positive sample data set for the vertical domain selection big model, where any positive sample data in the positive sample data set is a mapping of historical data of a set of input structures of historically calling the vertical domain selection big model to historical data of output structures, wherein the input structure includes query intent and user portrait, and the output structure includes vertical domain selection scores of multiple target vertical domains; Building a vertical domain selection macromodel based on the task scenario prompt words, wherein the task scenario prompt words are used to enable the vertical domain selection macromodel to obtain output data of the output structure based on the input data of the input structure; Inputting the current query intent and the current user portrait into the vertical domain selection macro model to obtain a vertical domain selection score for each currently available target vertical domain; According to a preset selection rule and a vertical domain selection score of each currently available target vertical domain, a final vertical domain for the current query intention of the user is selected from each currently available target vertical domain; The answer to the query question is determined according to the current query intention and the vertical domain big model corresponding to the final vertical domain.

13. The method for training a vertical domain large model according to claim 12, characterized in that: The method further comprises: Get the user's query question; Inputting the query question into an embedding model to output an embedding vector of the query question through the embedding model; Inputting the embedding vector into a vertical domain selection model to obtain a selection probability vector output by the vertical domain selection model; Determine, according to the selection probability vector, a current selection vector representing whether each target vertical domain is a final vertical domain to which the query question belongs; Determine the final vertical domain to which the user's query question belongs from each target vertical domain according to all current selection vectors; A response to the query question is determined based on the query question and the vertical domain macro model corresponding to the final vertical domain.

14. A device for training a large vertical model, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and, when executing the instructions, is able to implement the method for training a large vertical domain model according to any one of claims 1 to 11 or the method for training a large vertical domain model according to any one of claims 12 to 13.

15. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the method for training a vertical domain large model according to any one of claims 1 to 11 or the method for training a vertical domain large model according to any one of claims 12 to 13.

Images