Language model pre-training method and device, equipment and storage medium

Abstract

The invention provides a language model pre-training method and device, equipment and a storage medium, and can be applied to the field of artificial intelligence and the field of financial science and technology. The method comprises the steps that sample word data in an enterprise corpus sample are matched with additional word data included in a plurality of initial additional data tuples, target additional word data corresponding to the enterprise corpus sample are obtained, and the initial additional data tuples comprise the additional word data and paraphrase vectors corresponding to the additional word data; vectorizing the enterprise corpus sample to obtain a sample vector; performing similarity calculation on the sample vector and a paraphrasing vector corresponding to the target additional word data to obtain a target additional data tuple; fusing the sample vectors and paraphrasing vectors included in the target additional data tuple to obtain target sample vectors; and pre-training the language model based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and financial technology, and in particular to a language model pre-training method, apparatus, device and storage medium. Background Art

[0002] With the gradual development of natural language processing technology, more and more businesses can be processed by machines. If you want machines to accurately provide business processing for users, it is essential to make machines understand the user's language and understand the user's needs.

[0003] Based on the powerful capabilities of language models, language models are usually used to enable machines to recognize the language input by users, so that the machine can understand the user's needs and perform business processing for the user. However, in the process of realizing the concept of the present disclosure, the inventors found that the language model in the related technology can obtain good recognition of general knowledge or English knowledge, but it is usually difficult to correctly recognize and process Chinese language or enterprise-specific language. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a language model pre-training method, a business processing method, and an apparatus, device, medium and program product of the above method.

[0005] According to one aspect of the present disclosure, a language model pre-training method is provided, comprising:

[0006] Matching the sample word data in the enterprise corpus sample with the additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes the additional word data and a meaning vector corresponding to the additional word data;

[0007] Vectorize the enterprise corpus samples to obtain sample vectors;

[0008] Calculate the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain the target additional data tuple;

[0009] The sample vector is merged with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector;

[0010] The language model is pre-trained based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

[0011] According to an embodiment of the present disclosure, the above method further includes:

[0012] Convert each dictionary data in the dictionary data set into a key-value pair form to obtain a target dictionary data set, wherein the dictionary data set at least includes: a Chinese dictionary data set and an enterprise professional terminology data set, the key data in the key-value pair is the additional word data, and the value data is the interpretation of the additional word data;

[0013] For each target dictionary data in the target dictionary data set, segmenting and encoding the additional word data in the target dictionary data to obtain the additional word code corresponding to the additional word data;

[0014] Vectorize the meaning of the additional word data to obtain a meaning vector;

[0015] Based on the additional word data, the additional word encoding and the interpretation vector respectively corresponding to each target guiding data in the target guiding data set, an initial additional data tuple is obtained.

[0016] According to an embodiment of the present disclosure, sample word data in an enterprise corpus sample is matched with additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, including:

[0017] Matching the sample words of the enterprise corpus sample with the additional word data of each of the multiple initial additional data tuples respectively to obtain intermediate additional data tuples;

[0018] Based on the additional word data included in the intermediate additional data tuple, target additional word data is obtained.

[0019] According to an embodiment of the present disclosure, sample words of the enterprise corpus sample are matched with the additional word data of each of the multiple initial additional data tuples to obtain intermediate additional data tuples, including:

[0020] Segment the enterprise corpus samples to obtain multiple sample word data;

[0021] Matching each sample word data with the additional word data of each of the multiple initial additional data tuples to obtain multiple matching results;

[0022] Based on the matching results respectively corresponding to the plurality of sample word data, an intermediate additional data tuple is determined from the plurality of initial additional data tuples.

[0023] According to an embodiment of the present disclosure, similarity calculation is performed between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple, including:

[0024] Determining, from the intermediate additional data tuple, a paraphrase vector corresponding to the plurality of target additional word data;

[0025] Calculate the similarity between the interpretation vector corresponding to the target additional word data and the sample vector to obtain a similarity value corresponding to the interpretation vector;

[0026] Based on the similarity value, a target additional data tuple is determined from the intermediate additional data tuples.

[0027] According to an embodiment of the present disclosure, the sample vector and the interpretation vector respectively included in the target additional data tuple are fused to obtain a target sample vector, including:

[0028] Determine a target position of the interpretation vector included in the target additional data tuple in the sample vector;

[0029] Determining a fusion weight based on the number of intermediate additional data tuples and the number of target additional data tuples;

[0030] Based on the target position and the fusion weight, the sample vector and the interpretation vector included in each target additional data tuple are fused to obtain the target sample vector.

[0031] According to an embodiment of the present disclosure, the intermediate additional data tuple further includes an additional word code corresponding to the additional word data;

[0032] Wherein, determining the target position of the interpretation vector included in each of the plurality of target additional data tuples in the sample vector comprises:

[0033] Based on the additional word codes included in the intermediate additional data tuple, a target additional word code is obtained;

[0034] Encode the enterprise corpus samples to obtain sample codes;

[0035] Match the target additional word code with the sample code to obtain the position of the target additional word code in the sample code;

[0036] The position where the target additional word is encoded in the sample encoding is used as the target position.

[0037] Another aspect of the present disclosure provides a business processing method, including:

[0038] In response to receiving the enterprise corpus data input by the target object, inputting the enterprise corpus data into a language model pre-trained by a language model pre-training method, and outputting a business processing result corresponding to the enterprise corpus data;

[0039] The business processing result is sent to the target object.

[0040] Another aspect of the present disclosure provides a language model pre-training device, comprising:

[0041] A data matching module, used for matching the sample word data in the enterprise corpus sample with the additional word data respectively included in the plurality of initial additional data tuples, to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes the additional word data and the interpretation vector corresponding to the additional word data;

[0042] A sample vector determination module is used to vectorize enterprise corpus samples to obtain sample vectors;

[0043] A target tuple determination module is used to calculate the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple;

[0044] A vector fusion module, used to fuse the sample vector with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector;

[0045] The training module is used to pre-train the language model based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

[0046] Another aspect of the present disclosure provides a service processing device, including:

[0047] A result determination module, in response to receiving the enterprise corpus data input by the target object, inputs the enterprise corpus data into a language model pre-trained by a language model pre-training method, and outputs a business processing result corresponding to the enterprise corpus data;

[0048] A sending module is used to send the business processing result to the target object.

[0049] Another aspect of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0050] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0051] Another aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0052] According to the language model pre-training method provided by the present disclosure, additional word data tuples with interpretation vectors are added to jointly participate in the pre-training process of the language model, and the sample word data of the enterprise corpus sample is matched with the additional word data tuple, and multiple target additional word data corresponding to the enterprise corpus sample are obtained from multiple additional word tuples, and the similarity is calculated using the interpretation vectors of each of the multiple target additional word data and the sample vector of the enterprise corpus sample to obtain a relatively accurate interpretation vector for the enterprise corpus sample, and the target interpretation vector is fused with the sample vector to obtain a target sample vector, and the language model is pre-trained using the target sample vector with relatively accurate interpretation, so as to obtain a language model with good recognition ability for the enterprise corpus. Since the additional word data tuple including the interpretation vector that can explain the enterprise corpus sample and the enterprise corpus sample are used together in the pre-training process of the language model, and the interpretation vector in the additional word data tuple is fused with the enterprise corpus sample to obtain the target sample vector, and the language model is pre-trained using the target sample vector with relatively accurate interpretation, the technical problem of the poor recognition effect of the language model for Chinese language or enterprise-specific language is at least partially solved, and the technical effect of improving the recognition accuracy of Chinese language or enterprise-specific language and providing better services to users is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0054] Figure 1 A diagram schematically shows an application scenario of a language model pre-training method and a business processing method according to an embodiment of the present disclosure;

[0055] Figure 2 A flowchart of a language model pre-training method according to an embodiment of the present disclosure is schematically shown;

[0056] Figure 3 Schematically shows a flow chart of generating an initial additional data tuple according to an embodiment of the present disclosure;

[0057] Figure 4 A flowchart of a language model pre-training method according to another embodiment of the present disclosure is schematically shown;

[0058] Figure 5 A flowchart of a business processing method according to an embodiment of the present disclosure is schematically shown;

[0059] Figure 6 The structure block diagram of the language model pre-training device according to the embodiment of the present disclosure is schematically shown;

[0060] Figure 7 A schematic diagram of a structure of a service processing device according to an embodiment of the present disclosure is shown; and

[0061] Figure 8 A block diagram of an electronic device suitable for implementing a language model pre-training method and a business processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0062] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0063] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0064] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0065] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0066] It should be noted that the language model pre-training method and device, and the business processing method and device disclosed in the present invention can be used in the financial field and the artificial intelligence field, and can also be used in any field other than the financial field and the artificial intelligence field. There is no limitation on the application field of the language model pre-training method and device, and the business processing method and device disclosed in the present invention.

[0067] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0068] During the research process, it was found that with the gradual development of natural language processing technology, more and more services can be provided by machines. Based on the powerful capabilities of language models, language models are usually used to enable machines to process the language input by users, so that the machine can understand the user's language and understand the user's requirements, and then provide services to users.

[0069] With the development of the large model ecosystem, the Large Language Model Augmented with Meta-learned Approaches (Llama2) has become one of the most popular language models. Its powerful capabilities and wide application make it the tool of choice for all walks of life. Although the large model Llama2 is powerful, since most of Llama2's pre-training corpus comes from English and is general knowledge, it performs poorly in the recognition and processing of corpus in Chinese contexts, corporate or industry-specific terms, such as the banking industry's specific terms.

[0070] An embodiment of the present disclosure provides a language model pre-training method, comprising: matching sample word data in an enterprise corpus sample with additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes additional word data and a meaning vector corresponding to the additional word data; vectorizing the enterprise corpus sample to obtain a sample vector; calculating similarity between the sample vector and the meaning vector corresponding to the target additional word data to obtain a target additional data tuple; fusing the sample vector with the meaning vector respectively included in the target additional data tuple to obtain a target sample vector; pre-training a language model based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

[0071] Figure 1 The application scenario diagram of the language model pre-training method and the business processing method according to the embodiments of the present disclosure is schematically shown.

[0072] like Figure 1As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0073] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0074] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0075] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0076] It should be noted that the language model pre-training method and the business processing method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the language model pre-training device and the business processing device provided in the embodiments of the present disclosure can generally be arranged in the server 105. The language model pre-training method and the business processing method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the language model pre-training device and the business processing device provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0077] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0078] The following will be based on Figure 1 The scene described by Figure 2 to Figure 4The language model pre-training method of the disclosed embodiment is described in detail.

[0079] Figure 2 The flowchart of the language model pre-training method according to an embodiment of the present disclosure is schematically shown.

[0080] like Figure 2 As shown, the method includes operations S210 to S250.

[0081] In operation S210, sample word data in the enterprise corpus sample is matched with additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes additional word data and interpretation vectors corresponding to the additional word data.

[0082] According to the embodiments of the present disclosure, the enterprise corpus samples may include corpus samples collected from real enterprise business processing scenarios, or may be related corpus extracted from articles, etc., and there is no limitation on the type and collection method of the enterprise corpus samples.

[0083] According to an embodiment of the present disclosure, there may be multiple enterprise corpus samples.

[0084] According to the embodiments of the present disclosure, the enterprise corpus samples may not be labeled, and the performance of the language model may be improved by utilizing the inherent structure and relevance of the unlabeled enterprise corpus samples.

[0085] According to an embodiment of the present disclosure, the language model is not limited and may be Llama2.

[0086] According to an embodiment of the present disclosure, before matching the enterprise corpus sample with the initial additional data tuple, data processing such as dirty data processing and normalization processing may be performed on the enterprise corpus sample.

[0087] According to the embodiments of the present disclosure, dirty data processing can be performed on enterprise corpus samples to clean dirty data, such as removing Hypertext Markup Language (html) tags, emoticons, and special characters. Performing dirty data processing on enterprise corpus samples can improve the generalization ability and reliability of the language model pre-trained using the enterprise corpus samples, making it more suitable for actual scenarios, and high-quality enterprise corpus samples play a crucial role in the final effect of the pre-trained language model.

[0088] According to an embodiment of the present disclosure, the sample word data in the enterprise corpus sample can be matched with the additional word data in each initial additional data tuple to obtain a matching result, and the additional word data representing the matching result as the same as the sample word data is used as the target additional word data.

[0089] According to an embodiment of the present disclosure, the target additional word data may be one or more. Specifically, the number of target additional words may be determined according to the matching result between the sample word data and the additional word data.

[0090] According to an embodiment of the present disclosure, by matching sample word data in an enterprise corpus sample with additional word data in each initial additional data tuple, the initial additional data tuple corresponding to the enterprise corpus sample can be determined more accurately, that is, the interpretation vector that the sample word data in the enterprise corpus sample may correspond to is determined.

[0091] In operation S220, the enterprise corpus sample is vectorized to obtain a sample vector.

[0092] According to the embodiments of the present disclosure, there is no limitation on the method of vectorizing the enterprise corpus sample, and it can be any method of vectorizing the enterprise corpus sample, for example: using Llama2's tokenizer to segment and vectorize the enterprise corpus sample to obtain a sample vector s_veci corresponding to the enterprise corpus sample, where i represents the identifier of the sample.

[0093] According to an embodiment of the present disclosure, while the enterprise corpus sample is vectorized, the enterprise corpus sample can be encoded, that is, the enterprise corpus sample is segmented and digitized using the tokenizer of Llama2 to obtain a sample code s_encodei corresponding to the enterprise corpus sample, where i represents the identifier of the sample.

[0094] According to an embodiment of the present disclosure, the sample vector and the sample code may be combined and stored, that is, the sample vector and the sample code are combined into a code-vector tuple pair and then stored. The form of the code-vector tuple pair is not limited, for example: enterprise corpus sample 1: (s_encode1, s_vec1), enterprise corpus sample 2: (s_encode2, s_vec2).

[0095] According to the embodiments of the present disclosure, the enterprise corpus samples are vectorized and encoded, so that the subsequent similarity calculation and vector fusion process can be performed better and more accurately.

[0096] In operation S230, similarity calculation is performed between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple.

[0097] According to an embodiment of the present disclosure, the same target additional word data may be in different initial additional data tuples, that is, one target additional word data may correspond to multiple interpretation vectors.

[0098] According to an embodiment of the present disclosure, the interpretation vector corresponding to each target additional word data is calculated for similarity with the sample vector. Based on the obtained similarity value, a relatively accurate interpretation vector with respect to the sample vector can be obtained, that is, a relatively accurate interpretation for each sample word in the enterprise corpus sample, and the initial additional data tuple in which the relatively accurate interpretation vector is located is used as the target additional data tuple.

[0099] According to an embodiment of the present disclosure, by calculating the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain the target additional data tuple, a relatively accurate interpretation corresponding to the enterprise corpus sample can be determined, thereby achieving interpretation of the enterprise corpus sample, so that the language model can better understand the Chinese context and the meaning of the enterprise corpus.

[0100] In operation S240, the sample vector is merged with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector.

[0101] According to an embodiment of the present disclosure, different weights can be set to fuse sample vectors and interpretation vectors so that sample vectors that are difficult to understand have greater weights, thereby speeding up the overall training speed of the language model, and determining the position of the sample words explained by the interpretation vector based on the interpretation vector, that is, placing the interpretation vector at a suitable position in the sample vector, so that the language model will be more accurate for the pre-trained language model.

[0102] In operation S250 , a language model is pre-trained based on a target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

[0103] According to the language model pre-training method provided by the present disclosure, additional word data tuples with interpretation vectors are added to jointly participate in the pre-training process of the language model, and the sample word data of the enterprise corpus sample is matched with the additional word data tuple, and multiple target additional word data corresponding to the enterprise corpus sample are obtained from multiple additional word tuples, and the similarity is calculated using the interpretation vectors of each of the multiple target additional word data and the sample vector of the enterprise corpus sample to obtain a relatively accurate interpretation vector for the enterprise corpus sample, and the target interpretation vector is fused with the sample vector to obtain a target sample vector, and the language model is pre-trained using the target sample vector with relatively accurate interpretation, so as to obtain a language model with good recognition ability for the enterprise corpus. Since the additional word data tuple including the interpretation vector that can explain the enterprise corpus sample and the enterprise corpus sample are used together in the pre-training process of the language model, and the interpretation vector in the additional word data tuple is fused with the enterprise corpus sample to obtain the target sample vector, and the language model is pre-trained using the target sample vector with relatively accurate interpretation, the technical problem of the poor recognition effect of the language model for Chinese language or enterprise-specific language is at least partially solved, and the technical effect of improving the recognition accuracy of Chinese language or enterprise-specific language and providing better services to users is achieved.

[0104] Figure 3 The flowchart of generating an initial additional data tuple according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 3 As shown, generating the initial additional data tuple includes operations S310 to S340.

[0106] In operation S310, each dictionary data in the dictionary data set is converted into a key-value pair form to obtain a target dictionary data set, wherein the dictionary data set at least includes: a Chinese dictionary data set and an enterprise professional terminology data set, the key data in the key-value pair is the additional word data, and the value data is the interpretation of the additional word data.

[0107] In operation S320, for each target dictionary data in the target dictionary data set, the additional word data in the target dictionary data is segmented and encoded to obtain an additional word code corresponding to the additional word data.

[0108] In operation S330, the meaning of the additional word data is vectorized to obtain a meaning vector.

[0109] In operation S340, an initial additional data tuple is obtained based on the additional word data, the additional word code, and the interpretation vector respectively corresponding to each target guiding data in the target guiding data set.

[0110] According to an embodiment of the present disclosure, the dictionary data set may be a data set of dictionary data that can interpret Chinese data and enterprise-specific data, and the dictionary data set includes at least data from the following two data sets: a Chinese dictionary data set and an enterprise professional terminology data set.

[0111] According to an embodiment of the present disclosure, the dictionary data set may also include other types of data sets, such as a French dictionary.

[0112] According to an embodiment of the present disclosure, the Chinese dictionary data set includes: data representing the meaning of a plurality of Chinese words, sentences, etc.; the enterprise professional terminology data includes: data representing the meaning of enterprise terminology.

[0113] According to an embodiment of the present disclosure, before converting dictionary data into key-value pairs, the dictionary data may be subjected to text dirty data processing, text normalization, adding special tags, etc., wherein text normalization may include limiting text length.

[0114] According to an embodiment of the present disclosure, the target dictionary data set includes a plurality of target dictionary data.

[0115] According to the embodiments of the present disclosure, each dictionary data can be converted into a key-value pair form and stored to obtain a target dictionary data set, and additional word data with different interpretations can be stored as different target dictionary data, specifically, such as: {"field", "just like territory, an area under the jurisdiction of a specific object, which is inviolable."}; {"field", "the scope of ideology or social activities: the field of thought, the field of academia, the field of life, the field of science."}; {"position", "means money. It is often used in securities, stocks, and futures trading. For example, the long and short positions often mentioned in stocks and futures are actually the abbreviations of long positions and short positions."}.

[0116] According to an embodiment of the present disclosure, since the dictionary data set includes data in the Chinese dictionary data set, that is, data that can interpret Chinese, adding the initial additional data tuple obtained from the dictionary data set to the pre-training process of the language model can improve the Chinese representation performance of the language model.

[0117] According to an embodiment of the present disclosure, since the dictionary data set includes data in the enterprise professional terminology data set, that is, data that can interpret enterprise-specific words, adding the initial additional data tuple obtained from the enterprise professional terminology data set to the pre-training process of the language model can improve the language model's processing capabilities for enterprise tasks, such as: improving the processing capabilities for banking tasks.

[0118] According to the embodiments of the present disclosure, the method of segmenting and encoding each additional word data is not limited and may be any method that can obtain the additional word encoding, for example: using Llama2's tokenizer to segment and digitize the additional word data to obtain the additional word encoding corresponding to each additional word data.

[0119] According to the embodiments of the present disclosure, there is no limitation on the method of vectorizing the interpretation of the additional word data, and it may be any method that can obtain an interpretation vector, for example: inputting the interpretation of each additional word data into a vectorization model to obtain multiple interpretation vectors.

[0120] According to the embodiments of the present disclosure, the vectorization model is not limited and may be a semantic vector model (BAAI General Embedding, BGE).

[0121] According to an embodiment of the present disclosure, multiple additional words, additional word codes corresponding to each additional word data, and interpretation vectors corresponding to each additional word data may be stored in the form of a matrix.

[0122] According to an embodiment of the present disclosure, for example: the matrix of additional word data is k=[field, ..., bookmark], the matrix of additional word encoding is shown in the following formula (1), and the matrix of interpretation vector is shown in the following formula (2).

[0123] According to an embodiment of the present disclosure, the dimension of the matrix of the additional word encoding and the dimension of the matrix of the interpretation vector can be detected. When the dimensions of the above two matrices are different, the dimension of the matrix of the interpretation vector can be adjusted, as shown in the following formula (3).

[0124]

[0125]

[0126] Embedding(bge) D_L =AX D_B ; (3)

[0127] Among them, Tokenizer (llama) represents the matrix of additional word encoding, t i represents the additional word encoding of the i-th word in the matrix of additional word data, where V represents the data size of the dictionary data set, X D_B The matrix representing the interpretation vector, s i Represents the meaning vector corresponding to the i-th additional word data, Embeding(bge) D_L is the matrix of the interpretation vector after dimension adjustment, and A is the weight matrix used for dimension adjustment.

[0128] According to an embodiment of the present disclosure, since each element in the matrix of additional word data, the matrix of additional word encoding and the matrix of interpretation vector is in a one-to-one correspondence, an initial additional data tuple can be obtained by taking an element from each of the three matrices according to the element position.

[0129] According to an embodiment of the present disclosure, a plurality of initial additional data tuples may be stored in an external storage device to save storage space.

[0130] According to the embodiments of the present disclosure, a data set containing a dictionary data set that interprets Chinese data and enterprise-specific data is used to participate in the pre-training process of the language model, so that the language model can understand the meaning of Chinese vocabulary more accurately and accelerate the convergence speed of the secondary pre-training model. At the same time, it can overcome the poor performance of the language model in the Chinese context and enterprise-specific fields, the relatively weak understanding and processing capabilities, and the inability to meet the needs of understanding some idioms, slang or specific expressions that may be encountered when communicating with Chinese users.

[0131] According to an embodiment of the present disclosure, matching sample word data in an enterprise corpus sample with additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample may include the following operations.

[0132] The sample words of the enterprise corpus sample are matched with the additional word data of each of the multiple initial additional data tuples to obtain intermediate additional data tuples; based on the additional word data included in the intermediate additional data tuples, target additional word data are obtained.

[0133] According to an embodiment of the present disclosure, there may be multiple additional word data matching the sample word data, and there may be multiple intermediate additional data tuples.

[0134] According to an embodiment of the present disclosure, when there are multiple target additional word data, the target additional word data may be stored as a target additional word data set, and the set has a mapping relationship with the enterprise corpus sample.

[0135] According to an embodiment of the present disclosure, the target additional word data set may be traversed to remove duplicates.

[0136] According to an embodiment of the present disclosure, by matching sample word data and additional word data, an initial additional data tuple containing sample word data can be determined, the initial additional data tuple containing sample word data can be used as an intermediate additional data tuple, and target additional word data can be determined from the intermediate additional data tuple.

[0137] According to an embodiment of the present disclosure, a target additional word code, that is, a target additional word code corresponding to the sample word data, may also be determined from the intermediate additional data tuple.

[0138] According to an embodiment of the present disclosure, matching sample words of an enterprise corpus sample with respective additional word data of a plurality of initial additional data tuples to obtain intermediate additional data tuples may include the following operations.

[0139] The enterprise corpus sample is segmented to obtain a plurality of sample word data; each sample word data is matched with the respective additional word data of a plurality of initial additional data tuples to obtain a plurality of matching results; based on the matching results respectively corresponding to the plurality of sample word data, an intermediate additional data tuple is determined from the plurality of initial additional data tuples.

[0140] According to the embodiments of the present disclosure, there is no limitation on the word segmentation method of the enterprise corpus sample, and the enterprise corpus sample can be segmented in any manner, for example, using the jieba word segmentation tool to segment the enterprise corpus sample.

[0141] According to the embodiments of the present disclosure, the benefit of using the jieba word segmentation tool to segment enterprise corpus samples is to avoid the word segmentation method adopted by language models such as Llama2, which may result in the sample word data obtained after word segmentation being a single character or incorrect words, so that the sample word data cannot meet the matching requirements with the additional word data.

[0142] According to an embodiment of the present disclosure, the process of determining an intermediate additional data tuple and determining target additional word data and a target additional word code from the intermediate additional data tuple is as follows.

[0143] According to an embodiment of the present disclosure, for example, an enterprise corpus sample may be: the strategic significance of position management in the investment field, and the result after word segmentation is: position / management / in / investment / field / of / strategic significance, that is, position, management, in, investment, field, of, and strategic significance are all sample words of the enterprise corpus sample.

[0144] According to an embodiment of the present disclosure, multiple sample word data are matched with the additional word data of each of the multiple initial additional data tuples, and an intermediate additional data tuple can be determined from the multiple initial additional data tuples based on the matching results. Since the additional word data may be polysemous, the same additional word data may be in different intermediate additional data tuples, and there may be multiple intermediate additional data tuples, specifically, multiple intermediate additional data tuples, as shown in the following formula (4).

[0145] ("Position", Tokenizer(llama) D_L (index("position")),Embeding(bge) D_L (index("Position")) ......

[0147] ("field", Tokenizer(llama) D_L (index("domain")),Embeding(bge) D_L (index("definition 1"))("domain", Tokenizer(llama) D_L (index("domain")),Embeding(bge) D_L (index("Interpretation 2")); (4)

[0148] According to an embodiment of the present disclosure, target additional word data and target additional word codes can be obtained from the intermediate additional data tuple. The target additional word data and target additional word codes can be represented by sets as shown in the following formulas (5) and (6).

[0149] {"position",...,"field"}; (5)

[0150] {Tokenizer(llama) D_L ("position"), ..., Tokenizer (llama) D_L ("field")}; (6)

[0151] According to an embodiment of the present disclosure, the set of target additional word data and the set of target additional word codes may be deduplicated, and only one of the same target additional word data and target additional word code may be retained.

[0152] According to an embodiment of the present disclosure, performing similarity calculation between a sample vector and a definition vector corresponding to target additional word data to obtain a target additional data tuple may include the following operations.

[0153] Determine interpretation vectors corresponding to multiple target additional word data from the intermediate additional data tuple; perform similarity calculation on the interpretation vector corresponding to the target additional word data and the sample vector to obtain a similarity value corresponding to the interpretation vector; and determine the target additional data tuple from the intermediate additional data tuple based on the similarity value.

[0154] According to an embodiment of the present disclosure, the set of target additional word data can be traversed to calculate the similarity between the interpretation vector corresponding to each target additional word data and the sample vector, thereby obtaining the similarity value between each interpretation vector and the sample vector.

[0155] According to an embodiment of the present disclosure, the target interpretation vector with the highest similarity value or a similarity ranking exceeding a sorting threshold can be determined from the similarity values ​​between the interpretation vector corresponding to the same target additional word data and the sample vector, and the middle additional data tuple where the target interpretation vector is located can be used as the target additional data tuple. Specifically, it can be as shown below.

[0156] For example, in the enterprise corpus sample: "The strategic significance of position management in the investment field", the word "field" corresponds to two intermediate additional data tuples, that is, it contains two interpretations, interpretation A and interpretation B. Interpretation A is: "The scope of ideology or social activities: the field of thought, the field of academia, the field of life, and the field of science." Interpretation B is "Like territory, an area under the jurisdiction of a specific object, which is inviolable." And through the calculation of similarity values, it can be seen that the similarity value between the interpretation vector a of interpretation A and the sample vector is higher than the similarity value between the interpretation vector b of interpretation B and the sample vector, so the intermediate additional data tuple where interpretation A is located can be used as the target additional data tuple.

[0157] According to an embodiment of the present disclosure, by taking the middle additional data tuple where the interpretation vector with the highest similarity value corresponding to each target additional word data is located as the target additional data tuple, the enterprise corpus sample may correspond to multiple target additional data tuples.

[0158] According to the embodiments of the present disclosure, there is no limitation on the similarity calculation method, and it can be any method that can calculate the similarity between the sample vector and the interpretation vector, which can be as shown in the following formula (7).

[0159]

[0160] Among them, A represents the sample vector, B i represents the i-th interpretation vector, A·B represents the dot product of vector A and vector B, ||A|| and ||B|| represent the modulus of vector A and vector B respectively.

[0161] According to an embodiment of the present disclosure, as shown in formula (6), the similarity value may be cosine similarity, and the value range of cosine similarity is between -1 and 1. When the angle between two vectors is 0 degrees, the cosine similarity is 1, indicating that they are completely similar; when the angle between two vectors is 90 degrees, the cosine similarity is 0, indicating that they are completely dissimilar; when the angle between two vectors is 180 degrees, the cosine similarity is -1, indicating that they are completely opposite.

[0162] According to the embodiments of the present disclosure, by calculating the similarity between the sample vector and the interpretation vector, dynamic selection of the interpretation vector is achieved. In the case where the sample word is polysemous, the interpretation vectors that do not meet the conditions are screened out, so that the remaining interpretation vectors are relatively similar to the sample vectors. Even if the interpretation of each word in the enterprise corpus sample is more appropriate, allowing more accurate interpretation information to participate in pre-training can improve the language model's understanding of the Chinese context and enterprise business processing capabilities, thereby improving the accuracy of the language model.

[0163] According to an embodiment of the present disclosure, fusing the sample vector with the interpretation vectors respectively included in the target additional data tuple to obtain the target sample vector may include the following operations.

[0164] Determine the target position of the interpretation vector included in the target additional data tuple in the sample vector; determine the fusion weight based on the number of intermediate additional data tuples and the number of target additional data tuples; based on the target position and the fusion weight, fuse the sample vector and the interpretation vector included in each target additional data tuple to obtain the target sample vector.

[0165] According to the embodiments of the present disclosure, by determining the target position of each interpretation vector in the sample vector, the interpretation vector corresponding to each sample word can be mounted under the corresponding sample word, that is, the sample word is interpreted.

[0166] According to an embodiment of the present disclosure, by determining the fusion weight, the training weight for each target sample vector can be determined, that is, target sample vectors with more interpretation vectors, that is, more interpretation information, participate in the training, thereby speeding up the pre-training speed and improving the language model's ability to process Chinese contexts and enterprise business.

[0167] According to an embodiment of the present disclosure, the intermediate additional data tuple also includes an additional word code corresponding to the additional word data; determining the target position of the interpretation vector included in each of the multiple target additional data tuples in the sample vector may include the following operations.

[0168] Based on the additional word codes included in the intermediate additional data tuple, a target additional word code is obtained; the enterprise corpus sample is encoded to obtain a sample code; the target additional word code is matched with the sample code to obtain the position of the target additional word code in the sample code; the position of the target additional word code in the sample code is used as the target position.

[0169] According to an embodiment of the present disclosure, there may be multiple target additional word codes. By traversing multiple target additional word codes and matching each target additional word code with the sample code respectively, it is determined where the interpretation vector corresponding to the target additional word code should be mounted or fused.

[0170] According to an embodiment of the present disclosure, the fusion weight is determined based on the number of intermediate additional data tuples and the number of target additional data tuples, which can be shown in the following formula (8).

[0171]

[0172] Among them, β i Represents the fusion weight, count tp Represents the number of intermediate additional data tuples, count tp_new Represents the number of additional data tuples of the target.

[0173] According to an embodiment of the present disclosure, based on the target position and the fusion weight, the sample vector and the interpretation vector respectively included in each target additional data tuple are fused to obtain a target sample vector, which can be shown as the following formula (9).

[0174]

[0175] Among them, F represents the target sample vector, s_vec j Represents the vector at position j in the sample vector, Embeding(bge) D_L (i) j represents the interpretation vector corresponding to the vector at position j.

[0176] According to an embodiment of the present disclosure, when count tp_new with count tp The closer it is, the fewer polysemous words there are in the corporate corpus sample, the more certain the information is, and the easier it is for the model to learn to distinguish. Therefore, less interpretation information can be allowed to participate in the new pre-training, and the weight of its interpretation vector will become smaller. Conversely, the weight of the interpretation vector will become larger, allowing more interpretation information to participate in the pre-training, thereby speeding up the pre-training speed, improving the model's understanding of the Chinese context and its ability to handle banking business, and thus improving the accuracy of the model.

[0177] Figure 4 The following schematically shows a flow chart of a language model pre-training method according to another embodiment of the present disclosure.

[0178] like Figure 4 As shown, the method includes operations S410 to S450.

[0179] In operation S410, a plurality of initial additional data tuples are generated based on the Chinese dictionary dataset and the enterprise professional terminology dataset, wherein the initial additional data tuples include additional word data, additional word codes corresponding to the additional word data, and interpretation vectors corresponding to the additional word data.

[0180] In operation S420, the sample word data in the enterprise corpus sample is matched with the additional word data respectively included in the multiple initial additional data tuples, and an intermediate additional data tuple is determined from the multiple initial additional data tuples.

[0181] In operation S430, based on the target additional word data included in the intermediate additional data tuple, a similarity calculation is performed between the interpretation vector corresponding to the target additional word data and the sample vector of the enterprise corpus sample to obtain a target additional data tuple;

[0182] In operation S440, the sample vector is merged with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector;

[0183] In operation S450, a language model is pre-trained based on the target sample vector to obtain a pre-trained language model.

[0184] Figure 5 The flowchart schematically shows a service processing method according to an embodiment of the present disclosure.

[0185] Figure 5 As shown, the method includes operations S510 to S520.

[0186] In operation S510, in response to receiving the enterprise corpus data input by the target object, the enterprise corpus data is input into a language model pre-trained by a language model pre-training method, and a business processing result corresponding to the enterprise corpus data is output.

[0187] In operation S520, the business processing result is sent to the target object.

[0188] According to an embodiment of the present disclosure, a business processing result is related to the parameter adjustment of a language model. Different parameter adjustments result in different functions of the language model obtained, and different businesses that can be processed are also different. For example: by adjusting the parameters of a pre-trained language model, the pre-trained language model can realize a dialogue with a target object, that is, based on the enterprise corpus data input by the target object, the pre-trained language model can input an answer to the enterprise corpus data, and the answer is the business processing result. For another example: by adjusting the parameters of a pre-trained language model, the pre-trained language model can realize the expansion of the corpus, that is, based on the enterprise corpus data, the pre-trained language model can input an expanded text for the enterprise corpus data.

[0189] According to the embodiments of the present disclosure, by providing business processing for the target object through a pre-trained language model, a more accurate understanding of the target object's corporate corpus data can be achieved, so that the business processing results are also more accurate, thereby better meeting the target object's needs and improving user experience.

[0190] Based on the above-mentioned language model pre-training method and business processing method, the present disclosure also provides a language model pre-training method and business processing method device. Figure 6 and Figure 7 The device is described in detail.

[0191] Figure 6 The structural block diagram of the language model pre-training device according to an embodiment of the present disclosure is schematically shown.

[0192] like Figure 6 As shown, the language model pre-training device 600 of this embodiment includes a data matching module 610 , a sample vector determination module 620 , a target tuple determination module 630 , a vector fusion module 640 and a training module 650 .

[0193] The data matching module 610 is used to match the sample word data in the enterprise corpus sample with the additional word data respectively included in multiple initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes the additional word data and the interpretation vector corresponding to the additional word data.

[0194] The sample vector determination module 620 is used to vectorize the enterprise corpus sample to obtain a sample vector.

[0195] The target tuple determination module 630 is used to calculate the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain the target additional data tuple.

[0196] The vector fusion module 640 is used to fuse the sample vector with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector.

[0197] The training module 650 is used to pre-train the language model based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

[0198] According to an embodiment of the present disclosure, the language model pre-training device 600 also includes: a target data set determination module, a data encoding module, a data vectorization module and an initial tuple determination module.

[0199] The target data set determination module is used to convert each dictionary data in the dictionary data set into a key-value pair form to obtain a target dictionary data set, wherein the dictionary data set at least includes: a Chinese dictionary data set and an enterprise professional terminology data set, the key data in the key-value pair is the additional word data, and the value data is the interpretation of the additional word data.

[0200] The data encoding module is used to segment and encode the additional word data in the target dictionary data set for each target dictionary data, and obtain the additional word code corresponding to the additional word data.

[0201] The data vectorization module is used to vectorize the interpretation of the additional word data to obtain the interpretation vector.

[0202] The initial tuple determination module is used to obtain an initial additional data tuple based on the additional word data, additional word encoding and interpretation vector corresponding to each target guiding data in the target guiding data set.

[0203] According to an embodiment of the present disclosure, the data matching module 610 includes: an intermediate tuple determination submodule and a target data determination module.

[0204] The intermediate tuple determination submodule is used to match the sample words of the enterprise corpus sample with the additional word data of each of the multiple initial additional data tuples to obtain the intermediate additional data tuple.

[0205] The target data determination module is used to obtain target additional word data based on the additional word data included in the intermediate additional data tuple.

[0206] According to an embodiment of the present disclosure, the intermediate tuple determination submodule includes: a word segmentation unit, a matching unit, and a data determination unit.

[0207] The word segmentation unit is used to segment the enterprise corpus sample to obtain multiple sample word data.

[0208] The matching unit is used to match each sample word data with the additional word data of each of the multiple initial additional data tuples to obtain multiple matching results.

[0209] The data determining unit is used to determine the intermediate additional data tuple from the multiple initial additional data tuples based on the matching results respectively corresponding to the multiple sample word data.

[0210] According to an embodiment of the present disclosure, the target tuple determination module 630 includes: a vector determination submodule, a similarity calculation submodule and a target tuple determination submodule.

[0211] The vector determination submodule is used to determine the interpretation vectors corresponding to the plurality of target additional word data from the intermediate additional data tuple.

[0212] The similarity calculation submodule is used to calculate the similarity between the interpretation vector corresponding to the target additional word data and the sample vector to obtain a similarity value corresponding to the interpretation vector.

[0213] The target tuple determination submodule is used to determine the target additional data tuple from the intermediate additional data tuples based on the similarity value.

[0214] According to an embodiment of the present disclosure, the vector fusion module 640 includes: a position determination submodule, a fusion weight determination submodule, and a vector fusion submodule.

[0215] The position determination submodule is used to determine the target position of the interpretation vector included in the target additional data tuple in the sample vector.

[0216] The fusion weight determination submodule is used to determine the fusion weight based on the number of intermediate additional data tuples and the number of target additional data tuples.

[0217] The vector fusion submodule is used to fuse the sample vector and the interpretation vector respectively included in each target additional data tuple based on the target position and the fusion weight to obtain the target sample vector.

[0218] According to an embodiment of the present disclosure, the intermediate additional data tuple also includes an additional word code corresponding to the additional word data. The position determination submodule includes: a target code determination unit, a sample code determination unit, a first position determination unit, and a second position determination unit.

[0219] The target coding determination unit is used to obtain the target additional word coding based on the additional word coding included in the intermediate additional data tuple.

[0220] The sample code determination unit is used to encode the enterprise corpus sample to obtain the sample code.

[0221] The first position determination unit is used to match the target additional word code with the sample code to obtain the position of the target additional word code in the sample code.

[0222] The second position determination unit is used to encode the position of the target additional word in the sample code as the target position.

[0223] According to an embodiment of the present disclosure, any multiple modules in the data matching module 610, the sample vector determination module 620, the target tuple determination module 630, the vector fusion module 640 and the training module 650 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the data matching module 610, the sample vector determination module 620, the target tuple determination module 630, the vector fusion module 640 and the training module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the data matching module 610, the sample vector determination module 620, the target tuple determination module 630, the vector fusion module 640 and the training module 650 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0224] Figure 7 The structure block diagram of the service processing device according to the embodiment of the present disclosure is schematically shown.

[0225] like Figure 7 As shown, the service processing device 700 of this embodiment includes: a result determination module 710 and a sending module 720 .

[0226] The result determination module 710, in response to receiving the enterprise corpus data input by the target object, inputs the enterprise corpus data into the language model pre-trained by the language model pre-training method, and outputs the business processing result corresponding to the enterprise corpus data.

[0227] The sending module 720 is used to send the business processing result to the target object.

[0228] According to an embodiment of the present disclosure, any multiple modules in the result determination module 710 and the sending module 720 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the result determination module 710 and the sending module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the result determination module 710 and the sending module 720 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0229] Figure 8 A block diagram of an electronic device suitable for implementing a language model pre-training method and a business processing method according to an embodiment of the present disclosure is schematically shown.

[0230] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0231] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0232] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

[0233] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0234] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0235] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the language model pre-training method and business processing method provided in the embodiment of the present disclosure.

[0236] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0237] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0238] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0239] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0240] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0241] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0242] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A language model pre-training method, comprising: Matching sample word data in the enterprise corpus sample with additional word data respectively included in a plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes additional word data and a meaning vector corresponding to the additional word data; Vectorizing the enterprise corpus sample to obtain a sample vector; Calculate the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple; The sample vector and the interpretation vectors respectively included in the target additional data tuple are merged to obtain a target sample vector; The language model is pre-trained based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

2. The method according to claim 1, further comprising: Convert each dictionary data in the dictionary data set into a key-value pair form to obtain a target dictionary data set, wherein the dictionary data set at least includes: a Chinese dictionary data set and an enterprise professional terminology data set, the key data in the key-value pair is the additional word data, and the value data is the interpretation of the additional word data; For each target dictionary data in the target dictionary data set, segmenting and encoding the additional word data in the target dictionary data to obtain the additional word code corresponding to the additional word data; Vectorizing the interpretation of the additional word data to obtain the interpretation vector; Based on the additional word data, the additional word code and the interpretation vector respectively corresponding to each target guiding data in the target guiding data set, an initial additional data tuple is obtained.

3. The method according to claim 1, wherein: The step of matching the sample word data in the enterprise corpus sample with the additional word data respectively included in the plurality of initial additional data tuples to obtain target additional word data corresponding to the enterprise corpus sample includes: Matching the sample words of the enterprise corpus sample with the additional word data of each of the multiple initial additional data tuples respectively to obtain intermediate additional data tuples; Based on the additional word data included in the intermediate additional data tuple, target additional word data is obtained.

4. The method according to claim 3, wherein: The step of matching the sample words of the enterprise corpus sample with the additional word data of each of the plurality of initial additional data tuples to obtain intermediate additional data tuples comprises: Segmenting the enterprise corpus sample to obtain a plurality of sample word data; Matching each of the sample word data with the additional word data of each of the multiple initial additional data tuples to obtain multiple matching results; Based on the matching results respectively corresponding to the plurality of sample word data, an intermediate additional data tuple is determined from the plurality of initial additional data tuples.

5. The method according to claim 3, wherein: The step of calculating the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple includes: Determine, from the intermediate additional data tuple, a definition vector corresponding to a plurality of the target additional word data; Calculating the similarity between the interpretation vector corresponding to the target additional word data and the sample vector to obtain a similarity value corresponding to the interpretation vector; Based on the similarity value, a target additional data tuple is determined from the intermediate additional data tuples.

6. The method according to claim 3, wherein: The step of fusing the sample vector with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector comprises: Determine a target position of the interpretation vector included in the target additional data tuple in the sample vector; Determining a fusion weight based on the number of the intermediate additional data tuples and the number of the target additional data tuples; Based on the target position and the fusion weight, the sample vector is fused with the interpretation vector respectively included in each of the target additional data tuples to obtain a target sample vector.

7. The method according to claim 6, wherein: The intermediate additional data tuple also includes an additional word code corresponding to the additional word data; The step of determining the target position of the interpretation vectors respectively included in the plurality of target additional data tuples in the sample vector comprises: Obtaining a target additional word code based on the additional word code included in the intermediate additional data tuple; Encoding the enterprise corpus sample to obtain a sample code; Matching the target additional word code with the sample code to obtain the position of the target additional word code in the sample code; The position of the target additional word encoding in the sample encoding is used as the target position.

8. A business processing method, comprising: In response to receiving the enterprise corpus data input by the target object, the enterprise corpus data is input into the language model pre-trained by the language model pre-training method according to any one of claims 1 to 7, and the business processing result corresponding to the enterprise corpus data is output; The business processing result is sent to the target object.

9. A language model pre-training device, comprising: A data matching module, used for matching sample word data in the enterprise corpus sample with additional word data respectively included in a plurality of initial additional data tuples, to obtain target additional word data corresponding to the enterprise corpus sample, wherein the initial additional data tuple includes additional word data and a meaning vector corresponding to the additional word data; A sample vector determination module, used for vectorizing the enterprise corpus sample to obtain a sample vector; A target tuple determination module, used for calculating the similarity between the sample vector and the interpretation vector corresponding to the target additional word data to obtain a target additional data tuple; A vector fusion module, used for fusing the sample vector with the interpretation vectors respectively included in the target additional data tuple to obtain a target sample vector; The training module is used to pre-train the language model based on the target sample vector corresponding to the enterprise corpus sample to obtain a pre-trained language model.

10. A service processing device, comprising: A result determination module, in response to receiving the enterprise corpus data input by the target object, inputs the enterprise corpus data into the language model pre-trained by the language model pre-training method according to any one of claims 1 to 7, and outputs the business processing result corresponding to the enterprise corpus data; A sending module is used to send the business processing result to the target object.

11. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images