Language model training method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a language model training method and device, electronic equipment and storage medium, relating to artificial intelligence technical fields such as machine learning and natural language processing. The specific implementation scheme is: converting sample data in a non-text format into sample data in a text format; and training a language model based on the converted sample data in the text format. The present disclosure can effectively realize unified modeling of heterogeneous data and improve modeling efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as machine learning and natural language processing, and particularly to a method, apparatus, electronic device and storage medium for training a language model. Background Technology

[0002] In existing business applications, various data structures are used. For example, there are relatively simple and fixed data structures such as text and images, as well as complex structured data such as databases, tables, and knowledge graphs.

[0003] When performing specific business processing based on data, different data formats are typically processed in different ways. For example, when modeling language models based on data of different formats, it is difficult to use the same method for modeling because the recognition methods for different formats are different. Therefore, in existing technologies, developers model language models separately for different formats. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for training a language model.

[0005] According to one aspect of this disclosure, a method for training a language model is provided, comprising:

[0006] Convert non-text format sample data into text format sample data;

[0007] The language model is trained based on the sample data in the converted text format.

[0008] According to another aspect of this disclosure, a language model training apparatus is provided, comprising:

[0009] The conversion module is used to convert non-text format sample data into text format sample data;

[0010] The training module is used to train the language model based on sample data in the converted text format.

[0011] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0015] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described above and any possible implementation thereof.

[0016] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0017] According to the technology disclosed herein, unified modeling of heterogeneous data can be effectively achieved, thereby improving modeling efficiency.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0022] Figure 3 This is a schematic diagram based on the third embodiment of this disclosure;

[0023] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0024] Figure 5 This is a block diagram of an electronic device used to implement the methods of the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Obviously, the described embodiments are only some, not all, of the embodiments disclosed herein. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0027] It should be noted that the terminal devices involved in the embodiments of this disclosure may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers; the display devices may include, but are not limited to, personal computers, televisions, and other devices with display functions.

[0028] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] Given that in the current technology, different language models are used to model various data formats, resulting in high costs and low efficiency in developing and maintaining multiple language models for multiple data formats.

[0030] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; as shown Figure 1 As shown in the figure, this embodiment provides a method for training a language model, which may specifically include the following steps:

[0031] S101. Convert non-text format sample data into text format sample data;

[0032] S102. Train the language model based on the sample data in the converted text format.

[0033] The language model training method in this embodiment can be used to model language data from heterogeneous datasets. Specifically, during the modeling process, non-text format sample data can be converted into text format sample data. This achieves a unified format conversion of heterogeneous data, ensuring all data is converted into text format sample data. The converted text format sample data is then used to train the language model.

[0034] For example, the non-text format sample data in this embodiment may include image format sample data. It may also include structured data sample data, such as databases, tables, and knowledge graphs, etc.

[0035] Regardless of the non-text format of the sample data, this embodiment can adopt a unified processing method to convert the non-text format sample data into text format sample data. This enables unified modeling of sample data of different formats, and subsequent unified management and maintenance, which can effectively improve the management and maintenance efficiency of the language model.

[0036] The language model training method in this embodiment converts non-text format sample data into text format sample data; and trains the language model based on the converted text format sample data. This method can unify the format of heterogeneous data and perform unified modeling, avoiding the need to model, manage, and maintain data of different formats separately. It can effectively achieve unified modeling of heterogeneous data, save development, management, and maintenance costs, and thus effectively improve the efficiency of modeling, management, and maintenance.

[0037] Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure; as shown Figure 2 As shown in the figure, this embodiment provides a method for training a language model, which may specifically include the following steps:

[0038] S201. Detect whether the current sample data is in non-text format; if yes, proceed to step S202; otherwise, determine that the sample data is in text format, discard the sample data, and return to continue detecting the next sample data. The sample data in this embodiment is pre-mined. It may include positive sample data and negative sample data.

[0039] It should be noted that, in order to improve the accuracy of the mined samples, for example, before step S201, non-text format sample data of user clicks can be mined from online logs as positive sample data.

[0040] The technical solution of this embodiment can be applied to a scenario that includes non-textual sample data. To enable the language model to learn and adapt to this domain, the pre-trained language model can be fine-tuned using sample data from that domain. Since the purpose of this embodiment is to enable the language model to learn the ability to recognize sample data after converting non-textual sample data to text format, textual sample data is temporarily discarded and not considered in this scenario.

[0041] S202. Detect whether the non-text format sample data includes structured data. If yes, proceed to step S203; otherwise, proceed to step S204.

[0042] The structured data in this embodiment may include databases, tables, and knowledge graphs, etc., and is not limited thereto.

[0043] S203. Concatenate the data from the structured data and convert it into sample data in text format; proceed to step S206.

[0044] In this embodiment, no complex rules are set to convert structured data into text data. The converted text data in this embodiment can be understood as text data described in natural language, which is suitable for modeling natural language models.

[0045] In this embodiment, the various parts of the structured data are concatenated through simple splicing to convert the structured data into text format sample data. Considering that most data in computers is stored in simple structures such as strings and numbers, or complex structures such as JSON and tables, the most basic data structures such as numbers, strings, and lists can be easily converted to text format. For slightly more complex data structures such as JSON and tables, the following conversion method can be used:

[0046] For JSON data, the key and value information can be extracted and concatenated into text, for example:

[0047] The original structured data is:

[0048] Data = {

[0049] "key1":"value1",

[0050] "key2":"value2"

[0051] }

[0052] The converted text format data can be represented as:

[0053] Data="${key1}:${value1},${key2}:${value2}"

[0054] If the original structured data contains nested structures, you can transform the nested substructures using the method described above, and then append the parent node's key to the transformation result of the substructure.

[0055] For tabular data, the content of each cell can be concatenated with its header to obtain short text data, and then all the short texts from all cells can be concatenated sequentially. For example:

[0056] The original data is: data =

[0057] Value1 Value2 Value3 Value4

[0058] The converted text format data can be represented as:

[0059] data="${H1}:${Value1},${H2}:${Value2},${H2}:${Value2},${H2}:${Value2}"

[0060] Following the method described above, all structured data can be quickly converted into text format, which facilitates unified modeling based on text format data.

[0061] S204. Detect whether the non-text format sample data includes images. If yes, proceed to step S205; otherwise, discard the sample data, return to step S201, and start detecting the next sample data.

[0062] In other words, this embodiment uses non-text format sample data, including structured data and images, as an example. In practical applications, it can be extended to other non-text format sample data, such as semi-structured data, which includes both unstructured text data and structured data. In this case, the structured data part can be converted into text according to the method in this embodiment, and then concatenated with the text data part according to its position in the original data to obtain the converted plain text format sample data. Alternatively, in practical applications, it can be extended to other non-text format sample data, with the same principle, which will not be elaborated here. In this embodiment, other forms of non-text format sample data other than images and structured data are not considered and can be ignored and discarded, and will not be used for language modeling.

[0063] S205. Extract the text corresponding to the image to obtain sample data in text format; proceed to step S206.

[0064] For example, in this embodiment, sample data including images can be converted into text format sample data in the following two ways:

[0065] The first method uses Optical Character Recognition (OCR) to recognize the text in the image and obtain sample data in text format.

[0066] Specifically, if the image contains text, this method can be used to extract the corresponding text from the image, thereby converting the image-formatted sample data into text-formatted sample data.

[0067] The second method uses a pre-trained information extraction model to extract the text corresponding to the image, obtaining sample data in text format.

[0068] Specifically, if the image contains not only text but also other objects, or if the image does not contain text, a pre-trained information extraction model can be used to extract the text in the image, or the text corresponding to each object in the image, thereby converting the image-formatted sample data into text-formatted sample data.

[0069] Additionally, it should be noted that if the image only contains text, in addition to the first method described above, a pre-trained information extraction model can be used to extract the text from the image, thus converting the image-formatted sample data into text-formatted sample data. However, if the image also contains other objects besides text, the second method described above must be used to convert the image-formatted sample data into text-formatted sample data; the first method cannot be used.

[0070] Using any of the above methods, it is possible to accurately convert image-formatted sample data into text-formatted sample data.

[0071] S206. Based on the sample data in the converted text format, fine-tune the pre-trained language model.

[0072] The language model in this embodiment is a natural language processing model. By uniformly converting all non-textual sample data into textual sample data, unified modeling of all heterogeneous sample data can be achieved.

[0073] In practical applications, model training can include two phases: pre-training and fine-tuning. The pre-training phase involves training the model from scratch on a large dataset to achieve a good performance level. The fine-tuning phase, on the other hand, involves fine-tuning the pre-trained model using a dataset specific to a particular task or domain, adapting the model to that dataset. Fine-tuning saves time and computational resources while enabling the model to quickly achieve better results.

[0074] In this embodiment, when training the language model based on the sample data in the converted text format, it can be applied in the pre-training stage or the fine-tuning stage.

[0075] For example, if the source data consists of abundant text-formatted sample data, this embodiment can utilize this rich text-formatted sample data for pre-training the language model. Conversely, non-text-formatted sample data, when converted to text format, has a relatively smaller volume. This converted text-formatted sample data can be used during the fine-tuning stage to fine-tune the pre-trained language model. This not only shortens the training time but also ensures that the fine-tuned language model can effectively handle converted text-formatted sample data. In other words, the pre-trained language model in this embodiment can be pre-trained based on a large amount of natural language text.

[0076] In the semantic recall scenario, the language model in this embodiment can be a semantic retrieval module and / or a ranking model. A language model can be pre-trained using state-of-the-art natural language processing (NLP) training methods. Based on this model, many mainstream NLP tasks can achieve excellent results. For example, in the technical solution of this embodiment, converted text-formatted sample data can be used to fine-tune the pre-trained language model, achieving recall and ranking of all data.

[0077] Specifically, semantic retrieval and ranking models can be pre-trained using a large amount of weakly supervised data from text search scenarios. Then, during the fine-tuning phase, the pre-trained semantic retrieval and ranking models are applied to data of various formats. This allows niche scenarios lacking data to benefit from language models while also achieving better results with the help of large amounts of text data.

[0078] The language model training method in this embodiment, by adopting the above-described scheme, can convert non-text format sample data into text format sample data. Then, based on the converted text format sample data, the pre-trained language model can be fine-tuned, making the language model suitable for processing data in the converted text format. Therefore, the language model trained in this embodiment can be uniformly modeled, managed, and maintained based on sample data of all different formats. Compared with the existing method of separately modeling, managing, and maintaining sample data of different formats, this not only saves costs but also effectively improves the efficiency of modeling, management, and maintenance.

[0079] For example, in a specific domain scenario where the sample data is not in text format, the above method can be used. Figure 2 The technical solution of the illustrated embodiment initially fine-tunes the language model in this domain scenario based on weakly supervised data. For further iteration, the end-to-end performance of this language model in the domain scenario can be evaluated to identify areas where the language model performs poorly and then targeted optimization can be performed. For example, this could include the following aspects:

[0080] The first aspect is the robustness of the model. This can be specifically addressed in the above... Figure 2 Following the illustrated embodiment, the following steps are included:

[0081] (a1) Based on the test set of the preset scenario, check whether the accuracy of the trained language model meets the first preset requirement; if it does not meet the requirement, proceed to step (b1); otherwise, do not perform any operation for the time being.

[0082] (b1) Mining sample data from a pre-defined scenario;

[0083] (c1) Train the language model based on sample data from a preset scenario.

[0084] The preset scenario in this embodiment can be a scenario where sample data in non-text format is used to test the trained language model.

[0085] For example, the accuracy of the language model is tested using a test set from the preset scenario to see if it meets a first preset requirement. This first preset requirement can refer to a preset accuracy requirement, such as an accuracy of 90% or higher, or an accuracy of 95% or higher, etc. If the accuracy does not meet this requirement, it indicates that the language model needs further fine-tuning and optimization using sample data from the preset scenario. At this point, further mining of sample data from the preset scenario is used to train the language model. Alternatively, it can be said that based on the mined sample data from the preset scenario, the language model is further fine-tuned to adapt it to the preset scenario, thereby further improving the accuracy of the language model.

[0086] For example, in entity attribute question answering scenarios, language models may have problems such as entity matching errors and attribute range matching errors. In this case, certain strategies can be used to target and mine such data to further refine the language model and improve its accuracy.

[0087] Secondly, there's the issue of over-recall in the model. Taking a language model as an example for semantic retrieval, the corresponding... Figure 2 The fine-tuning in the illustrated embodiment refers to fine-tuning the semantic retrieval model. Specifically, this can be achieved through the above... Figure 2 Following the illustrated embodiment, the following steps are included:

[0088] (a2) Based on the semantic retrieval test set and the original semantic retrieval model, check whether the recall result of the semantic retrieval model is within the second preset requirement; if not, proceed to step (b2); otherwise, if it is, do not perform any operation for the time being.

[0089] (b2) Based on the recall results, the semantic retrieval model is trained.

[0090] The original semantic retrieval model in this embodiment was trained using sample data in text format as the source data. That is, in this embodiment, the original semantic retrieval model is used to test the trained semantic retrieval model.

[0091] Specifically, for any query in the semantic retrieval test set, the recall results of the original semantic retrieval model and the recall results of the trained semantic retrieval model can be obtained; the difference between the two recall results is compared to see if it falls within a second preset requirement. If the semantic retrieval model recalls search results that should not have been recalled, then based on this recall result, corresponding negative sample data needs to be constructed to train the semantic retrieval model for further fine-tuning, so that the semantic retrieval model no longer recalls results that should not have been recalled, thereby further improving the model's accuracy.

[0092] In practical applications, for simple queries in online logs, a well-trained semantic retrieval model can return accurate results, but it may also trigger some answers that shouldn't return results. In this case, as in this embodiment, the method of taking the difference (diff) between the model and the original semantic retrieval model trained on general text can be used to mine the recalled content as negative samples and add them to the training set to further refine the semantic retrieval model and improve its accuracy.

[0093] Thirdly, there's the issue of model scoring confidence. This problem can occur in models where the language model is used for semantic retrieval or ranking. For example, for some boundary samples, the language model may exhibit scoring drift and confidence issues. Based on this problem, specific solutions can be found in the above... Figure 2 Following the illustrated embodiment, the following steps are included:

[0094] (a3) Obtain multiple reference language models under different settings;

[0095] (b3) Based on the test samples of the test set and multiple reference models, check whether the processing capability of the language model meets the third preset requirement; if it does not meet the requirement, proceed to step (c3); otherwise, do not perform any operation for the time being.

[0096] (c3) Train the language model based on the test samples.

[0097] For example, to determine whether the language model's processing capability meets the third preset requirement, based on test samples in the test set and multiple reference models, we can obtain the corresponding test processing results of the language model, such as the language model's score for the test sample, based on the test samples. Then, based on the test samples and multiple reference language models, we obtain a comprehensive processing result. This comprehensive processing result can be understood as a combination of the processing results of multiple reference language models on the test sample. For example, we can obtain the scores of each reference language model for the test sample. Then, based on the scores of each reference language model for the test sample, we obtain a comprehensive score. The comprehensive score can be equal to the average of the scores of multiple reference language models, or other mathematical formulas can be used to obtain the comprehensive score based on the scores of each reference language model for the test sample. Finally, based on the comprehensive processing result and the test processing result, we check whether the language model's processing capability meets the third preset requirement. For example, we compare the difference (diff) between the language model's score for the test sample and the comprehensive score to see if it meets the third preset requirement, such as whether it is less than a preset threshold. If so, it indicates that the language model's score is reasonable and not significantly different from the comprehensive score. If the third preset requirement is not met, such as exceeding the preset threshold, it can be considered that the language model's scoring of the current test sample is unreasonable. In this case, test samples with incorrect scores by the language model can be identified and added to the training set. Only augmentation training is needed to improve the accuracy of the language model.

[0098] By conducting tests in the above three aspects, and when the requirements are not met, targeted mining of sample data is used to iterate the language model several times, resulting in an end-to-end language model that is adapted to the target domain scenario and can surpass the performance of the original system.

[0099] By employing the three enhanced training methods described above, the accuracy of the language model can be further improved, making it more suitable for the corresponding domain scenarios.

[0100] In existing technologies, for common data such as text and images, researchers invest significant effort in developing high-performing search algorithms for each data category. This is because such data has a uniform format and large volume, and targeted improvements in retrieval capabilities for this type of data can significantly enhance the final search results. However, structured data such as tables and knowledge graphs often have diverse formats, making it difficult to model them using a single approach. Furthermore, the scale of each data type is relatively small, resulting in less benefit from designing a single search algorithm for each. Consequently, the search performance for various structured data types is often inconsistent, becoming a bottleneck in the overall search business. The solution based on the embodiments of this disclosure allows for unified modeling, management, and maintenance based on sample data of various formats. For example, a language model can be trained using a large amount of text data. Then, non-text sample data from the domain scenario can be converted into text sample data, and the trained language model can be fine-tuned based on the converted text sample data to ensure its suitability for the domain scenario. In practical applications, non-textual sample data from various domain scenarios can be used to fine-tune the language model, enabling it to adapt to various domain scenarios and process data of various formats, making it highly practical.

[0101] The language model training method described in this embodiment can be implemented using the same simple architecture for both unstructured text data and structured or semi-structured data. Good results can be obtained by fine-tuning each type of data on a pre-trained language model, i.e., the base model, with only a small amount of data.

[0102] By employing the language model training method described in this embodiment, the development cost is low. Using only sample data in different text formats from various domains, the language model can be fine-tuned to quickly achieve model transfer across different domains. This eliminates the need to develop complex templates and rules based on business logic, significantly saving manpower and development costs. State-of-the-art pre-trained language models can be used for all types of data, and the advantages of text data can be transferred to other data types with smaller datasets, effectively improving the corresponding language processing performance.

[0103] Figure 3 This is a schematic diagram based on the third embodiment of this disclosure; as shown Figure 3 As shown, this embodiment provides a language model training device 300, including:

[0104] The conversion module 301 is used to convert non-text format sample data into text format sample data;

[0105] Training module 302 is used to train the language model based on sample data in the converted text format.

[0106] The language model training device 300 in this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0107] Figure 4 This is a schematic diagram based on the fourth embodiment of the present disclosure; as shown Figure 4 As shown, this embodiment provides a language model training device 400, including: [the aforementioned...] Figure 3 The same-named and same-function module conversion module 401 and training module 402.

[0108] In this embodiment, the conversion module 401 is used for:

[0109] If the non-text format sample data includes structured data, concatenate the data from the structured data and convert it into text format sample data; or

[0110] If the non-text format sample data includes images, extract the text corresponding to the images to obtain the text format sample data.

[0111] Furthermore, in one embodiment of this disclosure, the conversion module 401 is configured to:

[0112] Using optical character recognition (OCR) methods, text in images is identified to obtain sample data in text format; and / or

[0113] A pre-trained information extraction model is used to extract the text corresponding to the image, resulting in text-formatted sample data.

[0114] Furthermore, such as Figure 4 As shown, the language model training device 400 in this embodiment may further include:

[0115] Mining module 403 is used to mine non-text format sample data of user clicks from online logs as positive sample data.

[0116] Furthermore, such as Figure 4 As shown, the language model training device 400 in this embodiment may also include a detection module 404;

[0117] The detection module 404 is used to detect whether the accuracy of the trained language model meets the first preset requirement based on a test set of a preset scenario; the preset scenario is the application scenario of non-text format sample data.

[0118] The mining module 403 is also used to mine sample data of a preset scenario if the target is not met.

[0119] Training module 402 is used to train the language model based on sample data from a preset scenario.

[0120] Furthermore, in one embodiment of this disclosure,

[0121] The detection module 404 is also used to detect whether the recall result of the semantic retrieval model is within the second preset requirement based on the semantic retrieval test set and the original semantic retrieval model when the language model is a semantic retrieval model; the original semantic retrieval model is trained using sample data in text format as the source data;

[0122] Training module 402 is also used to train the semantic retrieval model based on the recall results if the object is not found.

[0123] Furthermore, in one embodiment of this disclosure, the language model training device 400 of this embodiment may also include an acquisition module 405;

[0124] Module 405 is used to acquire multiple reference language models under different settings;

[0125] The detection module 404 is also used to detect whether the processing capability of the language model meets the third preset requirement based on the test samples of the test set and multiple reference models;

[0126] Training module 402 is also used to train the language model based on the test samples if the target is not met.

[0127] Furthermore, in one embodiment of this disclosure, the detection module 404 is configured to:

[0128] Based on the test samples in the test set, obtain the test processing results corresponding to the language model;

[0129] Based on test samples and multiple reference language models, obtain comprehensive processing results;

[0130] Based on the comprehensive processing results and test results, the processing capability of the language model is tested to see if it meets the third preset requirement.

[0131] Furthermore, in one embodiment of this disclosure, the training module is used for:

[0132] The pre-trained language model is fine-tuned based on sample data in the converted text format.

[0133] The language model training device 400 in this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0134] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0138] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the methods of this disclosure. For example, in some embodiments, the methods of this disclosure may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the methods of this disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the methods of this disclosure by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a language model, used to model language models of heterogeneous data, comprising: The language model is pre-trained using sample data in text format; The language model includes a semantic retrieval model and / or ranking model in the semantic recall scenario; the text-formatted sample data is weakly supervised data; Non-text format sample data is converted into text format sample data; the text format sample data is text data described in natural language, suitable for modeling the language model that processes natural language; the non-text format sample data includes structured sample data, and when converting to text format sample data, the data in each part of the structured sample data is concatenated to obtain the text format sample data; the structured sample data includes databases, tables, knowledge graphs, or JSON format; for JSON format sample data, when converting to text format sample data, the contents of the included keys and values ​​are extracted and processed, and then concatenated into text; Based on the sample data in the converted text format, the pre-trained language model is fine-tuned so that the semantic retrieval model included in the language model can learn the ability to uniformly recall data in different formats in both text and non-text formats, and / or so that the ranking model included in the language model can learn the ability to uniformly rank data in different formats in both text and non-text formats.

2. The method according to claim 1, wherein, If the non-text format sample data includes images, the non-text format sample data is converted into text format sample data, including: Extract the text corresponding to the image to obtain sample data in the text format.

3. The method according to claim 2, wherein, Extract the text corresponding to the image to obtain sample data in the text format, including: Using optical character recognition (OCR) methods, the text in the image is identified to obtain sample data in the text format; and / or A pre-trained information extraction model is used to extract the text corresponding to the image, resulting in sample data in the text format.

4. The method according to claim 1, wherein, Before converting non-text format sample data to text format sample data, the method further includes: Mine non-text formatted sample data of user clicks from online logs as positive sample data.

5. The method according to claim 1, wherein, After training the language model based on the sample data in the converted text format, the process also includes: Based on a test set of a preset scenario, the accuracy of the trained language model is tested to see if it meets a first preset requirement; the preset scenario is a scenario in which non-text format sample data is applied. If the target is not met, then mine sample data from the preset scenario; The language model is trained based on sample data from the preset scenario.

6. The method according to claim 1, wherein, After training the language model based on the sample data in the converted text format, the process also includes: If the language model is a semantic retrieval model, based on the semantic retrieval test set and the original semantic retrieval model, it is detected whether the recall result of the semantic retrieval model is within the second preset requirement; the original semantic retrieval model is trained using sample data in text format as the source data; If not, the semantic retrieval model is trained based on the recall results.

7. The method according to claim 1, wherein, After training the language model based on the sample data in the converted text format, the process also includes: Obtain multiple reference language models under different settings; Based on the test samples of the test set and the multiple reference language models, the processing capability of the language model is tested to see if it meets the third preset requirement. If the target is not met, the language model is trained based on the test samples.

8. The method according to claim 7, wherein, Based on test samples from the test set and the multiple reference language models, the processing capability of the language model is tested to determine whether it meets the third preset requirement, including: Based on the test samples in the test set, obtain the test processing results corresponding to the language model; Based on the test samples and the multiple reference language models, a comprehensive processing result is obtained; Based on the comprehensive processing results and the test processing results, it is determined whether the processing capability of the language model meets the third preset requirement.

9. A language model training device for modeling language models of heterogeneous data, comprising: The training module is used to pre-train the language model using sample data in text format. The language model includes a semantic retrieval model and / or a ranking model in the semantic recall scenario; the text-formatted sample data is weakly supervised data; The conversion module is used to convert non-text format sample data into text format sample data. The text format sample data is text data described in natural language, suitable for modeling the language model that processes natural language. The non-text format sample data includes structured sample data. When converting to text format sample data, the data in each part of the structured sample data is concatenated to obtain the text format sample data. The structured sample data includes databases, tables, knowledge graphs, or JSON format. For JSON format sample data, when converting to text format sample data, the key and value contents are extracted and processed, and then concatenated into text. The training module is further configured to fine-tune the pre-trained language model based on the sample data in the converted text format, so that the semantic retrieval model included in the language model can learn the ability to uniformly recall data in different formats in text and non-text formats, and / or so that the ranking model included in the language model can learn the ability to uniformly rank data in different formats in text and non-text formats.

10. The apparatus according to claim 9, wherein, The conversion module is used for: If the non-text format sample data includes images, extract the text corresponding to the images to obtain the text format sample data.

11. The apparatus according to claim 10, wherein, The conversion module is used for: Using optical character recognition (OCR) methods, the text in the image is identified to obtain sample data in the text format; and / or A pre-trained information extraction model is used to extract the text corresponding to the image, resulting in sample data in the text format.

12. The apparatus according to claim 9, wherein, The device further includes: The mining module is used to extract non-text formatted sample data from online logs as positive sample data.

13. The apparatus according to claim 12, wherein, The device also includes a detection module; The detection module is used to detect whether the accuracy of the trained language model meets a first preset requirement based on a test set of a preset scenario; the preset scenario is a scenario in which sample data in non-text format is applied. The mining module is also used to mine sample data of the preset scenario if the target is not met. The training module is used to train the language model based on sample data from the preset scenario.

14. The apparatus according to claim 13, wherein: The detection module is further configured to, if the language model is a semantic retrieval model, detect whether the recall result of the semantic retrieval model is within the second preset requirement based on the semantic retrieval test set and the original semantic retrieval model; the original semantic retrieval model is trained using sample data in text format as the source data; The training module is also used to train the semantic retrieval model based on the recall results if the object is not found.

15. The apparatus according to claim 13, wherein, The device also includes an acquisition module; The acquisition module is used to acquire multiple reference language models under different settings; The detection module is also used to detect whether the processing capability of the language model meets the third preset requirement based on the test samples of the test set and the multiple reference language models; The training module is also used to train the language model based on the test samples if the target is not met.

16. The apparatus according to claim 15, wherein, The detection module is used for: Based on the test samples in the test set, obtain the test processing results corresponding to the language model; Based on the test samples and the multiple reference language models, a comprehensive processing result is obtained; Based on the comprehensive processing results and the test processing results, it is determined whether the processing capability of the language model meets the third preset requirement.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images