Artificial intelligence-based text processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a text processing method and device, equipment and storage medium, relating to the technical field of artificial intelligence and natural language processing. The method comprises: obtaining a target sentence text, and performing word segmentation processing on the target sentence text to obtain a target word group; inputting the target word group into a small network model to obtain a first word group embedding representation; inputting the first word group embedding representation into a nonlinear mapping model to obtain a second word group embedding representation; and obtaining a sentence embedding representation according to the second word group embedding representation, wherein the accuracy of the output result of the nonlinear mapping model can approach the output result of a pre-trained large network model, thereby making up for the problem of inaccurate output result of the small network model. Since the output result of the small network model is fitted to the output result of the large network model by using the simple structure of the nonlinear mapping model, the processing time will not increase significantly, and the balance between accuracy and timeliness is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text processing method and device based on artificial intelligence, an equipment and a storage medium. BACKGROUND

[0002] With the improvement of computing power and data volume, artificial intelligence technology has been booming in recent years and is widely used in various scenarios of the Internet. Natural language processing (NLP) as an important branch of artificial intelligence has a wide range of applications in classic question and answer system scenarios such as intelligent voice interaction, online customer service, knowledge search, and emotional chat. In the question and answer system, a natural language processing model is usually used to encode the semantics of the user input text, and then an answer to the user's question is given based on the semantic encoding content. Generally speaking, in the question and answer system, large natural language processing models (such as BERT, T5, etc.) have better performance than small natural language processing models (such as Word2Vec, GloVe, Fasttext, etc.). However, the model parameter volume of large natural language processing models is much larger than that of small natural language processing models. The landing of the question and answer system often involves the computing timeliness of the model. Although the increase in model parameter volume can improve the accuracy and recall rate of the question and answer system, it also increases the time consumption of text processing, thereby reducing the user experience of the question and answer system. Therefore, it is necessary to balance the timeliness and accuracy of natural language processing tasks. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a text processing method and device based on artificial intelligence, and an equipment and a storage medium, aiming to balance the timeliness and accuracy of natural language processing tasks.

[0004] In a first aspect, the embodiments of the present application propose a text processing method based on artificial intelligence, which comprises the following steps:

[0005] Obtaining a target sentence text and performing word segmentation processing on the target sentence text to obtain at least one target word group;

[0006] Inputting each target word group into a pre-trained small network model to obtain a first word group embedding representation corresponding to the target word group output by the small network model;

[0007] Inputting the first word group embedding representation corresponding to each target word group into a pre-trained nonlinear mapping model to obtain a second word group embedding representation corresponding to the word group output by the nonlinear mapping model;

[0008] obtaining a sentence embedding representation corresponding to the target sentence text according to a second word group embedding representation corresponding to each of the target word groups;

[0009] The pre-trained nonlinear mapping model is trained by the following steps: inputting training word groups into a pre-trained large network model and the pre-trained small network model respectively to obtain a third word group embedding representation output by the pre-trained large network model and a fourth word group embedding representation output by the pre-trained small network model, taking the fourth word group embedding representation as input data of an initial nonlinear mapping model, taking the third word group embedding representation as expected output data of the initial nonlinear mapping model, and training the initial nonlinear mapping model to obtain the pre-trained nonlinear mapping model.

[0010] With reference to the first aspect, in a possible implementation manner of the first aspect, after performing the word segmentation processing on the target sentence text, before inputting the word groups into the pre-trained small network model, the method further includes:

[0011] determining stop words and non-stop words from at least one initial word group obtained through the word segmentation processing;

[0012] determining the non-stop words as the target word groups to be input into the pre-trained small network model.

[0013] With reference to the first aspect, in a possible implementation manner of the first aspect, before inputting the training word groups into the pre-trained large network model and the pre-trained small network model respectively, the method further includes:

[0014] obtaining training sentence texts from a preset sentence text training set;

[0015] extracting a center word from the training sentence texts to determine the training word groups according to the center word.

[0016] With reference to the first aspect, in a possible implementation manner of the first aspect, the extracting a center word from the training sentence texts to determine the training word groups according to the center word includes:

[0017] extracting the center word and at least one adjacent word of the center word from the training sentence texts to take the center word and the adjacent word corresponding to the center word as the training word groups.

[0018] With reference to the first aspect, in a possible implementation manner of the first aspect, the taking the fourth word group embedding representation as input data of an initial nonlinear mapping model, taking the third word group embedding representation as expected output data of the initial nonlinear mapping model, and training the initial nonlinear mapping model include:

[0019] embedding representation of the fourth word group is input into the initial nonlinear mapping model to obtain a fifth word group embedding representation output by the initial nonlinear mapping model;

[0020] a loss value is calculated according to the third word group embedding representation and the fifth word group embedding representation;

[0021] whether a training end condition of the initial nonlinear mapping model is satisfied is determined according to the loss value;

[0022] if the training end condition of the initial nonlinear mapping model is satisfied, the pre-trained nonlinear mapping model is obtained according to current model parameters of the initial nonlinear mapping model;

[0023] if the training end condition of the initial nonlinear mapping model is not satisfied, model parameters of the initial nonlinear mapping model are adjusted, and the initial nonlinear mapping model is continuously trained based on the training word groups.

[0024] With reference to the first aspect, in a possible implementation manner of the first aspect, the number of the training word groups is a plurality, and the determining whether the training end condition of the initial nonlinear mapping model is satisfied according to the loss value includes:

[0025] a loss value average is calculated according to loss values of the respective training word groups;

[0026] if the loss value average is less than or equal to a preset loss threshold, it is determined that the training end condition of the initial nonlinear mapping model is satisfied;

[0027] if the loss value average is greater than the preset loss threshold, it is determined that the training end condition of the initial nonlinear mapping model is not satisfied.

[0028] With reference to the first aspect, in a possible implementation manner of the first aspect, the calculating the loss value according to the third word group embedding representation and the fifth word group embedding representation includes:

[0029] a similarity between the third word group embedding representation and the fifth word group embedding representation is calculated;

[0030] the loss value of the training word group is determined according to the similarity between the third word group embedding representation and the fifth word group embedding representation.

[0031] Secondly, the present application provides a text processing device based on artificial intelligence, which comprises:

[0032] a preprocessing module configured to acquire a target sentence text and perform word segmentation processing on the target sentence text to obtain at least one target word group;

[0033] The first prediction module is used to input each of the target word groups into a pre-trained small network model to obtain the first word group embedding representation corresponding to the target word group output by the small network model;

[0034] The second prediction module is used to input the first word group embedding representation corresponding to each of the target word groups into a pre-trained nonlinear mapping model to obtain the second word group embedding representation corresponding to the word group output by the nonlinear mapping model. The pre-trained nonlinear mapping model is trained through the following steps: inputting training word groups into a pre-trained large network model and a pre-trained small network model respectively to obtain the third word group embedding representation output by the pre-trained large network model and the fourth word group embedding representation output by the pre-trained small network model; using the fourth word group embedding representation as the input data of the initial nonlinear mapping model; using the third word group embedding representation as the expected output data of the initial nonlinear mapping model; and training the initial nonlinear mapping model to obtain the pre-trained nonlinear mapping model.

[0035] The determining module is used to obtain the statement embedding representation corresponding to the target statement text based on the second phrase embedding representation corresponding to each of the target phrases.

[0036] Thirdly, embodiments of this application propose an electronic device, including a memory and a processor, wherein the memory stores a computer program or instructions, and the processor executes the computer program or instructions to implement the artificial intelligence-based text processing method described in the first aspect above.

[0037] Fourthly, embodiments of this application propose a computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, implements the artificial intelligence-based text processing method described in the first aspect above.

[0038] The text processing method, apparatus, device, and storage medium based on artificial intelligence proposed in this application first acquires the target sentence text and performs word segmentation on the target sentence text to obtain at least one target word group; then, each of the target word groups is input into a pre-trained small network model to obtain a first word group embedding representation corresponding to the target word group output by the small network model; then, the first word group embedding representation corresponding to each of the target word groups is input into a pre-trained nonlinear mapping model to obtain a second word group embedding representation corresponding to the word group output by the nonlinear mapping model; finally, the sentence embedding representation corresponding to the target sentence text is obtained based on the second word group embedding representation corresponding to each of the target word groups. The solution in this application uses a combination of a small network model and a nonlinear mapping model to achieve natural language processing of text. The pre-trained nonlinear mapping model is trained through the following steps: training phrases are input into a pre-trained large network model and the pre-trained small network model respectively, resulting in a third phrase embedding representation output by the pre-trained large network model and a fourth phrase embedding representation output by the pre-trained small network model. The fourth phrase embedding representation is used as input data for the initial nonlinear mapping model, and the third phrase embedding representation is used as the expected output data for the initial nonlinear mapping model. The initial nonlinear mapping model is then trained to obtain the pre-trained nonlinear mapping model. The accuracy of the output result of the nonlinear mapping model trained in this way can approach the output result of the pre-trained large network model, compensating for the inaccuracy of the small network model's output result. Furthermore, since a simple nonlinear mapping model is used to fit the output result of the small network model to the output result of the large network model, the overall processing time does not increase significantly, achieving a balance between accuracy and timeliness in natural language processing tasks. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating an artificial intelligence-based text processing method provided in an embodiment of this application.

[0040] Figure 2 This is a schematic diagram of the architecture of the text representation model provided in the embodiments of this application;

[0041] Figure 3 This is a schematic diagram of the training process of the nonlinear mapping model provided in the embodiments of this application;

[0042] Figure 4 This is a schematic diagram of the training process of a nonlinear mapping model in one embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the structure of the artificial intelligence-based text processing device provided in the embodiments of this application;

[0044] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] First, let's analyze some of the terms used in this application:

[0049] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0050] Natural Language Processing (NLP) is a discipline that uses computer technology to analyze, understand, and process natural language. It utilizes computers as a powerful tool for language research, quantitatively analyzing linguistic information and providing language descriptions that can be used by both humans and computers. It comprises two main parts: Natural Language Understanding (NLU) and Natural Language Generation (NLG). NLP is primarily applied in machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and optical character recognition (OCR).

[0051] Embedding is a method of representing an object using a numerical vector. This object can be a word, an item, a movie, or something else. An item can be embedded because the distance between its vector and other item vectors reflects the similarity between those items. Furthermore, the distance vector between two vectors can even reflect the relationship between them.

[0052] Question Answering (QA) systems are an advanced form of information retrieval systems that answer user questions in natural language with accurate and concise natural language. QA systems include the following types: 1) Retrieval-based QA systems: Answers are derived from a fixed text corpus or the internet. The system retrieves answers by searching relevant documents and extracting relevant information. 2) Knowledge-based QA systems: The knowledge required to answer questions is stored in a structured format such as a database. The system first parses the question into a structured query, then retrieves relevant knowledge points and uses this knowledge to deduce the answer. 3) Frequently Asked Questions (FAQs) systems: These systems retrieve previously accumulated frequently asked questions to answer similar questions posed by users. 4) Reading Comprehension-based QA systems: These systems answer user questions by extracting text fragments from a given document or generating an answer.

[0053] Nonlinear mapping models refer to models where the mapping from input to output is nonlinear. As an example, a classic nonlinear mapping model is the RBF neural network, which includes an input layer, hidden layers, and an output layer. The transformation from the input space to the hidden layer space is nonlinear, while the transformation from the hidden layer space to the output layer space is linear. The basic idea of ​​an RBF network is to use RBFs as the "basis" of hidden units to construct the hidden layer space. This allows the input vector to be directly mapped to the hidden space without the need for weighted connections. Once the center point of the RBF is determined, this mapping relationship is also determined. The mapping from the hidden layer space to the output space is linear; that is, the network output is a linear weighted sum of the outputs of the hidden units, where the weights are the adjustable parameters of the network. The role of the hidden layers is to map the vector from a low-dimensional p to a high-dimensional h, thus making linearly inseparable vectors in the low dimension linearly separable in the high dimension—this is mainly due to the idea of ​​kernel functions. Therefore, the mapping from input to output is nonlinear, while the network output is linear with respect to the adjustable parameters. The weights of the network can be directly solved from the system of linear equations, which greatly speeds up the learning process and avoids local minima.

[0054] Stop words: In information retrieval, stop words are words or phrases that are automatically filtered out before or after processing natural language data (or text) to save storage space and improve search efficiency. These words are called stop words because if they are encountered during text processing, processing is immediately stopped and they are discarded. Discarding these words reduces the amount of indexing, increases retrieval efficiency, and usually improves search results. Stop words mainly include English characters, numbers, mathematical characters, punctuation marks, and frequently used single Chinese characters.

[0055] Artificial intelligence (AI) technology has flourished in recent years with the increase in computing power and data volume, and is widely used in various internet scenarios. Natural Language Processing (NLP), as an important branch of AI, has wide applications in classic question-answering systems such as intelligent voice interaction, online customer service, knowledge search, and emotion-based chat. Question-answering systems typically use NLP models to semantically encode user input text and then provide answers to user questions based on the semantically encoded content. Generally, in question-answering systems, large NLP models (such as BERT and T5) perform better than small NLP models (such as Word2Vec, GloVe, and Fasttext). However, large NLP models have significantly more parameters than small ones. The implementation of question-answering systems often involves the computational efficiency of the model. While increasing the number of model parameters can improve the accuracy and recall of the question-answering system, it also increases the time consumed in text processing, thus reducing the user experience. Therefore, it is necessary to strike a balance between the timeliness and accuracy of natural language processing tasks.

[0056] In view of this, embodiments of this application provide an artificial intelligence-based text processing method, apparatus, device, and storage medium, aiming to balance the timeliness and accuracy of natural language processing tasks.

[0057] The artificial intelligence-based text processing method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the artificial intelligence-based text processing method in this application embodiment is described.

[0058] The text processing method provided in this application relates to the fields of artificial intelligence and natural language processing. The AI-based text processing method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the AI-based text processing method, but is not limited to the above forms.

[0059] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0060] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0061] Please see Figure 1 , Figure 1 This is a flowchart illustrating an artificial intelligence-based text processing method provided in an embodiment of this application. Figure 1 As shown, the embodiment of this application provides an artificial intelligence-based text processing method, including but not limited to steps S110-S140, which will be described in turn below.

[0062] Step S110: Obtain the target sentence text and perform word segmentation on the target sentence text to obtain at least one target word group.

[0063] It is understandable that the target text is the object of the current natural language processing task. The target text can be the text obtained after speech recognition of the user's input question, or it can be the question text directly entered by the user.

[0064] It is understandable that word segmentation is required before natural language processing of the target text. Specifically, word segmentation of the target text to obtain at least one target word group includes: inputting the target text into a pre-trained word segmentation model to obtain at least one target word group output by the word segmentation model. It should be noted that the word segmentation model can be a general-domain word segmentation model or a specialized-domain word segmentation model. This application does not impose excessive limitations on the specific form of the word segmentation model or the training process.

[0065] In some embodiments, after segmenting the target sentence text and before inputting the phrase into a pre-trained small network model, the method further includes: determining stop words and non-stop words from at least one initial phrase obtained through segmentation; and determining the non-stop words as target phrases to be input into the pre-trained small network model.

[0066] Understandably, the target text is segmented to obtain at least one initial word group. Then, for each initial word group, it is determined whether it is a stop word. If it is a stop word, it is removed; if it is not a stop word, it is retained. The retained initial word groups are used as target word groups for input into a small network model. By removing stop words from the target text, the efficiency of text processing is improved.

[0067] Step S120: Input each target word group into the pre-trained small network model to obtain the first word group embedding representation corresponding to the target word group output by the small network model.

[0068] Among them, the small network model can be a text representation model based on prediction and static word embedding, such as Word2Vec, GloVe, Fasttext, etc. Here, the small network model is a model with fewer parameters and lower complexity compared to the large network model. The embodiments of this application do not limit the specific type of small network model.

[0069] It should be noted that the small network model used in step S120 is pre-trained. Specifically, it can be a small network model trained based on preset training text and labels. This application embodiment does not limit the training process of the small network model.

[0070] In this embodiment, the target word groups obtained by segmenting the target sentence text are input into a pre-trained small network model, so that the small network model outputs the first word group embedding representation corresponding to the target word group.

[0071] Step S130: Input the first word embedding representation corresponding to each target word group into the pre-trained nonlinear mapping model to obtain the second word embedding representation corresponding to the word group output by the nonlinear mapping model.

[0072] The nonlinear mapping model can be a nonlinear model such as an RBF neural network. In this embodiment, a pre-trained nonlinear mapping model maps the first word embedding representation output by a small network model to the second word embedding representation, thereby improving the performance of the word embeddings corresponding to the target word group and overcoming the problem of weak performance of word embeddings output by small network models. By using a nonlinear mapping model as an auxiliary model, a model with a small number of parameters can achieve computational performance that is close to that of a model with a large number of parameters. Furthermore, since a simple multi-layer nonlinear mapping model is used to fit the word embeddings output by the small network model to the word embeddings output by the large network model, the time cost does not increase significantly, thus solving the problem of computational efficiency.

[0073] Step S140: Obtain the statement embedding representation corresponding to the target statement text based on the second phrase embedding representation corresponding to each target phrase.

[0074] It is understandable that the second phrase embedding representations corresponding to each target phrase are combined to form the statement embedding representation corresponding to the target statement text.

[0075] Please see Figure 2 , Figure 2 A schematic diagram of the architecture of the text representation model provided in an embodiment of this application is shown. For example... Figure 2 As shown, this text representation model includes a preprocessing module, a small network model, and a nonlinear mapping model. The following section combines... Figure 2 The provided text representation model serves as an illustrative example of the text processing method in the embodiments of this application.

[0076] In one possible implementation, the text processing method of this application embodiment includes the following steps:

[0077] Step S210: Obtain the voice statement input by the user, and perform speech recognition on the voice statement to obtain the target statement text corresponding to the voice statement;

[0078] Step S220: Input the target sentence text into the preprocessing module, so that the preprocessing module performs word segmentation on the target sentence text based on the pre-trained word segmentation model to obtain multiple initial word groups, remove the stop words in the multiple initial word groups, and obtain multiple target word groups Word(n);

[0079] Step S230: Input multiple target word groups Word(n) into the small network model in sequence, so that the small network model outputs the first word group embedding representation Word_embedding'(n) corresponding to each target word group;

[0080] Step S240: Input the first word embedding representation Word_embedding'(n) corresponding to each target word group into the nonlinear mapping model, so that the nonlinear mapping model outputs the second word embedding representation Word_embedding"(n) corresponding to each target word group;

[0081] Step S250: Determine the statement embedding representation Text_embedding corresponding to the target statement text based on the second phrase embedding representation Word_embedding"(n) corresponding to each target phrase.

[0082] Please see Figure 3 , Figure 3 A schematic diagram illustrating the training process of the nonlinear mapping model provided in an embodiment of this application is shown. Figure 3 As shown, the pre-trained nonlinear mapping model of this application embodiment is obtained through the following steps S310 and S320:

[0083] Step S310: Input the training word groups into the pre-trained large network model and the pre-trained small network model respectively to obtain the third word group embedding representation output by the pre-trained large network model and the fourth word group embedding representation output by the pre-trained small network model.

[0084] The large-scale network model can be a natural speech model based on Transformer as the encoder, generally based on large-scale pre-trained models with dynamic word embeddings, such as BERT, ELMo, GPT, or T5. This application does not limit the specific type of the large-scale network model.

[0085] It should be noted that the large network model used in step S310 is pre-trained. Specifically, it can be a large network model trained based on preset training text and labels. This application embodiment does not limit the training process of the large network model.

[0086] In some embodiments, before inputting training word groups into a pre-trained large network model and a pre-trained small network model, the method further includes: obtaining training sentence text from a preset sentence text training set; and extracting center words from the training sentence text to determine training word groups based on the center words. The extraction of center words generally involves calculating the relevance of each word in the sentence text to measure the degree of relevance between each word and the sentence text, and then identifying and extracting the center words from the sentence text.

[0087] Specifically, extracting the central word from the training statement text to determine the training word group based on the central word includes: extracting the central word and at least one adjacent word of the central word from the training statement text, and using the central word and the adjacent word corresponding to the central word as the training word group.

[0088] For example, a fixed-size context window C can be set first, and K words near the center word can be obtained based on the context window C. These K words are the neighboring words of the center word, and then the center word and its neighboring words are used as training word groups.

[0089] Step S320: Use the fourth word group embedding representation as the input data of the initial nonlinear mapping model, use the third word group embedding representation as the expected output data of the initial nonlinear mapping model, and train the initial nonlinear mapping model to obtain a pre-trained nonlinear mapping model.

[0090] For example, the fourth word group embedding representation is used as the input data of the initial nonlinear mapping model, and the third word group embedding representation is used as the expected output data of the initial nonlinear mapping model. The training of the initial nonlinear mapping model may specifically include the following steps:

[0091] Step S321: Input the fourth word group embedding representation into the initial nonlinear mapping model to obtain the fifth word group embedding representation output by the initial nonlinear mapping model;

[0092] Step S322: Calculate the loss value based on the third phrase embedding representation and the fifth phrase embedding representation;

[0093] Step S323: Determine whether the training termination condition of the initial nonlinear mapping model is met based on the loss value;

[0094] Step S324: If the training termination condition of the initial nonlinear mapping model is met, obtain the pre-trained nonlinear mapping model based on the current model parameters of the initial nonlinear mapping model.

[0095] Step S325: If the training termination condition of the initial nonlinear mapping model is not met, adjust the model parameters of the initial nonlinear mapping model and continue to train the initial nonlinear mapping model based on the training phrases.

[0096] In some embodiments, there are multiple training word groups. Different training word groups are obtained from different training sentence texts in a preset sentence text training set. Specifically, the central word is extracted from each training sentence text, and then the training word group corresponding to each training sentence text is determined based on the central word and a preset context window. In this way, multiple training word groups can be obtained. Each training word group consists of a central word and adjacent words in the context of the central word.

[0097] When there are multiple training phrases, the training termination condition of the initial nonlinear mapping model can be determined based on the loss value, and may include:

[0098] Step S3231: Calculate the mean loss value based on the loss value of each training word group;

[0099] Step S3232: If the mean loss value is less than or equal to the preset loss threshold, determine that the training termination condition of the initial nonlinear mapping model is met.

[0100] Step S3233: If the mean loss value is greater than the preset loss threshold, it is determined that the training termination condition of the initial nonlinear mapping model is not met.

[0101] Understandably, the multiple training word groups mentioned above are first treated as a batch and sequentially input into a small network model and a large network model. The output data of the small network model is then input into the nonlinear mapping model. For each training word group, the loss value of that word is determined based on its word embedding representation in both the large network model and the nonlinear mapping model. Then, the loss values ​​of all words in the training word group are averaged to obtain the loss value of the training word group. Next, the loss values ​​of all training word groups are averaged to obtain the mean loss value. The mean loss value is compared with a preset loss threshold. If the mean loss value is less than or equal to the preset loss threshold, the training of the nonlinear mapping model ends; otherwise, the model parameters of the nonlinear mapping model are adjusted, and training is restarted based on the multiple training word groups.

[0102] For example, calculating the loss value based on the third and fifth word phrase embeddings may include the following steps: calculating the similarity between the third and fifth word phrase embeddings; and determining the loss value of the training word phrase based on the similarity between the third and fifth word phrase embeddings.

[0103] The similarity between the third and fifth word phrase embeddings can be determined by calculating the cosine distance between them. Generally, the smaller the cosine distance between the third and fifth word phrase embeddings (closer to 0), the more similar they are. In other words, the output of the nonlinear mapping model is closer to the output of the large network model.

[0104] Please see Figure 4 , Figure 4 A schematic diagram illustrating the training process of a nonlinear mapping model in one embodiment is shown. Figure 4 As shown, the training process of a nonlinear mapping model may include:

[0105] Step S410: Obtain training phrases, wherein the training phrases are phrases extracted from the training statement text through preset upper and lower windows, and the training phrases include the central word and several adjacent words near the central word;

[0106] Step S420: Input each word in the training phrase into the large network model and the small network model in sequence to obtain the phrase embedding representation Large_embedding output by the large network model and the phrase embedding representation Small_embedding output by the small network model.

[0107] Step S430: Input the word embedding representation Small_embedding output by the small network model into the nonlinear mapping model to obtain the Small_embedding' output by the nonlinear mapping model;

[0108] Step S440: Calculate the loss value based on the phrase embedding representation Large_embedding output by the large network model and the Small_embedding output by the nonlinear mapping model, and update the model parameters of the nonlinear mapping model based on the loss value until the preset model training termination condition is met.

[0109] Understandably, some downstream applications of natural language processing have high requirements for model computation time (e.g., question answering systems, where long computation time for word embeddings leads to a poor user experience). While using a model with a small number of parameters in downstream tasks results in shorter computation time, its performance is not as good as that of a large model. Therefore, this application aims to use an auxiliary model to enable a model with a small number of parameters to achieve computational performance that is close to that of a model with a large number of parameters. Since only a few nonlinear mapping layers are added to fit the two, the time cost will not increase significantly.

[0110] Please see Figure 5 In some possible embodiments of this application, an artificial intelligence-based text processing device 800 is also provided, which can implement the above-described artificial intelligence-based text processing method. The device 800 includes:

[0111] The preprocessing module 801 is used to acquire the target sentence text and perform word segmentation on the target sentence text to obtain at least one target word group;

[0112] The first prediction module 802 is used to input each target word group into a pre-trained small network model to obtain the first word group embedding representation corresponding to the target word group output by the small network model.

[0113] The second prediction module 803 is used to input the first word group embedding representation corresponding to each target word group into a pre-trained nonlinear mapping model to obtain the second word group embedding representation corresponding to the word group output by the nonlinear mapping model. The pre-trained nonlinear mapping model is trained through the following steps: inputting the training word groups into a pre-trained large network model and a pre-trained small network model respectively to obtain the third word group embedding representation output by the pre-trained large network model and the fourth word group embedding representation output by the pre-trained small network model; using the fourth word group embedding representation as the input data of the initial nonlinear mapping model; using the third word group embedding representation as the expected output data of the initial nonlinear mapping model; and training the initial nonlinear mapping model to obtain the pre-trained nonlinear mapping model.

[0114] The determination module 804 is used to obtain the statement embedding representation corresponding to the target statement text based on the second phrase embedding representation corresponding to each target phrase.

[0115] It should be noted that the specific implementation of the text processing device provided in this application is basically the same as the specific implementation of the text processing method described above, and will not be repeated here.

[0116] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned artificial intelligence-based text processing method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0117] Please see Figure 6 , Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device includes:

[0118] The processor 1010 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0119] The memory 1020 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010 using the text processing method of the embodiments of this application.

[0120] The input / output interface 1030 is used to implement information input and output;

[0121] The communication interface 1040 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0122] Bus 1050 transmits information between various components of the device (e.g., processor 1010, memory 1020, input / output interface 1030, and communication interface 1040);

[0123] The processor 1010, memory 1020, input / output interface 1030 and communication interface 1040 are connected to each other within the device via bus 1050.

[0124] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described artificial intelligence-based text processing method.

[0125] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0126] The text processing method, apparatus, device, and storage medium based on artificial intelligence proposed in this application first acquires the target sentence text and performs word segmentation on the target sentence text to obtain at least one target word group; then, each target word group is input into a pre-trained small network model to obtain a first word group embedding representation corresponding to the target word group output by the small network model; then, the first word group embedding representation corresponding to each target word group is input into a pre-trained nonlinear mapping model to obtain a second word group embedding representation corresponding to the word group output by the nonlinear mapping model; finally, the sentence embedding representation corresponding to the target sentence text is obtained based on the second word group embedding representation corresponding to each target word group. The solution in this application combines a small network model and a nonlinear mapping model to achieve natural language processing of text. The pre-trained nonlinear mapping model is trained through the following steps: training phrases are input into a pre-trained large network model and a pre-trained small network model, respectively, to obtain the third phrase embedding representation output by the pre-trained large network model and the fourth phrase embedding representation output by the pre-trained small network model. The fourth phrase embedding representation is used as the input data of the initial nonlinear mapping model, and the third phrase embedding representation is used as the expected output data of the initial nonlinear mapping model. The initial nonlinear mapping model is then trained to obtain the pre-trained nonlinear mapping model. The accuracy of the output result of the nonlinear mapping model trained in this way can approach the output result of the pre-trained large network model, compensating for the inaccuracy of the small network model's output result. Furthermore, since a simple nonlinear mapping model is used to fit the output result of the small network model to the output result of the large network model, the overall processing time does not increase significantly, achieving a balance between accuracy and timeliness in natural language processing tasks.

[0127] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0128] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0131] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0132] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: "a", "b", "c", "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A text processing method based on artificial intelligence, characterized in that, The method includes the following steps: Obtain the target statement text and perform word segmentation on the target statement text to obtain at least one target word group; Each of the target word groups is input into a pre-trained small network model to obtain the first word group embedding representation corresponding to the target word group output by the small network model; The first word embedding representation corresponding to each of the target word groups is input into a pre-trained nonlinear mapping model to obtain the second word embedding representation corresponding to the word group output by the nonlinear mapping model. The statement embedding representation corresponding to the target statement text is obtained based on the second phrase embedding representation corresponding to each of the target phrases; The pre-trained nonlinear mapping model is obtained through the following steps: inputting training word groups into a pre-trained large-scale network model and a pre-trained small-scale network model respectively to obtain a third word group embedding representation output by the pre-trained large-scale network model and a fourth word group embedding representation output by the pre-trained small-scale network model; using the fourth word group embedding representation as input data of the initial nonlinear mapping model; using the third word group embedding representation as the expected output data of the initial nonlinear mapping model; and training the initial nonlinear mapping model to obtain the pre-trained nonlinear mapping model.

2. The text processing method based on artificial intelligence according to claim 1, characterized in that, After segmenting the target sentence text, and before inputting the phrases into a pre-trained small network model, the method further includes: Determine stop words and non-stop words from at least one initial word group obtained through the word segmentation process; The non-stop words are identified as the target word groups to be input into the pre-trained small network model.

3. The text processing method based on artificial intelligence according to claim 1, characterized in that, Before inputting the training phrases into the pre-trained large network model and the pre-trained small network model, respectively, the method further includes: Obtain training statement text from a preset statement text training set; The training statement text is used to extract the central word, and the training word group is determined based on the central word.

4. The text processing method based on artificial intelligence according to claim 3, characterized in that, The step of extracting the central word from the training statement text, and determining the training word group based on the central word, includes: Extract the central word and at least one neighboring word from the training statement text, and use the central word and the neighboring word corresponding to the central word as the training word group.

5. The text processing method based on artificial intelligence according to claim 1, characterized in that, The step of using the fourth phrase embedding representation as input data to the initial nonlinear mapping model and the third phrase embedding representation as the expected output data to train the initial nonlinear mapping model includes: The fourth word group embedding representation is input into the initial nonlinear mapping model to obtain the fifth word group embedding representation output by the initial nonlinear mapping model; The loss value is calculated based on the third phrase embedding representation and the fifth phrase embedding representation; Determine whether the training termination condition of the initial nonlinear mapping model is met based on the loss value; If the training termination condition of the initial nonlinear mapping model is met, the pre-trained nonlinear mapping model is obtained based on the current model parameters of the initial nonlinear mapping model. If the training termination condition of the initial nonlinear mapping model is not met, the model parameters of the initial nonlinear mapping model are adjusted, and the initial nonlinear mapping model is trained again based on the training phrase.

6. The text processing method based on artificial intelligence according to claim 5, characterized in that, The number of training word groups is multiple, and the step of determining whether the training termination condition of the initial nonlinear mapping model is met based on the loss value includes: The mean loss value is calculated based on the loss value of each training word group. If the mean of the loss value is less than or equal to the preset loss threshold, the training termination condition of the initial nonlinear mapping model is determined to be met. If the mean of the loss values ​​is greater than the preset loss threshold, it is determined that the training termination condition of the initial nonlinear mapping model is not met.

7. The text processing method based on artificial intelligence according to claim 6, characterized in that, The step of calculating the loss value based on the third phrase embedding representation and the fifth phrase embedding representation includes: Calculate the similarity between the third phrase embedding representation and the fifth phrase embedding representation; The loss value of the training phrase is determined based on the similarity between the third phrase embedding representation and the fifth phrase embedding representation.

8. A text processing device based on artificial intelligence, characterized in that, The device includes: The preprocessing module is used to acquire the target sentence text and perform word segmentation on the target sentence text to obtain at least one target word group; The first prediction module is used to input each of the target word groups into a pre-trained small network model to obtain the first word group embedding representation corresponding to the target word group output by the small network model; The second prediction module is used to input the first word group embedding representation corresponding to each of the target word groups into a pre-trained nonlinear mapping model to obtain the second word group embedding representation corresponding to the word group output by the nonlinear mapping model. The pre-trained nonlinear mapping model is trained through the following steps: inputting training word groups into a pre-trained large network model and a pre-trained small network model respectively to obtain the third word group embedding representation output by the pre-trained large network model and the fourth word group embedding representation output by the pre-trained small network model; using the fourth word group embedding representation as the input data of the initial nonlinear mapping model; using the third word group embedding representation as the expected output data of the initial nonlinear mapping model; and training the initial nonlinear mapping model to obtain the pre-trained nonlinear mapping model. The determining module is used to obtain the statement embedding representation corresponding to the target statement text based on the second phrase embedding representation corresponding to each of the target phrases.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program or instructions, and the processor executing the computer program or instructions to implement the artificial intelligence-based text processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, which, when executed by a processor, implement the artificial intelligence-based text processing method as described in any one of claims 1 to 7.

Images