Text processing method, text processing apparatus, electronic device, and computer-readable storage medium
By filtering and using the latest search results as reference texts and combining the content with high similarity of candidate citations, the problem of low timeliness and controllability of text generated by large language models is solved, and the timeliness and authenticity of generated content is improved.
Patent Information
- Application Number
- PCT/CN2023/132040
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2023-11-16
- Publication Date
- 2024-11-14
AI Technical Summary
Due to the lack of information acquisition and self-update capabilities, the large language model has poor timeliness and cannot integrate new natural language knowledge in time, resulting in low limitations and controllability of generated content, and the problems of information disconnection and insufficient factuality.
By obtaining the query text, calling the search engine interface to obtain the latest text search results, filter out the answer text fragments that match the query text, determine the most relevant reference text, and use it to generate answer text for large language models, thereby improving timeliness and controllability, while filtering high-similar text from candidate citation text as citation content, added to the generated text to improve authenticity and credibility.
It improves the timeliness and controllability of content generated by large language models, solves the problems of information disconnection and insufficient factuality, and enhances the authenticity and credibility of generated texts.
Smart Images

Figure CN2023132040_14112024_PF_FP_ABST
Abstract
Description
Text processing method, text processing device, electronic device and computer-readable storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the Chinese patent application with application number 202310525850.2 and application date of May 11, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field
[0003] The present application relates to artificial intelligence technology, and in particular to a text processing method, a text processing device, an electronic device, and a computer-readable storage medium. Background Art
[0004] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) involves natural language, the language we use everyday, and is closely related to linguistics. It also involves key technologies in computer science, mathematics, and artificial intelligence for model training.
[0005] In the related art, large language models (LLMs) are usually used to learn and understand natural languages, and automatically generate corresponding texts based on given contexts in downstream tasks. Generally speaking, large language models are usually based on the Transformer architecture, with a huge number of model parameters and difficult to deploy and train. However, large language models do not have the ability to acquire information and update themselves. When the frequency of model updates is low, they can only generate text within a fixed existing natural language category, and the generated content has limitations and low controllability. Since large language models cannot integrate and learn new natural languages in a timely manner, there may be information disconnection between the generated text and the given text, and there will be great differences in nature at different times, resulting in poor timeliness of the generated text content.
[0006] Summary of the Invention
[0007] Embodiments of the present application provide a text processing method, a text processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the timeliness of text generated by a language model.
[0008] The technical solution of the embodiment of the present application is implemented as follows:
[0009] An embodiment of the present application provides a text processing method, which is performed by an electronic device and includes:
[0010] Get the query text;
[0011] Invoking a search engine interface based on the query text to obtain a plurality of text search results corresponding to the query text;
[0012] Acquire multiple answer text segments matching the query text from the multiple text search results;
[0013] Determining the relevance between the query text and the multiple answer text segments, and using the answer text segment corresponding to the maximum relevance as a reference text for the query text;
[0014] A first language model is called based on the query text and the reference text to obtain an answer text for the query text.
[0015] The present invention provides a text processing device, including:
[0016] An acquisition module configured to acquire query text;
[0017] The acquisition module is further configured to call a search engine interface based on the query text to obtain a plurality of text search results corresponding to the query text;
[0018] The acquisition module is further configured to acquire a plurality of answer text segments matching the query text from the plurality of text search results;
[0019] a determination module configured to determine the relevance between the query text and the plurality of answer text segments, and use the answer text segment corresponding to the maximum relevance as a reference text for the query text;
[0020] The calling module is configured to call a first language model based on the query text and the reference text to obtain an answer text of the query text.
[0021] An embodiment of the present application provides an electronic device, including:
[0022] Memory for storing computer-executable instructions or computer programs;
[0023] The processor is used to implement the text processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.
[0024] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the text processing method provided in the embodiment of the present application when executed by a processor.
[0025] An embodiment of the present application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the text processing method provided in the embodiment of the present application is implemented.
[0026] The embodiments of the present application have the following beneficial effects:
[0027] By obtaining the query text and calling the search engine interface, multiple text search results are obtained, answer text fragments matching the query text are screened out from the text search results, and reference texts are screened out from the answer text fragments based on the relevance to the query text; the query text and the screened reference text are related, and the language model generates the answer text corresponding to the query text through the query text and the matching reference text, so that there is no information disconnection between the query text and the answer text, which increases the controllability of the content generated by the language model; using the searched text as the reference text improves the accuracy of the generated question and answer text; calling the search engine interface for search through the query text, and then using the answer text related to the query text obtained by the search as the reference text of the language model, so that the language model can obtain timely information through the search engine interface in a timely manner, and generate text in combination with the reference text obtained by the real-time search, which can also avoid the difference in nature between the query text and the answer text due to the different generation time, overcome the timeliness limitations when generating text, and improve the timeliness of the content generated by the language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] FIG1 is a schematic diagram of the structure of a text processing system architecture provided by an embodiment of the present application;
[0029] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
[0030] 3A to 3L are flowcharts of a text processing method according to an embodiment of the present application;
[0031] FIG4 is a flowchart of a text processing method according to an embodiment of the present application;
[0032] FIG5 is a diagram of a process for extracting answer text fragments provided by an embodiment of the present application;
[0033] FIG6 is a diagram of a process for determining a relevance score between a query text and an answer text segment provided by an embodiment of the present application;
[0034] FIG7 is a flowchart of a process for citing and aligning generated texts according to an embodiment of the present application;
[0035] FIG8 is a diagram showing the prediction process of the semantic similarity matching model provided in an embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0039] It is understandable that in the embodiments of the present application, when user information and other related data are involved (for example, query text entered by the user through the terminal device), when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0041] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0042] 1) Large Language Models (LLMs), also known as language models, are machine learning models that can process and generate natural language, such as the Bidirectional Encoder Representation from Transformers (BERT) model and the Generate Pre-Training Model (GPT) model. Language model prediction tasks can include text classification, cloze tests, and question answering.
[0043] 2) Timeliness refers to the fact that the same thing can have significant differences in properties at different times. This difference is called timeliness. When large language models generate text content, poor timeliness means that the large language model cannot incorporate new language knowledge in real time, thereby updating the scope of language knowledge it possesses. As a result, the content of the generated text is not broad enough and has limitations.
[0044] 3) Bidirectional Encoder Representation from Transformers (BERT) model, a pre-trained text processing model that can encode text and can be used in various application scenarios such as text generation, determining text similarity, and text recognition.
[0045] Embodiments of the present application provide a text processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the timeliness of text generated by a language model.
[0046] Refer to Figure 1, which is a schematic diagram of the architecture of the text processing system 100 provided in an embodiment of the present application, including a terminal 500, a network 300 and a server 200. The terminal 500 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0047] In the first application scenario, various text editing or text processing application scenarios are running in the terminal 500. When the user enters the inquiry text to be answered in the text editing interface of the APP, the inquiry text is received and sent to the server 200 through the network 300. After receiving the inquiry text sent by the terminal, the server 200 uses the inquiry text as the query text and calls the search engine interface to obtain multiple text search results corresponding to the query text. Then, multiple answer text fragments that match the query text are obtained from the multiple text search results, and then the answer text fragments are screened according to the relevance to obtain the reference text. Finally, based on the query text, the language model is called to obtain the answer text corresponding to the query text from the reference text, and the obtained answer text is returned to the terminal 500 through the network 300 and displayed on the text editing interface of the corresponding APP in the terminal 500.
[0048] In the second application scenario, after the user enters a query text requiring an answer in the text editing interface of the terminal application, the terminal can directly process the query text. Specifically, the query text is used as the query text, and the search engine interface is called to obtain multiple text search results corresponding to the query text. Then, multiple answer text fragments matching the query text are obtained from the multiple text search results. The answer text fragments are then filtered based on relevance to obtain reference text. Finally, based on the query text, the language model is called to obtain the answer text corresponding to the query text from the reference text, and the answer text is directly displayed in the text editing interface.
[0049] In some embodiments, the server 200 shown in Figure 1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 500 shown in Figure 1 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0050] The embodiments of the present application can be implemented with the help of artificial intelligence (AI) technology, which is a theory, method, technology and application system for using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0051] Taking the server provided in the embodiments of this application as an example, a server cluster can be deployed in the cloud, thereby opening up artificial intelligence cloud services (AI as a Service, AIaaS) to users or developers. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall. All users or developers can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.
[0052] For example, the server in the cloud is encapsulated with a text processing program provided by an embodiment of the present application. The user calls the text processing service in the cloud service through a terminal device (the terminal device runs a text editing application), so that the server deployed in the cloud calls the encapsulated text processing program, receives the consultation text to be answered input by the user, uses the consultation text as the query text, and calls the search engine interface to obtain multiple text search results corresponding to the query text. Then, multiple answer text fragments that match the query text are obtained from the multiple text search results, and then the answer text fragments are screened according to the relevance to obtain the reference text. Finally, based on the query text, the language model is called to obtain the answer text corresponding to the query text from the reference text, so that the answer text is directly displayed on the terminal device.
[0053] Referring to Figure 2, Figure 2 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present application. The electronic device 400 can be implemented as a server in the first application scenario described above, or as a terminal in the second application scenario described above. The electronic device 400 shown in Figure 2 includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 400 are coupled together via a bus system 440. It is understandable that the bus system 440 is used to implement connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 440 in Figure 2.
[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0055] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.
[0056] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.
[0057] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.
[0058] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;
[0059] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420 . Exemplary network interfaces 420 include Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).
[0060] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented in software. FIG2 shows a text processing apparatus 453 stored in memory 450. This apparatus 453 can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 4531, a determination module 4532, and a call module 4533. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.
[0061] In some embodiments, the terminal or server can implement the text processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, or it can be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.
[0062] The text processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.
[0063] See Figure 3A, which is a flow chart of the text processing method provided in an embodiment of the present application. The execution entity can be the server in the above-mentioned first application scenario or the terminal device in the above-mentioned second application scenario. The execution entity will not be repeated below, and will be explained in combination with the steps shown in Figure 3A.
[0064] In step 101, a query text is obtained.
[0065] In some embodiments, the query text is generally related to the corresponding application scenario. For example, it can be the context and prompt words given in the application scenario, or the consultation text that needs to be answered. It can also be a search keyword or a search term composed of multiple search keywords corresponding to the search system scenario. The query text is primarily used to obtain the latest search information to extract answer text fragments that match the query text, providing corresponding reference text for the first language model.
[0066] In step 102, a search engine interface is called based on the query text to obtain a plurality of text search results corresponding to the query text.
[0067] For example, a search engine interface is invoked based on the query text to perform a search operation on the Internet or a specific database. The search results can come from web pages, social platforms, etc. In the search result list obtained by the search engine interface, the search results are sorted from shortest to longest based on the time difference between the generation time and the current time, with the search results corresponding to the most recent generation time displayed first.
[0068] In some embodiments, referring to FIG. 3B , step 102 shown in FIG. 3A may be implemented by following steps 1021 to 1022 , which are described in detail below.
[0069] In step 1021, a search engine interface is called based on the query text, so that the search engine interface searches for multiple text search results related to the query text in a manner sorted by generation time.
[0070] After obtaining the query text, the relevant search engine interface is called to obtain multiple text search results based on the query text. The search engine interface can be an interface of a search server, a search interface called from a relevant terminal browser, or other engines or functional interfaces with search capabilities. The search method of the search engine interface is to sort by generation time, that is, the search information with the latest generation time is sorted and displayed at the top. The text search results can generally be in the form of plain text or rich media. For the latter, the text can be extracted, generally including a body text and the corresponding text title.
[0071] In step 1022, a plurality of text search results related to the query text are obtained from a search engine interface.
[0072] After calling the search engine interface to search the query text, multiple text search results are obtained. Then, multiple text search results related to the query text are obtained from the search engine interface in sequence according to the sorting method of the generation time of the text search results, that is, multiple text search results related to the query text and with the latest generation time are obtained.
[0073] The embodiment of the present application utilizes the query text and obtains the latest search information related to the query text from the search engine interface according to the principle of timeliness first, so as to be used for subsequent screening and obtaining reference text, thereby ensuring the timeliness of the reference text information.
[0074] Continuing to refer to FIG. 3A , in step 103 , a plurality of answer text segments matching the query text are obtained from the plurality of text search results.
[0075] Among the main text included in the text search results, only part of the text can be used as the answer to the query text, and the other part is text information irrelevant to the query text, so it is necessary to extract these text fragments that can be used as answers from the main text included in the text search results.
[0076] In some embodiments, referring to FIG. 3C , FIG. 3C is a flow chart of a text processing method provided in an embodiment of the present application. Step 103 shown in FIG. 3A can be implemented through steps 1031 to 1035 of FIG. 3C , which will be described in detail below.
[0077] In step 1031 , the text search results are segmented into a plurality of candidate reference text segments of fixed length.
[0078] Only a portion of the body text included in the text search results can be considered as a matching answer to the query text. Therefore, the body text in each text search result is segmented according to a fixed length to obtain multiple candidate reference text segments. The fixed length of the candidate reference text segments can be set based on the text size or number of segments in the actual text search results, for example, 500. The fixed length is positively correlated with the text size or number of segments in the actual text search results.
[0079] In step 1032 , the matching scores between the query text and the plurality of candidate reference text segments are obtained, as well as the starting probability of each element in the candidate reference text segment being the starting position and the ending probability of each element in the candidate reference text segment being the ending position of the answer text segment.
[0080] After segmenting the body text, to determine candidate citation text segments that match the query text, the query text can be combined with each candidate citation text segment and input into the relevant model for processing to predict the matching scores between the query text and the multiple candidate citation text segments. Considering that the answer text segment of the query text is selected from the candidate citation text segments, it is determined whether each element in the candidate citation text segment can serve as the starting element and ending element of the answer text segment to determine the starting and ending positions of the answer text segment within the candidate citation text segments.
[0081] In some embodiments, referring to FIG3D , FIG3D is a flow chart of a text processing method provided in an embodiment of the present application. Step 1032 shown in FIG3C can be implemented through steps 10321 to 10322 of FIG3D , which will be described in detail below.
[0082] In step 10321 , multiple candidate reference text segments are respectively combined with the query text into text pairs.
[0083] In some embodiments, to predict the matching scores between the query text and multiple candidate reference text segments, it is necessary to determine the similarity between the query text and the candidate reference text segments. In this embodiment of the present application, multiple candidate reference text segments are respectively combined with the query text into text pairs, which are then used to predict the similarity between the query text and the candidate reference text segments in the text pairs using a subsequent second language model.
[0084] In step 10322, the second language model is called based on each text pair to obtain the matching scores between the query text and multiple candidate reference text fragments, as well as the starting probability and ending probability of each element in the candidate reference text fragment as the starting position of the answer text fragment.
[0085] Continuing with the above embodiment, after the query text and the candidate reference text fragments are respectively combined into text pairs, the text pairs are processed by calling the second language model to obtain matching scores between the query text and multiple candidate reference text fragments. The matching score represents the similarity between the semantics of the candidate reference text and the semantics of the query text. The higher the matching score, the closer the semantics.
[0086] Obtaining whether each element in the candidate reference text fragment in the text pair is suitable as the starting element and the ending element of the answer text fragment can be achieved in the following way: determining the starting probability of each element in the candidate reference text fragment as the starting position and the ending probability of the ending position of the answer text fragment, selecting the element with the highest starting probability as the starting position, and selecting the element with the highest ending probability as the ending position.
[0087] In some embodiments, referring to FIG3E , FIG3E is a flow chart of a text processing method provided in an embodiment of the present application. Step 10322 shown in FIG3D can be implemented through steps 103221 to 103225 of FIG3E , which will be described in detail below.
[0088] In step 103221, each element in the query text and the candidate reference text segment is taken as a regular tag, and the regular tags are spliced into a first tag sequence.
[0089] Before calling the second language model to process the text pair, the embedded feature vector of the text pair is obtained. Therefore, each element (token) in the query text and the candidate text fragment is treated as a regular token. Regular tokens are used to represent text elements, as opposed to start tokens and segmentation tokens. All regular tokens are then concatenated to form the first token sequence. An element can be a character, word, or short sentence in the text. This is because some text may not express specific semantic information if it is forcibly distinguished by word. Finally, a start token (CLS) is inserted at the beginning of the first token sequence.
[0090] In step 103222, embedding processing is performed on the first marker sequence to obtain an embedded feature vector of the first marker sequence.
[0091] Continuing with the above embodiment, after obtaining the first token sequence with the start tag inserted, the first token sequence is then embedded (embedded) and mapped into an embedded feature vector for input into the second language model for processing. The embedding process for the first token sequence can be implemented by calling a layer of a neural network or a text embedding model. The embedding process is to convert the discretized text into a dense continuous vector, that is, to map the text to a point in the vector space. Ultimately, the semantic information of the text in the semantic space is converted into a vector for subsequent encoding processing of the vector by the second language model.
[0092] In step 103223, the second language model is called to perform encoding processing based on the embedded feature vector of the first token sequence to obtain the semantic feature vector of the first token sequence.
[0093] Continuing with the above embodiment, after obtaining the embedded feature vector of the first token sequence through embedding processing, the second language model is called to encode the embedded feature vector, thereby obtaining a semantic feature vector for each token (including start tags and regular tags) in the first token sequence. The second language model can be a BERT model with a 12-layer structure (BERT-12-Layer), or other text encoders or text encoding models with a multi-layer structure.
[0094] In step 103224, mapping processing is performed on the semantic feature vector of the start tag in the semantic feature vector of the first tag sequence to obtain a matching score between the query text and the candidate reference text segment.
[0095] Considering that the semantic feature vector of the start tag is more fair than that of other elements in the text pair, and integrates the semantic information of each element in the text pair more fairly, it can be considered that the semantic feature vector of the start tag includes the semantic information of the entire text pair.
[0096] In this embodiment, the semantic feature vector corresponding to the start tag is input into a fully connected layer for mapping, thereby obtaining a matching score between the query text and the candidate reference text fragment. The mapping process can be viewed as a prediction and classification process, and the activation function of the fully connected layer can be a Softmax function, which is used to map the corresponding semantic feature vector to a predicted classification probability, which is the matching score.
[0097] In step 103225, the semantic feature vector corresponding to each regular tag in the semantic feature vector of the first tag sequence is mapped to obtain the starting probability of each element in the candidate reference text segment as the starting position of the answer text segment and the ending probability of the ending position.
[0098] While determining the matching scores between the query text and the candidate reference text fragments, it is also determined which text fragments in the candidate reference text fragments in the text pair can be used as answer text fragments, that is, predicting the starting probability of the starting position and the ending probability of the ending position of each element in the candidate reference text fragment as being suitable as the answer text fragment. Specifically, the semantic feature vector corresponding to each regular tag in the candidate reference text fragment is input into a fully connected layer for mapping processing. Among them, two position vectors are preset in the fully connected layer, namely the starting position vector and the ending position vector. The inner product is calculated by taking the inner product of the semantic feature vector corresponding to each regular tag input with the starting position vector and the ending position vector, and then the two inner product results are mapped to two classification probabilities through the activation function of the fully connected layer, wherein the activation function of the fully connected layer can be a Softmax function, and the two classification probabilities are the corresponding starting probability and ending probability.
[0099] In an embodiment of the present application, by analyzing the starting probability of the elements in the candidate reference text fragment as the starting position of the answer text fragment and the ending probability of the elements as the ending position of the answer text fragment, and determining the starting position and ending position of the answer text fragment corresponding to the candidate reference text fragment based on the probability, the redundancy of the answer text fragment is reduced, the correlation between the answer text fragment and the query text is made higher, and the accuracy of determining the answer text fragment is improved.
[0100] Continuing to refer to FIG. 3C , in step 1033 , the best matching segment is determined.
[0101] The matching scores between the query text and each candidate reference text segment are determined, and the candidate reference text segment corresponding to the maximum matching score is determined according to the matching scores as the optimal matching segment.
[0102] In step 1034 , the start element and the end element in the best matching segment are determined.
[0103] After determining the optimal matching segment, determine which portion of the text segment in the optimal matching segment belongs to the answer text segment. Specifically, determine the starting probability of each element in the optimal matching segment as the starting position of the answer text segment, and use the element with the maximum starting probability in the optimal matching segment as the starting element of the answer text segment. Similarly, determine the ending probability of each element in the optimal matching segment as the ending position of the answer text segment, and use the element with the maximum ending probability in the optimal matching segment as the ending element of the answer text segment.
[0104] In step 1035 , the portion of the best matching segment between the start element and the end element is used as the answer text segment that matches the query text.
[0105] After determining the start element and end element of the answer text segment from the optimal matching segment, the text part between the start element and end element corresponding to the optimal matching segment is determined as the answer text segment that matches the query text.
[0106] For example, the query text is "What's the temperature today", and the optimal matching segment with the highest matching score determined from the candidate reference text segments is "The weather is nice, the temperature is twenty degrees, and it's cloudy". At this time, it is necessary to determine which part of the text segment in the optimal matching segment can be used as the answer text segment. Since when predicting the matching score, the start probability and end probability of each element in the candidate reference text segment being suitable as the start position and end position of the answer text segment are also predicted, according to the start probability predicted for each element in the optimal matching segment, it is determined that the start probability predicted for the element "气" is the highest, so the element "气" is used as the start element of the answer text segment. Similarly, according to the end probability predicted for each element in the optimal matching segment, it is determined that the end probability predicted for the element "度" is the highest, so the element "度" is used as the end element of the answer text segment. Thus, the start element and end element of the answer text segment are determined from the optimal matching segment, and the text part between the start element "气" and the end element "度" in the optimal matching segment is used as the answer text segment, and thus the answer text segment is determined to be "The temperature is twenty degrees". Through this process, each text search result will ultimately be screened to obtain an answer text segment.
[0107] Continuing to refer to FIG. 3A, in step 104, the relevance between the query text and multiple answer text segments is determined, and the answer text segment corresponding to the maximum relevance is used as the reference text of the query text.
[0108] Each text search result will ultimately be screened to obtain an answer text segment, and multiple answer text segments are screened. The specific screening process is as follows: The relevance between the query text and multiple answer text segments is determined, and the answer text segment corresponding to the maximum relevance is determined as the reference text of the query text.
[0109] Relevance is an index used to characterize the relevance between the content of the query text and the answer text segment, and can be characterized by the probability that the features of the query text and the features of the answer text segment belong to the same type.
[0110] In some embodiments, referring to FIG. 3F, FIG. 3F is a schematic flowchart of the text processing method provided in an embodiment of the present application. "Determining the relevance between the query text and multiple answer text segments" in step 104 shown in FIG. 3A can be implemented through steps 1041 to 1044 in FIG. 3F, which will be specifically described below.
[0111] In step 1041, each element in the query text, the answer text segment, and the title text is treated as a regular tag, and the regular tags of each element are spliced into a second tag sequence.
[0112] For each answer text fragment, each element in the query text and the answer text fragment is treated as a regular tag, and then the regular tags of each element are spliced into a second tag sequence, wherein the regular tags between the query text and the answer text fragment are connected with a split tag, and a start tag is inserted at the head of the second tag sequence.
[0113] In some embodiments, the title text of the answer text segment can also be extracted as part of the second tag sequence. Therefore, each element in the query text, the answer text segment, and the title text can be treated as a regular tag, and the regular tags of each element can be concatenated to form the second tag sequence. Because the title text generally expresses strong semantic features, it helps in the subsequent determination of relevance and improves the accuracy of the relevance calculation.
[0114] In step 1042, embedding processing is performed on the second marker sequence to obtain an embedded feature vector of the second marker sequence.
[0115] Continuing with the above embodiment, after obtaining the second token sequence, embedding processing is performed on the second token sequence to obtain an embedded feature vector of the second token sequence for subsequent encoding by the second language model. The embedding processing of the second token sequence can be implemented by calling a neural network layer (fully connected layer), a multi-layer perceptron (MLP), or other text embedding models.
[0116] In step 1043, the third language model is called to encode the embedded feature vector of the second token sequence to obtain a semantic feature vector of the second token sequence.
[0117] Continuing with the above embodiment, a third language model is invoked to encode the embedded feature vector of the second token sequence, thereby obtaining a semantic feature vector for the second token sequence. The semantic feature vector of the second token sequence is also the semantic feature vector corresponding to the start marker of the second token sequence, because the semantic encoding result corresponding to the start marker position can represent the semantic encoding result of the entire second token sequence. The third language model here can be a cross-encoder, a BERT model, or other non-cross-encoder or text encoder with an unlimited number of network layers and network structure.
[0118] In step 1044, the semantic feature vectors of the second tag sequence are mapped to obtain the relevance between the query text and the answer text segment.
[0119] Following the above embodiment, after the semantic feature vector corresponding to the start mark of the second mark sequence is used as the semantic feature vector of the entire second mark sequence, a multi-layer perceptron is called to map the semantic feature vector. Specifically, the semantic feature vector corresponding to the start mark position is mapped to a predicted classification probability through the multi-layer perceptron. The predicted classification probability is the correlation between the query text and the answer text fragment, which is denoted as S q,t+p .
[0120] Through the above embodiment, the relevance (S) of the query text and each answer text segment is determined. q,t+p ), the maximum value of S q,t+p The corresponding answer text fragment determines the reference text corresponding to the query text.
[0121] Continuing to refer to FIG. 3A , in step 105 , a first language model is called based on the query text and the reference text to obtain an answer text for the query text.
[0122] Through the above embodiment, a reference text corresponding to the query text is finally obtained. At this time, the first language model can be called to process the query text and the reference text to predict the answer text corresponding to the query text from the reference text.
[0123] In some embodiments, referring to FIG. 3G , FIG. 3G is a flow chart of a text processing method provided in an embodiment of the present application. Step 105 shown in FIG. 3A can be implemented through steps 1051 to 1052 of FIG. 3G , which will be described in detail below.
[0124] In step 1051 , a first language model is called based on the query text and the reference text to perform prediction processing on the query text to determine the starting element and the ending element of the answer text in the reference text.
[0125] After determining the reference text from the answer text fragment, the reference text can be used as a reference material to guide the first language model to predict the input answer text to generate the corresponding answer text. Specifically, the query text and the reference text are input into the first language model for prediction processing to determine the starting element and the ending element of the answer text in the reference text, thereby obtaining the starting position and the ending position of the answer text in the reference text.
[0126] In some embodiments, referring to FIG3H , FIG3H is a flow chart of a text processing method provided in an embodiment of the present application. Step 1051 shown in FIG3G can be implemented through steps 10511 to 10515 of FIG3H , which will be described in detail below.
[0127] In step 10511, each element in the query text and the reference text is treated as a regular tag, and the regular tags are concatenated into a third tag sequence.
[0128] In some embodiments, since the language model cannot make predictions directly based on the original text, it is still necessary to determine the semantic feature vector of the text. First, the embedded feature vector of the text is obtained. Each element in the query text and the reference text is treated as a regular token. The regular tokens of each element are then concatenated into a third token sequence. The regular tokens between the query text and the reference text are connected by a split token, and a start token (CLS) is inserted at the head of the third token sequence.
[0129] In step 10512, embedding processing is performed on the third marker sequence to obtain an embedded feature vector of the third marker sequence.
[0130] Continuing with the above embodiment, the concatenated third token sequence is embedded to obtain an embedded feature vector of the third token sequence for subsequent encoding by the first language model. Similarly, embedding the third token sequence can be implemented by calling a neural network layer (fully connected layer), a multi-layer perceptron, or other text embedding models.
[0131] In step 10513, the first language model is called to perform encoding processing based on the embedded feature vector of the third token sequence to obtain a semantic feature vector of each regular token in the third token sequence.
[0132] Continuing with the above embodiment, the encoder of the first language model is invoked to encode the embedded feature vector of the third token sequence, thereby obtaining a semantic feature vector corresponding to each regular token in the third token sequence. The first language model can be any large language model that is capable of fully learning and understanding the semantic features of a given text and generating text that matches the given text based on the semantic features of the given text context.
[0133] In step 10514, the semantic feature vector corresponding to each regular tag is mapped to obtain each element of the reference text as the start probability of the start element and the end probability of the end element of the answer text.
[0134] Following the above embodiment, after obtaining the semantic feature vector of each regular tag in the third tag sequence, considering that the embodiment of the present application uses the existing reference text as the reference material for the query text, it is only necessary to use the first language model to predict the answer text that matches the query statement from the reference text. Specifically, the semantic feature vector of each regular tag in the third tag sequence is mapped to two predicted classification probabilities, and the inner product of the semantic feature vector corresponding to each regular tag with the starting position vector and the ending position vector is calculated through the preset starting position vector and the ending position vector in the fully connected layer of the first language model. The two inner product results are then mapped to two classification probabilities through the activation function of the fully connected layer. The two classification probabilities are respectively the starting probability of the corresponding elements of the regular tags of the reference text as the starting element of the answer text, and the ending probability of the ending element, wherein the activation function of the fully connected layer can be a normalization function (Softmax function).
[0135] In step 10515, the element corresponding to the maximum starting probability is used as the starting element of the answer text, and the element corresponding to the maximum ending probability is used as the ending element of the answer text.
[0136] After determining the two classification probabilities of the corresponding elements of each regular tag in the third tag sequence, the element corresponding to the starting probability of the maximum value in the third tag sequence is used as the starting element of the answer text, and the element corresponding to the ending probability of the maximum value is used as the ending element of the answer text, thereby determining the specific position of the answer text in the reference text.
[0137] Continuing to refer to FIG. 3G , in step 1052 , the text between the start element and the end element of the reference text is used as the answer text of the query text.
[0138] After determining the specific position of the answer text in the reference text, the text segment between the start element and the end element in the reference text is then determined as the answer text, which serves as the final predicted text of the first language model, that is, the generated text of the first language model.
[0139] Exemplarily, the query text is "What's the temperature today?", and the corresponding reference text is "It's sunny today, and the temperature is 20 degrees". Each character of the query text and the reference text is used as a regular token, and they are concatenated with a separator token to obtain a third token sequence. Then, a start token is inserted at the head of the sequence, resulting in "[CLS]What's the temperature today?[SEP]It's sunny today, and the temperature is 20 degrees". Next, the third token sequence is embedded to obtain the corresponding embedded feature vectors, which are input into the first language model for encoding processing to obtain the semantic feature vectors corresponding to each regular token. Then, the semantic feature vectors of each regular token are mapped to two classification probabilities, namely the start probability of being the start position of the answer text and the end probability of being the end position of the answer text. It is determined that the start probability mapped from the semantic feature vector corresponding to the element "qi" in the reference text is the largest, which can be used as the start position of the answer text. Correspondingly, the end probability mapped from the semantic feature vector corresponding to the element "du" in the reference text is the largest, which can be used as the end position of the answer text. Thus, the text between the element "qi" and the element "du" in the reference text is used as the answer text, that is, "the temperature is 20 degrees" is used as the answer text to the query text "What's the temperature today?".
[0140] Through the embodiments of the present application, the query text is obtained, the search engine interface is called to generate multiple latest text search results with priority given to timeliness. Then, the answer text fragments relevant to the query text are extracted from the text search results, and screening is performed to determine the text fragment with the highest relevance to the query text from the multiple answer text fragments as the reference text, which serves as the reference material for the first language model to predict and generate text. Then, the first language model is called to predict the answer text corresponding to the query text from the reference text, enabling the first language model to answer the query text based on the latest reference material, thereby generating the corresponding answer text. Moreover, the first language model predicts the answer text from the fixedly generated reference text, increasing the controllability of the content generated by the first language model and improving the timeliness of the generated content.
[0141] In some embodiments, the first language model, the second language model, and the third language model can be integrated into a large language model, and the first language model, the second language model, and the third language model are trained separately.
[0142] In some embodiments, referring to FIG. 3I, FIG. 3I is a schematic flowchart of the text processing method provided by the embodiments of the present application. After the step 105 shown in FIG. 3A, the steps 106 to 108 of FIG. 3I can also be executed, which will be specifically described below.
[0143] In step 106, multiple candidate reference texts are obtained.
[0144] After the language model in the related art generates an answer text based on the query text, considering that the answer text may be less than authentic and contain fabricated content, relevant reference text is added to the corresponding answer text as the text source of the answer text, thereby improving the authenticity of the answer text generated by the first language model and resolving the issue of possible fabrication. The corresponding candidate reference text is obtained from the answer text, where the candidate reference text includes the material used to cite in the answer text. The method for obtaining the candidate reference text can call a search engine interface based on the answer text to search for text search content related to the answer text, or call a text generation model to randomly generate multiple texts based on the semantics of the answer text.
[0145] In step 107 , the answer text is segmented into a plurality of answer text segments, and each candidate quotation text is segmented into a plurality of quotation text segments.
[0146] Following the above embodiment, after obtaining the candidate quoting text, it is determined whether the candidate quoting text matches the answer text before it can be used as a quoting material. The method of determining whether the candidate quoting text matches the answer text is to determine the text similarity between the two.
[0147] In an embodiment of the present application, considering that the text size of the answer text and the candidate reference text may be large, the amount of calculation required to directly determine the text similarity is too large, so the answer text is divided into multiple answer text segments greater than a fixed length, and each candidate reference text is divided into multiple reference text segments greater than a fixed length. This is achieved by determining the similarity of the text segments, and reference materials can also be added to each answer text segment of the answer text as reference text.
[0148] In step 108 , a reference text segment that matches at least one answer text segment is determined, and the reference text segment that matches at least one answer text segment is inserted into the answer text.
[0149] In some embodiments, referring to FIG3J , FIG3J is a flow chart of a text processing method provided in an embodiment of the present application. The “determining a reference text fragment that matches at least one answer text fragment” in step 108 shown in FIG3I can be implemented by following steps 1081A to 1082A, which are described in detail below.
[0150] In step 1081A, multiple answer text segments and multiple reference text segments are combined in pairs to form multiple candidate text pairs.
[0151] After segmenting the answer text and candidate quote texts, multiple text segments are obtained. Considering the need to add quotes to each answer text segment, each answer text segment is combined with each quote text segment to form multiple candidate text pairs. In other words, multiple answer text segments and multiple quote text segments are paired together. This allows the similarity between the answer text segments and the quote text segments in the candidate text pairs to be determined.
[0152] For example, a certain answer text is segmented into 20 answer text segments, and each of the 10 candidate reference texts is segmented into 5 reference text segments. After each answer text segment is combined with each reference text segment, the number of candidate text pairs finally obtained is 1000 (20x10x5).
[0153] In step 1082A, at least one matching text pair is identified from the plurality of candidate text pairs, and it is determined that the reference text segment in the matching text pair matches the answer text segment in the matching text pair.
[0154] After combining the answer text fragments and the reference text fragments into multiple candidate text pairs, by determining the similarity between the two text fragments in the candidate text pair, it is possible to identify whether the answer text fragment matches the reference text fragment, so as to identify at least one matching text pair from the multiple candidate text pairs, thereby determining that the reference text fragment in the matching text pair matches the answer text fragment in the matching text pair.
[0155] In some embodiments, considering that there are a large number of answer text fragments and reference text fragments, the number of candidate text pairs formed is too large. In order to improve text processing efficiency and reduce the computational complexity of subsequent steps, before identifying matching text pairs from multiple candidate text pairs, multiple candidate text pairs can be preliminarily filtered to filter out obviously unmatched or unimportant candidate text pairs.
[0156] In some embodiments, the method for preliminary filtering candidate text pairs can be text keyword recognition. Because quotations are generally added only to text containing keywords in the answer text segment, as keywords convey the most semantic information, while other non-keyword text is unimportant and does not require quotations, preliminary filtering can eliminate candidate text pairs that do not contain keywords.
[0157] The specific filtering process based on text keyword recognition is that, for each candidate text pair, the keyword recognition model is called to perform keyword recognition on the answer text segment and the reference text segment in the candidate text pair. When at least one of the answer text segment and the reference text segment in the candidate text pair fails to recognize a keyword, it means that both the answer text segment and the reference text segment are unimportant texts without key semantic information, or one of the answer text segment and the reference text segment fails to recognize a keyword, which indicates that there is an obvious semantic mismatch between the two. Then, these unimportant texts and candidate text pairs with obvious semantic mismatches are directly filtered out. Among them, the keyword recognition model can be a text ranking algorithm (Text Rank), a term frequency-inverse document frequency method (TF-IDF), a rapid automatic keyword extraction algorithm (RAKE), or a combination of the three methods.
[0158] In some embodiments, the method for preliminary filtering of candidate text pairs is based on the similarity matching method of text segment embedding (sentence embedding).Specific filtering process is, for each candidate text pair, calling text encoding model respectively to encode the answer text segment and the reference text segment in the candidate text pair, obtain the embedding feature vector of the answer text segment and the reference text segment, wherein, the embedding feature vector of the answer text segment can be the embedding feature vector of the segment as a whole, or the embedding feature vector of the sub-segment (the answer text segment is further divided), the embedding feature vector of the reference text segment can be the embedding feature vector of the segment as a whole, or the embedding feature vector of the sub-segment (the reference text segment is further divided).Then determine the similarity between the embedding feature vector of the answer text segment and the embedding feature vector of the reference text segment, wherein the similarity between the embedding feature vectors can be the similarity between the embedding feature vectors of the segment as a whole, or the similarity between the embedding feature vectors of the sub-segment.Thus can determine the similarity of each candidate text pair, the embodiment of the present application is by setting a similarity threshold value, such as the similarity of the candidate text pair is less than the similarity threshold value, delete the corresponding candidate text pair.
[0159] For example, in a candidate text pair, the embedded feature vector of the answer text fragment can be the embedded feature vector of the entire fragment, such as embedded feature vector A, and the embedded feature vector of the reference text fragment in the corresponding candidate text pair can be the embedded feature vector of the entire fragment, such as embedded feature vector B. Then, the similarity between embedded feature vector A and embedded feature vector B can be determined. When the similarity is less than the similarity threshold, the corresponding candidate text pair is deleted.
[0160] In other examples, the answer text segment in the candidate text pair can be further divided to obtain multiple answer text sub-segments, and then the text encoding model is called to encode the multiple answer text sub-segments, thereby obtaining the embedded feature vectors of each answer text sub-segment, denoted as A1, A2,…, An. Correspondingly, the reference text segment in the candidate text pair is further divided to obtain multiple reference text sub-segments, and then the text encoding model is called to encode the multiple reference text sub-segments, thereby obtaining the embedded feature vectors of each reference text sub-segment, denoted as B1, B2,…, Bn, and then the similarity between A1 and B1, the similarity between A2 and B2, and the similarity between An and Bn are determined respectively. When one of the similarities is less than the similarity threshold, it means that the answer text segment does not match the reference text segment, and the corresponding candidate text pair is deleted.
[0161] In an embodiment of the present application, before identifying a matching text pair from multiple candidate text pairs, the multiple candidate text pairs can be preliminarily filtered, so that obviously mismatched or unimportant candidate text pairs can be filtered out first, reducing the amount of calculation in subsequent steps and improving text processing efficiency.
[0162] In some embodiments, referring to FIG3K , FIG3K is a flow chart of a text processing method provided in an embodiment of the present application. The “identifying at least one matching text pair from a plurality of candidate text pairs” in step 1082A shown in FIG3J can be implemented through steps 10821A to 10825A of FIG3J , which will be described in detail below.
[0163] In step 10821A, the fourth language model is called to encode the answer text segment and the reference text segment in the candidate text pair respectively to obtain the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment.
[0164] In some embodiments, determining the similarity between two text segments in a candidate text pair can be achieved by invoking a language model. For each candidate text pair, a fourth language model is invoked to encode the answer text segment and the reference text segment in the candidate text pair, respectively, to obtain an embedded feature vector for the answer text segment and an embedded feature vector for the reference text segment. The fourth language model can be a text-based BERT model (Sentence-BERT) or another text encoding model.
[0165] In step 10822A, the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment are pooled to obtain the semantic feature vector of the answer text segment and the semantic feature vector of the reference text segment.
[0166] Since the text length or text scale of the answer text fragment and the quoted text fragment may be quite different, the embedded feature vectors of the answer text fragment and the quoted text fragment can be pooled separately through a pooling layer to make the dimension of the embedded feature vector of the answer text fragment the same as the dimension of the embedded feature vector of the quoted text fragment, which is more conducive to the subsequent determination of the similarity of the vectors and reduces the amount of calculation. After the pooling layer performs pooling processing, the semantic feature vector of the answer text fragment and the semantic feature vector of the quoted text fragment are obtained.
[0167] In step 10823A, the semantic feature vector of the reference text segment and the semantic feature vector of the answer text segment are concatenated to obtain a fused feature vector of the candidate text pair.
[0168] Continuing from the above embodiment, after determining the semantic feature vector of the answer text segment and the semantic feature vector of the reference text segment, considering that the dimensions of the two semantic feature vectors are the same, the two semantic feature vectors can be directly spliced to obtain a fused feature vector of the text pair.
[0169] In step 10824A, the fused feature vector is mapped to obtain the semantic similarity between the answer text segment and the reference text segment in the candidate text pair.
[0170] Continuing from the above embodiment, after obtaining the fused feature vector of each candidate text pair, a layer of neural network (fully connected layer) or a multi-layer perceptron can be called to map the fused feature vector to map the fused feature vector into a classification probability. This classification probability can be used as the semantic similarity between the answer text fragment and the reference text fragment in the candidate text pair, wherein the activation function of the fully connected layer can be the Softmax function.
[0171] In some embodiments, the semantic feature vector of the answer text fragment and the semantic feature vector of the reference text fragment do not need to be spliced, that is, there is no need to construct a fusion feature vector, but the two semantic feature vectors are directly processed, for example, cosine similarity or edit distance is calculated as semantic similarity.
[0172] In step 10825A, candidate text pairs whose semantic similarity is greater than a semantic similarity threshold are determined as matching text pairs.
[0173] For each candidate text pair, after determining the semantic similarity of the answer text segment and the reference text segment in the candidate text pair, the candidate text pairs are screened using a preset semantic similarity threshold, and text pairs with a semantic similarity greater than the semantic similarity threshold are determined as matching text pairs. The semantic similarity threshold can be set based on the number of candidate text pairs, and the set semantic similarity threshold is positively correlated with the number of candidate text pairs. That is, when the number of candidate text pairs is large, the semantic similarity threshold can be set larger, and when the number of candidate text pairs is small, the semantic similarity threshold can be set smaller. For example, when the number of candidate text pairs is too large, such as 1,000 candidate text pairs, the semantic similarity threshold can be set higher, such as 0.9. When the number of text pairs is small, such as only 500 candidate text pairs, the semantic similarity threshold can be set lower, such as 0.7.
[0174] In some embodiments, referring to FIG3L , FIG3L is a flow chart of a text processing method provided in an embodiment of the present application. The step 108 shown in FIG3I of "inserting a reference text fragment that matches at least one answer text fragment into the answer text" can be implemented by following steps 1081B to 1082B, which are described in detail below.
[0175] In step 1081B, in response to the number of at least one answer text segment being less than or equal to a first number threshold, the reference text segments matched by the at least one answer text segment are respectively inserted into positions after the at least one answer text segment is matched.
[0176] In some embodiments, by determining the matching text pairs, the quoted text segments in the matching text pairs can be used as the citation material for the corresponding answer text segments. Specifically, the answer text segments that match the quoted text segments are determined from the answer text based on the matching text pairs, so that the citation position of each quoted text segment in the answer text can be determined.
[0177] Considering that there may be a large number of answer text fragments in the answer text, it would be too computationally expensive to add reference materials to each answer text fragment. In the embodiment of the present application, a first number threshold of answer text fragments is set. When the number of answer text fragments in the answer text is less than or equal to the first number threshold, the reference text fragments are spliced after the corresponding answer text fragments in the answer text according to the matching text pairs as reference materials. The reference materials can be expressed in the form of specific symbolic identifiers, such as text superscripts or text subscripts, and some reference symbols, such as "[]" or "||", can also be used. These reference symbols can be used to quickly locate and find the corresponding reference materials.
[0178] In step 1082B, in response to the number of reference text segments matched by at least one answer text segment being greater than a first quantity threshold, the reference text segments matched by at least one answer text segment are collectively inserted at the end of the answer text.
[0179] Continuing from the above embodiment, when the number of answer text fragments in the answer text is greater than the first quantity threshold, the reference text fragments that match at least one answer text fragment are inserted into the end of the entire answer text in sequence as the reference material of the answer text, wherein the reference text fragments may be inserted in any order.
[0180] In some embodiments, considering that each answer text segment and the quoted text segment are combined in pairs, an answer text segment may have multiple quoted text segments that can be combined, that is, the answer text segment may have multiple quoted materials. However, if the number of quoted materials may be large, it may be detrimental to the embodiment of the answer text and increase the text size and reading volume. Therefore, before inserting the quoted text segment that matches the answer text segment, the embodiment of the present application limits the number of quoted materials of the answer text segment by setting a second quantity threshold. When the number of quoted text segments matched by at least one answer text segment is greater than the second quantity threshold, the at least one answer text segment and the matched quoted text segment are sorted in descending order according to the similarity between them, and starting from the head in the descending sorting result, the quoted text segments that match the answer text segment with a set number or a set proportion are used as the text segments to be inserted. Finally, these text segments to be inserted are inserted into the position after the corresponding answer text segment.
[0181] For example, an answer text segment is matched with 6 quoted text segments. The 6 quoted text segments can be inserted after the answer text segment in order from large to small according to the semantic similarity between the quoted text segments and the answer text segment. Therefore, the number of quoted texts in the answer text segment is 6. At this time, by setting the second quantity threshold to 2, the number of quoted text segments is greater than the second quantity threshold, and the set number (for example, 2) or the preset proportion (for example, half, 50%) of quoted text segments are used as text segments to be inserted, and finally inserted into the position after the answer text segment.
[0182] In some embodiments, step 1081B and step 1082B shown in FIG. 3L may be executed in parallel, and there is no particular order of execution.
[0183] Through the embodiments of the present application, texts with high similarity to the text generated by the first language model (i.e., the answer text) are screened out from the provided candidate quoted texts as quoted materials, and quoted content is added to the relevant answer text fragments in the generated text. On the one hand, this can solve the problem that the content generated by the first language model has no quoted information, making it difficult to judge the authenticity and further understand it. On the other hand, using the quoted content as reference material for the text generated by the first language model also improves the authenticity and credibility of the text generated by the first language model, and solves the problem that the original generated text has low factuality and credibility and may be fabricated.
[0184] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.
[0185] In some application scenarios, such as search systems and conversational systems, related technologies often use large language models to learn and understand natural language and automatically generate high-quality text based on a given context or question. For example, these models can be used for code writing, essay generation, role-playing, and other functions, significantly improving search information acquisition efficiency and conversation quality. However, large language models are typically based on a transformer-encoder architecture, where multiple attention layers are stacked in very deep neural networks. These models can have hundreds of millions of parameters and are difficult to deploy and train. Large language models lack information acquisition and self-update capabilities. When model updates are infrequent, they can only generate text within a fixed, existing natural language domain, resulting in poor controllability. Because the model cannot promptly incorporate and learn new natural language knowledge, the generated text can easily become disconnected from the given text, resulting in poor timeliness. Furthermore, this poor timeliness can lead to insufficient factuality, low credibility, and fabrication in the generated text, which can impact the performance of large language models.
[0186] Based on this scenario, an embodiment of the present application provides a text processing method that collects timely search information to screen out corresponding answer texts as reference texts for the model. Then, based on the given text, the large language model is guided to generate controllable text content, thereby allowing the model to learn new search knowledge through the reference text in a timely manner, thereby improving the timeliness of the generated text. Furthermore, by screening out texts with high similarity to the generated text from the provided candidate texts as reference texts, the reference content is added to the generated text content, thereby improving the authenticity and credibility of the generated text, and resolving the problem that the original generated text is factual, unreliable, and may be fabricated.
[0187] See Figure 4, which is a processing flow chart of the text processing method provided in an embodiment of the present application, which specifically includes six steps: obtaining query text, obtaining query results, extracting answer text fragments, determining the relevance score of the query text and the answer text fragments, calling a large language model to generate text, and quoting and aligning the generated text. The specific implementation of each step process will be described in detail in conjunction with Figure 4 below.
[0188] Referring to FIG. 4 , in step 401 , a query text is obtained.
[0189] The query text (query) is generally related to the corresponding application scenario. For example, a conversation scenario can be a search keyword related to the conversation scenario topic or a search term composed of multiple search keywords. It is used to obtain the latest search information and extract the corresponding answer text fragment, providing the corresponding reference text for the large language model. Among them, the processing framework diagram for obtaining the latest search information based on the query text and extracting the answer text fragment can be seen in Figure 5, which is a processing process diagram for extracting the answer text fragment provided in an embodiment of the present application.
[0190] Continuing to refer to FIG. 4 , in step 402 , a search engine interface is called to obtain search results.
[0191] As shown in Figure 5, after obtaining the query text, the relevant search engine interface can be called to obtain the search results, wherein the search engine interface can be a relevant terminal browser, search server or other engine and function interface with search function. The search results are generally a plain text, including a body text and a corresponding text title.
[0192] In some embodiments, the search results of the search engine interface are sorted by generation time, that is, the search information with the latest generation time is sorted at the front, and then the 10 relevant text search results with the latest generation time are obtained from the search engine interface in reverse order of generation time, that is, the "top-10 text" shown in Figure 5.
[0193] Continuing to refer to FIG. 4 , in step 403 , the answer text segment is extracted.
[0194] In the text included in the text search results, only part of the text is the answer to the query text, and the rest is worthless text information irrelevant to the query text. First, the answer text fragment (Passage) will be extracted from the text search results.
[0195] As shown in Figure 5, the 10 relevant text search results (top-10 main texts) are used as the main text. For each input main text, it is first cut into text segments of 500 words each, as shown in Figure 5, text segment 1, text segment 2, and so on until text segment n. Then, each text segment is input into the 12-layer BERT model (i.e., BERT-12-layer, equivalent to the second language model mentioned above) together with the query text. The BERT model calculates the matching score (QS Score) of each text segment and the query text. Since the text cannot be directly input into the BERT model for processing, each text element (Token) in the query text and the text segment is first treated as a regular token. All regular tokens are spliced together through the start tag (CLS) and the segmentation tag (SEP), and then embedded to obtain the embedded feature vector of the spliced text. The embedded feature vector is input into the 12-layer BERT model for encoding to obtain a semantic feature vector. The semantic feature vector corresponding to the start tag (CLS) is input into the fully connected layer and mapped to the matching score (QS Score) between the text segment and the query text, wherein the activation function of the fully connected layer can be a Softmax function. When calling the BERT model to encode the embedded feature vector of the spliced text, the starting probability (Start Score) of each text element (Token) of the text segment in the spliced text is predicted to be suitable as the starting element of the answer text segment and the ending probability (End Score) of the ending element of the answer text segment. Specifically, after encoding the embedded feature vector (Token Represent, Token Rep) of each text element in the text segment, the encoded semantic feature vector is mapped to the corresponding probability of being the starting element of the answer text segment and the probability of the ending element through the fully connected layer.
[0196] As shown in Figure 5, after each text segment and query text are concatenated and subjected to the above operations, the text segment with the highest matching score (Max QS Score) between the text segment and the query text is determined. In this text segment, the element with the highest starting probability (Max Start Score) suitable as the starting element of the answer text segment is screened out and determined as the starting element of the answer text segment. At the same time, the element with the highest ending probability (Max End Score) suitable as the ending element of the answer text segment is screened out and determined as the ending element of the answer text. Finally, the text between the starting element and the ending element in the text segment is used as the final answer text segment. In this way, the 10 text search results (top-10 main texts), that is, the 10 main texts, can be extracted to obtain the corresponding 10 answer text segments.
[0197] Continuing to refer to FIG. 4 , in step 404 , the relevance scores of the query text and the answer text segment are determined.
[0198] After obtaining 10 answer text fragments, the answer text fragments related to the query text are screened as reference texts for the large language model, specifically by determining the relevance scores of the query text and the answer text fragments respectively.
[0199] The process of determining the relevance score of the query text and the answer text fragment can be seen in Figure 6. Figure 6 is a process diagram for determining the relevance score of the query text and the answer text fragment provided by an embodiment of the present application. First, for each answer text fragment, each element in the query text (Query shown in Figure 6), the title of the answer text fragment (Title shown in Figure 6) and the answer text fragment (Passage shown in Figure 6) is used as a regular tag. The regular tags between the three are spliced with a segmentation tag (SEP) to obtain a spliced text tag sequence, and a start tag (CLS) is inserted at the head of the spliced text tag sequence. The spliced text tag sequence is embedded to obtain the spliced text embedded feature vector and then input into the cross-encoder 601 (Cross-Encoder) for encoding to obtain the corresponding semantic feature vector. The cross-encoder here is equivalent to the third language model mentioned above. Finally, the semantic feature vector corresponding to the start tag (CLS) is input into the multi-layer perceptron 602 and mapped to the relevance score of the query text and the answer text fragment, denoted as S q,t+p , where the correlation score is the correlation score fitted by the multilayer perceptron.
[0200] In some embodiments, the cross encoder does not specifically limit the model type and topology of the deep neural network and can be replaced with various other model structures, such as using a non-cross encoder. Alternatively, the number of network model layers can be increased or decreased, or the structure of the deep network model can be adjusted based on the actual needs of the model. The cross encoder is trained using 50 million text pairs consisting of query text and answer text fragments. The training data can be obtained through data mining or manual precise annotation.
[0201] After determining the relevance scores of the 10 answer text fragments with the query text, the answer text fragments can be filtered using the relevance score threshold. The answer text fragments with relevance scores less than the relevance score threshold are filtered out, and the remaining answer text fragments are used as reference texts for the large language model.
[0202] Continuing with FIG4 , in step 405 , the large language model is called to generate text.
[0203] After determining the reference text, the reference text and the query text are input into the large language model for prediction. The large language model here is equivalent to the first language model mentioned above. The specific prediction process is as follows: First, each element in the reference text and the query text is treated as a regular token. The regular tokens between the reference text and the query text are spliced together using a segmentation token (SEP) to form a spliced sequence. A start token (CLS) is inserted at the head of the spliced sequence. The spliced sequence is then embedded to obtain an embedded feature vector for the spliced sequence. The embedded feature vector is then input into the large language model for encoding, ultimately obtaining a semantic feature vector for the spliced sequence. Each element semantic feature vector is mapped to the starting probability of the starting element of the answer text as the query text, and the ending probability of the ending element. The element with the highest starting probability is determined as the starting element of the answer text, and the element with the highest ending probability is determined as the ending element of the answer text. This predicts the starting and ending positions of the answer text of the query text in the reference text. The text between the starting and ending positions of the reference text is determined as the answer text, that is, the generated text of the large language model for the query text and output.
[0204] Through the embodiments of the present application, a query text is obtained and a search engine interface is called to generate multiple, latest text search results that prioritize timeliness. Then, answer text fragments that match the query text are extracted from the text search results and filtered to determine the text fragment with the highest relevance to the query text from the multiple answer text fragments. This is used as the reference text for the large language model. The large language model is then called to predict the answer text corresponding to the query text from the reference text and generate the corresponding text, thereby increasing the controllability of the content generated by the large language model and improving the timeliness of the generated content.
[0205] Continuing with FIG4 , in step 406 , reference alignment is performed on the generated text.
[0206] Considering that the text generated by the large language model in the related art may have problems such as insufficient factuality, low credibility, and fabrication, which will affect the performance of the large language model, the embodiment of the present application selects texts with high similarity to the generated text from the provided candidate texts as quoted texts, and adds the quoted content to the generated text content to achieve quote alignment, improve the authenticity and credibility of the generated text, and solve the problems of the original generated text having low factuality and credibility and possible fabrication.
[0207] See Figure 7, which is a flowchart of the process of citation alignment of generated text provided by an embodiment of the present application. The specific process of citation alignment will be introduced in conjunction with Figure 7.
[0208] In step 701 , the answer text and candidate texts are obtained, and the process proceeds to steps 702 and 705 .
[0209] After the large language model predicts the corresponding answer text based on the query text, it can obtain search text related to the query text through search, or call a text mining model based on the query text to mine and obtain the search text, and use these search texts as candidate texts, where the candidate texts are used as reference texts for the answer text.
[0210] In step 702, the sentence segments are segmented and combined in pairs.
[0211] After obtaining the answer text and candidate text, the answer text and candidate text are divided into multiple clauses, that is, the text is divided into multiple segments to be matched. The segments to be matched must include the basic semantic information of the text. The segment length cannot be too short, for example, the length cannot be less than 10. Because if the segment is too short, there is too little reference information for subsequent modules to use, causing the subsequent modules to fail and the semantic extraction effect cannot be guaranteed. The specific process is: the text is segmented using punctuation marks. If the length of the segmented segment is less than 10, it is merged with the adjacent segment to form a longer segment. The above process is repeated until the length of all segments to be matched exceeds 10 or the segment to be matched has included the entire input text.
[0212] After the segmentation process is complete, two sets of matching segments are obtained, corresponding to the answer text and the candidate text. The two sets of segmented clauses (matching segments) are combined in pairs to obtain multiple candidate clause pairs. Specifically, each segmented clause in the answer text is combined with each segmented clause in the candidate text one by one to obtain multiple candidate clause pairs.
[0213] For example, the answer text is segmented into 20 clauses, there are 10 candidate texts in total, and each candidate text is segmented into 5 clauses. After the two groups of segmented clauses are paired one by one, the total number of candidate clause pairs obtained is 1000 (20x10x5).
[0214] In step 703, candidate clause pairs are filtered.
[0215] Filtering the matching clause pairs mainly involves preliminary screening of the multiple candidate clause pairs obtained to filter out obviously mismatched or unimportant sentences, thereby reducing the computational complexity of subsequent processing.
[0216] In some embodiments, the filtering method can be a key fragment identification based on keywords. Specifically, some unsupervised recognition algorithms are called to respectively identify the keywords of the answer text and the corresponding clauses of the candidate text in the candidate clause pair, and the clauses including the keywords are identified as key text fragments. Among all the candidate clause pairs, if one of them is not a key text fragment, it is filtered, which means that the keywords of the two clauses in the candidate clause pair are different, and the candidate clause pair is filtered. Among them, the unsupervised recognition algorithm can be a graph-based text processing and sorting algorithm, such as a text sorting algorithm (Text Rank), a term frequency-inverse document frequency method (Term Frequency-Inverse Document Frequency, TF-IDF) or a rapid automatic keyword extraction algorithm (Rapid Automatic Keyword Extraction, RAKE). The three methods can also be used in combination to more accurately extract keywords in the text.
[0217] In some embodiments, the filtering method can be a similarity matching method based on text segment embedding (sentence embedding), which can specifically be to determine the edit distance of the two clauses in the clause pair, or to determine the longest common subsequence of the two clauses, or to use the above methods at the same time. When the edit distance is less than the distance threshold or the longest common subsequence is greater than a certain sequence number, it is determined that there are clause similarities. Taking into account that the similarity effect of determining the text segment embedding of long sentences is poor, and the length standard of long sentences is not easy to specify, the two clauses in the candidate clause pair can be further segmented to obtain multiple short sentences, and then it is determined whether there is a similar short sentence. If there is a similar short sentence, the corresponding candidate clause pair is retained, and if not, the candidate clause pair is filtered.
[0218] In step 704, the similarity of the candidate clause pairs is determined.
[0219] Determine whether a clause in the candidate text can constitute a reference to a clause in the answer text. More specifically, determine whether the semantics of the candidate text clause in the candidate clause pair are similar to those of the answer text clause or whether the semantics of the former include the latter. If they are similar or the semantics of the former include the latter, then the candidate clause pair can be determined to be a matching clause pair. Specifically, a matching clause pair can be determined from multiple candidate clause pairs. The semantic similarity of the two clauses can be calculated using a semantic similarity matching model. When the calculated semantic similarity exceeds a similarity threshold, it indicates that the semantics of the answer text clause are similar to those of the candidate text clause, that is, the candidate text clause in the candidate clause pair can be used as a reference to the corresponding answer text clause. The candidate clause pair can then be determined to be a matching clause pair. The similarity threshold can be set according to actual needs, for example, it can be 0.8.
[0220] In some embodiments, the semantic similarity matching model (equivalent to the fourth language model above) can specifically be a Sentence-BERT model, and the model prediction process can be seen in Figure 8, which is a prediction process diagram of the semantic similarity matching model provided in an embodiment of the present application. As shown in Figure 8, two sentences A and sentence B to be detected are first input into the model, and after encoding processing by the hidden layer of the BERT model, they are input into the pooling layer for pooling (Pooling) to map sentence A to an embedding vector u, and at the same time map sentence B to an embedding vector v, where the vector dimensions of u and v are the same. The embedding vector u and the embedding vector v are then vector spliced to obtain a fusion vector output by the hidden layer, expressed as "|uv|". Among them, the specific method of splicing can be to sum the elements in the embedding vector u to the elements in the corresponding positions of the embedding vector v. The embedded vector u, the embedded vector v, and the fused vector are directly concatenated, and the concatenated features are input into a fully connected layer for prediction classification (Softmax classifier), thereby predicting the semantic similarity between sentences A and B. The activation function of the fully connected layer is the Softmax function, and the semantic similarity can be the minimum edit distance similarity between vectors, the Euclidean distance similarity, or the cosine similarity (cosine-similarity).
[0221] In step 705, matching is performed based on the reference markers in the answer text.
[0222] After determining that the clause of the candidate text in the matching clause pair can be used as the reference text of the clause of the corresponding answer text, the clause of the candidate text can be inserted as the reference text into the end of the clause of the corresponding answer text based on the reference mark of each clause in the answer text and using the matching clause pair.
[0223] Steps 702 to 704 and step 705 may be executed in parallel or in series, which is not limited in the embodiment of the present application.
[0224] In step 706, post-processing is performed on the matching clause pairs.
[0225] After all matching clause pairs that meet the similarity threshold are determined from the candidate clause pairs, the clauses of the candidate text in the matching clause pairs can be used as reference texts for the clauses of the corresponding answer texts, and the position and number of the reference text serial numbers can be adjusted according to the answer texts.
[0226] Taking into account that the clauses of the candidate text and the clauses of the answer text are combined in pairs to obtain matching clause pairs, it is possible that a clause of the answer text has multiple clauses of the candidate text as reference texts, that is, a clause of the answer text has multiple reference texts. Then, according to the semantic similarity of the corresponding matching clause pairs, they are sorted, and the clause of the candidate text with the highest semantic similarity to the clause of the answer text is directly added to the end of the clause of the answer text. The remaining reference texts (clauses of the candidate text) are sorted in order according to the size of the semantic similarity, thereby completing the post-processing of the matching clause pairs.
[0227] For example, if clause A in the answer text has multiple candidate clauses (such as clause B and clause C) as citation texts, and the semantic similarity between clauses A and C is 0.9, and the semantic similarity between clauses A and B is 0.8, clause C will be added directly to the end of clause A as the citation text, and clause B will also be placed after clause C as the citation text. At the same time, you can adjust clause C of the candidate text to citation 1 and clause B of the candidate text to citation 2, and then set the number of citations for clause A in the answer text to 2.
[0228] In step 707, a reply text including the quoted text is generated.
[0229] After adding reference text to all clauses of the answer text, the large language model generates the answer text based on the query text. Each clause in the answer text will have a corresponding reference text. The corresponding reference text content can be quickly queried based on the sequence number of the reference text.
[0230] Through the embodiments of the present application, texts with high similarity to the generated texts of the large language model (i.e., the answer texts) are screened out from the provided candidate texts as quoted texts, so that quoted content is added to each clause in the generated text, thereby improving the authenticity and credibility of the texts generated by the large language model.
[0231] In summary, the embodiment of the present application obtains the query text and calls the search engine interface to generate multiple latest text search results with timeliness priority, then extracts the answer text fragments related to the query text from the text search results, and performs screening to determine the text fragment with the highest relevance to the query text from the multiple answer text fragments as the reference text of the large language model. Then, the large language model is called to predict the answer text corresponding to the query text from the reference text and generate the corresponding text, thereby increasing the controllability of the content generated by the large language model and improving the timeliness of the generated content. Subsequently, the text with high similarity to the generated text (i.e., the answer text) of the large language model is screened from the provided candidate text as the reference text, so that the reference content is added to each clause in the generated text. On the one hand, it can solve the problem that the content generated by the large language model has no reference information and is difficult to judge true or false and further understand. On the other hand, using the reference content as reference material for the text generated by the large language model also improves the authenticity and credibility of the text generated by the large language model, and solves the problem that the original generated text is factual and has low credibility and may be fabricated. In addition, the text processing method provided in the embodiment of the present application is applied to the search system, which can significantly improve the efficiency of obtaining search information and improve the experience of the search system by quickly filtering reference materials from the search results through query text.
[0232] The following further describes an exemplary structure of the text processing device 453 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in FIG2 , the software modules stored in the text processing device 453 in the memory 450 may include:
[0233] The acquisition module 4531 is configured to obtain the query text; call the search engine interface based on the query text to obtain multiple text search results corresponding to the query text; obtain multiple answer text fragments matching the query text from the multiple text search results; the determination module 4532 is configured to determine the relevance of the query text to the multiple answer text fragments, and use the answer text fragment corresponding to the maximum relevance as the reference text of the query text; the calling module 4533 is configured to call the first language model based on the query text and the reference text to obtain the answer text of the query text.
[0234] In some embodiments, the acquisition module 4531 is further configured to call a search engine interface based on the query text, so that the search engine interface searches for multiple text search results related to the query text in order of generation time; and obtain multiple text search results related to the query text from the search engine interface.
[0235] In some embodiments, the acquisition module 4531 is further configured to divide the text search results into multiple candidate reference text segments of fixed length; obtain the matching scores between the query text and the multiple candidate reference text segments, and the starting probability of each element in the candidate reference text segment as the starting position and the ending probability of the ending position of the answer text segment; determine the optimal matching segment, wherein the optimal matching segment is the candidate reference text segment corresponding to the matching score with the maximum value; determine the starting element and the ending element in the optimal matching segment, wherein the starting element is the element corresponding to the starting probability with the maximum value, and the ending element is the element corresponding to the ending probability with the maximum value; and use the part of the optimal matching segment between the starting element and the ending element as the answer text segment matching the query text.
[0236] In some embodiments, the acquisition module 4531 is further configured to combine multiple candidate reference text fragments with the query text into text pairs; call the second language model based on each text pair to obtain the matching scores between the query text and the multiple candidate reference text fragments, as well as the starting probability of each element in the candidate reference text fragment as the starting position and the ending probability of the ending position of the answer text fragment.
[0237] In some embodiments, the acquisition module 4531 is further configured to treat each element in the query text and the candidate reference text fragment as a regular tag, and splice the regular tags of each element into a first tag sequence, wherein a start tag is inserted at the head of the first tag sequence; embed the first tag sequence to obtain an embedded feature vector of the first tag sequence; call the second language model for encoding based on the embedded feature vector of the first tag sequence to obtain a semantic feature vector of the first tag sequence; map the semantic feature vector of the start tag in the semantic feature vector of the first tag sequence to obtain a matching score between the query text and the candidate reference text fragment; map the semantic feature vector corresponding to each regular tag in the semantic feature vector of the first tag sequence to obtain the starting probability of each element in the candidate reference text fragment as the starting position of the answer text fragment and the ending probability of the ending position.
[0238] In some embodiments, the determination module 4532 is configured to treat each element in the query text, the answer text fragment, and the title text as a regular tag, and splice the regular tags of each element into a second tag sequence, wherein a start tag is inserted at the head of the second tag sequence, and the regular tags between the query text and the answer text fragment, and between the answer text fragment and the title text are connected with separation tags; embed the second tag sequence to obtain an embedded feature vector of the second tag sequence; call a third language model to encode the embedded feature vector of the second tag sequence to obtain a semantic feature vector of the second tag sequence; and map the semantic feature vector of the second tag sequence to obtain the relevance of the query text and the answer text fragment.
[0239] In some embodiments, the calling module 4533 is configured to call the first language model based on the query text and the reference text to perform predictive processing on the query text to determine the starting element and the ending element of the answer text in the reference text; and use the text between the starting element and the ending element of the reference text as the answer text of the query text.
[0240] In some embodiments, the calling module 4533 is further configured to treat each element in the query text and the reference text as a regular token, and to concatenate the regular tokens into a third token sequence, wherein a start token is inserted at the head of the third token sequence, and the regular tokens between the query text and the reference text are connected by a separator token;
[0241] The third token sequence is embedded to obtain an embedded feature vector of the third token sequence; the first language model is called to perform encoding processing based on the embedded feature vector of the third token sequence to obtain a semantic feature vector of each regular token in the third token sequence; the semantic feature vector corresponding to each regular token is mapped to obtain each element of the reference text as a starting probability of the starting element and an ending probability of the ending element of the answer text; the element corresponding to the maximum starting probability is used as the starting element of the answer text, and the element corresponding to the maximum ending probability is used as the ending element of the answer text.
[0242] In some embodiments, the determination module 4532 is further configured to obtain multiple candidate quotation texts, wherein the candidate quotation texts include materials for quoting in the answer text; divide the answer text into multiple answer text segments, and divide each candidate quotation text into multiple quotation text segments; determine a quotation text segment that matches at least one answer text segment, and insert the quotation text segment that matches at least one answer text segment into the answer text.
[0243] In some embodiments, the determination module 4532 is further configured to combine multiple answer text fragments and multiple reference text fragments in pairs to form multiple candidate text pairs; identify at least one matching text pair from the multiple candidate text pairs, and determine that the reference text fragment in the matching text pair matches the answer text fragment in the matching text pair.
[0244] In some embodiments, the determination module 4532 is further configured to perform keyword recognition on the answer text segment and the reference text segment in the candidate text pair respectively; in response to no keyword being recognized from at least one of the answer text segment and the reference text segment, the candidate text pair is deleted.
[0245] In some embodiments, the determination module 4532 is further configured to encode the answer text segment and the reference text segment in the candidate text pair respectively to obtain the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment; determine the similarity between the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment; and delete the candidate text pair in response to the similarity being less than a similarity threshold.
[0246] In some embodiments, the determination module 4532 is further configured to call the fourth language model to encode the answer text segment and the candidate reference text segment in the candidate text pair respectively to obtain the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment; perform pooling processing on the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment respectively to obtain the semantic feature vector of the answer text segment and the semantic feature vector of the reference text segment; perform splicing processing on the semantic feature vector of the reference text segment and the semantic feature vector of the answer text segment to obtain the fused feature vector of the candidate text pair; perform mapping processing on the fused feature vector to obtain the semantic similarity of the answer text segment and the reference text segment in the candidate text pair; determine the candidate text pair whose semantic similarity is greater than the semantic similarity threshold as a matching text pair.
[0247] In some embodiments, the determination module 4532 is further configured to, in response to the number of at least one answer text segment being less than or equal to a first quantity threshold, insert the reference text segments that match at least one answer text segment into positions after the at least one answer text segment is matched; in response to the number of reference text segments that match at least one answer text segment being greater than a first quantity threshold, uniformly insert the reference text segments that match at least one answer text segment into the end of the answer text.
[0248] In some embodiments, the determination module 4532 is further configured to, in response to the number of at least one answer text fragment being greater than a second quantity threshold, sort the at least one answer text fragment and the matched reference text fragment in descending order according to the similarity between the at least one answer text fragment and the matched reference text fragment, starting from the head in the descending sorting result, and use the reference text fragments that match a set number or a set proportion of the answer text fragments as the text fragments to be inserted.
[0249] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the text processing method described in the embodiment of the present application.
[0250] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the text processing method provided by an embodiment of the present application, for example, the text processing method shown in Figures 3A-3L.
[0251] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.
[0252] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0253] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).
[0254] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.
[0255] In summary, through the embodiment of the present application, the query text is obtained and the search engine interface is called to generate multiple text search results with high timeliness. Then, the answer text fragments relevant to the query text are extracted from the text search results, and screening is performed to determine the text fragment with the highest relevance to the query text from the multiple answer text fragments as the reference text of the large language model. Then the large language model is called to predict the answer text corresponding to the query text from the reference text and generate the corresponding text, thereby increasing the controllability of the content generated by the large language model and improving the timeliness of the generated content. The text with high similarity to the generated text (i.e., the answer text) of the large language model is screened out from the provided candidate reference text as the reference text, so that the reference content is added to each clause in the generated text. On the one hand, it can solve the problem that the content generated by the large language model has no reference information and is difficult to judge true or false and to further understand. On the other hand, the reference content is used as the reference material for the text generated by the large language model, which also improves the authenticity and credibility of the text generated by the large language model, and solves the problem that the original generated text is factual and has low credibility and may be fabricated. In addition, the text processing method provided in the embodiment of the present application is applied to the search system, which can significantly improve the efficiency of obtaining search information and improve the experience of the search system by quickly filtering reference materials from the search results through query text.
[0256] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.
Claims
1. A text processing method, the method being executed by an electronic device, the method comprising: Get the query text; Calling a search engine interface based on the query text to obtain a plurality of text search results corresponding to the query text; Acquire multiple answer text segments matching the query text from the multiple text search results; Determine the relevance of the query text to the multiple answer text segments respectively, and use the answer text segment corresponding to the maximum relevance as the reference text of the query text; A first language model is called based on the query text and the reference text to obtain an answer text for the query text.
2. The method according to claim 1, wherein: The step of calling a search engine interface based on the query text to obtain a plurality of text search results corresponding to the query text includes: Calling a search engine interface based on the query text so that the search engine interface searches for multiple text search results related to the query text in a manner sorted by generation time; A plurality of text search results related to the query text are obtained from the search engine interface.
3. The method according to claim 1, wherein: The step of obtaining a plurality of answer text segments matching the query text from the plurality of text search results includes: The following processing is performed for each text search result: Segmenting the text search results into a plurality of candidate reference text segments of fixed length; Obtaining matching scores between the query text and the plurality of candidate reference text segments, respectively, and a starting probability of each element in the candidate reference text segment being a starting position and an ending probability of each element in the candidate reference text segment being an answer text segment; Determine the best matching segment, wherein the best matching segment is the candidate reference text segment corresponding to the matching score with the maximum value; Determine a start element and an end element in the best matching segment, wherein the start element is the element corresponding to the start probability of the maximum value, and the end element is the element corresponding to the end probability of the maximum value; The portion of the best matching segment between the start element and the end element is used as the answer text segment matching the query text.
4. The method according to claim 3, wherein: The obtaining of the matching scores between the query text and the plurality of candidate reference text segments, and the starting probability of each element in the candidate reference text segment being the starting position and the ending probability of each element in the candidate reference text segment being the ending position of the answer text segment, comprises: Combining the plurality of candidate reference text segments with the query text into text pairs respectively; Based on each of the text pairs, a second language model is called to obtain matching scores between the query text and the multiple candidate reference text segments, as well as a starting probability of each element in the candidate reference text segment being a starting position and an ending probability of each element in the candidate reference text segment being an answer text segment.
5. The method according to claim 4, wherein: The method of calling the second language model based on each of the text pairs to obtain the matching scores between the query text and the plurality of candidate reference text segments and the starting probability of each element in the text segment being the starting position and the ending probability of the ending position of the answer text segment comprises: For each candidate quoted text segment, the following processing is performed: Taking each element in the query text and the candidate reference text fragment as a regular tag, and splicing the regular tags of each element into a first tag sequence, wherein a start tag is inserted at the head of the first tag sequence; Performing embedding processing on the first tag sequence to obtain an embedded feature vector of the first tag sequence; Calling the second language model to perform encoding processing based on the embedded feature vector of the first tag sequence to obtain a semantic feature vector of the first tag sequence; Mapping the semantic feature vector of the start tag in the semantic feature vector of the first tag sequence to obtain a matching score between the query text and the candidate reference text segment; The semantic feature vector corresponding to each of the conventional tags in the semantic feature vector of the first tag sequence is mapped to obtain the starting probability of each element in the candidate reference text segment as the starting position of the answer text segment and the ending probability of the ending position.
6. The method according to any one of claims 1 to 5, wherein: The answer text segment includes a title text, and determining the relevance of the query text to the multiple answer text segments respectively includes: For each answer text fragment, perform the following processing: Taking each element in the query text, the answer text segment, and the title text as a regular tag, splicing the regular tags of each element into a second tag sequence, wherein a start tag is inserted at the head of the second tag sequence, and the regular tags between the query text and the answer text segment, and between the answer text segment and the title text are connected with a separator tag; Performing embedding processing on the second tag sequence to obtain an embedded feature vector of the second tag sequence; Calling a third language model to encode the embedded feature vector of the second tag sequence to obtain a semantic feature vector of the second tag sequence; Mapping processing is performed on the semantic feature vector of the second tag sequence to obtain the relevance between the query text and the answer text segment.
7. The method according to any one of claims 1 to 6, wherein: The calling of the first language model based on the query text and the reference text to obtain an answer text of the query text includes: Based on the query text and the reference text, a first language model is called to perform prediction processing on the query text to determine the starting element and the ending element of the answer text in the reference text; The text between the start element and the end element of the reference text is used as the answer text of the query text.
8. The method according to claim 7, wherein: The calling of the first language model to perform prediction processing on the query text and the reference text to determine the starting element and the ending element of the answer text in the reference text includes: Taking each element in the query text and the reference text as a regular tag, splicing the regular tags into a third tag sequence, wherein a start tag is inserted at the head of the third tag sequence, and the regular tags between the query text and the reference text are connected with a separator tag; Performing embedding processing on the third marker sequence to obtain an embedded feature vector of the third marker sequence; Calling the first language model to perform encoding processing based on the embedded feature vector of the third tag sequence to obtain a semantic feature vector of each regular tag in the third tag sequence; Mapping the semantic feature vector corresponding to each conventional mark to obtain each element of the reference text as the starting probability of the starting element of the answer text and the ending probability of the ending element; The element corresponding to the starting probability of the maximum value is used as the starting element of the answer text, and the element corresponding to the ending probability of the maximum value is used as the ending element of the answer text.
9. The method according to any one of claims 1 to 8, wherein: After calling the first language model based on the query text and the reference text to obtain the answer text of the query text, the method further includes: Acquire a plurality of candidate quoted texts, wherein the candidate quoted texts include materials for quoting in the answer text; The answer text is divided into a plurality of answer text segments, and each of the candidate quoted texts is divided into a plurality of quoted text segments; A reference text segment that matches at least one answer text segment is determined, and the reference text segment that matches the at least one answer text segment is inserted into the answer text.
10. The method according to claim 9, wherein: The step of determining at least one reference text segment that matches the answer text segment comprises: Combining the multiple answer text segments and the multiple reference text segments in pairs to form multiple candidate text pairs; At least one matching text pair is identified from the plurality of candidate text pairs, and it is determined that the reference text segment in the matching text pair matches the answer text segment in the matching text pair.
11. The method according to claim 10, wherein: The step of identifying at least one matching text pair from the plurality of candidate text pairs comprises: Calling a fourth language model to encode the answer text segment and the candidate reference text segment in the candidate text pair, respectively, to obtain an embedded feature vector of the answer text segment and an embedded feature vector of the reference text segment; Performing pooling processing on the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment respectively to obtain the semantic feature vector of the answer text segment and the semantic feature vector of the reference text segment; Concatenate the semantic feature vector of the reference text segment with the semantic feature vector of the answer text segment to obtain a fused feature vector of the candidate text pair; Mapping the fused feature vector to obtain semantic similarity between the answer text segment and the reference text segment in the candidate text pair; The candidate text pairs whose semantic similarity is greater than a semantic similarity threshold are determined as matching text pairs.
12. The method according to claim 10, wherein: Before identifying a matching text pair from the plurality of candidate text pairs, the method further includes: For each candidate text pair, perform the following processing: Performing keyword recognition on the answer text segment and the quoted text segment in the candidate text pair respectively; In response to not identifying a keyword from at least one of the answer text segment and the reference text segment, Delete the candidate text pair.
13. The method according to claim 10, wherein: Before identifying a matching text pair from the plurality of candidate text pairs, the method further includes: For each candidate text pair, perform the following processing: Encoding the answer text segment and the reference text segment in the candidate text pair respectively to obtain an embedded feature vector of the answer text segment and an embedded feature vector of the reference text segment; Determining the similarity between the embedded feature vector of the answer text segment and the embedded feature vector of the reference text segment; In response to the similarity being less than a similarity threshold, the candidate text pair is deleted.
14. The method according to any one of claims 9 to 13, wherein: The step of inserting the reference text segment that matches the at least one answer text segment into the answer text comprises: In response to the number of the at least one answer text segment being less than or equal to a first number threshold, inserting reference text segments that match the at least one answer text segment into positions after the at least one answer text segment, respectively; In response to the number of the at least one answer text segment being greater than the first number threshold, the reference text segments respectively matched by the at least one answer text segment are uniformly inserted into the end of the answer text.
15. The method according to any one of claims 9 to 13, wherein: Before inserting the reference text segment matching the at least one answer text segment into the answer text, the method further comprises: In response to the number of reference text fragments matched by the at least one answer text fragment being greater than a second quantity threshold, the at least one answer text fragment and the matched reference text fragment are sorted in descending order according to the similarity between them, and starting from the head in the descending sorting result, a set number or a set proportion of reference text fragments matched by the answer text fragments are used as text fragments to be inserted.
16. A text processing device, comprising: An acquisition module configured to acquire a query text; The acquisition module is further configured to call a search engine interface based on the query text to obtain a plurality of text search results corresponding to the query text; The acquisition module is further configured to acquire a plurality of answer text segments matching the query text from the plurality of text search results; A determination module configured to determine the relevance between the query text and the plurality of answer text segments, respectively, and use the answer text segment corresponding to the maximum relevance as a reference text for the query text; The calling module is configured to call the first language model based on the query text and the reference text to obtain an answer text for the query text.
17. An electronic device, comprising: Memory for storing computer executable instructions or computer programs; A processor, configured to implement the text processing method according to any one of claims 1 to 15 when executing the computer executable instructions or computer programs stored in the memory.
18. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implements the text processing method according to any one of claims 1 to 15.
19. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program, when executed by a processor, implements the text processing method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Answer extraction method and device, computer equipment and storage medium
CN112347229A
Method and system for generating fine-grained image from stacked text based on semantic constraints
CN112348911A
Question and answer index generation method and device, question and answer model processing method and device and storage medium
CN114328796A
Text matching method and device, storage medium and terminal
CN114661861A
Text processing method and device, electronic equipment and computer readable storage medium
CN116340467A
Cited By
Multi-information element semantic combination method and system based on large language model
CN119599023A
The method for adaptive automatic selection of text content search modes based on entity distribution in generative artifical intelligence systems
RS20250781A3
