Text processing method and device and electronic equipment

Abstract

The invention discloses a text processing method and device and electronic equipment. The method comprises the steps that in response to the received initial inquiry text, the inquiry intention of an inquiry object is determined based on the initial inquiry text, and the inquiry object is used for representing an object sending the initial inquiry text; based on the inquiry intention, determining a question and answer database matched with the initial inquiry text; inputting the inquiry intention, the initial inquiry text and data in a question-answer database into a text question-answer model, and selecting a target text from the question-answer database by utilizing the text question-answer model; a reply text corresponding to the initial inquiry text is generated based on the target text, and the text structure of the reply text is the same as that of the initial inquiry text. The information query method and device solve the technical problem that in the related art, the information query efficiency is low.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text processing method, device and electronic device. Background Art

[0002] As the level of enterprise informatization increases, the amount and complexity of data generated by enterprises in their daily operations may show explosive growth. These data not only include information such as finance, meeting minutes, system management documents, employee assessments, but also cover multi-dimensional data such as production behavior and operating conditions. The corresponding data stored in the database usually has a large amount of data and the data is complex and diverse, resulting in low information query efficiency in the actual information query process.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a text processing method, device and electronic device to at least solve the technical problem of low efficiency in information query in related technologies.

[0005] According to one aspect of an embodiment of the present invention, a text processing method is provided, including: in response to receiving an initial inquiry text, determining the inquiry intention of an inquiry object based on the initial inquiry text, wherein the inquiry object is used to characterize the object that sends the initial inquiry text; based on the inquiry intention, determining a question and answer database that matches the initial inquiry text; inputting the inquiry intention, the initial inquiry text, and data in the question and answer database into a text question and answer model, and selecting a target text from the question and answer database using the text question and answer model; and generating a reply text corresponding to the initial inquiry text based on the target text, wherein a text structure of the reply text is the same as a text structure of the initial inquiry text.

[0006] Furthermore, the inquiry intention includes: knowledge query intention, and the question and answer database includes: a knowledge base corresponding to the knowledge query intention; the inquiry intention, the initial inquiry text and the data in the question and answer database are input into a text question and answer model, and the target text is selected from the question and answer database using the text question and answer model, including: analyzing the initial inquiry text based on the knowledge query intention, and determining inquiry keywords from the initial inquiry text, wherein the inquiry keywords are used to reflect the needs of the inquiry object; adjusting the initial inquiry text based on the inquiry keywords to obtain a target inquiry text; matching the target inquiry text with multiple initial texts contained in the question and answer database to obtain a text matching result; and determining the target text from the multiple initial texts based on the text matching result.

[0007] Furthermore, the target query text is matched with multiple initial texts contained in the question and answer database to obtain a text matching result, including: obtaining at least one index vector corresponding to different initial texts; vectorizing the target query text to obtain a target text vector; matching at least one index vector with the target text vector to obtain a text matching result.

[0008] Furthermore, the above method also includes: performing level identification on text titles contained in different initial texts to obtain title levels corresponding to different text titles; dividing the initial text based on the title level and the text title to obtain at least one text segment; performing feature extraction on at least one text segment to obtain text segment features corresponding to at least one text segment; based on the text segment features, generating at least one index vector corresponding to different initial texts.

[0009] Furthermore, based on the text segment features, at least one index vector corresponding to different initial texts is generated, including: based on the text segment features, constructing a unique character corresponding to at least one text segment, wherein the unique character is used to reflect the source information and title content of at least one text segment, and the unique characters corresponding to different text segments are different; based on the unique character, constructing a character vector corresponding to at least one text segment; and summarizing the character vectors corresponding to different text segments to obtain at least one index vector corresponding to different initial texts.

[0010] Furthermore, the inquiry intention includes: data retrieval intention, and the question and answer database includes: a database corresponding to the data retrieval intention; inputting the inquiry intention, the initial inquiry text and the data in the question and answer database into the text question and answer model, and selecting the target text from the question and answer database using the text question and answer model, including: obtaining the historical conversation text entered by the inquiry object within the historical time period; constructing a structural query statement corresponding to the initial inquiry text based on the data retrieval intention and the initial inquiry text; adjusting the structural query statement based on the historical conversation text to obtain the target query statement; searching the data text contained in the database based on the target query statement to obtain the target text.

[0011] Furthermore, determining the inquiry intention of the inquiry object based on the initial inquiry text includes: performing semantic recognition on the initial inquiry text to obtain the initial text semantics; inputting the initial text semantics and multiple text intentions into an intention recognition model, and using the intention recognition model to determine the intention confidence corresponding to different text intentions; and determining the inquiry intention from multiple text intentions based on the intention confidence.

[0012] Furthermore, a reply text corresponding to the initial inquiry text is generated based on the target text, including: performing text recognition on the target text to obtain a text recognition result; matching the text recognition result with a plurality of sensitive words included in a sensitive word list to obtain a text matching result; adjusting the target text based on the text matching result to obtain an adjusted text; inputting the adjusted text and the initial inquiry text into a text generation model, and using the text generation model to generate a reply text.

[0013] According to another aspect of an embodiment of the present invention, a text processing device is also provided, including: an intention determination module, which is used to determine the inquiry intention of the inquiry object based on the initial inquiry text in response to receiving the initial inquiry text, wherein the inquiry object is used to characterize the object sending the initial inquiry text; a database acquisition module, which is used to determine the question and answer database matching the initial inquiry text based on the inquiry intention; a text selection module, which is used to input the inquiry intention, the initial inquiry text and the data in the question and answer database into a text question and answer model, and select a target text from the question and answer database using the text question and answer model; a text output module, which is used to generate a reply text corresponding to the initial inquiry text based on the target text, wherein the text structure of the reply text is the same as the text structure of the initial inquiry text.

[0014] Furthermore, the inquiry intention includes: knowledge query intention, and the question and answer database includes: a knowledge base corresponding to the knowledge query intention; the text selection module is also used to: analyze the initial inquiry text based on the knowledge query intention, and determine the inquiry keywords from the initial inquiry text, wherein the inquiry keywords are used to reflect the needs of the inquiry object; adjust the initial inquiry text based on the inquiry keywords to obtain the target inquiry text; match the target inquiry text with multiple initial texts contained in the question and answer database to obtain a text matching result; and determine the target text from the multiple initial texts based on the text matching result.

[0015] Furthermore, the text selection module is also used to: obtain at least one index vector corresponding to different initial texts; vectorize the target query text to obtain a target text vector; match at least one index vector with the target text vector to obtain a text matching result.

[0016] Furthermore, the above-mentioned device also includes: a level identification module, which is used to perform level identification on text titles contained in different initial texts to obtain title levels corresponding to different text titles; a text segment division module, which is used to divide the initial text based on the title level and the text title to obtain at least one text segment; a feature extraction module, which is used to perform feature extraction on at least one text segment to obtain text segment features corresponding to at least one text segment; and a vector generation module, which is used to generate at least one index vector corresponding to different initial texts based on the text segment features.

[0017] Furthermore, the vector generation module is also used to: construct a unique character corresponding to at least one text segment based on text segment features, wherein the unique character is used to reflect the source information and title content of at least one text segment, and different text segments correspond to different unique characters; construct a character vector corresponding to at least one text segment based on the unique character; and summarize the character vectors corresponding to different text segments to obtain at least one index vector corresponding to different initial texts.

[0018] Furthermore, the inquiry intention includes: data retrieval intention, and the question and answer database includes: a database corresponding to the data retrieval intention; the text selection module is also used to: obtain the historical conversation text entered by the inquiry object within the historical time period; based on the data retrieval intention and the initial inquiry text, construct a structural query statement corresponding to the initial inquiry text; adjust the structural query statement based on the historical conversation text to obtain a target query statement; search the data text contained in the database based on the target query statement to obtain the target text.

[0019] Furthermore, the intent determination module is also used to: perform semantic recognition on the initial inquiry text to obtain the initial text semantics; input the initial text semantics and multiple text intentions into the intent recognition model, and use the intent recognition model to determine the intent confidence corresponding to different text intentions; based on the intent confidence, determine the inquiry intent from multiple text intentions.

[0020] Furthermore, the text output module is also used to: perform text recognition on the target text to obtain a text recognition result; match the text recognition result with multiple sensitive words included in the sensitive word list to obtain a text matching result; adjust the target text based on the text matching result to obtain an adjusted text; input the adjusted text and the initial query text into the text generation model, and use the text generation model to generate a reply text.

[0021] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0025] According to another aspect of the embodiments of the present invention, a computer program is further provided. When the computer program is executed by a processor, the methods in the embodiments of the present invention are implemented.

[0026] In an embodiment of the present invention, in response to receiving an initial inquiry text, the inquiry intention of the inquiry object is determined based on the initial inquiry text; based on the inquiry intention, a question and answer database matching the initial inquiry text is determined; the inquiry intention, the initial inquiry text and the data in the question and answer database are input into a text question and answer model, and a target text is selected from the question and answer database using the text question and answer model; a reply text corresponding to the initial inquiry text is generated based on the target text. By selecting a question and answer database that can meet the needs of the inquiry object according to the inquiry object's current inquiry intention, and combining the inquiry intention and the initial inquiry text at the same time, the text question and answer model is used to search the data contained in the question and answer database to obtain the corresponding target text, the time spent on searching for data irrelevant to the needs of the inquiry object can be reduced, thereby improving the efficiency of querying the initial inquiry text, and further solving the technical problem of low efficiency in information query in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 is a flowchart of a text processing method according to an embodiment of the present invention;

[0029] Figure 2 is a structural block diagram of a text processing system according to an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a text processing process according to an embodiment of the present application;

[0031] Figure 4 is a flowchart of a text conversion slicing process according to an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of a document segmentation structure according to an embodiment of the present application;

[0033] Figure 6It is a structural block diagram of a text processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] According to an embodiment of the present invention, a method embodiment of text processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 1 is a flowchart of a text processing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0038] Step S102, in response to receiving the initial inquiry text, determining the inquiry intention of the inquiry object based on the initial inquiry text.

[0039] The inquiry object is used to represent the object that sends the initial inquiry text.

[0040] The initial inquiry text may refer to a text that can reflect the needs of the inquiry object for information query, such as a request for data retrieval, a message for knowledge search, etc., but is not limited thereto. The inquiry object may refer to the object that sends the initial inquiry text, such as a user, a computer with information acquisition function, etc., but is not limited thereto.

[0041] In an optional solution of this embodiment, when the inquirer needs to query the required information from these data, the amount of data stored in the database may be too large and the data may be complicated and diverse, resulting in the text question answering model used for information query being unable to quickly and accurately output the required information. Considering the inquirer's needs for different types of data, which usually correspond to different inquiry intentions, for example, for unstructured data such as meeting minutes and system management documents, the inquirer usually performs natural language query, data understanding and other query operations, and the corresponding text question answering model needs to deeply understand the query text entered by the inquirer and the relevance between these unstructured data to ensure that the selected target text can match the query text; for structured data such as production results and finance, the inquirer usually performs management operations such as data retrieval and data update, and the corresponding text question answering model only needs to ensure that the selected target text is complete and consistent with the query text. Therefore, in order to improve the efficiency of the text question answering model in performing information query based on the inquiry text input by the inquirer, the text processing system can first divide different types of data to construct different databases according to the inquiry intentions that the inquirer may raise. For example, a corresponding knowledge base can be constructed for unstructured knowledge text to facilitate the inquirer to perform knowledge search, and a corresponding database can be constructed for structured data text to facilitate the inquirer to perform data retrieval.

[0042] Based on this, in order to improve the efficiency of the text question and answer model in searching the inquiry text, after receiving the initial inquiry text sent by the inquiry object, the text processing system can first identify the intent of the received initial inquiry text, such as extracting multiple keywords related to the intent from the initial inquiry text, and then classifying the initial inquiry text according to these multiple keywords to determine the current inquiry intent of the inquiry object, and perform information query through the text question and answer model based on the determined inquiry intent.

[0043] Step S104: determining a question-answer database that matches the initial query text based on the query intention.

[0044] In an optional scheme of this embodiment, in order to avoid information query errors, after determining the inquiry intention of the inquiry object, the text processing system can first determine the database containing the information required by the inquiry object based on the inquiry intention, so as to obtain a question and answer database that matches the initial inquiry text.

[0045] Step S106, input the inquiry intention, the initial inquiry text and the data in the question and answer database into the text question and answer model, and use the text question and answer model to select the target text from the question and answer database.

[0046] In an optional solution of this embodiment, after determining the question-and-answer database for information query, the text processing system can use the text question-and-answer model to match the initial query text with the data in the question-and-answer database, so as to filter out the text currently needed by the inquirer from the question-and-answer database, that is, the above-mentioned target text. Considering that the simple initial query text may not reflect the actual needs of the inquirer, when selecting the above-mentioned target text from the question-and-answer database, the above-mentioned identified query intention can be further introduced, and the query intention can be used to further improve the actual needs of the inquirer contained in the initial query text, so as to improve the matching degree between the text selected from the question-and-answer database by the text question-and-answer model and the actual needs of the inquirer.

[0047] Step S108, generating a response text corresponding to the initial inquiry text based on the target text.

[0048] The text structure of the reply text is the same as the text structure of the initial inquiry text.

[0049] In an optional solution of this embodiment, after selecting the target text, the text processing model can output the target text for the inquiry object to view. In order to facilitate the user to understand the target text, the text processing system can generate a reply text with the same text structure as the initial inquiry text based on the obtained target text, and output the reply text for the user to view.

[0050] In an embodiment of the present invention, in response to receiving an initial inquiry text, the inquiry intention of the inquiry object is determined based on the initial inquiry text; based on the inquiry intention, a question and answer database matching the initial inquiry text is determined; the inquiry intention, the initial inquiry text and the data in the question and answer database are input into a text question and answer model, and a target text is selected from the question and answer database using the text question and answer model; a reply text corresponding to the initial inquiry text is generated based on the target text. By selecting a question and answer database that can meet the needs of the inquiry object according to the inquiry object's current inquiry intention, and combining the inquiry intention and the initial inquiry text at the same time, the text question and answer model is used to search the data contained in the question and answer database to obtain the corresponding target text, the time spent on searching for data irrelevant to the needs of the inquiry object can be reduced, thereby improving the efficiency of querying the initial inquiry text, and further solving the technical problem of low efficiency in information query in related technologies.

[0051] Furthermore, the inquiry intention includes: knowledge query intention, and the question and answer database includes: a knowledge base corresponding to the knowledge query intention; the inquiry intention, the initial inquiry text and the data in the question and answer database are input into a text question and answer model, and the target text is selected from the question and answer database using the text question and answer model, including: analyzing the initial inquiry text based on the knowledge query intention, and determining inquiry keywords from the initial inquiry text, wherein the inquiry keywords are used to reflect the needs of the inquiry object; adjusting the initial inquiry text based on the inquiry keywords to obtain a target inquiry text; matching the target inquiry text with multiple initial texts contained in the question and answer database to obtain a text matching result; and determining the target text from the multiple initial texts based on the text matching result.

[0052] In an optional solution of this embodiment, as described above, the inquiry intention determined from the initial inquiry text may at least include: knowledge query intention, that is, the inquiry object currently has a need to query unstructured knowledge text. Based on this, the question-answer database determined by the text processing system according to the inquiry intention may be a database for storing unstructured knowledge text, that is, the above-mentioned knowledge base corresponding to the knowledge query intention. When selecting the target text, the text processing system may first analyze the initial inquiry text according to the knowledge query intention to determine the keywords that can accurately reflect the needs of the inquiry object from the initial inquiry text, that is, the above-mentioned inquiry keywords, and then use the determined inquiry keywords to adjust the initial inquiry text to increase the weight of the needs of the inquiry object in the initial inquiry text, so as to obtain a target inquiry text that better reflects the needs of the inquiry object. For example, the text processing system may construct a new description content for the inquiry keyword and add the description content to the start area or end area of ​​the initial text keyword to obtain the above-mentioned target inquiry text, or may directly use the inquiry keyword to generate a new text description and use the new text description as the target inquiry text. After obtaining the target inquiry text, the text processing system can further match the target inquiry text with multiple initial texts contained in the question and answer database to obtain the above-mentioned text matching results. Finally, the text processing system can filter out the above-mentioned target text from multiple initial texts based on the matched text matching results. For example, the text with the highest matching degree in the text matching results can be determined as the target text to improve the efficiency of determining the target text, or one or more initial texts with a matching degree greater than a preset threshold can be used together as the above-mentioned target text to avoid the situation where only viewing a single text causes the inquiring party to be unable to obtain the desired text content.

[0053] Furthermore, the target query text is matched with multiple initial texts contained in the question and answer database to obtain a text matching result, including: obtaining at least one index vector corresponding to different initial texts; vectorizing the target query text to obtain a target text vector; matching at least one index vector with the target text vector to obtain a text matching result.

[0054] In an optional scheme of the present embodiment, when matching the target query text with multiple initial texts contained in the question and answer database, in order to improve the matching efficiency, the text processing system may first obtain at least one index vector corresponding to different initial texts, and vectorize the target query text to obtain the corresponding target text vector, and finally match the at least one index vector obtained with the determined target text vector to obtain the above-mentioned text matching result.

[0055] Furthermore, the above method also includes: performing level identification on text titles contained in different initial texts to obtain title levels corresponding to different text titles; dividing the initial text based on the title level and the text title to obtain at least one text segment; performing feature extraction on at least one text segment to obtain text segment features corresponding to at least one text segment; based on the text segment features, generating at least one index vector corresponding to different initial texts.

[0056] In an optional solution of the present embodiment, in order to avoid errors when performing vector matching, the text processing system can determine at least one index vector corresponding to different initial texts based on the text content contained in different initial texts. Based on this, the text processing system can first perform level recognition on the text titles contained in different initial texts to determine the title levels corresponding to different text tags. For example, if a text contains Title 1, and Title 2 and Title 3 are set in parallel in the content of Title 1, it can be determined that the title level corresponding to Title 1 is Level 1, and the title levels corresponding to Title 2 and Title 3 are Level 2. After determining the title levels corresponding to different text titles, the text processing system can further divide the initial text according to the title level and the text title to obtain at least one text segment corresponding to different initial texts, and then perform feature extraction on the at least one divided text segment to obtain text segment features corresponding to different text segments, such as features such as keywords, title sources, and title names contained in different text segments, and finally generate at least one index vector corresponding to different initial texts based on the extracted text segment features.

[0057] It should be noted that the content corresponding to the above-mentioned title levels is only an exemplary display. The specific title level configuration and text segment division rules can be set by the staff according to actual conditions and are not limited here.

[0058] Furthermore, based on the text segment features, at least one index vector corresponding to different initial texts is generated, including: based on the text segment features, constructing a unique character corresponding to at least one text segment, wherein the unique character is used to reflect the source information and title content of at least one text segment, and the unique characters corresponding to different text segments are different; based on the unique character, constructing a character vector corresponding to at least one text segment; and summarizing the character vectors corresponding to different text segments to obtain at least one index vector corresponding to different initial texts.

[0059] In an optional scheme of the present embodiment, when generating at least one index vector corresponding to different initial texts according to text segment features, in order to reflect the difference between different text segments and ensure that the index vectors constructed for different initial texts are different, the text processing system can first construct a unique character corresponding to the above-mentioned at least one text segment according to the extracted text segment features, so as to use the unique character to reflect the source information and title content of at least one text segment, based on which the unique characters constructed for different text segments are different. After constructing the unique character, the text processing system can further use the unique character to construct a character vector corresponding to at least one text segment, and finally summarize the character vectors corresponding to different text segments to obtain at least one index vector corresponding to different initial texts.

[0060] Furthermore, the inquiry intention includes: data retrieval intention, and the question and answer database includes: a database corresponding to the data retrieval intention; inputting the inquiry intention, the initial inquiry text and the data in the question and answer database into the text question and answer model, and selecting the target text from the question and answer database using the text question and answer model, including: obtaining the historical conversation text entered by the inquiry object within the historical time period; constructing a structural query statement corresponding to the initial inquiry text based on the data retrieval intention and the initial inquiry text; adjusting the structural query statement based on the historical conversation text to obtain the target query statement; searching the data text contained in the database based on the target query statement to obtain the target text.

[0061] In an optional scheme of this embodiment, corresponding to the aforementioned knowledge query intention, the query intention obtained by identifying the initial query text may also include: data retrieval intention, that is, the query object currently has the need to query structured data text. Based on this, the question and answer database determined by the text processing system according to the query intention may be a database for storing structured knowledge text, that is, the above-mentioned database corresponding to the data retrieval intention. When selecting the target text, the text processing system may first obtain the historical conversation text entered by the query object in the historical time period, and at the same time, according to the data retrieval intention and the initial query text, construct a structured query statement for data query, and then use the historical conversation text to adjust the structured query statement to obtain the above-mentioned target query statement, so that the text question answering model can accurately understand the data that the query object currently needs to retrieve through the target query statement, and finally use the target query statement to search the data text contained in the database, so as to obtain a target text with higher accuracy.

[0062] Furthermore, determining the inquiry intention of the inquiry object based on the initial inquiry text includes: performing semantic recognition on the initial inquiry text to obtain the initial text semantics; inputting the initial text semantics and multiple text intentions into an intention recognition model, and using the intention recognition model to determine the intention confidence corresponding to different text intentions; and determining the inquiry intention from multiple text intentions based on the intention confidence.

[0063] In an optional scheme of the present embodiment, in order to accurately determine the inquiry intention of the inquirer, the text processing system may first perform semantic recognition on the initial inquiry text to obtain the corresponding initial text semantics, and then input the initial text semantics and multiple pre-set text intentions into the intention recognition model, so as to use the intention recognition model to determine the intention confidence corresponding to different text intentions. Finally, based on the determined intention confidence, the text processing system can determine the above-mentioned inquiry intention from multiple text intentions. For example, the text processing system can directly determine the text intention with the highest intention confidence as the above-mentioned inquiry intention to improve the efficiency of determining the inquiry intention, or can display multiple text intentions with higher intention confidence to the inquirer in the operation interface, and the inquirer can actively select the current inquiry intention to improve the accuracy of determining the inquiry intention.

[0064] Furthermore, a reply text corresponding to the initial inquiry text is generated based on the target text, including: performing text recognition on the target text to obtain a text recognition result; matching the text recognition result with a plurality of sensitive words included in a sensitive word list to obtain a text matching result; adjusting the target text based on the text matching result to obtain an adjusted text; inputting the adjusted text and the initial inquiry text into a text generation model, and using the text generation model to generate a reply text.

[0065] In an optional scheme of the present embodiment, when generating a reply text based on the target text, the text processing system may first perform text recognition on the target text to obtain a corresponding text recognition result, and then match the text recognition result with multiple sensitive words included in the sensitive word list to obtain a corresponding text matching result, thereby determining whether there are sensitive words in the text recognition result that cannot be displayed to the inquirer. If not, the facial text may be kept unchanged. If so, the text processing system may adjust the target text according to the corresponding text matching result, such as replacing the sensitive words in the target text to obtain the above-mentioned adjusted text. Finally, the text processing system may input the adjusted text or the unchanged target text into the text generation model in combination with the initial inquiry text entered by the inquirer to obtain a reply text that the inquirer can understand.

[0066] It should be noted that the multiple models mentioned in the above content, such as text question and answer models, intent recognition models, text generation models, etc., can all be pre-trained large-scale language models. The specific model parameters can be set by the staff according to actual conditions and are not limited here.

[0067] For ease of understanding, Figure 2 is a structural block diagram of a text processing system according to an embodiment of the present application. Figure 2 As shown, the text processing system may include an operation interface, in which the inquirer may input an initial inquiry text. The operation interface may be divided into two parts, a "question-and-answer interface" and a "management interface". The inquirer may input an initial inquiry instruction through the question-and-answer interface or the management interface to obtain the required content.

[0068] The interface gateway can serve as a bridge between the internal and external interfaces of the system, processing and forwarding requests and data from the question-and-answer interface and the management interface, while ensuring the security and correctness of the data.

[0069] The text processing part can include modules such as "context management", "intention recognition", "system Q&A" and "data Q&A". Among them, the context management module can be used to store and manage context information in multiple rounds of conversations to ensure that the intelligent assistant can understand the continuity of the conversation and the user's intention. The intention recognition module can be used to parse the natural language input by the user, accurately determine whether the user needs to obtain system information or business data, and provide direction for subsequent processing. The system Q&A module can be used when the user's intention is to point to system consultation. This module uses RAG technology to retrieve the enterprise knowledge base and provide accurate system information or document links. The data Q&A module can be used to process the user's data query needs, convert natural language into SQL query statements through Text2SQL technology, call the database interface to execute the query and return the results.

[0070] The knowledge base management part can include modules such as "knowledge base management", "document conversion" and "document fragmentation management". Among them, the knowledge base management module can be used to store and maintain the content of the enterprise's business management and system documents, including document classification, labeling, retrieval and update. The document conversion module can be used to convert the company's existing DOC format documents into Markdown format for easy storage, retrieval and processing, and improve the readability and management efficiency of the knowledge base. The document fragmentation management module can be used to segment the converted Markdown document into multiple small fragments, and each fragment is vectorized and indexed to improve retrieval efficiency and accuracy.

[0071] Various types of enterprise data and documents that can be accessed and processed, such as "knowledge data", "rules and regulations", "announcements", "meeting minutes", "production ledgers", "operating data", "revenue costs", "task items", "production safety", "human resources" and "assessment reports", etc. These data and documents constitute the knowledge base or database basis for text queries.

[0072] Figure 3 is a schematic diagram of a text processing process according to an embodiment of the present application, such as Figure 3 As shown, after receiving the initial inquiry text, the text processing system can first perform intent recognition on the initial inquiry text to obtain the inquiry intention of the inquiry object. If the inquiry intention is a data retrieval intention, such as the retrieval of business management data, the text processing system can obtain the corresponding business management database, and build a Text2Sql query statement by calling the query tool to query the data currently required by the inquiry object from the experience management database. For example, the text processing system can combine the schema definition information of the database table (table name, field, comment, etc.) and the model for generating SQL statements to extract relevant information according to the user's query requirements, generate and execute SQL statements, and obtain query results. At the same time, the multi-round dialogue manager maintains the dialogue history, understands user needs, and continuously optimizes SQL statements.

[0073] If the inquiry intention is a knowledge query intention, such as querying corporate knowledge, the text processing system can obtain the corresponding corporate knowledge base and perform knowledge retrieval through RAG, match the initial inquiry text with the data in the enterprise's direct library, and select corporate knowledge that can meet the needs of the inquiry object from the enterprise knowledge database based on the matching results. If the text processing system does not recognize the inquiry intention of the inquiry object, it can directly feedback to the inquiry object in the operation interface that the inquiry intention is not currently recognized, and the inquiry object can choose the current inquiry intention by itself. For example, the text processing system can first load the knowledge base to vectorize the knowledge base documents and use efficient retrieval tools for rapid retrieval to find content fragments related to the user's questions. Then use the predefined sensitive word list to perform necessary desensitization processing on the retrieved knowledge fragments. Finally, the processed content is combined with the prompt word to obtain an anthropomorphic professional field answer through a large language model.

[0074] Figure 4This is a flowchart of a text conversion and slicing process according to an embodiment of the present application. Considering that most of the documents in enterprises are basically in doc format, there is no uniform standard for doc format, which increases the difficulty of LLM model parsing. Therefore, after obtaining the document to be converted, the text processing system needs to convert the doc format into a Markdown file with a standard format. The steps can be as follows:

[0075] (1) doc to html conversion: load the doc document content through an appropriate document reading tool, and run the conversion command to convert the read doc content into html format.

[0076] (2) Convert HTML to Markdown: Use appropriate tools to convert HTML documents into Markdown format, and decide whether to retain image content based on needs, thereby generating standardized Markdown text.

[0077] After completing the format conversion, in order to improve the efficiency of document retrieval and avoid the impact of document segmentation on the original logic of the text during the RAG process, it is necessary to slice the document according to a certain logic and associate the slices with indexes. The steps can be as follows:

[0078] (1) Content segmentation: Split the Markdown document into chapters and paragraphs. Install the Python markdown library to identify the chapters and paragraphs of the markdown document. First, find the lines starting with "#" to identify the titles of different levels, and use regular expressions to identify the titles in the article. The corresponding pseudo code is as follows:

[0079]

[0080]

[0081] Then check the content under the current title, and stop extracting when encountering the next title of the same level or higher level. If there is already content, save the current chapter, otherwise set a new chapter title, and different chapter contents are saved as different files.

[0082] (2) Index association: The fragmented article content is multi-level associated according to the title index, and a fragment list and a prefix list (used to record the hierarchical relationship of the title) are set. By traversing each fragment, the function extracts the title and content and generates an object containing context and source information. In the process, a unique source string is generated for each fragment, indicating which title the content comes from. The content of each fragment is associated with the upper-level title and the corresponding title content, forming a complete hierarchical structure.

[0083] Figure 5 is a schematic diagram of a document segmentation structure according to an embodiment of the present application. After the document is divided by title, a connection structure between different titles and different contents can be obtained. For example, the titles contained in the document may be a first-level title: Title 1, a second-level title: Title 21, Title 22 and Title 23, a third-level title: Title 3, and the corresponding contents may be Content 1, Content 21, Content 22, Content 23, Content 3, such as Figure 5 As shown, Title 1 can correspond to Content 1, and can include Content 21, Content 22, and Content 23; Title 21 can correspond to Content 21, and can include Content 3; Title 22 can correspond to Content 22; Title 23 can correspond to Content 23; and Title 3 can correspond to Content 3.

[0084] According to an embodiment of the present invention, a text processing device embodiment is provided. It should be noted that the device can be used to execute the above-mentioned text processing method. Figure 6 is a structural block diagram of a text processing device according to an embodiment of the present application. Figure 6 As shown, the device may include: an intention determination module 602 , a database acquisition module 604 , a text selection module 606 and a text output module 608 .

[0085] Among them, the intention determination module 602 is used to determine the inquiry intention of the inquiry object based on the initial inquiry text in response to receiving the initial inquiry text, wherein the inquiry object is used to characterize the object sending the initial inquiry text; the database acquisition module 604 is used to determine the question and answer database matching the initial inquiry text based on the inquiry intention; the text selection module 606 is used to input the inquiry intention, the initial inquiry text and the data in the question and answer database into the text question and answer model, and select the target text from the question and answer database using the text question and answer model; the text output module 608 is used to generate a reply text corresponding to the initial inquiry text based on the target text, wherein the text structure of the reply text is the same as the text structure of the initial inquiry text.

[0086] Furthermore, the inquiry intention includes: knowledge query intention, and the question and answer database includes: a knowledge base corresponding to the knowledge query intention; the text selection module is also used to: analyze the initial inquiry text based on the knowledge query intention, determine the inquiry keywords from the initial inquiry text, wherein the inquiry keywords are used to reflect the needs of the inquiry object, adjust the initial inquiry text based on the inquiry keywords, and obtain the target inquiry text; match the target inquiry text with multiple initial texts contained in the question and answer database to obtain a text matching result; and determine the target text from the multiple initial texts based on the text matching result.

[0087] Furthermore, the text selection module is also used to: obtain at least one index vector corresponding to different initial texts; vectorize the target query text to obtain a target text vector; match at least one index vector with the target text vector to obtain a text matching result.

[0088] Furthermore, the above-mentioned device also includes: a level identification module, which is used to perform level identification on text titles contained in different initial texts to obtain title levels corresponding to different text titles; a text segment division module, which is used to divide the initial text based on the title level and the text title to obtain at least one text segment; a feature extraction module, which is used to perform feature extraction on at least one text segment to obtain text segment features corresponding to at least one text segment; and a vector generation module, which is used to generate at least one index vector corresponding to different initial texts based on the text segment features.

[0089] Furthermore, the vector generation module is also used to: construct a unique character corresponding to at least one text segment based on text segment features, wherein the unique character is used to reflect the source information and title content of at least one text segment, and different text segments correspond to different unique characters; construct a character vector corresponding to at least one text segment based on the unique character; and summarize the character vectors corresponding to different text segments to obtain at least one index vector corresponding to different initial texts.

[0090] Furthermore, the inquiry intention includes: data retrieval intention, and the question and answer database includes: a database corresponding to the data retrieval intention; the text selection module is also used to: obtain the historical conversation text entered by the inquiry object within the historical time period; based on the data retrieval intention and the initial inquiry text, construct a structural query statement corresponding to the initial inquiry text; adjust the structural query statement based on the historical conversation text to obtain a target query statement; search the data text contained in the database based on the target query statement to obtain the target text.

[0091] Furthermore, the intent determination module is also used to: perform semantic recognition on the initial inquiry text to obtain the initial text semantics; input the initial text semantics and multiple text intentions into the intent recognition model, and use the intent recognition model to determine the intent confidence corresponding to different text intentions; based on the intent confidence, determine the inquiry intent from multiple text intentions.

[0092] Furthermore, the text output module is also used to: perform text recognition on the target text to obtain a text recognition result; match the text recognition result with multiple sensitive words included in the sensitive word list to obtain a text matching result; adjust the target text based on the text matching result to obtain an adjusted text; input the adjusted text and the initial query text into the text generation model, and use the text generation model to generate a reply text.

[0093] An embodiment of the present application further provides an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention when running.

[0094] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0095] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in various embodiments of the present invention when executed by a processor.

[0096] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0097] The embodiments of the present application further provide a computer program, which implements the methods in the above-mentioned embodiments of the present invention when executed by a processor.

[0098] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text processing method, characterized in that: include: In response to receiving the initial inquiry text, determining the inquiry intention of the inquiry object based on the initial inquiry text, wherein the inquiry object is used to represent the object that sent the initial inquiry text; Based on the inquiry intention, determining a question-answer database matching the initial inquiry text; Inputting the inquiry intention, the initial inquiry text and the data in the question-answer database into a text question-answer model, and selecting a target text from the question-answer database using the text question-answer model; A reply text corresponding to the initial inquiry text is generated based on the target text, wherein the text structure of the reply text is the same as the text structure of the initial inquiry text.

2. The method according to claim 1, characterized in that The inquiry intention includes: knowledge query intention, and the question-answering database includes: a knowledge base corresponding to the knowledge query intention; inputting the inquiry intention, the initial inquiry text, and the data in the question-answering database into a text question-answering model, and selecting a target text from the question-answering database using the text question-answering model, including: Analyzing the initial inquiry text based on the knowledge query intention, and determining inquiry keywords from the initial inquiry text, wherein the inquiry keywords are used to reflect the needs of the inquiry object; Adjusting the initial query text based on the query keywords to obtain a target query text; Matching the target query text with a plurality of initial texts contained in the question-answer database to obtain a text matching result; The target text is determined from the plurality of initial texts based on the text matching result.

3. The method according to claim 2, characterized in that Matching the target query text with a plurality of initial texts included in the question-answer database to obtain text matching results, including: Obtain at least one index vector corresponding to different initial texts; Performing vectorization processing on the target query text to obtain a target text vector; The at least one index vector is matched with the target text vector to obtain the text matching result.

4. The method according to claim 3, characterized in that The method further comprises: Performing level recognition on text titles contained in different initial texts to obtain the title levels corresponding to different text titles; Divide the initial text based on the title level and the text title to obtain at least one text segment; Performing feature extraction on the at least one text segment to obtain a text segment feature corresponding to the at least one text segment; Based on the text segment feature, the at least one index vector corresponding to different initial texts is generated.

5. The method according to claim 4, characterized in that Generating the at least one index vector corresponding to different initial texts based on the text segment feature includes: Based on the text segment feature, construct a unique character corresponding to the at least one text segment, wherein the unique character is used to reflect the source information and title content of the at least one text segment, and different text segments correspond to different unique characters; Based on the unique character, construct a character vector corresponding to the at least one text segment; The character vectors corresponding to different text segments are aggregated to obtain the at least one index vector corresponding to different initial texts.

6. The method according to claim 1, characterized in that The inquiry intention includes: data retrieval intention, and the question-answering database includes: a database corresponding to the data retrieval intention; inputting the inquiry intention, the initial inquiry text, and the data in the question-answering database into a text question-answering model, and selecting a target text from the question-answering database using the text question-answering model, including: Obtain the historical conversation text entered by the query object within the historical time period; Based on the data retrieval intention and the initial query text, construct a structured query statement corresponding to the initial query text; Adjusting the structured query statement based on the historical conversation text to obtain a target query statement; The data text contained in the database is searched based on the target query statement to obtain the target text.

7. The method according to any one of claims 1 to 6, characterized in that: Determining the inquiry intention of the inquiry object based on the initial inquiry text includes: Performing semantic recognition on the initial query text to obtain the semantics of the initial text; Inputting the initial text semantics and multiple text intentions into an intention recognition model, and using the intention recognition model to determine the intention confidences corresponding to different text intentions; The query intent is determined from the multiple text intents based on the intent confidence.

8. The method according to any one of claims 1 to 6, characterized in that: Generating a reply text corresponding to the initial inquiry text based on the target text includes: Performing text recognition on the target text to obtain a text recognition result; Matching the text recognition result with multiple sensitive words included in the sensitive word list to obtain a text matching result; Adjusting the target text based on the text matching result to obtain an adjusted text; The adjustment text and the initial inquiry text are input into a text generation model, and the reply text is generated using the text generation model.

9. A text processing device, characterized in that: include: an intention determination module, configured to, in response to receiving an initial inquiry text, determine an object intention of an inquiry object based on the initial inquiry text, wherein the inquiry object is used to represent an object that sends the initial inquiry text; A database acquisition module, used to acquire a question-answer database matching the initial query text based on the object intention; A text selection module, used for inputting the initial inquiry text and the question-answer database into a text question-answer model, and selecting a target reply text from the question-answer database using the text question-answer model; A text output module is used to output the target reply text.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 8 when running.

Images