Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method. The method comprises the following steps: acquiring a current question text; based on the current question text, at least one first word segmentation set is determined, the first word segmentation set comprises at least one target question segmentation word, and the target question segmentation word is a proper noun appearing in the current question text; a proper noun dictionary corresponding to the first word segmentation set is obtained, knowledge matching processing is conducted on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set, and a second word segmentation set is obtained and comprises at least one target problem word segmentation and knowledge data corresponding to the target problem word segmentation; the proper noun dictionary comprises proper nouns and knowledge data corresponding to the proper nouns; and determining a prompt text corresponding to the current question text based on the current question text and the second word segmentation set. The accuracy of the prompt text is improved, and then the output accuracy of the large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] Existing large-scale language models have made significant progress in the field of natural language processing through deep learning technology. However, these large-scale language models still face limitations in knowledge acquisition and reasoning in certain application scenarios. In order to make up for this gap, researchers have begun to explore ways to integrate large-scale language models with external knowledge bases. For example, when a language model needs to solve a problem in a specific field, it can query an external knowledge base to obtain the latest data or insights from experts in the field, thereby improving the accuracy and comprehensiveness of answering questions. However, the current mainstream solution for large-scale language model plug-in knowledge bases is to retrieve knowledge base content based on user questions, and use a semantic recall model to sort by semantic similarity or perform full-text retrieval to obtain plug-in knowledge. When relying on multiple knowledge and the knowledge differences are large, it will not be possible to retrieve and hit diverse and different knowledge at the same time, resulting in the problem of poor accuracy of the acquired knowledge, which in turn affects the accuracy of the output of large language models. Summary of the invention

[0003] The embodiment of the present invention provides a data processing method, which aims to solve the problem of poor accuracy in retrieving knowledge in a large language model plug-in knowledge base, so as to improve the output accuracy of the large language model. The corresponding knowledge data is matched for the target question segmentation in the current question text through at least one first segmentation set and a corresponding proper noun dictionary of the current question text, the accuracy of the segmentation is ensured through at least one first segmentation set, and the accuracy of the knowledge data is ensured through the proper noun dictionary corresponding to the first segmentation set, so that the current question text obtains more fine-grained and more accurate knowledge data at the segmentation level, improves the accuracy of the prompt text, and further improves the output accuracy of the large language model.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising the following steps:

[0005] Get the current question text;

[0006] Based on the current question text, at least one first segmentation set is determined, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text;

[0007] Acquire a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentations and knowledge data corresponding to the target question segmentations, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns;

[0008] Based on the current question text and the second word segmentation set, a prompt text corresponding to the current question text is determined, and the prompt text is used to request answer data corresponding to the current question text from a large language model.

[0009] Optionally, determining at least one first word segmentation set based on the current question text includes:

[0010] Acquire a proper noun list, the proper noun list including proper nouns and part-of-speech tagging data corresponding to the proper nouns;

[0011] Based on the proper noun list, the current question text is subjected to lexical analysis to obtain at least one first word segmentation set, each of which corresponds to a type of part-of-speech tagging data.

[0012] Optionally, obtaining a proper noun list includes:

[0013] Obtain the proper nouns in each vertical field;

[0014] Determining part-of-speech tagging data corresponding to the proper noun based on the vertical field to which the proper noun belongs;

[0015] The proper noun list is constructed based on the proper nouns and the part-of-speech tagging data corresponding to the proper nouns.

[0016] Optionally, the performing lexical analysis on the current question text based on the proper noun list to obtain at least one first word segmentation set includes:

[0017] Importing the proper noun list into a preset lexical analysis tool to obtain a target lexical analysis tool;

[0018] Performing lexical analysis on the current question text by using the target lexical analysis tool to obtain a plurality of target question segmentations, each target question segmentation corresponding to one part-of-speech tagging data;

[0019] The plurality of target question word segments are classified and processed according to the part-of-speech tagging data to obtain at least one first word segmentation set.

[0020] Optionally, each of the proper noun dictionaries corresponds to a type of part-of-speech tagging data, and obtaining the proper noun dictionary corresponding to the first word segmentation set includes:

[0021] Determining a correspondence between the first word segmentation set and the proper noun dictionary based on the part-of-speech tagging data corresponding to the first word segmentation set and the part-of-speech tagging data corresponding to the proper noun dictionary;

[0022] Based on the correspondence between the first word segmentation set and the proper noun dictionary, a proper noun dictionary corresponding to the first word segmentation set is determined.

[0023] Optionally, before determining the proper noun dictionary corresponding to the first word segmentation set, the method further includes:

[0024] Acquiring knowledge data corresponding to the proper noun;

[0025] Based on the proper nouns, the knowledge data corresponding to the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, the proper noun dictionary corresponding to the part-of-speech tagging data is constructed.

[0026] Optionally, performing knowledge matching processing on the first word segmentation set through a proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set includes:

[0027] Matching the target question segmentation in the first segmentation set with the proper noun in the proper noun dictionary corresponding to the first segmentation set to obtain a target proper noun corresponding to the target question segmentation;

[0028] Determine the knowledge data corresponding to the target question segmentation by using the knowledge data corresponding to the target proper noun;

[0029] Based on the target question segmentations and the knowledge data corresponding to the target question segmentations, a second segmentation set is constructed.

[0030] In a second aspect, an embodiment of the present invention further provides a data processing device, the data processing device comprising:

[0031] The first acquisition module is used to obtain the current question text;

[0032] A first processing module is used to determine at least one first segmentation set based on the current question text, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text;

[0033] A second processing module is used to obtain a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentation and knowledge data corresponding to the target question segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns;

[0034] The third processing module is used to determine the prompt text corresponding to the current question text based on the current question text and the second word segmentation set, and the prompt text is used to request the answer data corresponding to the current question text from the large language model.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method provided in the embodiment of the present invention when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the embodiment of the invention are implemented.

[0037] In an embodiment of the present invention, a current question text is obtained; based on the current question text, at least one first word segmentation set is determined, the first word segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text; a proper noun dictionary corresponding to the first word segmentation set is obtained, and the first word segmentation set is subjected to knowledge matching processing through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, the second word segmentation set includes at least one target question segmentation, and knowledge data corresponding to the target question segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns; based on the current question text and the second word segmentation set, a prompt text corresponding to the current question text is determined, and the prompt text is used to request answer data corresponding to the current question text from a large language model. By using at least one first word segmentation set of the current question text and a corresponding proper noun dictionary, the corresponding knowledge data is matched for the target question word segmentation in the current question text. By using at least one first word segmentation set, the accuracy of the word segmentation is ensured. By using a proper noun dictionary corresponding to the first word segmentation set, the accuracy of the knowledge data is ensured. This enables the current question text to obtain more fine-grained and more accurate knowledge data at the word segmentation level, thereby improving the accuracy of the prompt text and further improving the output accuracy of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0040] Figure 2 is a flow chart of an intelligent dialogue method provided by an embodiment of the present invention;

[0041] Figure 3 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;

[0042] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, Figure 1 : is a flow chart of a data processing method provided by an embodiment of the present invention, the data processing method comprising the steps of:

[0045] 101. Get the current question text.

[0046] In an embodiment of the present invention, the data processing method can be applied to an intelligent dialogue platform or a localization plug-in, and the intelligent dialogue platform can be constructed by a server or a distributed server. The intelligent dialogue platform includes a user-side interface, a data processing application, a knowledge database, a large language model or a calling interface of a large language model. The user's input text can be obtained through the user-side interface, and the input text can be processed into a prompt text of a large language model through a combination of a data processing application and a knowledge database. The prompt text is input into the large language model for processing (or the prompt text can be input into the large language model for processing through the calling interface of the large language model), and the output result of the large language model is obtained, and the output result of the large language model is returned to the user through the user-side interface, thereby completing the dialogue with the user. The localization plug-in can be directly installed on the user side, and the localization plug-in can include a user-side interface, a data processing application, and a knowledge database. The localization plug-in can only output the prompt text, which is used to submit the prompt text to the large language model. Of course, the localization plug-in can also include an interface of a large language model, and the prompt text is input into the large language model through the interface of the large language model.

[0047] The user inputs the corresponding question through the user terminal, and the user terminal uploads the question input by the user to the intelligent dialogue platform in text form, so that the intelligent dialogue platform obtains the current question text. The above-mentioned current question text may include the content of the question asked by the user in the current round during the current dialogue process. In a possible embodiment, the above-mentioned current question text may also include the historical dialogue content during the current dialogue process, and the above-mentioned historical dialogue content may include the user's historical questions and the historical answers of the intelligent dialogue platform.

[0048] 102. Determine at least one first word segmentation set based on the current question text.

[0049] In an embodiment of the present invention, after obtaining the current question text, the current question text can be segmented to obtain a segmentation result of the current question text, wherein the segmentation result includes multiple question segmentations, and the multiple question segmentations can be classified to divide the question segmentations into proper nouns and non-proper nouns, and the proper nouns are added to a set as target question segmentations to form a first segmentation set. The above segmentation processing can be implemented using existing segmentation tools, such as SnowNLP, NLTK, LangSegment, and other segmentation tools.

[0050] Furthermore, for proper nouns, they can be further classified. The types of proper nouns can be related to vertical fields. Different vertical fields correspond to different types of proper nouns, such as physics, football star, blockchain, astronomy, education, etc. Correspondingly, the above proper nouns can be proper nouns in the field of physics, football star, blockchain, astronomy, education, etc. Proper nouns of the same type can be divided into the same set to obtain at least one first participle set, and each first participle set represents a type of problem participle set.

[0051] The first word segmentation set includes at least one target question word segmentation, and the target question word segmentation is a proper noun that appears in the current question text. The number of the first word segmentation sets can be determined according to the number of types of proper nouns. If the number of types of proper nouns is one, the number of the first word segmentation sets is also one. If the number of types of proper nouns is multiple, the number of the first word segmentation sets is also multiple, that is, the first word segmentation sets correspond to the types of proper nouns one by one.

[0052] 103. Obtain a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set using the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set.

[0053] In the embodiment of the present invention, after obtaining the first word segmentation set, the proper noun dictionary corresponding to the first word segmentation set can be obtained. It can be understood that the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns. The proper noun dictionary can be a proper noun dictionary corresponding to all types of proper nouns. In this case, all types of proper nouns can share one proper noun dictionary. The proper noun dictionary can also be a proper noun dictionary corresponding to one type of proper noun. In this case, one type of proper noun can only use one corresponding proper noun dictionary.

[0054] The above-mentioned proper noun dictionary can be stored in a knowledge database, and when in use, the corresponding proper noun dictionary can be searched from the knowledge database. After obtaining the proper noun dictionary corresponding to the first word segmentation set, the target problem segmentation in the first word segmentation set can be compared with the proper nouns in the proper noun dictionary, so as to match the target problem segmentation to the corresponding knowledge data, and then form a set including the target problem segmentation and the knowledge data corresponding to the target problem segmentation, which is the second word segmentation set. When there are multiple first word segmentation sets, a first word segmentation set can obtain a second word segmentation set after knowledge matching processing, that is, a first word segmentation set can correspond to a second word segmentation set. Of course, the above-mentioned second word segmentation set can also be a large set. When there are multiple first word segmentation sets, all the sets of the first word segmentation sets after knowledge matching processing are merged to obtain a second word segmentation set including all target problem segmentations and the knowledge data corresponding to the target problem segmentations.

[0055] The second word segmentation set includes at least one target question word segmentation and knowledge data corresponding to the target question word segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns.

[0056] 104. Based on the current question text and the second word segmentation set, determine a prompt text corresponding to the current question text.

[0057] In an embodiment of the present invention, after obtaining the second word segmentation set corresponding to the current question text, the target question word segmentation in the second word segmentation set and the knowledge data corresponding to the target question word segmentation can be filled into the prompt template. At the same time, the current question text is also filled into the prompt template, thereby obtaining a prompt text including knowledge data.

[0058] In a possible embodiment, for question segmentation words of non-proper nouns, a common noun dictionary can be obtained from a knowledge database, and knowledge matching processing is performed on the question segmentation words of non-proper nouns through the common noun dictionary to obtain knowledge data corresponding to the question segmentation words of non-proper nouns, and the knowledge data corresponding to the question segmentation words of non-proper nouns are also filled in the above-mentioned prompt template, thereby obtaining a prompt text including the knowledge data.

[0059] The prompt text is used to request the large language model for answer data corresponding to the current question text. After obtaining the prompt text, the prompt text can be used as the input of the large language model (in this case, a large language model is deployed in the intelligent dialogue platform), and the prompt text can also be input into the large language model through the interface of the large language model (in this case, the large language model may not be deployed in the intelligent dialogue platform, but only the interface of the large language model).

[0060] The above prompt template looks like this:

[0061] "Refer to the following glossary for relevant knowledge:

[0062] {word:definition}

[0063] Please answer the user's question:

[0064] {question}".

[0065] For example, the current question text is "What is the relationship between quantum computing and quantum mechanics?". The target question words are "quantum computing" and "quantum mechanics". The knowledge data of the target question word "quantum computing" is: "Quantum computing is a new computing model that follows the laws of quantum mechanics to control quantum information units for computing. Unlike classical computing, quantum computing follows the laws of quantum mechanics. It is a new computing model that can break through the bottleneck of classical computing power. Quantum computers, as devices that perform quantum computing tasks, use quantum bits (qubits) as basic computing units. In quantum computing, based on the principle of quantum superposition, different states of quantum bits can be stored and processed simultaneously."; The knowledge data of the target question word "quantum mechanics" is: "Quantum mechanics (Quantum Mechanics) is a physics theory. It is a branch of physics that studies the laws of motion of microscopic particles in the material world. It mainly studies the basic theories of the structure and properties of atoms, molecules, condensed matter, as well as atomic nuclei and elementary particles. Together with the theory of relativity, it constitutes the theoretical basis of modern physics. Quantum mechanics is not only one of the basic theories of modern physics, but also widely used in disciplines such as chemistry and many modern technologies.", then the prompt text can be as follows:

[0066] "Refer to the following glossary for relevant knowledge:

[0067] Quantum computing: Quantum computing is a new computing model that follows the laws of quantum mechanics to control quantum information units for computing. Unlike classical computing, quantum computing follows the laws of quantum mechanics and is a new computing model that can break through the bottleneck of classical computing power.

[0068] Quantum Mechanics: Quantum Mechanics is a physics theory. It is a branch of physics that studies the laws of motion of microscopic particles in the material world. It mainly studies the basic theories of the structure and properties of atoms, molecules, condensed matter, as well as atomic nuclei and elementary particles. Together with the theory of relativity, it constitutes the theoretical basis of modern physics. Quantum mechanics is not only one of the basic theories of modern physics, but also widely used in disciplines such as chemistry and many modern technologies.

[0069] Please answer the user's question:

[0070] What is the relationship between quantum computing and quantum mechanics? ".

[0071] For another example, the current question text is "How are blockchain technology and smart contracts applied in the financial industry?". The target question segmentation words are "blockchain" and "smart contract". The knowledge data of the target question segmentation word "blockchain" is: "Blockchain is a decentralized distributed ledger that is block-chain stored, cannot be tampered with, is secure and reliable. It combines distributed storage, peer-to-peer transmission, consensus mechanism, cryptography and other technologies to record transactions and information through a growing data block chain (Blocks) to ensure data security and transparency."; The knowledge data of the target question segmentation word "smart contract" is: "A smart contract is a computer protocol designed to disseminate, verify or execute contracts in an information-based manner. Smart contracts allow trusted transactions without a third party, and these transactions are traceable and irreversible. The concept of smart contracts was first proposed by Nick Szabo in 1994.". Then the prompt text can be as follows:

[0072] "Refer to the following glossary for relevant knowledge:

[0073] Blockchain: Blockchain is a decentralized distributed ledger that is tamper-proof, secure and reliable. It combines distributed storage, peer-to-peer transmission, consensus mechanism, cryptography and other technologies to record transactions and information through a growing data block chain to ensure data security and transparency.

[0074] Smart Contract: A smart contract is a computer protocol designed to communicate, verify or execute contracts in an information-based manner. Smart contracts allow for trusted transactions without a third party, which are traceable and irreversible. The concept of smart contracts was first proposed by Nick Szabo in 1994.

[0075] Please answer the user's question:

[0076] How are blockchain technology and smart contracts used in the financial industry? ".

[0077] For another example, the current question text is "The new word "Mozi Sky Survey" in 2023 has Mozi's name, is it related to WFIRST?". The target question segmentation words are "Mozi Sky Survey" and "WFIRST". The knowledge data of the target question segmentation word "Mozi Sky Survey" is: "Refers to the "Large Field Survey Telescope" officially put into use on September 17, 2023. It is the only large-aperture and large-field optical time-domain survey telescope in the world so far. More than two thousand years ago, Mozi first discovered the principle that light propagates in a straight line, and conducted experiments similar to pinhole imaging. He is known as the "world's first person in optics." The name "Mozi" not only contains the scientific spirit of seeking knowledge and truth, but also embodies mankind's expectations for exploring the universe."; The knowledge data of the target question segmentation word "WFIRST" is: "WFIRST was designed by NASA and many astronomers to discover new planets and study dark energy (which may be the key to solving the mystery of the expansion of the universe). WFIRS Built on the foundation of the Kepler telescope, with a wide field of view and high resolution, WFIRST will search for new planets with the help of gravitational microlensing, which is related to Einstein's theory of relativity and allows telescopes to find planets orbiting stars thousands of light-years away from Earth (much farther than other planet-hunting techniques). But the effect only works when the gravity of a planet or star bends the light of another star, so the microlensing effect of any given planet or star is only visible for a few hours every few million years. In view of this, WFIRST will conduct long-term continuous monitoring of 100 million stars in the center of the Milky Way. If all goes according to plan, the depth of field of each photo taken by WFIRST will be 100 times that of the Hubble Telescope. ", the prompt text can be as follows:

[0078] "Refer to the following glossary for relevant knowledge:

[0079] Mozi Sky Survey: refers to the "Large Field Survey Telescope" which will be officially put into use on September 17, 2023. It is the world's only large-aperture and large-field optical time-domain survey telescope. More than 2,000 years ago, Mozi first discovered the principle that light travels in a straight line, and conducted experiments similar to pinhole imaging. He is known as the "world's first optics person". The name "Mozi" not only contains the scientific spirit of seeking knowledge and truth, but also embodies mankind's expectations for exploring the universe.

[0080] WFIRST: WFIRST was designed by NASA and many astronomers to discover new planets and study dark energy (which may be the key to solving the mystery of the expansion of the universe). WFIRST is built on the basis of the Kepler telescope, with a wide field of view and high resolution. WFIRST will use the gravitational microlensing effect to find new planets. The microlensing effect is related to Einstein's theory of relativity. It enables telescopes to find planets orbiting stars thousands of light years away from Earth (much farther than other planet-hunting techniques). But the effect only works when the gravity of a planet or star bends the light of another star, so the microlensing effect of any given planet or star can only be seen for a few hours every few million years. In view of this, WFIRST will conduct long-term continuous monitoring of 100 million stars in the center of the Milky Way. If all goes according to plan, the depth of field of each photo taken by WFIRST will be 100 times that of the Hubble telescope.

[0081] Please answer the user's question:

[0082] The new term in 2023, "Mozi Sky Survey", has the name of Mozi in it. Does it have anything to do with WFIRST? "

[0083] In an embodiment of the present invention, a current question text is obtained; based on the current question text, at least one first word segmentation set is determined, the first word segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text; a proper noun dictionary corresponding to the first word segmentation set is obtained, and the first word segmentation set is subjected to knowledge matching processing through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, the second word segmentation set includes at least one target question segmentation, and knowledge data corresponding to the target question segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns; based on the current question text and the second word segmentation set, a prompt text corresponding to the current question text is determined, and the prompt text is used to request answer data corresponding to the current question text from a large language model. By using at least one first word segmentation set of the current question text and a corresponding proper noun dictionary, the corresponding knowledge data is matched for the target question word segmentation in the current question text. By using at least one first word segmentation set, the accuracy of the word segmentation is ensured. By using a proper noun dictionary corresponding to the first word segmentation set, the accuracy of the knowledge data is ensured. This enables the current question text to obtain more fine-grained and more accurate knowledge data at the word segmentation level, thereby improving the accuracy of the prompt text and further improving the output accuracy of the large language model.

[0084] It is understandable that in the specific implementation of this application, user data, conversation data, knowledge data and other related data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data, as well as the training, deployment and calling of word segmentation tools and large language models, must comply with relevant laws, regulations and standards of relevant countries and regions.

[0085] Optionally, in the step of determining at least one first word segmentation set based on the current question text, a proper noun list can be obtained, the proper noun list including proper nouns and part-of-speech tagging data corresponding to the proper nouns; based on the proper noun list, the current question text is subjected to lexical analysis to obtain at least one first word segmentation set, each first word segmentation set corresponding to a type of part-of-speech tagging data.

[0086] In an embodiment of the present invention, the correspondence between the proper nouns and the part-of-speech tagging data is recorded in the above-mentioned proper noun list, and each proper noun has a part-of-speech tagging data corresponding thereto. The above-mentioned part-of-speech tagging data is related to the vertical field, and different vertical fields are represented by different part-of-speech tagging data. The type of the above-mentioned part-of-speech tagging data corresponds to the type of the vertical field. Therefore, the type of the proper noun can be represented by the type of the part-of-speech tagging data (the type can also be identified by a value). The above-mentioned part-of-speech tagging data can also be called a part-of-speech tag.

[0087] The above proper noun list records the correspondence between proper nouns and part-of-speech tagging data. Therefore, the current question text can be subjected to lexical analysis through the proper noun list. The above lexical analysis can be implemented by using existing word segmentation tools and part-of-speech tagging tools. The current question text is subjected to word segmentation through the word segmentation tool to obtain question segmentation. The question segmentation is tagged with part-of-speech through the part-of-speech tagging tool. The target of part-of-speech tagging is the part-of-speech tagging data in the proper noun list. The above part-of-speech tagging tool can be an existing part-of-speech tagging tool, such as ICTCLAS, HanLP, NLTK and other part-of-speech tagging tools.

[0088] Through the list of proper nouns, the current question text is lexically analyzed to obtain question segmentations with part-of-speech tagging data. The question segmentations are classified according to the part-of-speech tagging data to obtain question segmentations corresponding to different part-of-speech tagging data. The question segmentations corresponding to the same part-of-speech tagging data are divided into a set to obtain a corresponding first segmentation set, and then at least one first segmentation set can be obtained, and each first segmentation set corresponds to a type of part-of-speech tagging data.

[0089] Optionally, in the step of obtaining a list of proper nouns, proper nouns in various vertical fields may be obtained; based on the vertical fields to which the proper nouns belong, part-of-speech tagging data corresponding to the proper nouns may be determined; and based on the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, a list of proper nouns may be constructed.

[0090] In an embodiment of the present invention, keywords of various vertical fields and knowledge data corresponding to the keywords can be collected to construct a vertical field proprietary knowledge system, keywords of vertical field knowledge can be sorted out as proper nouns, and a proper noun list corresponding to the proper nouns of all vertical fields can be constructed.

[0091] Specifically, we can first build a proper noun knowledge system and define a proper noun dictionary object consisting of three elements: dict_element = [word, pos, definition]. Among them, the first element word is the proper noun vocabulary itself; the second element pos (Part of Speech) is the part-of-speech tag of the custom-annotated proper noun, that is, the part-of-speech tagging data; the third element definition is the proper noun knowledge or term explanation, that is, knowledge data. The first element word and the second element pos in the proper noun dictionary object are taken out to construct a proper noun list. Of course, in a possible embodiment, only the first element word in the proper noun dictionary object can be taken out to construct a proper noun list.

[0092] By constructing a proper noun list, the proper nouns in the current question text can be accurately obtained, thereby improving the accuracy of knowledge data.

[0093] Optionally, in the step of performing lexical analysis on the current question text based on the proper noun list to obtain at least one first word segmentation set, the proper noun list can be imported into a preset lexical analysis tool to obtain a target lexical analysis tool; performing lexical analysis on the current question text through the target lexical analysis tool to obtain multiple target question segmentations, each target question segmentation corresponding to a part-of-speech tagging data; and classifying the multiple target question segmentations according to the part-of-speech tagging data to obtain at least one first word segmentation set.

[0094] In an embodiment of the present invention, after obtaining a list of proper nouns, the list of proper nouns can be imported into a preset lexical analysis tool, so that the lexical analysis tool can perform lexical analysis on the current question text according to the list of proper nouns to obtain question segmentations corresponding to the part-of-speech tagging data.

[0095] For example, the lexical analysis tool may be a LAC lexical analysis tool, which is used to segment the current question text, and segment the question words from a large number of question words with part-of-speech tagging data: [word1:pos1], [word2:pos2], ..., [word n :pos n ]} to filter out the problem word segmentation corresponding to the specific part-of-speech tagging data to obtain the proper noun; for example, to obtain the custom physics proper noun, filter the word segmentation {[word a1 :PHY],[word b1 :PHY],...,[word k1 :PHY]}, which is a physics term, and can also filter the word participle where pos=FBS{[word a2 :FBS],[word b2 :FBS],...,[word k2 :FBS]}, which is a proper noun for football stars. a1 :PHY],[word b1 :PHY],...,[word k1 :PHY]} is a first participle set. a2 :FBS],[word b2 :FBS],...,[word k2 :FBS]} is also a first participle set.

[0096] Exemplarily, the part-of-speech list of the above LAC lexical analysis tool is shown in Table 1 below:

[0097] Table 1

[0098]

[0099] Proprietary parts of speech can be customized according to the vertical industry, as shown in Table 2 below:

[0100] Table 2

[0101] Part-of-speech tags Part of Speech Meaning PHY Physics terms EDU Education industry terms FBS Football star proper noun

[0102] By importing the proper noun list into the lexical analysis tool, the lexical analysis tool can perform part-of-speech tagging according to the proper noun list, thereby improving the accuracy of the part-of-speech tagging data of the target question segmentation.

[0103] Optionally, each proper noun dictionary corresponds to a type of part-of-speech tagging data. In the step of obtaining the proper noun dictionary corresponding to the first word segmentation set, the correspondence between the first word segmentation set and the proper noun dictionary can be determined based on the part-of-speech tagging data corresponding to the first word segmentation set and the part-of-speech tagging data corresponding to the proper noun dictionary. Based on the correspondence between the first word segmentation set and the proper noun dictionary, the proper noun dictionary corresponding to the first word segmentation set can be determined.

[0104] In the embodiment of the present invention, after the knowledge system of each vertical field is constructed, the proper noun dictionary corresponding to the knowledge system of each vertical field is stored in the knowledge database, and each proper noun dictionary has corresponding part-of-speech tagging data.

[0105] After obtaining the first word segmentation set, according to the part-of-speech tagging data of the first word segmentation set, a proper noun dictionary having the same word tagging data as the first word segmentation set is searched in the knowledge database, and the proper noun dictionary is determined as the proper noun dictionary corresponding to the first word segmentation set. It can be understood that in the knowledge database, the above-mentioned part-of-speech tagging data and the proper noun dictionary can constitute a key-value pair, with the part-of-speech tagging data as the key and the proper noun dictionary as the value, and then the proper noun dictionary corresponding to the first word segmentation set is searched by means of the key-value pair.

[0106] Specifically, in the knowledge database, if there is a proper noun dictionary with the same word annotation data as the first word segmentation set, it is determined that there is a corresponding relationship between the first word segmentation set and the proper noun dictionary, so that the proper noun dictionary is determined as the proper noun dictionary corresponding to the first word segmentation set. If there is no proper noun dictionary with the same word annotation data as the first word segmentation set, it means that it may be the first word segmentation set corresponding to a new vertical field, and the proper noun dictionary in the knowledge database can be updated.

[0107] Optionally, before the step of determining the proper noun dictionary corresponding to the first word segmentation set, knowledge data corresponding to the proper nouns can also be obtained; based on the proper nouns, the knowledge data corresponding to the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, a proper noun dictionary corresponding to the part-of-speech tagging data is constructed.

[0108] In an embodiment of the present invention, in the process of constructing the knowledge system of each vertical field, the keywords of each vertical field and the knowledge data corresponding to the keywords can be collected to construct the proprietary knowledge system of the vertical field. The keywords of the vertical field knowledge can be sorted out as proper nouns, and the proper noun dictionary object is defined to be composed of three elements dict_element = [word, pos, definition]. Among them, the first element word is the proper noun vocabulary itself; the second element pos (Part of Speech) is the part-of-speech tag of the custom-annotated proper noun, that is, the part-of-speech tagging data; the third element definition is the proper noun knowledge or term explanation, that is, knowledge data. For the same second element pos, the first element word and the third element definition in the proper noun dictionary object can be taken out to construct a proper noun dictionary. Then, the second element pos in the proper noun dictionary object is used as the part-of-speech tagging data corresponding to the proper noun dictionary. The proper noun dictionary and the corresponding part-of-speech tagging data are stored in the knowledge database. Of course, the proper noun dictionary object with the same second element pos can also be used to construct the proper noun dictionary of the corresponding vertical field.

[0109] Optionally, in the step of performing knowledge matching processing on the first word segmentation set through the dictionary of proper nouns corresponding to the first word segmentation set to obtain the second word segmentation set, the target problem segmentations in the first word segmentation set can be matched with the proper nouns in the dictionary of proper nouns corresponding to the first word segmentation set to obtain the target proper nouns corresponding to the target problem segmentations; the knowledge data corresponding to the target proper nouns can be determined to correspond to the target problem segmentations; and the second word segmentation set can be constructed based on the target problem segmentations and the knowledge data corresponding to the target problem segmentations.

[0110] In an embodiment of the present invention, after obtaining the first word segmentation set and the proper noun dictionary corresponding to the first word segmentation set, the target problem segmentation in the first word segmentation set can be matched with the proper nouns in the proper noun dictionary, so as to determine the proper noun corresponding to the target problem segmentation in the proper noun dictionary, and use the knowledge data corresponding to the proper noun corresponding to the target problem segmentation as the knowledge data of the target problem segmentation.

[0111] After obtaining the knowledge data of the target question segmentation, the target question segmentation and the corresponding knowledge data are associated to construct a second segmentation set.

[0112] For example, according to the first word segmentation set {word a ,word b ,...,word k} to the proper noun dictionary and perform a full match query based on the dictionary object word to find the proper noun dictionary object {dict_element a ,dict_element b ,...,dict_element k}, after obtaining the proper noun dictionary object corresponding to the target question word, extract the third element dict_element in the proper noun dictionary object as the knowledge data corresponding to the target question word, and associate the target question word with the corresponding knowledge data, by {[word a :definition a ],[word b :definition b ],...,[word k :definition k ]} constitute the knowledge set of proper nouns involved in this problem, that is, the second participle set.

[0113] Table reference Figure 2 , Figure 2 is a flow chart of an intelligent dialogue method provided by an embodiment of the present invention, such as Figure 2 As shown, during the conversation with the user, the user's current question is captured to obtain the question text, the question text and the list of proper nouns are input into the lexical analysis tool, the question text is segmented and extracted through the lexical analysis tool to obtain the proper nouns corresponding to the question text, the proper nouns are subjected to knowledge extraction in the proper noun dictionary to obtain the proper noun knowledge, the proper nouns and the proper noun knowledge are used to construct a prompt sentence in combination with the current question and historical question and answer data, the constructed prompt sentence is input into the large language model (LLM), the LLM question and answer reasoning is performed, and the answer to the question corresponding to the current question is output.

[0114] When a large language model is performing question-answering, it enables the large language model to simultaneously obtain diverse and widely different external knowledge for a question, thereby improving the accuracy and comprehensiveness of the large language model's question answers. By constructing a proprietary knowledge system in a vertical field, and combing the keywords of the vertical field knowledge into proper nouns, and then constructing a word segmentation lexicon for proper nouns by customizing the part-of-speech tags of proper nouns, the question text is analyzed using a lexical analysis tool or a part-of-speech tagging tool combined with the word segmentation lexicon for proper nouns, and the required diverse and widely different external knowledge points involved in the question text are identified; then, through the constructed proprietary knowledge system, the corresponding external knowledge of multiple knowledge points is accurately obtained, and together with the question itself, a prompt sentence is constructed as input to the large language model for question-answering reasoning, so that the large language model can simultaneously obtain diverse and widely different knowledge for a question, thereby improving the accuracy and comprehensiveness of the large language model's question answers.

[0115] By managing proprietary knowledge in vertical fields, and then customizing lexical analysis tools or part-of-speech tagging tools to identify the diverse and different external knowledge points required in the question text, combining the external knowledge of the knowledge points enables the large language model to simultaneously obtain diverse and different knowledge for a question, thereby improving the accuracy and comprehensiveness of the large language model's question answers.

[0116] like Figure 3 As shown, an embodiment of the present invention provides a data processing device, the data processing device comprising:

[0117] The first acquisition module 301 is used to acquire the current question text;

[0118] A first processing module 302 is used to determine at least one first segmentation set based on the current question text, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text;

[0119] The second processing module 303 is used to obtain a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentation and knowledge data corresponding to the target question segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns;

[0120] The third processing module 304 is used to determine the prompt text corresponding to the current question text based on the current question text and the second word segmentation set, and the prompt text is used to request the large language model for answer data corresponding to the current question text.

[0121] Optionally, the first processing module 302 is also used to obtain a list of proper nouns, which includes proper nouns and part-of-speech tagging data corresponding to the proper nouns; based on the list of proper nouns, the current question text is subjected to lexical analysis to obtain at least one first word segmentation set, each of which corresponds to a type of part-of-speech tagging data.

[0122] Optionally, the first processing module 302 is also used to obtain proper nouns in each vertical field; based on the vertical field to which the proper nouns belong, determine the part-of-speech tagging data corresponding to the proper nouns; based on the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, construct the proper noun list.

[0123] Optionally, the first processing module 302 is also used to import the proper noun list into a preset lexical analysis tool to obtain a target lexical analysis tool; perform lexical analysis on the current question text through the target lexical analysis tool to obtain multiple target question participles, each target question participle corresponding to one part-of-speech tagging data; classify the multiple target question participles according to the part-of-speech tagging data to obtain at least one first participle set.

[0124] Optionally, each of the proper noun dictionaries corresponds to a type of part-of-speech tagging data, and the second processing module 303 is further used to determine the correspondence between the first word segmentation set and the proper noun dictionary based on the part-of-speech tagging data corresponding to the first word segmentation set and the part-of-speech tagging data corresponding to the proper noun dictionary; based on the correspondence between the first word segmentation set and the proper noun dictionary, determine the proper noun dictionary corresponding to the first word segmentation set.

[0125] Optionally, the device further comprises:

[0126] A second acquisition module is used to acquire knowledge data corresponding to the proper noun;

[0127] A construction module is used to construct the proper noun dictionary corresponding to the part-of-speech tagging data based on the proper noun, the knowledge data corresponding to the proper noun and the part-of-speech tagging data corresponding to the proper noun.

[0128] Optionally, the second processing module 303 is also used to match the target problem segmentation in the first segmentation set with the proper nouns in the proper noun dictionary corresponding to the first segmentation set to obtain the target proper noun corresponding to the target problem segmentation; determine the knowledge data corresponding to the target proper noun from the knowledge data corresponding to the target problem segmentation; and construct a second segmentation set based on the target problem segmentation and the knowledge data corresponding to the target problem segmentation.

[0129] like Figure 4 As shown, an embodiment of the present invention further provides an electronic device, including a processor, and the processor can execute any one of the above data processing methods.

[0130] Specifically, it includes a processor 401 and a memory 402, and a computer program for executing the data processing method which is stored in the memory 402 and can be run on the processor 401, wherein:

[0131] The processor 401 runs the computer program of the data processing method stored in the memory 402 to perform the following steps:

[0132] Get the current question text;

[0133] Based on the current question text, at least one first segmentation set is determined, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text;

[0134] Acquire a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentations and knowledge data corresponding to the target question segmentations, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns;

[0135] Based on the current question text and the second word segmentation set, a prompt text corresponding to the current question text is determined, and the prompt text is used to request answer data corresponding to the current question text from a large language model.

[0136] Optionally, the determining of at least one first word segmentation set based on the current question text performed by the processor 401 includes:

[0137] Acquire a proper noun list, the proper noun list including proper nouns and part-of-speech tagging data corresponding to the proper nouns;

[0138] Based on the proper noun list, the current question text is subjected to lexical analysis to obtain at least one first word segmentation set, each of which corresponds to a type of part-of-speech tagging data.

[0139] Optionally, the acquiring of the proper noun list performed by the processor 401 includes:

[0140] Obtain the proper nouns in each vertical field;

[0141] Determining part-of-speech tagging data corresponding to the proper noun based on the vertical field to which the proper noun belongs;

[0142] The proper noun list is constructed based on the proper nouns and the part-of-speech tagging data corresponding to the proper nouns.

[0143] Optionally, the processor 401 performs lexical analysis on the current question text based on the proper noun list to obtain at least one first word segmentation set, including:

[0144] Import the proper noun list into the preset lexical analysis tool to obtain the target lexical analysis tool;

[0145] Performing lexical analysis on the current question text by using the target lexical analysis tool to obtain a plurality of target question segmentations, each target question segmentation corresponding to one part-of-speech tagging data;

[0146] The plurality of target question word segments are classified and processed according to the part-of-speech tagging data to obtain at least one first word segmentation set.

[0147] Optionally, each of the proper noun dictionaries corresponds to a type of part-of-speech tagging data, and the obtaining of the proper noun dictionary corresponding to the first word segmentation set performed by the processor 401 includes:

[0148] Determining a correspondence between the first word segmentation set and the proper noun dictionary based on the part-of-speech tagging data corresponding to the first word segmentation set and the part-of-speech tagging data corresponding to the proper noun dictionary;

[0149] Based on the correspondence between the first word segmentation set and the proper noun dictionary, a proper noun dictionary corresponding to the first word segmentation set is determined.

[0150] Optionally, before determining the proper noun dictionary corresponding to the first word segmentation set, the method executed by the processor 401 further includes:

[0151] Acquiring knowledge data corresponding to the proper noun;

[0152] Based on the proper nouns, the knowledge data corresponding to the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, the proper noun dictionary corresponding to the part-of-speech tagging data is constructed.

[0153] Optionally, the processor 401 performs knowledge matching processing on the first word segmentation set by using a proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, including:

[0154] Matching the target question segmentation in the first segmentation set with the proper noun in the proper noun dictionary corresponding to the first segmentation set to obtain a target proper noun corresponding to the target question segmentation;

[0155] Determine the knowledge data corresponding to the target question segmentation by using the knowledge data corresponding to the target proper noun;

[0156] Based on the target question segmentations and the knowledge data corresponding to the target question segmentations, a second segmentation set is constructed.

[0157] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the data processing method or the application-side data processing method provided by the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0159] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises the following steps: Get the current question text; Based on the current question text, at least one first segmentation set is determined, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text; Acquire a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentations and knowledge data corresponding to the target question segmentations, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns; Based on the current question text and the second word segmentation set, a prompt text corresponding to the current question text is determined, and the prompt text is used to request answer data corresponding to the current question text from a large language model.

2. The data processing method according to claim 1, characterized in that: The step of determining at least one first word segmentation set based on the current question text includes: Acquire a proper noun list, the proper noun list including proper nouns and part-of-speech tagging data corresponding to the proper nouns; Based on the proper noun list, the current question text is subjected to lexical analysis to obtain at least one first word segmentation set, each of which corresponds to a type of part-of-speech tagging data.

3. The data processing method according to claim 2, characterized in that: The obtaining of a proper noun list includes: Obtain the proper nouns in each vertical field; Determining part-of-speech tagging data corresponding to the proper noun based on the vertical field to which the proper noun belongs; The proper noun list is constructed based on the proper nouns and the part-of-speech tagging data corresponding to the proper nouns.

4. The data processing method according to claim 2, characterized in that: The step of performing lexical analysis on the current question text based on the proper noun list to obtain at least one first word segmentation set includes: Importing the proper noun list into a preset lexical analysis tool to obtain a target lexical analysis tool; Performing lexical analysis on the current question text by using the target lexical analysis tool to obtain a plurality of target question segmentations, each target question segmentation corresponding to one part-of-speech tagging data; The plurality of target question word segments are classified and processed according to the part-of-speech tagging data to obtain at least one first word segmentation set.

5. The data processing method according to claim 2, characterized in that: Each of the proper noun dictionaries corresponds to a type of part-of-speech tagging data, and obtaining the proper noun dictionary corresponding to the first word segmentation set includes: Determining a correspondence between the first word segmentation set and the proper noun dictionary based on the part-of-speech tagging data corresponding to the first word segmentation set and the part-of-speech tagging data corresponding to the proper noun dictionary; Based on the correspondence between the first word segmentation set and the proper noun dictionary, a proper noun dictionary corresponding to the first word segmentation set is determined.

6. The data processing method according to claim 3, characterized in that: Before determining the proper noun dictionary corresponding to the first word segmentation set, the method further includes: Acquiring knowledge data corresponding to the proper noun; Based on the proper nouns, the knowledge data corresponding to the proper nouns and the part-of-speech tagging data corresponding to the proper nouns, the proper noun dictionary corresponding to the part-of-speech tagging data is constructed.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The step of performing knowledge matching processing on the first word segmentation set by using a proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set includes: Matching the target question segmentation in the first segmentation set with the proper noun in the proper noun dictionary corresponding to the first segmentation set to obtain a target proper noun corresponding to the target question segmentation; Determine the knowledge data corresponding to the target question segmentation word by using the knowledge data corresponding to the target proper noun; Based on the target question segmentations and the knowledge data corresponding to the target question segmentations, a second segmentation set is constructed.

8. A data processing device, characterized in that: The data processing device comprises: The first acquisition module is used to obtain the current question text; A first processing module is used to determine at least one first segmentation set based on the current question text, wherein the first segmentation set includes at least one target question segmentation, and the target question segmentation is a proper noun that appears in the current question text; A second processing module is used to obtain a proper noun dictionary corresponding to the first word segmentation set, and perform knowledge matching processing on the first word segmentation set through the proper noun dictionary corresponding to the first word segmentation set to obtain a second word segmentation set, wherein the second word segmentation set includes at least one of the target question segmentation and knowledge data corresponding to the target question segmentation, and the proper noun dictionary includes proper nouns and knowledge data corresponding to the proper nouns; The third processing module is used to determine the prompt text corresponding to the current question text based on the current question text and the second word segmentation set, and the prompt text is used to request the answer data corresponding to the current question text from the large language model.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data processing method according to any one of claims 1 to 7 are implemented.

Images