Question answering method and device based on enhanced search, equipment and medium
By performing character-level and semantic-level matching filtering in the database, multilingual knowledge points are obtained and unified, cross-language document search is facilitated, and problems of low matching accuracy and high computational volume in cross-language document search are solved, and Q&A is improved Response speed and accuracy of the big model.
Patent Information
- Application Number
- CN202411312623.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-06
AI Technical Summary
During the document search process of cross-language documents, it leads to problems with low matching accuracy and high computational cost. Especially in the big model question and answer scenario, documents from different languages are difficult to match effectively.
By performing character-level and semantic-level matching filtering in the database, we can obtain the initial screening and fine-screen material knowledge points, and obtain the bound material slices based on these knowledge points, generate prompt information and enter the Q&A big model to generate answers, and use standard languages Unify multilingual knowledge points, facilitate quick and accurate matching.
It improves the accuracy of matching the questions to be answered and the knowledge points of the material, reduces the amount of calculations in the answer generation process, and improves the response speed and accuracy of the question-and-answer model to answer questions.
Smart Images

Figure CN119088943A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence, intelligent search and large model technology. Background Art
[0002] In a question-and-answer scenario where a large model is used to generate answers to user questions, it is often necessary to search for document materials associated with the question so that they can be input into the large model to generate answers. Since the document materials cover a wide range of areas, it is inevitable that documents in different languages will appear.
[0003] When searching for documents across languages and then inputting them into a large model to generate answers, the existence of documents in different languages will lead to problems such as low matching accuracy and high computational complexity. Summary of the invention
[0004] The present disclosure provides a question-answering method, apparatus, device, and medium based on enhanced search.
[0005] According to one aspect of the present disclosure, a question-answering method based on enhanced search is provided, the method comprising:
[0006] Get questions to be answered;
[0007] Match the questions to be answered with the material knowledge points stored offline in the database at the character level to obtain the preliminary screening material knowledge points;
[0008] Perform semantic matching screening on the questions to be answered in the primary screening material knowledge points to obtain the refined screening material knowledge points;
[0009] Obtain bound material slices from the database based on carefully screened material knowledge points;
[0010] Generate prompt information based on material slices and questions to be answered;
[0011] Input the prompt information into the question-answering model for processing to generate the answer;
[0012] The database stores bound material knowledge points and material slices offline; each material slice has at least two languages; the material knowledge points are in a standard language, and the material knowledge points in the standard language are obtained by translating material slices in a non-standard language.
[0013] According to another aspect of the present disclosure, a question-answering device based on enhanced search is provided, the device comprising:
[0014] A module for obtaining questions to be answered, used to obtain questions to be answered;
[0015] The initial screening material knowledge point acquisition module is used to match and screen the questions to be answered with the material knowledge points stored offline in the database at the character level to obtain the initial screening material knowledge points;
[0016] A module for obtaining knowledge points of fine screening materials is used to match and screen the questions to be answered at the semantic level among the knowledge points of the preliminary screening materials to obtain the knowledge points of fine screening materials;
[0017] The material slice acquisition module is used to obtain bound material slices from the database according to the carefully screened material knowledge points;
[0018] A prompt information generation module is used to generate prompt information according to material slices and questions to be answered;
[0019] The answer generation module is used to input the prompt information into the question-answering model for processing to generate the answer;
[0020] The database stores bound material knowledge points and material slices offline; each material slice has at least two languages; the material knowledge points are in a standard language, and the material knowledge points in the standard language are obtained by translating material slices in a non-standard language.
[0021] According to another aspect of the present disclosure, there is also provided an electronic device, including:
[0022] at least one processor; and
[0023] a memory communicatively connected to at least one processor; wherein,
[0024] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute any one of the enhanced search-based question-answering methods provided in the embodiments of the present disclosure.
[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the enhanced search-based question-answering methods provided by the embodiments of the present disclosure.
[0026] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, it implements any one of the enhanced search-based question-answering methods provided by the embodiments of the present disclosure.
[0027] According to the technology disclosed in the present invention, it is possible to translate and process materials in multiple languages in the offline stage, standardize the languages of knowledge points, and facilitate fast and accurate matching of knowledge points in the same language during online question and answer processing, thereby improving the accuracy of matching the questions to be answered with the knowledge points in the materials and reducing the amount of calculation in the answer generation process.
[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0030] Figure 1 is a schematic diagram of a question-answering method based on enhanced search provided according to an embodiment of the present disclosure;
[0031] Figure 2 is a schematic diagram of another question-answering method based on enhanced search provided according to an embodiment of the present disclosure;
[0032] Figure 3A is a schematic diagram of a question-answering method based on enhanced search provided according to an embodiment of the present disclosure;
[0033] Figure 3B is another question-answering flow chart based on enhanced search provided according to an embodiment of the present disclosure;
[0034] Figure 3C is a schematic diagram of processing material slices according to an embodiment of the present disclosure;
[0035] Figure 4 is a structural diagram of a question-answering device based on enhanced search provided according to an embodiment of the present disclosure;
[0036] Figure 5 It is a block diagram of an electronic device used to implement the question-answering method based on enhanced search according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0038] Figure 1It is a schematic diagram of a question-answering method based on enhanced search provided according to an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the situation of cross-language question-answering based on a large model, that is, the application of retrieval-augmented generation technology in question-answering scenarios. In related RAG technologies, it is necessary to search in the material library based on input information to obtain materials that match the input information, match the materials and the input information, and input them into the large model together to generate the required results. In the question-answering scenario, the input information is usually a question raised by the user, and the material is the content that can be used to answer the question, such as various literature materials. For text questions and materials, the matched materials and questions will be input into a large language model (LLM). The large language model can perform semantic analysis on the questions and materials and generate answers. The method can be executed by a question-answering device based on enhanced search, which can be implemented in hardware and / or software and can be configured in an electronic device. Reference Figure 1 , the method specifically includes the following:
[0039] S101. Obtain questions to be answered.
[0040] Among them, unanswered questions refer to user questions that need to be answered.
[0041] Specifically, the questions to be answered may be obtained through a user interaction interface.
[0042] S102, performing character-level matching screening on the question to be answered and the material knowledge points stored offline in the database to obtain preliminary screening material knowledge points.
[0043] Among them, the material knowledge points can be sentences or words that can represent independent semantics in the material. It should be noted that the language of the material knowledge points is the standard language. The standard language refers to the language specified in a specific question-and-answer scenario, which can be determined based on the language of a large number of questions to be answered. For example, if the language of a large number of questions to be answered is Chinese, the standard language is Chinese. For example, if the language of a large number of questions to be answered is English, the standard language is English. The standard language of the database can be set. Usually, one database has one standard language. Different databases can also be set to correspond to different standard languages to support questions to be answered in different languages. The preliminary screening material knowledge points refer to the knowledge points obtained after the material knowledge points are matched and screened at the character level; optionally, the preliminary screening material knowledge points are all material knowledge points in the standard language, including material knowledge points obtained from materials in the standard language and / or material knowledge points extracted from materials in non-standard languages and translated.
[0044] Specifically, based on a preset character matching algorithm, the question to be answered is matched at the character level with the material knowledge points stored offline in the database to obtain a character matching degree, and the material knowledge points with a character matching degree greater than a matching degree threshold are used as preliminary screening material knowledge points. Among them, the preset character matching algorithm can be pre-set according to actual business needs. For example, the preset character matching algorithm can be a KMP algorithm (The Knuth-Morris-Pratt Algorithm), which is not specifically limited in the embodiments of the present disclosure. The matching degree threshold refers to a numerical value used to define the degree of matching between the question to be answered and the matching material knowledge point, which can be determined through a large number of experiments or pre-set based on the experience of those skilled in the art. The embodiments of the present disclosure do not specifically limit it.
[0045] It is understandable that by matching the questions to be answered with the material knowledge points stored offline in the database at the character level, material knowledge points that are completely irrelevant to the questions to be answered can be filtered out, thereby reducing the amount of calculation in the answer generation process and improving the response speed of the question-answering model to the questions to be answered.
[0046] S103, performing semantic level matching screening on the questions to be answered in the primary screening material knowledge points to obtain the refined screening material knowledge points.
[0047] Among them, the refined screening material knowledge points refer to the knowledge points remaining after the initial screening material knowledge points are screened at the semantic level through matching.
[0048] Semantic matching can be achieved in one or more ways, and the semantic matching results can be comprehensively screened, such as semantic search or semantic vector model matching.
[0049] Specifically, based on the multilingual semantic matching model, the questions to be answered and the knowledge points of each preliminary screening material are vectorized to obtain a question vector and multiple preliminary screening knowledge point vectors; the semantic similarity between the question vector and each preliminary screening knowledge point vector is calculated respectively, and the preliminary screening material knowledge points corresponding to the preliminary screening knowledge point vectors whose semantic similarity is greater than the semantic similarity threshold are used as fine screening material knowledge points. It should be noted that the preliminary screening material knowledge points correspond to the preliminary screening knowledge point vectors one by one. Among them, the multilingual semantic matching model can be pre-set according to actual business needs. For example, the multilingual semantic matching model can be an XLM model (cross-lingual pre-training model, Cross-lingual Language Model Pretraining), an XLM-R model (large-scale unsupervised cross-lingual representation model, Large-scale Unsupervised Cross-lingual Representation) or an m-BERT model (cross-lingual model, Multi-lingual BERT). Among them, BERT (Bidirectional Encoder Representations from Transformers) is a bidirectional encoder representation method, also known as a bidirectional transformer model, which is a pre-trained language model based on the Transformer architecture.
[0050] It can be understood that after the question to be answered is matched and screened at the character level with the material knowledge points stored offline in the database to obtain the preliminary screened material knowledge points, the question to be answered is again matched and screened at the semantic level among the preliminary screened material knowledge points, taking into account the user intention behind the question to be answered and the semantic meaning behind the material knowledge points, so that the obtained refined material knowledge points are more accurate, so that the knowledge points most relevant to the question to be answered can be retrieved from the database.
[0051] S104, obtaining bound material slices from a database according to the carefully screened material knowledge points.
[0052] It should be noted that the database stores bound material knowledge points and material slices offline. One fine-screening material knowledge point is bound to one material slice, and one material slice can be bound to multiple fine-screening material knowledge points.
[0053] Specifically, for each fine screening material knowledge point, the material slice bound to the fine screening material knowledge point is obtained from the database using the knowledge point identifier of the fine screening material knowledge point as an index. Similarly, the material slice bound to each fine screening material knowledge point can be obtained.
[0054] S105: Generate prompt information based on the material slices and the questions to be answered.
[0055] The prompt information is generated together with the material slices and the questions to be answered, which can be parsed and processed by the big model when generating answers. The specific format of the prompt information can be set according to the input format required by the big model.
[0056] Specifically, a prompt information template is obtained, the material slice is added to the material area of the prompt information template, and the question to be answered is added to the question area of the prompt information template, thereby obtaining the prompt information. The prompt information template can be pre-set according to actual business needs. For example, the prompt information template can be as follows:
[0057] The following information is known:
[0058] Material slice 1
[0059] Material slice 2
[0060] …
[0061] Material slice n
[0062] Please interpret the following questions:
[0063] Questions to be answered
[0064] The above “Know the following information” and “Please answer the following questions” can be used as template statements of the prompt information template to limit each content area. Of course, different large models can also have different template settings. For example, if the material slices are in different languages, the material areas of different languages can also be further distinguished in the material area.
[0065] It is understandable that adding material slices and questions to be answered to the corresponding areas of the prompt information template can quickly generate prompt information, thereby improving the efficiency of the Q&A model in responding to questions to be answered. At the same time, the use of material slices and questions to be answered in the Q&A model has been distinguished in advance in the prompt information template, which helps the Q&A model to answer questions to be answered, thereby improving the accuracy of the Q&A model's answers.
[0066] S106: Input the prompt information into the question-answering model for processing to generate an answer.
[0067] Specifically, the question-answering big model refers to a question-answering big model that has been trained in advance according to actual business needs. It can usually be a large language model, which is determined through pre-training and business fine-tuning training.
[0068] In the disclosed embodiment, the database stores bound material knowledge points and material slices offline; the languages of each material slice include at least two; the language of the material knowledge points is a standard language, and the material knowledge points in the standard language are obtained by translating the material slices in the non-standard language.
[0069] It should be noted that some knowledge points of materials in standard languages are obtained by translating material slices in non-standard languages. Among them, non-standard languages refer to languages other than standard languages. It should be noted that the languages of material slices include at least two. For example, the standard language is Chinese, and non-standard languages include: English, French, Japanese, Chinese and any other non-Chinese languages.
[0070] In an optional embodiment, translating material slices in a non-standard language to obtain material knowledge points in a standard language may be: based on a text segmentation method, processing the material slices in a non-standard language into one or more material knowledge points in a non-standard language; wherein the text segmentation method may be pre-set according to actual business needs, for example, the text segmentation method may be a sentence segmentation method or a paragraph segmentation method; based on a first language recognition model, identifying the language to which the material knowledge points in a non-standard language belong, and using a dedicated translator corresponding to the language to translate the material knowledge points in a non-standard language into material knowledge points in a standard language. wherein the first language recognition model may be obtained by training the model using sample knowledge points corresponding to the language.
[0071] It can be understood that converting material knowledge points in non-standard languages into material knowledge points in standard languages and storing them in the database enriches the material knowledge points in the database, improves the matching success rate between the questions to be answered and the material knowledge points in the database, and facilitates more comprehensive retrieval of material knowledge points related to the questions to be answered.
[0072] The disclosed embodiment scheme obtains the question to be answered; matches and screens the question to be answered with the material knowledge points stored offline in the database at the character level to obtain the preliminary screening material knowledge points; matches and screens the question to be answered at the semantic level in the preliminary screening material knowledge points to obtain the refined screening material knowledge points; obtains the bound material slices from the database according to the refined screening material knowledge points; generates prompt information according to the material slices and the question to be answered; inputs the prompt information into the question-answering model for processing to generate answers; wherein, the database stores the bound material knowledge points and material slices offline; the languages of each material slice include at least two; the language of the material knowledge points is the standard language, and the material knowledge points of the standard language are obtained by translating the material slices of the non-standard language. The above technical scheme directly stores the material knowledge points of the standard language offline in the database, unifies the language of the question to be answered and the material knowledge points, and facilitates the retrieval of more knowledge points related to the question to be answered, thereby improving the accuracy of the matching between the question to be answered and the material knowledge points. At the same time, the questions to be answered and the material knowledge points stored offline in the database are first matched and screened at the character level, and then at the semantic level, filtering out the material knowledge points that are completely irrelevant to the questions to be answered, reducing the amount of calculation in the answer generation process, and improving the response speed of the large question model to the questions to be answered. In addition, the knowledge points most relevant to the questions to be answered can be retrieved from the database.
[0073] Figure 2 This is a schematic diagram of another question-answering method based on enhanced search provided in accordance with an embodiment of the present disclosure. This embodiment is an optional solution proposed on the basis of the above embodiment, which further optimizes "matching and screening the questions to be answered at the semantic level in the preliminary screening material knowledge points to obtain the refined screening material knowledge points". It should be noted that for the parts not described in detail in the embodiment of the present disclosure, reference can be made to the relevant descriptions of other embodiments. Figure 2 The question-answering method based on enhanced search provided in this embodiment includes:
[0074] S201. Obtain questions to be answered.
[0075] S202, performing character-level matching screening on the question to be answered and the material knowledge points stored offline in the database to obtain preliminary screening material knowledge points.
[0076] S203: If the initially screened material knowledge points are material knowledge points obtained through translation, the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the initially screened material knowledge points are determined from the database.
[0077] Among them, non-standard language material knowledge points refer to material knowledge points in a non-standard language, and correspondingly, standard language material knowledge points refer to material knowledge points in a standard language.
[0078] Specifically, if the initial screening material knowledge points are material knowledge points obtained through translation, the knowledge point identifiers of the initial screening material knowledge points are used as indexes to obtain the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the initial screening material knowledge points from the database.
[0079] Exemplarily, if the standard language is Chinese, the initial screening material knowledge point A is a translated material knowledge point, and the knowledge point identifier of the initial screening material knowledge point A is knowledge point identifier 1, then using knowledge point identifier 1 as an index, obtain the non-standard language material knowledge point corresponding to the initial screening material knowledge point A from the database: "Lung organoids can be divided into airway and alveolar organoids[13–15]", and the translated standard language material knowledge point corresponding to the initial screening material knowledge point A: "Lung organoids can be divided into airway organoids and alveolar organoids[13–15]".
[0080] It can be understood that, when the preliminary screening material knowledge points are material knowledge points obtained through translation, the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the preliminary screening material knowledge points are determined from the database, covering the expression methods of the preliminary screening material knowledge points in different languages, enhancing the comprehensiveness of the preliminary screening material knowledge points, and facilitating subsequent cross-language question and answer.
[0081] S204. Integrate non-standard language material knowledge points and standard language material knowledge points to form multi-language material knowledge points.
[0082] Among them, multilingual material knowledge points refer to knowledge points that cover the initial screening material knowledge points expressed in multiple languages.
[0083] Specifically, the non-standard language material knowledge points corresponding to the initial screening material knowledge points can be spliced with the standard language material knowledge points corresponding to the initial screening material knowledge points in a certain splicing order, thereby obtaining multi-language material knowledge points. It should be noted that the embodiment of the present disclosure does not specifically limit the splicing order.
[0084] To give a specific example, if the non-standard language material knowledge point corresponding to the initial screening material knowledge point 1 is "Airway organoids originate from basal cells[8,16], which proliferate and differentiate to ciliated, goblet, and club cells in cyst-like structures.", and the standard language material knowledge point corresponding to the initial screening material knowledge point 1 is "Airway organoids originate from basal cells[8,16], which proliferate and differentiate into ciliated cells, goblet cells, and club cells in cyst-like structures.", then the non-standard language material knowledge point and the standard language material knowledge point corresponding to the initial screening material knowledge point 1 can be spliced in the order of non-standard language material knowledge points first and standard language material knowledge points later, so as to obtain the multi-language material knowledge point corresponding to the initial screening material knowledge point 1, namely, "Airway organoids originate from basal cells[8,16], which proliferate and differentiate to ciliated, goblet, and club cells in Airway organoids originate from basal cells[8,16], which proliferate and differentiate into ciliated cells, goblet cells, and club cells in cyst-like structures.
[0085] It is understandable that the fusion of non-standard language material knowledge points and standard language material knowledge points effectively solves the errors in grammatical structure, grammatical order or semantics of material knowledge points caused by inaccurate translation models.
[0086] S205, input the multilingual material knowledge points and the questions to be answered into the semantic vector matching model to convert them into semantic vectors and perform vector matching.
[0087] Among them, the semantic vector matching model is the m-BERT model, which is trained using multilingual corpus samples.
[0088] Specifically, the multilingual material knowledge points and the questions to be answered are input into the m-BERT model, and the multilingual material knowledge points are converted into multilingual material knowledge point vectors through the m-BERT model, and the questions to be answered are converted into question vectors, and the question vectors are matched with the multilingual material knowledge point vectors. The multilingual material knowledge points include different language expressions of the same knowledge point, so the semantic content represented after conversion to vectors is richer and more comprehensive, and when matched with the question vectors to be answered, the matching results can be more accurate.
[0089] S206. Screening is performed according to the vector matching results to determine the knowledge points of the refined screening materials.
[0090] Specifically, the multilingual material knowledge points whose vector matching results are successfully matched are used as fine-screened material knowledge points.
[0091] S207, obtaining bound material slices from a database according to the carefully screened material knowledge points.
[0092] In an optional embodiment, in order to further improve the retrieval accuracy of enhanced retrieval, before obtaining bound material slices from the database based on the refined material knowledge points, it also includes: inputting each refined material knowledge point and the question to be answered into a vector sorting model to convert them into semantic vectors respectively, and sorting them according to the similarity of the semantic vectors; and screening each refined material knowledge point according to the sorting results and sorting filter conditions.
[0093] Among them, the sorting and filtering conditions can be pre-set according to actual business needs or according to experimental experience, and the present disclosure does not make specific limitations on them.
[0094] Specifically, each refined material knowledge point and the question to be answered are input into the vector sorting model, and each refined material knowledge point is converted into a first semantic vector through the vector sorting model, and the question to be answered is converted into a second semantic vector; the similarity between the second semantic vector and each first semantic vector is calculated, and the refined material knowledge points are sorted according to the similarity to obtain the sorting result; the refined material knowledge points that meet the sorting filter conditions in the sorting result are deleted to obtain the filtered refined material knowledge points.
[0095] S208: Generate prompt information based on the material slices and the questions to be answered.
[0096] S209: Input the prompt information into the question-answering model for processing to generate an answer.
[0097] The disclosed embodiment obtains questions to be answered; matches and screens the questions to be answered with material knowledge points stored offline in a database at the character level to obtain preliminary screening material knowledge points; if the preliminary screening material knowledge points are translated material knowledge points, the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the preliminary screening material knowledge points are determined from the database; the non-standard language material knowledge points and the standard language material knowledge points are integrated to form multi-language material knowledge points; the multi-language material knowledge points and the questions to be answered are input into a semantic vector matching model to be converted into semantic vectors respectively and vector matching is performed; screening is performed according to the vector matching results to determine refined screening material knowledge points; bound material slices are obtained from the database according to the refined screening material knowledge points; prompt information is generated according to the material slices and the questions to be answered; the prompt information is input into the question-answering large model for processing to generate answers. The above technical solution integrates non-standard language material knowledge points with standard language material knowledge points to obtain multi-language material knowledge points, so that the database contains different language expressions of material knowledge points, which is convenient for subsequent more accurate enhanced search based on the questions to be answered. There is no need to translate the questions to be answered into expressions in different languages for enhanced retrieval, nor is there a need to form prompt words in different languages, thereby improving the response speed of the question-answering model to the questions to be answered. At the same time, after the character-level matching screening of the questions to be answered and the material knowledge points stored offline in the database, the semantic-level matching screening is performed, which reduces the amount of vector calculation in the answer generation process.
[0098] Figure 3A It is a schematic diagram of another question-answering method based on enhanced search provided according to an embodiment of the present disclosure. This embodiment is an optional solution proposed on the basis of the above embodiment, and further adds an offline generation process of the database. It should be noted that for the parts not described in detail in the embodiment of the present disclosure, reference can be made to the relevant descriptions of other embodiments. Figure 3A The question-answering method based on enhanced search provided in this embodiment includes:
[0099] S301: Split the document to be processed according to the slicing rules to form material slices.
[0100] Among them, the document to be processed refers to the document that needs to be processed; optionally, the document to be processed includes but is not limited to e-books, online articles, academic papers and journals. Slicing rules refer to the rules used to split the document to be processed; optionally, the slicing rules include slicing setting information, such as slice size, slice identifier and slice order. It should be noted that the slicing rules can be pre-set according to actual business needs. For example, the slicing rules can be sentence slicing rules, paragraph slicing rules or chapter slicing rules. For example, the slicing rules can be recursive character text slicing rules, which are not specifically limited in the embodiments of the present disclosure. Material slicing refers to the slices obtained after the document to be processed is split.
[0101] Exemplarily, the document to be processed is split according to sentence slicing rules to obtain multiple material slices.
[0102] Exemplarily, the document to be processed is split according to paragraph slicing rules to obtain multiple material slices.
[0103] In an optional embodiment, before splitting the document to be processed according to the slicing rules to form material slices, it is also possible to: use a set language recognition algorithm to identify the language of the document to be processed, and use the identified language as the language of the material slices of the document to be processed.
[0104] The language recognition algorithm is set to include at least one of the following: a language recognition model and a regular expression. The language recognition model is obtained by training the model with sample documents corresponding to the language.
[0105] It is understandable that before splitting the document to be processed, the language of the document to be processed is determined first, rather than splitting the document to be processed into material knowledge points and then determining the language of each material knowledge point, which speeds up the translation speed of translating material knowledge points in non-standard languages into material knowledge points in standard languages.
[0106] S302: Slice the material into one or more material knowledge points.
[0107] Specifically, if the material knowledge point is a vocabulary with independent semantics, the material slice is processed into one or more material knowledge points with the help of a material slicing model; wherein the material slicing model can be obtained by training a machine learning model through sample material slices annotated with word labels.
[0108] In an optional embodiment, if the material knowledge point is a sentence with independent semantics, the material slice is split using a set sentence separator of a standard language to obtain one or more sentences; the split sentences are split again based on a sentence splitting model of at least one non-standard language, and the sentences after the second split are used as material knowledge points.
[0109] The sentence separators may be preset according to actual business needs, for example, the sentence separators include but are not limited to period, comma, semicolon, exclamation mark and question mark. The sentence splitting model may be preset according to actual business needs, for example, the sentence splitting model may be an NLTK model (Natural Language Toolkit); wherein the NLTK model may be trained using sample sentences corresponding to non-standard languages.
[0110] Specifically, if the material knowledge point is a sentence with independent semantics, the material slice is split using the set sentence separator of the standard language to obtain one or more split sentences; the non-standard language involved in the split sentence is determined, and the split sentence is split again using sentence splitting models of different non-standard languages to obtain the second-split sentence, that is, the material knowledge point; if the language of the split sentence is the standard language, the split sentence is directly used as the material knowledge point.
[0111] To give a specific example, if the standard language is Chinese, and the non-standard languages involved in the split sentences include English, Japanese and French, the English sentence splitting model, Japanese sentence splitting model and French sentence splitting model are used respectively to perform secondary splitting on the split sentences, thereby obtaining the secondary split sentences, namely the material knowledge points.
[0112] It can be understood that after the material slices are split using the set sentence separators of the standard language to obtain the split sentences, the split sentences are split again using the sentence splitting model of the non-standard language involved in the split sentences, thereby avoiding the incorrect splitting of sentences or vocabulary in the non-standard language in the material slices, thereby ensuring the semantic integrity and accuracy of the material knowledge points.
[0113] S303: Translate the material knowledge points in the non-standard language to obtain the material knowledge points in the standard language.
[0114] Specifically, a multilingual translation model can be used to translate material knowledge points in non-standard languages to obtain material knowledge points in standard languages. The multilingual translation model can be obtained by training a machine learning model with sample material knowledge points in different languages.
[0115] In an optional embodiment, for material knowledge points in a non-standard language, a language-corresponding translation model is used for translation to obtain material knowledge points in a standard language; wherein the language-corresponding translation model is obtained by training with language-corresponding samples, and the language-corresponding samples are word samples.
[0116] Specifically, for material knowledge points in non-standard languages, a language recognition model is used to identify the target language of the material knowledge points in the non-standard language; a translation model corresponding to the target language is used to translate the material knowledge points in the non-standard language to obtain material knowledge points in the standard language.
[0117] It is understandable that, compared with the technical solution of using a general multilingual translation model to translate material knowledge points in non-standard languages to obtain material knowledge points in standard languages, using a dedicated translation model to translate material knowledge points in non-standard languages to obtain material knowledge points in standard languages improves the accuracy of translation and can reduce the scale requirements of the translation model. Since the initial screening at the character level and then the fine screening at the semantic level are performed when generating questions and answers online, the translation accuracy requirements for the material knowledge points translated in the database are not high. For example, the grammatical structure is not more accurate than a high degree. Therefore, the translation model can only use word samples as training samples, and there is no need to use a large number of sentence samples or even document samples for training. This can significantly reduce the cost of the translation model.
[0118] In an optional embodiment, before translating the material knowledge points in a non-standard language, it is also possible to: use regular expressions to identify each material knowledge point to determine the language to which it belongs.
[0119] Specifically, for each material knowledge point, a regular expression is used to identify the special character set in the material knowledge point, and the language of the material knowledge point is determined by these special character sets. For example, if the material knowledge point does not include any Chinese characters, it is considered a non-standard language.
[0120] In an optional embodiment, for each material knowledge point, a regular expression may be used to identify a specific grammatical structure or specific vocabulary in the material knowledge point, and the language to which the material knowledge point belongs may be determined through these specific grammatical structures or specific vocabulary.
[0121] In an optional embodiment, for each material knowledge point, a regular expression can also be used to identify a special character set and a specific grammatical structure or a specific vocabulary in the material knowledge point, and the language to which the material knowledge point belongs can be determined by combining the special character set and the specific grammatical structure (or specific vocabulary).
[0122] It is understandable that before translating material knowledge points in non-standard languages, the language to which each material knowledge point belongs is first determined through regular expressions so that the material knowledge points in non-standard languages can be screened out, thereby only translating the material knowledge points in non-standard languages, reducing unnecessary translation and speeding up the translation of material knowledge points in non-standard languages.
[0123] S304: Establish a binding relationship between each material knowledge point and the material slice, and store it in a database.
[0124] Specifically, a binding relationship between each material knowledge point and the material slice may be established by creating an association table, and the created association table is stored in a database.
[0125] S305: Obtain questions to be answered.
[0126] S306: Match and screen the question to be answered with the material knowledge points stored offline in the database at the character level to obtain preliminary screening material knowledge points.
[0127] S307, performing semantic level matching screening on the questions to be answered in the primary screening material knowledge points to obtain the refined screening material knowledge points.
[0128] S308. Obtain bound material slices from the database according to the carefully screened material knowledge points.
[0129] S309: Generate prompt information based on the material slices and the questions to be answered.
[0130] S310: Input the prompt information into the question-answering model for processing to generate an answer.
[0131] The database stores bound material knowledge points and material slices offline; each material slice has at least two languages; the material knowledge points are in a standard language, and the material knowledge points in the standard language are obtained by translating material slices in a non-standard language.
[0132] The disclosed embodiment forms material slices by splitting the document to be processed according to the slicing rules; processing the material slices into one or more material knowledge points; translating the material knowledge points in the non-standard language to obtain the material knowledge points in the standard language; establishing a binding relationship between each material knowledge point and the material slices, and storing them in a database; obtaining the question to be answered; matching and screening the question to be answered with the material knowledge points stored offline in the database at the character level to obtain the preliminary screening material knowledge points; matching and screening the question to be answered in the preliminary screening material knowledge points at the semantic level to obtain the refined screening material knowledge points; obtaining bound material slices from the database according to the refined screening material knowledge points; generating prompt information according to the material slices and the question to be answered; and inputting the prompt information into the question-answering big model for processing to generate an answer. The above technical solution realizes offline processing of documents to be processed in different languages, converts material knowledge points in non-standard languages into material knowledge points in standard languages in advance, establishes a binding relationship between each material knowledge point and material slice, and stores them in the database, unifying the language of the questions to be answered and the material knowledge points, making it easier to retrieve more knowledge points related to the questions to be answered, thereby improving the accuracy of matching the questions to be answered and the material knowledge points.
[0133] The question-answering process based on enhanced search provided in the embodiment of the present disclosure can be referred to Figure 3B As shown in the figure, the Q&A process based on enhanced search mainly includes two stages, namely, the database offline preparation stage and the online Q&A stage. The steps of the database offline preparation stage include: slicing the documents to be processed; the specific processing of material slicing can be referred to Figure 3C As shown, material knowledge points of standard languages are formed and stored in the database, and the relationship between material knowledge points and material slices is bound. The steps of the online question-and-answer stage include: obtaining the question to be answered (query) raised by the user; character matching the question to be answered with the material knowledge points in the database to obtain preliminary screening material knowledge points; then performing mixed retrieval on the initial material knowledge points to obtain refined screening material knowledge points. The mixed retrieval may include but is not limited to vector retrieval and the best matching algorithm BM25; re-sorting the refined screening material knowledge points to further obtain sorted and filtered refined screening material knowledge points; obtaining bound material slices based on the refined screening material knowledge points; generating prompt information based on the material slices and the question to be answered; and inputting the prompt information into the question-and-answer model to generate answers. Reference Figure 3C As shown, after the document is split into material slices, the material slices can be further divided into sentences; non-Chinese sentences can be identified based on regular algorithms; and then the non-text sentences are translated into Chinese sentences using a translation model; the Chinese sentences directly obtained from the material slices and the translated Chinese sentences are used as standard language material knowledge points and stored in the database as slice information.
[0134] Figure 4 1 is a schematic diagram of a question-answering device based on enhanced search according to an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the case of cross-language question-answering based on a large model. The device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 4 The question-answering device 400 based on enhanced search specifically includes the following:
[0135] The to-be-answered question acquisition module 401 is used to acquire the to-be-answered question;
[0136] The initial screening material knowledge point acquisition module 402 is used to match and screen the question to be answered with the material knowledge points stored offline in the database at the character level to obtain the initial screening material knowledge points;
[0137] A refined screening material knowledge point acquisition module 403 is used to perform semantic level matching screening on the questions to be answered in the primary screening material knowledge points to obtain refined screening material knowledge points;
[0138] The material slice acquisition module 404 is used to acquire bound material slices from the database according to the carefully screened material knowledge points;
[0139] A prompt information generating module 405 is used to generate prompt information according to the material slices and the questions to be answered;
[0140] The answer generation module 406 is used to input the prompt information into the question-answering model for processing to generate an answer;
[0141] The database stores bound material knowledge points and material slices offline; each material slice has at least two languages; the material knowledge points are in a standard language, and the material knowledge points in the standard language are obtained by translating material slices in a non-standard language.
[0142] The disclosed embodiment scheme obtains the question to be answered; matches and screens the question to be answered with the material knowledge points stored offline in the database at the character level to obtain the preliminary screening material knowledge points; matches and screens the question to be answered at the semantic level in the preliminary screening material knowledge points to obtain the refined screening material knowledge points; obtains the bound material slices from the database according to the refined screening material knowledge points; generates prompt information according to the material slices and the question to be answered; inputs the prompt information into the question-answering model for processing to generate answers; wherein, the database stores the bound material knowledge points and material slices offline; the languages of each material slice include at least two; the language of the material knowledge points is the standard language, and the material knowledge points of the standard language are obtained by translating the material slices of the non-standard language. The above technical scheme directly stores the material knowledge points of the standard language offline in the database, unifies the language of the question to be answered and the material knowledge points, and facilitates the retrieval of more knowledge points related to the question to be answered, thereby improving the accuracy of the matching between the question to be answered and the material knowledge points. At the same time, the questions to be answered and the material knowledge points stored offline in the database are first matched and screened at the character level, and then at the semantic level, filtering out the material knowledge points that are completely irrelevant to the questions to be answered, reducing the amount of calculation in the answer generation process, and improving the response speed of the large model to the questions to be answered. In addition, the knowledge points most relevant to the questions to be answered can be retrieved from the database.
[0143] In an optional implementation, the fine screening material knowledge point acquisition module 403 is specifically used for:
[0144] If the initial screening material knowledge points are material knowledge points obtained through translation, then determining the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the initial screening material knowledge points from the database;
[0145] Integrate knowledge points of non-standard language materials and knowledge points of standard language materials to form multi-language material knowledge points;
[0146] Input the knowledge points of multilingual materials and questions to be answered into the semantic vector matching model to convert them into semantic vectors and perform vector matching respectively;
[0147] Screening is performed based on the vector matching results to determine the knowledge points of the refined screening materials.
[0148] In an optional implementation, the semantic vector matching model is an m-BERT model, and the semantic vector matching model is trained using multilingual corpus samples.
[0149] In an optional implementation, the enhanced search-based question-answering device 400 further includes:
[0150] The vector sorting module is used to input each of the finely screened material knowledge points and the questions to be answered into the vector sorting model before obtaining the bound material slices from the database according to the finely screened material knowledge points, so as to convert them into semantic vectors respectively, and sort them according to the similarity of the semantic vectors;
[0151] The fine screening material knowledge point screening module is used to screen the fine screening material knowledge points according to the sorting results and sorting filtering conditions.
[0152] In an optional implementation, the prompt information generating module 405 is specifically used for:
[0153] Obtain a prompt information template, add the material slices to the material area of the prompt information template, and add the questions to be answered to the question area of the prompt information template to generate prompt information.
[0154] In an optional implementation, the enhanced search-based question-answering device 400 further includes a database offline generation module, and the database offline generation module includes:
[0155] The material slice forming unit is used to split the document to be processed according to the slicing rules to form material slices;
[0156] A material slicing processing unit, used for processing the material slices into one or more material knowledge points;
[0157] A material knowledge point translation unit is used to translate material knowledge points in non-standard languages to obtain material knowledge points in standard languages;
[0158] The binding relationship establishing unit is used to establish the binding relationship between each material knowledge point and the material slice, and store it in the database.
[0159] In an optional implementation manner, the material knowledge point is a sentence with independent semantics, and the material slicing processing unit is specifically used for:
[0160] Split the material slices using the set sentence separator of the standard language to obtain one or more sentences;
[0161] The split sentences are split again based on at least one sentence splitting model of a non-standard language, and the sentences after the second split are used as material knowledge points.
[0162] In an optional implementation, the database offline generation module further includes:
[0163] The material knowledge point language determination unit is used to use regular expressions to identify each material knowledge point before translating the material knowledge point in a non-standard language to determine the language to which it belongs.
[0164] In an optional implementation manner, the material knowledge point translation unit is specifically used for:
[0165] For material knowledge points in non-standard languages, use the translation model corresponding to the language to translate them to obtain material knowledge points in the standard language;
[0166] The language-corresponding translation model is obtained by training with language-corresponding samples, and the language-corresponding samples are word samples.
[0167] In an optional implementation, the database offline generation module further includes:
[0168] The document language identification unit is used to split the document to be processed according to the slicing rules. Before forming material slices, the document to be processed is identified by using a set language identification algorithm, and the identified language is used as the language of the material slices of the document to be processed.
[0169] In an optional implementation, the language recognition algorithm is set to include at least one of the following: a language recognition model and a regular expression.
[0170] A question-and-answer device based on enhanced search provided by the technical solution of the embodiments of the present disclosure can execute the question-and-answer method based on enhanced search provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the question-and-answer method based on enhanced search.
[0171] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0172] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0173] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0174] like Figure 5As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0175] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0176] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a question-and-answer method based on enhanced search. For example, in some embodiments, the question-and-answer method based on enhanced search may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the question-and-answer method based on enhanced search described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the enhanced search-based question-answering method in any other appropriate manner (eg, by means of firmware).
[0177] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0178] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0179] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0181] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0182] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.
[0183] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.
[0184] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.
[0185] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.
[0186] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A question-answering method based on enhanced search, comprising: Get questions to be answered; Match the question to be answered with the material knowledge points stored offline in the database at the character level to obtain preliminary screening material knowledge points; The questions to be answered are matched and screened at the semantic level in the primary screening material knowledge points to obtain the refined screening material knowledge points; Acquire bound material slices from the database according to the finely screened material knowledge points; Generate prompt information according to the material slices and the questions to be answered; Inputting the prompt information into the question-answering model for processing to generate an answer; The database offline stores bound material knowledge points and material slices; the languages of each material slice include at least two; the language of the material knowledge points is a standard language, and the material knowledge points in the standard language are obtained by translating the material slices in a non-standard language.
2. The method according to claim 1, wherein: The question to be answered is matched and screened at the semantic level in the primary screening material knowledge points to obtain the refined screening material knowledge points, including: If the initially screened material knowledge points are material knowledge points obtained through translation, then determining the non-standard language material knowledge points and the translated standard language material knowledge points corresponding to the initially screened material knowledge points from the database; Integrating the non-standard language material knowledge points with the standard language material knowledge points to form multi-language material knowledge points; Input the multilingual material knowledge points and the questions to be answered into a semantic vector matching model to convert them into semantic vectors and perform vector matching respectively; Screening is performed based on the vector matching results to determine the knowledge points of the refined screening materials.
3. The method according to claim 2, wherein: The semantic vector matching model is an m-BERT model, and the semantic vector matching model is trained using multilingual corpus samples.
4. The method according to claim 1, before obtaining the bound material slices from the database according to the finely screened material knowledge points, further comprises: Inputting each of the carefully screened material knowledge points and the questions to be answered into a vector sorting model to convert them into semantic vectors respectively, and sorting them according to the similarity of the semantic vectors; According to the sorting results and sorting filter conditions, each of the finely screened material knowledge points is screened.
5. The method according to claim 1, wherein: Generating prompt information according to the material slice and the question to be answered includes: A prompt information template is obtained, the material slice is added to the material area of the prompt information template, and the question to be answered is added to the question area of the prompt information template to generate the prompt information.
6. The method according to any one of claims 1 to 5, wherein: The offline generation process of the database includes: Split the document to be processed according to the slicing rules to form material slices; Processing the material slices into one or more material knowledge points; Translate the knowledge points of materials in non-standard languages to obtain the knowledge points of materials in standard languages; A binding relationship between each material knowledge point and the material slice is established and stored in the database.
7. The method according to claim 6, wherein: If the material knowledge point is a statement with independent semantics, slicing the material into one or more material knowledge points includes: Splitting the material slices using a set sentence separator of a standard language to obtain one or more sentences; The split sentences are split again based on at least one sentence splitting model of a non-standard language, and the sentences after the second split are used as material knowledge points.
8. The method according to claim 6, before translating the material knowledge points in the non-standard language, further comprises: Regular expressions are used to identify knowledge points of each material to determine the language to which it belongs.
9. The method according to claim 6, wherein: Translating knowledge points of materials in non-standard languages to obtain knowledge points of materials in standard languages includes: For material knowledge points in non-standard languages, use the translation model corresponding to the language to translate them to obtain material knowledge points in the standard language; The language-corresponding translation model is obtained by training with language-corresponding samples, and the language-corresponding samples are word samples.
10. The method according to claim 6, before splitting the document to be processed according to the slicing rules to form material slices, further comprises: For the document to be processed, a set language recognition algorithm is used to perform language recognition, and the recognized language is used as the language of the material slice of the document to be processed.
11. The method according to claim 10, wherein: The language recognition algorithm is set to include at least one of the following: a language recognition model and a regular expression.
12. A question-answering device based on enhanced search, comprising: A module for obtaining questions to be answered, used to obtain questions to be answered; A preliminary screening material knowledge point acquisition module is used to match and screen the question to be answered with the material knowledge points stored offline in the database at the character level to obtain the preliminary screening material knowledge points; A refined screening material knowledge point acquisition module is used to perform semantic level matching screening on the questions to be answered in the primary screening material knowledge points to obtain refined screening material knowledge points; A material slice acquisition module, used for acquiring bound material slices from the database according to the finely screened material knowledge points; A prompt information generating module, used for generating prompt information according to the material slices and the questions to be answered; An answer generation module, used for inputting the prompt information into the question-answering model for processing to generate an answer; The database offline stores bound material knowledge points and material slices; the languages of each material slice include at least two; the language of the material knowledge points is a standard language, and the material knowledge points in the standard language are obtained by translating the material slices in a non-standard language.
13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the enhanced search-based question-answering method described in any one of claims 1-11.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the enhanced search-based question-answering method according to any one of claims 1-11.
15. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the enhanced search-based question-answering method according to any one of claims 1-11.
