An intelligent question and answer processing method and device, an electronic device, and a storage medium

Abstract

The application discloses an intelligent question and answer processing method and device, electronic equipment and storage medium, and relates to the technical field of artificial intelligence. Metadata is constructed based on data information of data in an obtained data set. Data content in the data set is converted into a digital vector. A plurality of data records are obtained based on the corresponding storage of the data content, data information, metadata and digital vector in the data set. A BM25 retriever is used to obtain a first candidate result related to a user query, a first relevance score of the first candidate result, a second candidate result related to the user query and a second relevance score of the second candidate result, and then determine a comprehensive score. A target candidate result is determined based on the comprehensive score. Prompt words of an intelligent question and answer model are constructed based on the target candidate result and the user query. The prompt words are used to output a query result corresponding to the user query by using the intelligent question and answer model. In this way, the accuracy of question and answer is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an intelligent question and answer processing method and device, electronic equipment and a storage medium. BACKGROUND

[0002] An earth observation / remote sensing data platform aggregates large-scale data sets from different organizations and units and covering various application fields, and data retrieval and question answering are usually performed based on the earth observation / remote sensing data platform.

[0003] In the related art, a document is first sliced, each long document is divided to obtain multiple segments, each segment is vector encoded to obtain multiple vector segments, and the multiple vector segments are stored in a vector database. A user question is vector encoded to obtain a question vector, candidate vector segments matching the question vector are obtained based on keyword retrieval and vector retrieval, a target vector segment matching the question vector is determined based on a preset rule, and a large model generates an answer based on the question vector and the target vector segment matching the question vector.

[0004] However, the above method for data retrieval and question answering has the following problems: the user cannot verify the authenticity of the answer because the answer has no source; the subsequent program processing is not facilitated because the output format is chaotic; and the current prompt words are not suitable for Chinese + professional field scenarios because they are mostly applicable to English scenarios. SUMMARY

[0005] Therefore, the present application provides an intelligent question and answer processing method and device, electronic equipment and a storage medium to solve the problem of low accuracy of the generated answer in the prior art question and answer processing method.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] The first aspect of the present application provides an intelligent question and answer processing method, comprising:

[0008] Obtaining data information of data in a data set;

[0009] Constructing metadata based on the data information;

[0010] Converting data content in the data set into a digital vector;

[0011] Corresponding storage of the data content, the data information, the metadata and the digital vector in the data set to obtain multiple data records;

[0012] Using a BM25 algorithm to construct a BM25 retriever based on the data records;

[0013] Based on the user query, a BM25 retriever is used to obtain a first candidate result related to the user query and a first relevance score of the first candidate result;

[0014] Based on the query vector corresponding to the user query and the data record, a second candidate result related to the user query is obtained, and a second relevance score of the second candidate result is obtained;

[0015] Based on the first relevance score and the second relevance score, a comprehensive score is determined;

[0016] Based on the comprehensive score, a target candidate result is determined;

[0017] Based on the target candidate result and the user query, a prompt word of an intelligent question and answer model is constructed;

[0018] Based on the prompt word, the intelligent question and answer model is used to output a query result corresponding to the user query.

[0019] In an optional embodiment, based on the first relevance score and the second relevance score, the comprehensive score is determined by using the following formula:

[0020] ;

[0021] wherein, the comprehensive score is represented by S, the weight parameter is represented by w, the first relevance score is represented by s1, and the second relevance score is represented by s2.

[0022] In an optional embodiment, after obtaining a plurality of data records based on the storage of data content, data information, metadata and digital vectors in the data set, the method further comprises:

[0023] If the metadata in the data record includes a data identifier, the data record is processed for deduplication using the data identifier;

[0024] If the metadata in the data record does not include a data identifier, a content identifier is determined from the data content, and the data record is processed for deduplication based on the content identifier;

[0025] If the data content does not include a content identifier, a hash value is obtained by performing a hash calculation on the data content, and the data record is processed for deduplication based on the hash value.

[0026] In an optional embodiment, after determining the target candidate result based on the comprehensive score, the method further comprises generating a result record based on the target candidate result, wherein the result record includes a data identifier, data time information, data summary points, a data download link and the data identifier;

[0027] The prompt word of the intelligent question and answer model is constructed based on the target candidate result and the user query, including:

[0028] The prompt word of the intelligent question and answer model is constructed based on the result record and the user query.

[0029] In an optional embodiment, after the prompt word of the intelligent question and answer model is constructed based on the target candidate result and the user query, further comprising:

[0030] Setting a constraint condition;

[0031] Based on the prompt word, the intelligent question and answer model is used to output the query result corresponding to the user query, including:

[0032] Based on the constraint condition and the prompt word, the intelligent question and answer model is used to output the query result corresponding to the user query.

[0033] In an optional embodiment, the data content in the data set is converted into a digital vector, including:

[0034] The data content in the data set is converted into a digital vector in a preset batch size.

[0035] In an optional embodiment, after the data in the data set is converted into a digital vector, further comprising:

[0036] The digital vector is normalized.

[0037] The second aspect of the present application provides an intelligent question and answer processing device, characterized in that, comprising:

[0038] An acquisition module is configured to acquire data information of data in a data set;

[0039] A first construction module is configured to construct metadata based on the data information;

[0040] A conversion module is configured to convert data content in the data set into a digital vector;

[0041] A first determination module is configured to obtain a plurality of data records based on the data content in the data set, the data information, the metadata, and the corresponding storage of the digital vector;

[0042] A second construction module is configured to construct a BM25 retriever based on the data records by using a BM25 algorithm;

[0043] A second determination module is configured to obtain a first candidate result related to a user query and a first relevance score of the first candidate result by using the BM25 retriever based on the user query;

[0044] The third determining module is configured to obtain second candidate results related to the user query and second relevance scores of the second candidate results based on the user query corresponding query vector and the data records.

[0045] The fourth determining module is configured to determine a comprehensive score based on the first relevance score and the second relevance score.

[0046] The fifth determining module is configured to determine a target candidate result based on the comprehensive score.

[0047] The third constructing module is configured to construct a prompt word of the intelligent question and answer model based on the target candidate result and the user query.

[0048] The output module is configured to output a query result corresponding to the user query by using the intelligent question and answer model based on the prompt word.

[0049] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method of the first aspect.

[0050] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0051] Compared with the prior art, the intelligent question and answer processing method provided by the present application constructs metadata based on the data information of the obtained data in the data set; converts the data content in the data set into a digital vector; obtains a plurality of data records based on the corresponding storage of the data content, the data information, the metadata and the digital vector in the data set; obtains the first candidate result related to the user query, the first relevance score of the first candidate result, the second candidate result related to the user query and the second relevance score of the second candidate result by using the BM25 retriever; determines the target candidate result based on the comprehensive score; constructs the prompt word of the intelligent question and answer model based on the target candidate result and the user query; and outputs the query result corresponding to the user query by using the intelligent question and answer model based on the prompt word. In this way, the accuracy of the question and answer is improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1A flowchart of an intelligent question answering processing method provided by an embodiment of the present application is shown in FIG. 1.

[0054] Figure 2 A structural block diagram of an intelligent question answering processing device provided by an embodiment of the present application is shown in FIG. 2.

[0055] Figure 3 A structural block diagram of an electronic device for implementing an intelligent question answering processing method provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0056] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0057] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0058] It should be understood that in the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects. "Including A, B, and / or C" means including any one or any two or three of A, B, and C.

[0059] It should be understood that in the embodiments of the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B", or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but also can be determined according to A and / or other information.

[0060] To solve the technical problems in the related art, the embodiments of the present application provide an intelligent question answering processing method and device, an electronic device, and a storage medium.

[0061] The intelligent question answering processing method provided by the embodiments of the present application can be executed by an electronic device. The electronic device can be a terminal or a server. The terminal can be a terminal device such as a smartphone, a tablet computer, or a notebook computer. The server can be a physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. It can be understood that the present application does not specifically limit the subject that executes the intelligent question answering processing method.

[0062] The technical solutions of the present application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments described below are used to explain the technical solutions of the present application and are not limited to actual use.

[0063] To solve the technical problems in the related art, the embodiments of the present application provide an intelligent question answering processing method, as shown in Figure 1 Figure 1 A flowchart of the intelligent question answering processing method provided by the embodiments of the present application. It should be noted that the steps shown can be executed in a different logical order than that shown in the method flowchart. The method can include the following steps S101-S111.

[0064] Step S101: Obtain data information of data in a data set.

[0065] In an optional embodiment, the data set in the embodiments of the present application includes data from different organizations and units and covering various application fields.

[0066] In a specific embodiment, the data can be earth observation / remote sensing data.

[0067] ​In an alternative embodiment, before obtaining the data information of the data in the data set, the following steps can also be performed: requesting a session token from an interface of a data platform (the data platform can be an earth observation / remote sensing data platform) with a preconfigured key, the session token being used to prove the legitimacy of the identity of the user. In this way, the dataset list interface provided by the data platform can be subsequently called to send a data set list request to the data platform, the data set list request carrying the session token. The data set list returned by the data platform is received, the data set list including a data set identifier, and the data set corresponding to the data set identifier is obtained according to the data set identifier. Then, the data information of the data in the data set is obtained from the corresponding data set.

[0068] In an alternative embodiment, the data information includes at least one of the following information: data identifier, data title, data keyword, data description, data time information, and data download link. In addition, the data information can also include other information, which is not limited by the present application.

[0069] In a specific embodiment, the data identifier is represented by id (identifier), the data title is represented by title (title), the data keyword is represented by keyword (keyword), the data description is represented by description (description), the data time information is represented by timeInfo (time information, Time Information), and the data download link is represented by download Url (download uniform resource locator, Download Uniform Resource Locator).

[0070] In a more specific embodiment, the data information of the data r can be represented as: .

[0071] In an alternative embodiment, the data information can be stored in JSON (JavaScript Object Notation) format so as to be parsed by a machine later.

[0072] In another alternative embodiment, the intelligent question and answer processing method provided by the embodiments of the present application further includes the following steps:

[0073] Retrying and recording the abnormal response to ensure the minimum available data set.

[0074] In the embodiments of the present application, considering that the abnormal situation may occur in the process of obtaining the data information of the data in the data set. Therefore, the abnormal response is retried and recorded to ensure the minimum available data set.

[0075] In a specific embodiment, retrying and recording the abnormal response ensures that the minimum available dataset specifically includes the following steps:

[0076] When the data information of the dataset data fails to be acquired, retrying a preset number of times; if the normal state is not restored after retrying the preset number of times, the dataset list interface, error code, and abnormal time of the response exception are recorded for subsequent troubleshooting. If there is a missing field, other fields can also be retained to generate "incomplete but usable" data information.

[0077] By retrying and recording the abnormal response, the minimum available dataset can improve system availability and automatically recover temporary failures without user awareness. Even if some failures occur, "defective but useful" results can be obtained. Recording the exception enables problem traceability.

[0078] Step S102: Constructing metadata based on the data information.

[0079] It should be noted that the metadata refers to information about the data.

[0080] In this embodiment, the metadata is constructed to ensure the traceability of subsequent retrieval and reference.

[0081] In an optional embodiment, the metadata includes at least one of the following information: data identification, data download link, dataset name of the data, and data source.

[0082] In a specific embodiment, the metadata can be expressed as follows: Wherein, Application Programming Interface index represents the dataset name of the data, and represents the data source.

[0083] Step S103: Converting the data content in the dataset into a digital vector.

[0084] In an optional embodiment, converting the data content in the dataset into a digital vector includes:

[0085] Converting the data content in the dataset into a digital vector in a preset batch size.

[0086] In an optional embodiment, the data content can be in the form of documents, web pages, etc.

[0087] In a specific embodiment, a vector conversion model is loaded, and the vector conversion model is used to convert the data content in the dataset into a digital vector in a preset batch size.

[0088] In a specific embodiment, the vector conversion model can be a BAAI / bge-large-zh model, a bge-m3 model, and a gte-large-zh model, and can also be other vector conversion models, which are not limited in the present application. It should be noted that the BAAI / bge-large-zh model, the bge-m3 model, and the gte-large-zh model are common models for natural language processing tasks, which are not described in detail in the present application.

[0089] In a specific embodiment, the batch size is represented by BATCH_SIZE, for example, BATCH_SIZE=64, BATCH_SIZE=32, and BATCH_SIZE=128, etc.

[0090] In another specific embodiment, the model is migrated to a GPU (Graphics Processing Unit) to accelerate batch encoding when the GPU is detected.

[0091] In a more specific embodiment, the document is cut into multiple pieces of text, and then the vector conversion model is used to process a preset batch size of pieces at a time (instead of processing one piece at a time), and efficiently convert them into semantic vectors for subsequent retrieval or matching.

[0092] In another optional embodiment, after converting the data content in the data set into a digital vector, it further includes:

[0093] The digital vector is normalized.

[0094] In a specific embodiment, the digital vector is L2 normalized.

[0095] In this embodiment, the digital vector is L2 normalized to improve the stability and efficiency of the similarity.

[0096] Step S104: Based on the corresponding storage of the data content, the data information, the metadata, and the digital vector in the data set, a plurality of data records are obtained.

[0097] In an optional embodiment, a vector database is established, which can be a lightweight and open source vector database. For example, the vector database can be Chroma, FAISS (Facebook AI Similarity Search), Milvus, Open Search k-NN (Open Search k-Nearest Neighbors), and Vespa k-NN (Vespa k-Nearest Neighbors). The above vector databases support index structures such as HNSW (Hierarchical Navigable Small World), which are not limited by the present application.

[0098] In a specific embodiment, a Persistent Client of the Chroma vector database is used to write data to the local disk in real time, so that even if the program is closed or the machine is restarted, the data will not be lost. For example, a directory A is specified, and data is stored in the directory. Further, a collection for storing earth observation data is created in the directory, for example, the collection can be a collection named earth_observation_data.

[0099] The data identifier can be expressed in the form of a stable prefix + self-incrementing index, where the prefix refers to a string of specified characters placed at the beginning of the number, and stable means that the same rule is used to generate the identifier each time. For example, the identifier can be expressed as: api_#, where api indicates that the data is obtained through an API, _# is a separator used to emphasize the subsequent number. The self-incrementing index ensures the uniqueness of the identifier and avoids duplication of the identifier.

[0100] Through the above operation, a mapping structure of "text-vector-metadata" is formed in the vector database, supporting robust deduplication and source reference based on unique identifiers.

[0101] In another optional embodiment, after completing the batch writing of data, the collection count and time consumption can be displayed. The collection count is used to confirm whether the data is completely written, and the time consumption is used to represent the time length of the entire data writing process. In this way, a monitorable and maintainable engineering data warehousing process is realized.

[0102] In another alternative embodiment, the intelligent question and answer processing method provided by the embodiments of the present application further includes the following steps: if the metadata in the data record includes a data identifier, performing deduplication processing on the data record by using the data identifier; if the metadata in the data record does not include a data identifier, determining a content identifier from the data content, and performing deduplication processing on the data record based on the content identifier; and if the data content does not include a content identifier, performing hash calculation on the data content to obtain a hash value, and performing deduplication processing on the data record based on the hash value.

[0103] In this embodiment, by performing deduplication processing on the data record, the repeated or highly similar candidate results are avoided to improve the retrieval quality and user experience; the invalid data occupying space is reduced to reduce the index volume and storage cost; and the retrieval speed is accelerated and the delay is reduced.

[0104] Step S105: constructing a BM25 retriever based on the data record by using the BM25 algorithm.

[0105] It should be noted that the BM (Best Matching) 25 retriever is a probabilistic retrieval model for evaluating the relevance of search terms and documents.

[0106] In an alternative embodiment, the BM25 retriever is constructed based on the data content, the data title, the data description and the data keywords.

[0107] In a specific embodiment, the BM25 retriever is constructed based on the data content, the data title, the data description and the data keywords, and specifically includes the following steps:

[0108] The data title, the data description and the data keywords are concatenated into a string in a structured manner and assigned to the page_content (text content) field. It should be noted that the page_content is a string type field, which is the input of the BM25 retriever.

[0109] In this embodiment, only the data title, the data description and the data keywords are written into the page_content field, and other redundant / noisy fields are not written into the page_content field, thereby reducing the index volume and term noise. JSON or other structured manner is used as the data content instead of simple string concatenation, the word order and field boundary are stable, which is conducive to the score repeatability when repeatedly constructed. The metadata separately saves the data identifier and the data download link instead of mixing them into the inverted text, thereby reducing meaningless term inflation.

[0110] Step S106: obtaining the first candidate result related to the user query and the first relevance score of the first candidate result by using the BM25 retriever based on the user query.

[0111] In an alternative embodiment, based on the user query, a BM25 retriever is used to obtain the first candidate results related to the user query and the first relevance scores of the first candidate results, specifically including the following steps:

[0112] Based on the user query, a BM25 retriever is used to obtain the first candidate results related to the user query and the first relevance scores of the first candidate results, wherein the first relevance score can be represented by .

[0113] It should be noted that the first candidate results are any data records in the above-mentioned plurality of data records.

[0114] In an alternative embodiment, the similarity can be cosine similarity, inner product, or Euclidean distance.

[0115] Step S107: Based on the query vector corresponding to the user query and the data records, the second candidate results related to the user query and the second relevance scores of the second candidate results are obtained.

[0116] In an alternative embodiment, based on the query vector corresponding to the user query and the data records, the second candidate results related to the user query and the second relevance scores of the second candidate results are obtained, specifically including the following steps:

[0117] The user query is converted into a query vector using a Sentence Embedding model; and the first N items of second candidate results most similar to the query vector are found in the vector database. The second relevance scores of the second candidate results are determined based on the similarity between the second candidate results and the query vector.

[0118] In a more specific embodiment, the Sentence Embedding model can be at least one of the following models: all-MiniLM-L6-v2, text2vec-base-chinese, and bge-small-zh-v1.5, in addition to other models, which are not limited by the present application. It should be noted that all-MiniLM-L6-v2, text2vec-base-chinese, and bge-small-zh-v1.5 are common models for converting text into vector representation, which are not described in detail herein.

[0119] In an alternative embodiment, based on the query vector corresponding to the user query and the data records, the second candidate results related to the user query and the second relevance scores of the second candidate results are obtained, including the following steps:

[0120] Based on the query vector corresponding to the user query, the data time information, the data keyword, the data description, the data identifier, and the data title, a second candidate result related to the user query and a second relevance score of the second candidate result are obtained.

[0121] It should be noted that the second candidate result is any data record in the plurality of data records.

[0122] In this embodiment, when performing vector conversion, only the data time information, the data keyword, the data description, the data identifier, and the data title are input into the vector conversion model, reducing semantic dilution and vector dimension noise. After being serialized into a JSON string according to a fixed key set, the vector conversion is performed, the text structure is stable, and the text structure can be repeatedly constructed; the Persistent Client of the Chroma vector database is used, and the link does not depend on cloud embedding / hosting vector services, meeting the privacy and controllable cost. The data identifier, data download link, data source, and are stored synchronously with the digital vector, so that the retrieval result can be directly output without second database query, and the information can be verified and downloaded.

[0123] Step S108: determining a comprehensive score based on the first relevance score and the second relevance score.

[0124] In an optional embodiment, the comprehensive score is determined based on the first relevance score and the second relevance score by using the following formula:

[0125] (1);

[0126] wherein, the comprehensive score is represented by S, the weight parameter is represented by w, the first relevance score is represented by S1, and the second relevance score is represented by S2.

[0127] It should be noted that different scenarios can set different . For example, in the short query and proper name scenario, the can be set to be larger.

[0128] In another optional embodiment, the comprehensive score is determined by using a reverse ranking fusion or Z-score (Z-score).

[0129] In another optional embodiment, before determining the comprehensive score based on the first relevance score and the second relevance score, the first relevance score and the second relevance score can be normalized.

[0130] Step S109: determining a target candidate result based on the comprehensive score.

[0131] In an optional embodiment, the candidate results are ranked according to the comprehensive scores, and the top M candidate results are selected as target candidate results.

[0132] In another optional embodiment, after determining the target candidate results based on the comprehensive scores, the method further includes the following steps: generating a result record based on the target candidate results, the result record including data identification, data time information, data summary points, data download links, and data identification. It should be noted that the data summary points represent the most critical and core information points in the data content. In addition, the result record can be adapted according to actual conditions, and the present application does not limit it.

[0133] In this embodiment, by generating a result record including only data identification, data time information, data summary points, data download links, and data identification, it can be adapted to the context window of the generation stage in the subsequent.

[0134] Step S110: constructing prompt words of the intelligent question and answer model based on the target candidate results and the user query.

[0135] In an optional embodiment, the intelligent question and answer model can be an LLM (Large Language Model).

[0136] In an optional embodiment, in the case of generating a result record based on the target candidate results, the prompt words of the intelligent question and answer model are constructed based on the result record and the user query.

[0137] In another optional embodiment, after constructing the prompt words of the intelligent question and answer model based on the target candidate results and the user query, the method further includes setting a constraint condition.

[0138] In a specific embodiment, the constraint condition can be "not to fabricate, the subsequent output query results need to carry data sources and data download links, if no query results are output, search suggestions need to be given".

[0139] Step S111: outputting the query results corresponding to the user query based on the prompt words using the intelligent question and answer model.

[0140] In an optional embodiment, when the constraint condition is set, the query results corresponding to the user query are output based on the prompt words using the intelligent question and answer model, including:

[0141] The query results corresponding to the user query are output based on the constraint condition and the prompt words using the intelligent question and answer model.

[0142] In another optional embodiment, a large model running platform such as an Ollama tool, a vLLM (vectorized Large Language Model inference engine), and a TGI (Text Generation Inference) is used to call an intelligent question and answer model, a model temperature of the intelligent question and answer model is set to a preset temperature, a maximum output length is set to a preset length, and a preset stop word is set. For example, the preset temperature can be 0.2, 0.3, and 0.5, and the preset length can be 200, 300, and 600. The present application does not limit the specific content of the preset temperature, the preset length, and the preset stop word. In this way, the model can ensure concise answers, avoid going off topic, and have fewer hallucinations.

[0143] In another optional embodiment, the format of the returned query result can be a fixed format. The fixed format can be a JSON format. By returning a structured query result, the query result has the advantages of being "clear to see", "easy to find", "easy to review", and "integrable". Among them, clear to see means that the user can know the relevant information of the query result, such as what the data source is, when the data time information is, and what the data summary points are; easy to find means that the user can obtain the data content based on the data download link; easy to review means that it can be directly traced; and integrable means that structured data is suitable for multiple systems.

[0144] In another optional embodiment, for the case where the query result cannot be given, an executable search suggestion and a log identifier can be output to facilitate the background to record logs and optimize the search strategy.

[0145] Corresponding to the intelligent question and answer processing method provided by the embodiments of the present application, the embodiments of the present application also provide an intelligent question and answer processing device, as shown in Figure 2 The intelligent question and answer processing device includes:

[0146] The acquisition module 201 is configured to acquire data information of data in the data set.

[0147] The first construction module 202 is configured to construct metadata based on the data information.

[0148] The conversion module 203 is configured to convert the data content in the data set into a digital vector.

[0149] The first determination module 204 is configured to obtain a plurality of data records based on the corresponding storage of the data content, the data information, the metadata, and the digital vector in the data set.

[0150] The second construction module 205 is configured to construct a BM25 retriever based on the data records by using a BM25 algorithm.

[0151] The second determination module 206 is configured to obtain a first candidate result related to the user query and a first relevance score of the first candidate result by using the BM25 retriever based on the user query.

[0152] The third determination module 207 is configured to obtain a second candidate result related to the user query and a second relevance score of the second candidate result based on the query vector corresponding to the user query and the data records.

[0153] The fourth determination module 208 is configured to determine a comprehensive score based on the first relevance score and the second relevance score.

[0154] The fifth determination module 209 is configured to determine a target candidate result based on the comprehensive score.

[0155] The third construction module 210 is configured to construct a prompt word of the intelligent question and answer model based on the target candidate result and the user query.

[0156] The output module 211 is configured to output a query result corresponding to the user query by using the intelligent question and answer model based on the prompt word.

[0157] Corresponding to the intelligent question and answer processing method provided in the embodiments of the present application, the embodiments of the present application further provide an electronic device for executing the intelligent question and answer processing method. Figure 3 As shown in the figure, the electronic device includes a processor 301 and a memory 302 for storing the program of the intelligent question and answer processing method. After the device is powered on and the processor runs the program of the intelligent question and answer processing method, the following steps are executed:

[0158] Obtain data information of data in a data set;

[0159] Construct metadata based on the data information;

[0160] Convert data content in the data set into a digital vector;

[0161] Corresponding storage of the data content, the data information, the metadata and the digital vector in the data set to obtain a plurality of data records;

[0162] Construct a BM25 retriever based on the data records by using a BM25 algorithm;

[0163] Obtain a first candidate result related to the user query and a first relevance score of the first candidate result by using the BM25 retriever based on the user query;

[0164] obtaining a second candidate result related to the user query and a second relevance score of the second candidate result based on the query vector corresponding to the user query and the data records;

[0165] determining a comprehensive score based on the first relevance score and the second relevance score;

[0166] determining a target candidate result based on the comprehensive score;

[0167] constructing a prompt word of an intelligent question and answer model based on the target candidate result and the user query;

[0168] outputting a query result corresponding to the user query by using the intelligent question and answer model based on the prompt word.

[0169] Corresponding to the intelligent question and answer processing method provided by the embodiments of the present application, the embodiments of the present application also provide a computer readable storage medium, which stores a program of an intelligent question and answer processing method, the program is run by a processor, and the following steps are executed:

[0170] obtaining data information of data in a data set;

[0171] constructing metadata based on the data information;

[0172] converting data content in the data set into a digital vector;

[0173] storing the data content, the data information, the metadata and the digital vector in the data set correspondingly to obtain a plurality of data records;

[0174] constructing a BM25 retriever based on the data records by using a BM25 algorithm;

[0175] obtaining a first candidate result related to the user query and a first relevance score of the first candidate result based on the user query by using the BM25 retriever;

[0176] obtaining a second candidate result related to the user query and a second relevance score of the second candidate result based on the query vector corresponding to the user query and the data records;

[0177] determining a comprehensive score based on the first relevance score and the second relevance score;

[0178] determining a target candidate result based on the comprehensive score;

[0179] constructing a prompt word of an intelligent question and answer model based on the target candidate result and the user query;

[0180] outputting a query result corresponding to the user query by using the intelligent question and answer model based on the prompt word.

[0181] Corresponding to the intelligent question and answer processing method provided by the embodiments of the present application, the embodiments of the present application also provide a computer program containing instructions, when the program is executed by a computer, the instructions make the computer execute the following steps:

[0182] Obtaining data information of data in the data set;

[0183] Constructing metadata based on the data information;

[0184] Converting data content in the data set into a digital vector;

[0185] Based on the corresponding storage of the data content, the data information, the metadata and the digital vector in the data set, obtaining a plurality of data records;

[0186] Using the BM25 algorithm, constructing a BM25 retriever based on the data records;

[0187] Based on the user query, using the BM25 retriever, obtaining a first candidate result related to the user query and a first relevance score of the first candidate result;

[0188] Based on the query vector corresponding to the user query and the data record, obtaining a second candidate result related to the user query and a second relevance score of the second candidate result;

[0189] Determining a comprehensive score based on the first relevance score and the second relevance score;

[0190] Determining a target candidate result based on the comprehensive score;

[0191] Constructing a prompt word of an intelligent question and answer model based on the target candidate result and the user query;

[0192] Based on the prompt word, outputting a query result corresponding to the user query by using the intelligent question and answer model.

[0193] It should be noted that the detailed description of the intelligent question and answer processing device, the electronic device and the computer readable storage medium provided by the embodiments of the present application can refer to the related description of the intelligent question and answer processing method provided by the embodiments of the present application, which will not be repeated here.

[0194] Although the present application discloses the above with preferred embodiments, it is not intended to limit the present application, any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be limited by the scope defined by the claims of the present application.

[0195] In a typical configuration, the electronic device includes one or more processors (Central Processing Unit), input / output interfaces, network interfaces and memories.

[0196] Memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) and / or a non-volatile memory, such as a read only memory (ROM) or a flash memory, in a computer readable medium. Memory is an example of computer readable media.

[0197] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of storage media of a computer include, but are not limited to, a programmable analog module, a static random access memory, a dynamic random access memory, other types of random access memory, a read only memory, an electrically erasable programmable read only memory, a flash memory or other memory technology, a compact disc read only memory, a digital video disc or other optical storage, a magnetic cassette tape, a magnetic tape disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carriers.

[0198] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer available storage media (including, but not limited to, disk memory, compact disc read only memory, optical storage or the like) including computer usable program code.

[0199] Although the present application is disclosed with reference to the preferred embodiments above, it is not intended to limit the present application, and any person skilled in the art who learns the present application can make possible changes and modifications without departing from the spirit and scope of the present application, and the protection scope of the present application should be defined by the scope of the claims of the present application.

Claims

1. An intelligent question answering processing method, characterized in that, The method comprises: obtaining data information of data in a data set; constructing metadata based on the data information; converting data content in the data set into a digital vector; corresponding storage based on the data content in the data set, the data information, the metadata and the digital vector, to obtain a plurality of data records; using a BM25 algorithm, constructing a BM25 retriever based on the data records; based on a user query, using the BM25 retriever to obtain a first candidate result related to the user query and a first relevance score of the first candidate result; based on a query vector corresponding to the user query and the data records, obtaining a second candidate result related to the user query and a second relevance score of the second candidate result; determining a comprehensive score based on the first relevance score and the second relevance score; determining a target candidate result based on the comprehensive score; constructing a prompt word of an intelligent question and answer model based on the target candidate result and the user query; setting a constraint condition; based on the constraint condition and the prompt word, outputting a query result corresponding to the user query by using the intelligent question and answer model. 2.The intelligent question answering processing method of claim 1, wherein, The comprehensive score is determined based on the first relevance score and the second relevance score using the following formula: ； wherein the denotes a combined score, the denotes a weight parameter, the denotes the first correlation score, the denotes the second correlation score. 3.The intelligent question answering processing method of claim 1, wherein, After obtaining a plurality of data records by corresponding storage based on the data content in the data set, the data information, the metadata and the digital vector, the method further comprises: if the metadata in the data record includes a data identifier, performing deduplication processing on the data record using the data identifier; if the metadata in the data record does not include a data identifier, determining a content identifier from the data content, and performing deduplication processing on the data record based on the content identifier; if the data content does not include the content identifier, performing hash calculation on the data content to obtain a hash value, and performing deduplication processing on the data record based on the hash value. 4.The intelligent question answering processing method of claim 1, wherein, After determining the target candidate result based on the comprehensive score, the method further comprises generating a result record based on the target candidate result, wherein the result record comprises a data identifier, data time information, data summary points, a data download link and a data identifier. The prompt word of the intelligent question and answer model is constructed based on the result record and the user query. The data content in the data set is converted into a digital vector, which comprises: 5.The intelligent question answering processing method of claim 1, wherein, converting the data content in the data set into a digital vector with a preset batch size. After converting the data in the data set into a digital vector, the method further comprises: 6.The intelligent question answering processing method of claim 1, wherein, normalizing the digital vector. The method comprises:

7. An intelligent question answering processing apparatus characterized by comprising: an acquisition module for obtaining data information of data in a data set; a first construction module for constructing metadata based on the data information; a conversion module for converting data content in the data set into a digital vector; a first determination module for obtaining a plurality of data records based on corresponding storage of the data content in the data set, the data information, the metadata and the digital vector; ​ The second construction module is configured to construct a BM25 retriever based on the data records by using a BM25 algorithm; The second determination module is configured to obtain a first candidate result related to the user query and a first relevance score of the first candidate result by using the BM25 retriever based on the user query; The third determination module is configured to obtain a second candidate result related to the user query and a second relevance score of the second candidate result based on a query vector corresponding to the user query and the data records; The fourth determination module is configured to determine a comprehensive score based on the first relevance score and the second relevance score; The fifth determination module is configured to determine a target candidate result based on the comprehensive score; The third construction module is configured to construct a prompt word of an intelligent question and answer model based on the target candidate result and the user query; The output module is configured to output a query result corresponding to the user query by using the intelligent question and answer model based on the prompt word; The apparatus further includes modules configured to perform the following operations: setting a constraint condition; The output module is configured to output a query result corresponding to the user query by using the intelligent question and answer model based on the constraint condition and the prompt word.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the intelligent question and answer processing method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the intelligent question and answer processing method in any one of claims 1-6.

Images