A data retrieval method, device, equipment, medium and product

Abstract

The present invention relates to the field of data processing technology, and specifically discloses a data retrieval method, device, equipment, medium and product. The method includes: constructing a query data pool according to the user classification of the questioning user; determining the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determining the initial retrieval sample set in the query data pool based on the user preference distance index; based on the user preference distance index of each question sample and the outlier situation of the answer, using the question samples in the query data pool to update the initial retrieval sample set to obtain the retrieval sample set; generating the final retrieval result according to the retrieval sample set and the user question. By introducing user preference information, this scheme can adapt to language ambiguity problems caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval; through the diversified update of the retrieval sample set, the flexibility of data retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data retrieval method, device, equipment, medium and product. Background Art

[0002] At present, structured data retrieval mainly relies on specific structured retrieval languages. Although this method is powerful, it limits the wide application and flexibility of data analysis tools. Taking the field of economics and finance as an example, for financial practitioners with non-technical backgrounds, it is difficult to understand and write these structured retrieval languages, which not only raises the threshold for use, but also reduces the efficiency and scope of data analysis. In addition, according to the different characteristics of financial business, the field of economics and finance can be subdivided into multiple fields. In these fields, the diversity of user backgrounds leads to the possibility that they may use non-standardized languages ​​to express their needs for the same data. This makes the existing data retrieval methods often face semantic ambiguity and inaccurate intent recognition when processing the automatic conversion of natural language to structured queries, thus affecting the accuracy of data retrieval. Summary of the invention

[0003] The present invention provides a data retrieval method, device, equipment, medium and product to achieve accuracy and flexibility of data retrieval.

[0004] According to one aspect of the present invention, there is provided a data retrieval method, the method comprising:

[0005] Build a query data pool based on the user classification of the user asking the question;

[0006] Determine the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine the initial retrieval sample set in the query data pool based on the user preference distance index;

[0007] Based on the user preference distance index of each question sample and the outlier situation of the answer, the initial retrieval sample set is updated using the question samples in the query data pool to obtain the retrieval sample set;

[0008] Generate the final retrieval results based on the retrieval sample set and user questions.

[0009] According to another aspect of the present invention, there is provided a data retrieval device, the device comprising:

[0010] A query data pool building module is used to build a query data pool according to the user classification of the user asking the question;

[0011] A retrieval sample set initialization module is used to determine the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine the initial retrieval sample set in the query data pool based on the user preference distance index;

[0012] A retrieval sample set updating module is used to update the initial retrieval sample set based on the user preference distance index and answer outlier of each question sample and to obtain a retrieval sample set by using the question samples in the query data pool;

[0013] The retrieval result generation module is used to generate the final retrieval results based on the retrieval sample set and user questions.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data retrieval method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data retrieval method described in any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, a computer program product is provided. The computer program product comprises a computer program. When the computer program is executed by a processor, the data retrieval method according to any embodiment of the present invention is implemented.

[0020] The technical solution of the embodiment of the present invention is to construct a query data pool according to the user classification of the questioning user; determine the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine the initial retrieval sample set in the query data pool based on the user preference distance index; based on the user preference distance index of each question sample and the outlier situation of the answer, update the initial retrieval sample set using the question samples in the query data pool to obtain the retrieval sample set; generate the final retrieval result according to the retrieval sample set and the user question. This solution constructs a query data pool by using the user classification of the user, and then introduces the user preference information to construct the initial retrieval sample set, which can adapt to the language ambiguity caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval; then based on the outlier situation of the answer to the question sample, update the initial retrieval sample set using the question samples in the query data pool to obtain the retrieval sample set, and generate the final retrieval result based on the user question and the retrieval sample set, thereby improving the flexibility of data retrieval.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a data retrieval method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a data retrieval method provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of a data retrieval device provided according to Embodiment 3 of the present invention;

[0026] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the data retrieval method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 A flowchart of a data retrieval method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of realizing accurate data retrieval. The method can be executed by a data retrieval device. The data retrieval device can be implemented in the form of hardware and / or software. The data retrieval device can be configured in an electronic device. Figure 1 As shown, a data retrieval method provided in this embodiment includes the following steps:

[0031] S110 , constructing a query data pool according to the user classification of the user asking the question.

[0032] Among them, user classification can refer to the type classification of users according to the characteristics of financial services. For example, corresponding user classification can be set for users according to the financial fields or business types involved. In one embodiment, the financial field classification can include general fields, banking fields, insurance fields, securities fields, regulatory fields, and financial technology fields; business type classification can include: sales, approval, risk control, finance, and comprehensive, etc.

[0033] The query data pool can be understood as a set of question sample identifiers (IDs) used for the subsequent construction of a retrieval sample set, where a question sample can refer to historical questions collected based on historical user questions, and each question sample has a unique question ID.

[0034] In an embodiment of the present invention, a number of question samples can be extracted from a pre-configured set of preset historical databases based on the user classification of the questioning user to construct a corresponding query data pool. In a specific embodiment, a corresponding target historical database can be found in the preset historical database set according to the user classification of the questioning user, and then the query data pool can be constructed using the question samples in the target historical database; further, in order to increase the diversity of question samples, a number of question samples can also be randomly extracted from other historical databases in the preset historical database set that do not match the user classification to construct the query data pool.

[0035] S120: Determine a user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine an initial retrieval sample set in the query data pool based on the user preference distance index.

[0036] Among them, user preference information can be understood as the historical behavior and demand information of the user asking the question. The user preference information can at least include the form of a user preference matrix. Utilizing user preference information can help extract question samples with a high degree of similarity to the user's question from the query data pool, thereby improving the accuracy of the final retrieval results.

[0037] The user preference distance index can be used to measure the similarity between the user's question and the question samples in the query data pool. The smaller the user preference distance index is, the higher the similarity between the user's question and the question samples is.

[0038] The initial retrieval sample set may refer to a set of question samples that have only been extracted by document similarity but not updated by diversification, that is, each question sample in the initial retrieval sample set is highly relevant to the user's question, but lacks as diverse question samples as possible. The retrieval document may refer to the question sample.

[0039] In an embodiment of the present invention, the user preference information of the questioning user can be introduced in the process of data retrieval, and the user preference information is used to determine the user preference distance index corresponding to each question sample in the query data pool, and then m question samples are extracted from the query data pool (the number of question samples is M) based on the user preference distance index to construct an initial retrieval sample set. In one embodiment, the method of determining the initial retrieval sample set in the query data pool based on the user preference distance index can be: sorting the question samples in the query data pool in ascending order according to the user preference distance index, and then extracting the top-m question samples to construct the initial retrieval sample set.

[0040] In another embodiment, the method for determining the initial retrieval sample set in the query data pool based on the user preference distance index can also be: based on the user preference distance index of each question sample in the query data pool, a pre-configured preset retrieval document extraction probability calculation formula can be called to determine the extraction probability of each question sample, and then based on the extraction probability, m question samples are randomly extracted from the query data pool to construct an initial retrieval sample set.

[0041] This embodiment can adapt to language ambiguity problems caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval by introducing the user's preference information during the data retrieval process.

[0042] S130 , based on the user preference distance index of each question sample and the outlier situation of the answer, the initial retrieval sample set is updated using the question samples in the query data pool to obtain a retrieval sample set.

[0043] Among them, the answer outlier can be understood as the deviation between the answer (i.e., the answer) corresponding to the question sample and the overall answer corresponding to all question samples.

[0044] The retrieval sample set may refer to a set of question samples obtained after document similarity extraction and diversified updating. In this case, the question samples in the retrieval sample set are the question samples to be output in the final retrieval results corresponding to the user's question.

[0045] In the embodiment of the present invention, in the process of data retrieval, in addition to the document similarity requirement, it is usually desired to increase the diversity of question samples within a certain similarity range, such as some question samples with similar meanings to the user's questions but different expressions, or some question samples with different meanings to the user's questions but may belong to the potential needs of the questioning user, etc. The addition of these diverse question samples can make the data retrieval system more flexible and more adaptable to various user needs. Based on this, in each iteration update cycle of the initial retrieval sample set, the present embodiment can randomly extract a first target question sample and a second target question sample from the query data pool and the initial retrieval sample set, respectively, and then determine the sample passing probability corresponding to the first target question sample based on the user preference distance index and the answer outlier situation corresponding to the first target question sample and the second target question sample respectively; then a random number generator can be used to randomly generate a probability value, if the probability value is less than the sample passing probability of the aforementioned first target question sample, then the second target question sample in the initial retrieval sample set is replaced by the first target question sample, otherwise the second target question sample is kept unchanged; finally, when the maximum preset number of iterations is reached, the currently updated initial retrieval sample set is used as the final retrieval sample set.

[0046] S140: Generate final search results based on the search sample set and the user's question.

[0047] The final search result may refer to the search result finally returned to the user who asked the question, that is, the answer to the user's question.

[0048] In an embodiment of the present invention, after obtaining the retrieval sample set, the answer corresponding to each question sample can be found in the historical (question-answer pair) database according to the question ID of each question sample in the retrieval sample set, and then the user's question and the answer corresponding to each question sample can be spliced ​​into the final retrieval result according to the pre-configured preset question-answer splicing format, and it is fed back to the user who asked the question.

[0049] The technical solution of the embodiment of the present invention is to construct a query data pool according to the user classification of the questioning user; determine the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine the initial retrieval sample set in the query data pool based on the user preference distance index; based on the user preference distance index of each question sample and the outlier situation of the answer, update the initial retrieval sample set using the question samples in the query data pool to obtain the retrieval sample set; generate the final retrieval result according to the retrieval sample set and the user question. This solution constructs a query data pool by using the user classification of the user, and then introduces the user preference information to construct the initial retrieval sample set, which can adapt to the language ambiguity caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval; then based on the outlier situation of the answer to the question sample, update the initial retrieval sample set using the question samples in the query data pool to obtain the retrieval sample set, and generate the final retrieval result based on the user question and the retrieval sample set, thereby improving the flexibility of data retrieval.

[0050] Further, based on the above-mentioned embodiment of the invention, in S110, a query data pool is constructed according to the user classification of the questioning user, including:

[0051] S1101, obtaining the user classification of the user who asked the question, and searching for a corresponding target historical database in a preset historical database set according to the user classification;

[0052] S1102, randomly extracting a first number of question samples from each other historical database in the preset historical database set that does not match the user classification;

[0053] S1103: Construct a query data pool based on all question samples in the target historical database and question samples extracted from other historical databases.

[0054] The preset historical database set may refer to a pre-configured set of several historical databases, each of which corresponds to a user category, and the historical database contains several pieces of question-answer pair data collected based on the user's historical questions, and each piece of question-answer pair data contains two parts: a question (question) and an answer, wherein the answer includes the database table information involved in the answer and the generated query statement. Furthermore, the historical database may also contain a question vector and an answer vector after vectorization processing of each piece of question-answer pair data.

[0055] In an embodiment of the present invention, after receiving a question from a user who asks a question, a user classification of the user who asks the question can be obtained. For example, the corresponding user classification can be determined based on the default account classification information of the user who asks the question, and then the user classification can be used to find the corresponding associated target historical database in the preset historical database set. Then, a first number of question samples can be randomly extracted from each other historical database that does not match the user classification. Finally, based on the question ID of each question sample in the target historical database and the question ID of the question sample randomly extracted from each other historical database, a corresponding query data pool G is constructed.

[0056] It is understandable that the user asking the question can also select question samples by specifying a historical database to construct a query data pool for this data retrieval; if the user does not specify, the default method of steps S1101 to S1103 is used to construct the query data pool.

[0057] Further, based on the above-mentioned embodiment of the invention, in S120, according to the user question and user preference information of the questioning user, a user preference distance index of each question sample in the query data pool is determined, and an initial retrieval sample set is determined in the query data pool based on the user preference distance index, including:

[0058] S1201, obtaining a user preference matrix corresponding to the user who asked the question, and extracting a question vector of each question sample in the query data pool;

[0059] S1202, calling a preset vectorization model to convert the user question into a user question vector;

[0060] S1203, based on the user preference matrix, the user question vector and the question vector of each question sample, calling a preset user preference distance index calculation formula to determine the user preference distance index of each question sample;

[0061] S1204, based on the preference distance index of each user, calling the preset retrieval document extraction probability calculation formula to determine the extraction probability of each question sample;

[0062] S1205. Randomly select a second number of question samples from the query data pool according to each extraction probability to construct an initial retrieval sample set.

[0063] Among them, the preset vectorization model may refer to a pre-configured model for converting text data into vector form. The preset vectorization model can be used to convert text questions and answers into question vectors and answer vectors in vector form. Exemplarily, the preset vectorization model may include but is not limited to: Word2Vec model, Glove model, etc.

[0064] The user preference matrix can be understood as a matrix that represents the user's historical behavior preference information configured in advance for each user. The user preference matrix H can be a The matrix is ​​a symmetric matrix of size, and each matrix element is independent of each other; at the same time, if the user asking the question is a new user, the unit matrix can be used as the user preference matrix.

[0065] In the embodiment of the present invention, the process of constructing the initial search sample set specifically includes:

[0066] Step 1: Obtain the user preference matrix H pre-configured for the current questioning user, and obtain the question vector corresponding to each question sample in the query data pool G (k is the question ID corresponding to the question sample).

[0067] Step 2: Use the preset vectorization model to process the user question Q and obtain the corresponding user question vector .

[0068] Step 3: Call the preset user preference distance index calculation formula to determine the user preference distance index of each question sample Specifically, the preset user preference distance index calculation formula can be defined as follows:

[0069]

[0070] Where T represents the transposition symbol.

[0071] Step 4: Call the preset retrieval document extraction probability calculation formula to determine the extraction probability of each question sample. Specifically, since the number of question samples contained in the query data pool G is fixed, the preset retrieval document extraction probability calculation formula can be defined as:

[0072]

[0073] In the formula, represents the initial retrieval sample set, and the initial retrieval sample set The number of problem samples contained in is m; The standard deviation of the normal distribution parameter. It represents the exclusion threshold of problem samples. As a hyperparameter of the data retrieval system, it can be dynamically determined based on the actual business scenario or the number of problem samples contained in the query data pool G.

[0074] Step 5: According to the probability of each extraction Randomly select a second number of question samples from the query data pool G to construct an initial retrieval sample set Specifically, the threshold value can be used in the query data pool G. Filter the question samples with too large user preference distance index, and then divide the probability interval into Divide into multiple probability sub-intervals. For example, there are 3 question samples in the query data pool G after threshold filtering, and the corresponding question IDs are ID1, ID2 and ID3. Assuming that the extraction probability of ID1 is 0.1, the extraction probability of ID2 is 0.3, and the extraction probability of ID3 is 0.6, then the probability interval can be Divide into (corresponding to ID1), (corresponding to ID2) and (corresponding to ID3) 3 probability sub-intervals; then a random number generator can be used to randomly generate a random number between 0 and 1, and then the random number can be mapped to a certain probability sub-interval corresponding to the above, so that the problem sample corresponding to the mapped probability sub-interval is used as the initial retrieval sample set Repeat the above random sampling process until the initial retrieval sample set The number of problem samples in is the specified second number m.

[0075] Further, based on the above-mentioned embodiment of the invention, in S130, based on the user preference distance index of each question sample and the outlier situation of the answer, the initial retrieval sample set is updated using the question samples in the query data pool to obtain the retrieval sample set, including:

[0076] S1301, based on the answer vector of each question sample, calling a preset answer outlier deviation calculation formula to determine the answer outlier deviation of each question sample;

[0077] S1302, based on the outlier deviation of each answer, calling a preset document answer probability calculation formula to determine the document answer probability of each question sample;

[0078] S1303, initializing the number of iterations of the retrieval sample set;

[0079] S1304. In each round of iteration, a first target question sample is randomly selected from the query data pool and a second target question sample is randomly selected from the initial retrieval sample set. Based on the user preference distance index and document answer probability corresponding to the first target question sample and the second target question sample, respectively, the preset sample is called to determine the sample passing probability corresponding to the first target question sample through the probability calculation formula. When the first random probability is less than the sample passing probability, the second target question sample in the initial retrieval sample set is replaced by the first target question sample, and the number of iterations of the retrieval sample set is updated; wherein the first random probability is a probability value randomly generated by a random number generator;

[0080] S1305: When the number of iterations of the retrieval sample set reaches a maximum preset number of iterations, the current initial retrieval sample set is used as the retrieval sample set.

[0081] In the embodiment of the present invention, the initial retrieval sample set The process of performing diversified updates to obtain the retrieval sample set W specifically includes:

[0082] Step 1: Call the preset answer outlier deviation calculation formula to determine the answer outlier deviation of each question sample. Specifically, the answer vector of question sample k is The corresponding answer is outlier deviation It can be defined as follows:

[0083]

[0084] Step 2: Call the preset document answer probability calculation formula to determine the document answer probability of each question sample. Specifically, since the number of question samples contained in the query data pool G is fixed, the preset document answer probability calculation formula can be defined as:

[0085]

[0086] In the formula, represents the parameters of the exponential distribution; Represents a very small positive number to prevent the denominator from being zero.

[0087] Step 3: Initialize and set the number of iterations F for retrieving the sample set.

[0088] Step 4: In each iteration, randomly select a first target question sample from the query data pool G And from the initial retrieval sample set A second target problem sample k is randomly selected from the sample k, and then based on the first target problem sample Corresponding user preference distance index and document answer probability , and the user preference distance index corresponding to the second target problem sample k and document answer probability , call the following preset sample to determine the first target problem sample through the probability calculation formula The corresponding sample passing probability :

[0089]

[0090] Then, a random number generator can be used to randomly generate a random number between 0 and 1, that is, the first random probability. When the first random probability is less than the first target problem sample The corresponding sample passing probability When , new samples are received and the initial retrieval sample set is updated , that is, in the initial retrieval sample set In the example, the second target problem sample k is replaced by the first target problem sample Otherwise, keep the second target question sample k unchanged and retrieve the sample set in the initial After updating, the number of iterations F of the retrieval sample set is increased by 1.

[0091] It is important to understand that from the initial retrieval sample set The method of extracting the second target question sample can be either random extraction each time or traversal extraction in order of question ID, which is not specifically limited in this embodiment.

[0092] Step 5: When the number of iterations F of the retrieval sample set reaches the maximum preset number of iterations When the current initial retrieval sample set As the final retrieval sample set W. Among them, the maximum preset number of iterations Corresponding settings can be made according to actual needs and system resource limitations, and this embodiment does not impose any restrictions on this.

[0093] Further, based on the above-mentioned embodiment of the invention, generating the final search result according to the search sample set and the user's question in S140 includes:

[0094] S1401, extracting the answer corresponding to each question sample in the retrieval sample set;

[0095] S1402. Concatenate the user's question and the answers to each question sample into a final search result according to a preset question-answer concatenation format.

[0096] Among them, the preset question and answer splicing format can be understood as a fixed format for splicing user questions and answers, so that the final search results obtained after processing are clearer and easier to understand.

[0097] In an embodiment of the present invention, after the retrieval sample set W is updated, the corresponding question answer can be found in the historical database according to the question ID of each question sample in the retrieval sample set W, and then the user's question and the answer corresponding to each question sample can be spliced ​​into the final retrieval result according to the pre-configured preset question-answer splicing format, and the result is fed back to the user who asked the question. In one embodiment, the user's question and the answer corresponding to each question sample can be spliced ​​using a large language model to generate the final retrieval result.

[0098] Further, based on the above-mentioned embodiment of the invention, after generating the final search result, the method further includes:

[0099] According to a preset matrix element update range threshold, updating element values ​​of a third number of elements in the user preference matrix of the user preference information to obtain a user preference matrix to be updated;

[0100] Based on the user preference matrix, the user preference matrix to be updated, the retrieval sample set and the query data pool, the preset matrix is ​​called to determine the matrix passing probability corresponding to the user preference matrix to be updated through the probability calculation formula, and when the second random probability is less than the matrix passing probability, the user preference matrix corresponding to the questioning user is replaced with the user preference matrix to be updated.

[0101] The preset matrix element update range threshold may refer to each element in the user preference matrix H. To avoid the matrix element value being updated too much and causing historical forgetting, each element The update range needs to be limited to the threshold Inside, threshold It can be a preset value or each element in the user preference matrix H The standard deviation of .

[0102] The user preference matrix to be updated may refer to the user preference matrix after the matrix element values ​​are updated. , the updated element in the matrix is ,in From the range Randomly generated.

[0103] In the embodiment of the present invention, after the final search result is generated, the query data pool G, the search sample set W and the user question Q involved in this data search can be used as the training data of the user preference matrix H corresponding to the user asking the question, so as to decide whether to update the user preference matrix H to the user preference matrix to be updated. Specifically, the third number of elements can be randomly extracted from the user preference matrix H. , and then update the range threshold according to the preset matrix elements For the above elements Randomly update the element values ​​to generate the user preference matrix to be updated ,in ; Then based on the original user preference matrix H and the user preference matrix to be updated , retrieve the sample set W and query the data pool G, call the following preset matrix to determine the user preference matrix to be updated through the probability calculation formula The corresponding matrix passes the probability:

[0104]

[0105] Next, a random number generator can be used to randomly generate a random number between 0 and 1, that is, a second random probability. When the second random probability is less than the user preference matrix to be updated The corresponding matrix passes the probability When , H is updated to , and apply the new user preference matrix when the corresponding questioning user initiates data retrieval next time .

[0106] In one embodiment, feedback from the user who asked the question regarding the final search result may also be received to determine whether the search result needs to be updated to the search sample set W, and the updated search sample set W is used as training data for the user preference matrix H.

[0107] Furthermore, based on the above-mentioned embodiments of the invention, it may also include a construction process of a preset history database set and a user history retrieval database.

[0108] Specifically, based on the collected historical questions of users, historical databases corresponding to user categories can be constructed respectively, and the historical databases can be saved in a preset historical database set, where each data in the historical database contains the following five types of information: question ID, original question text, question vector, original answer text and answer vector, and the original question text and the original answer text need to be limited to 512 words.

[0109] Furthermore, each piece of data in the user history retrieval database R contains the following five types of information: question vector, query data pool G, retrieval sample set W, user ID and user category, where each user corresponds to a user category. For each stored question sample, the relevant question ID can be annotated from the constructed history database using a variety of methods such as retrieval technology and manual annotation.

[0110] Embodiment 2

[0111] Figure 2This is a flow chart of a data retrieval method provided in Example 2 of the present invention. Based on the above examples, this example provides an implementation of a data retrieval method that can implement structured language retrieval of financial information based on user preferences combined with Bayesian inference. Figure 2 As shown, a data retrieval method provided by Embodiment 2 of the present invention specifically includes the following steps:

[0112] S210: receiving user questions and constructing a query data pool according to the user classification of the user asking the question.

[0113] S220: Initialize and construct an initial retrieval sample set based on the user preference matrix.

[0114] In an embodiment of the present invention, the preferences and needs of a specific user can be introduced into the data retrieval system in the form of parameters (i.e., a user preference matrix). When the user is a new user, the unit matrix can be used as the user preference matrix H, which will be subsequently updated based on the user's retrieval history.

[0115] Furthermore, based on the user preference matrix, the user question vector and the question vector of each question sample, the user preference distance index corresponding to each question vector can be determined; then, the extraction probability of each question sample can be determined based on the defined preset retrieval document extraction probability calculation formula; finally, based on the extraction probability, a second number of question samples are randomly extracted from the query data pool G to construct an initial retrieval sample set. The specific process can be referred to the above embodiment and will not be described in detail here.

[0116] S230: Update the initial search sample set based on the diversity of answers to user search documents to obtain a search sample set.

[0117] In the embodiment of the present invention, in the process of data retrieval, in addition to the document similarity requirement, the system usually hopes to increase the diversity of question samples within a certain similarity range. Therefore, this embodiment uses the answer vector of the question sample to define the answer outlier deviation, and based on this, defines the document answer probability of each question sample. In each iteration of And from the initial retrieval sample set A second target problem sample k is randomly selected from the sample, and the first target problem sample is further defined based on Bayesian inference The corresponding sample passing probability :

[0118]

[0119] Finally, a random number generator is used to randomly generate a random number between 0 and 1, namely the first random probability. When the first random probability is less than the first target problem sample The corresponding sample passing probability When , new samples are received and the initial retrieval sample set is updated , and the final retrieval sample set W is obtained after the iteration.

[0120] S240, splicing the user's questions and the answers corresponding to each question sample in the retrieval sample set into the final retrieval result according to the preset question and answer splicing format.

[0121] S250: Update the user history search database.

[0122] In an embodiment of the present invention, after the final search result is returned to the user who asked the question, the user or the data retrieval system can decide whether to retain the question vector Q, query data pool G, search sample set W, user ID and user classification involved in this query based on the final search result. If retention is required, the above information will be updated to the user history search database R.

[0123] S260: Update the user preference matrix using the user history retrieval database.

[0124] In the embodiment of the present invention, the query data pool G, the retrieval sample set W and the user question Q involved in this data retrieval can be used as the training data of the user preference matrix H corresponding to the questioning user, so as to determine whether to update the user preference matrix H to the user preference matrix to be updated , and the new user preference matrix is ​​applied when the corresponding questioning user initiates data retrieval next time The specific updating process of the user preference matrix is ​​described in the above embodiment and will not be described in detail here.

[0125] The technical solution of the embodiment of the present invention receives user questions and constructs a query data pool according to the user classification of the user who asked the question; constructs an initial retrieval sample set based on the user preference matrix initialization; updates the initial retrieval sample set based on the diversity of answers to user retrieval documents to obtain a retrieval sample set; splices the user questions and the answers corresponding to each question sample in the retrieval sample set into the final retrieval result according to a preset question-answer splicing format; updates the user history retrieval database; and updates the user preference matrix using the user history retrieval database. This solution constructs a query data pool using the user classification of the user, and then introduces the user preference information to construct the initial retrieval sample set, which can adapt to language ambiguity problems caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval; at the same time, the diversified update of the retrieval sample set can enhance the flexibility of data retrieval, thereby adapting to a variety of user needs.

[0126] Embodiment 3

[0127] Figure 3 This is a schematic diagram of the structure of a data retrieval device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0128] A query data pool building module 31 is used to build a query data pool according to the user classification of the user asking the question;

[0129] The retrieval sample set initialization module 32 is used to determine the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine the initial retrieval sample set in the query data pool based on the user preference distance index;

[0130] A retrieval sample set updating module 33 is used to update the initial retrieval sample set based on the user preference distance index and answer outlier of each question sample and to obtain a retrieval sample set by using the question samples in the query data pool;

[0131] The retrieval result generating module 34 is used to generate the final retrieval result according to the retrieval sample set and the user's question.

[0132] The technical solution of the embodiment of the present invention is to construct a query data pool according to the user classification of the questioning user through the query data pool construction module; the retrieval sample set initialization module determines the user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determines the initial retrieval sample set in the query data pool based on the user preference distance index of each question sample and the outlier situation of the answer, and obtains the retrieval sample set by using the question samples in the query data pool; the retrieval result generation module generates the final retrieval result according to the retrieval sample set and the user question. This solution constructs the query data pool by using the user classification of the user, and then introduces the user preference information to construct the initial retrieval sample set, which can adapt to the language ambiguity caused by factors such as the user's language habits and industry background, thereby improving the accuracy of data retrieval; then based on the outlier situation of the answer of the question sample, the question sample in the query data pool is used to update the initial retrieval sample set to obtain the retrieval sample set, and the final retrieval result is generated based on the user question and the retrieval sample set, thereby improving the flexibility of data retrieval.

[0133] Further, based on the above-mentioned embodiment of the invention, the query data pool construction module 31 includes:

[0134] A target history database search unit is used to obtain the user classification of the questioning user, and search for a corresponding target history database in a preset history database set according to the user classification;

[0135] A question sample extraction unit, configured to randomly extract a first number of question samples from each other historical database in a preset historical database set that does not match the user classification;

[0136] The query data pool construction unit is used to construct a query data pool based on all question samples in the target historical database and question samples extracted from other historical databases.

[0137] Further, based on the above-mentioned embodiment of the invention, the sample set retrieval initialization module 32 includes:

[0138] A user preference matrix and question vector acquisition unit, used to acquire the user preference matrix corresponding to the questioning user, and to extract the question vector of each question sample in the query data pool;

[0139] A user question conversion unit, used to call a preset vectorization model to convert the user question into a user question vector;

[0140] A user preference distance index determination unit, configured to determine the user preference distance index of each question sample by calling a preset user preference distance index calculation formula based on the user preference matrix, the user question vector and the question vector of each question sample;

[0141] A sample extraction probability determination unit is used to determine the extraction probability of each question sample by calling a preset retrieval document extraction probability calculation formula based on each user preference distance index;

[0142] The initial retrieval sample set construction unit is used to randomly select a second number of question samples from the query data pool according to each extraction probability to construct an initial retrieval sample set.

[0143] Further, based on the above-mentioned embodiment of the invention, the retrieval sample set updating module 33 includes:

[0144] An answer outlier deviation determination unit is used to determine the answer outlier deviation of each question sample by calling a preset answer outlier deviation calculation formula based on the answer vector of each question sample;

[0145] A document answer probability determination unit, used to determine the document answer probability of each question sample by calling a preset document answer probability calculation formula based on the outlier deviation of each answer;

[0146] An iteration number initialization unit, used to initialize the number of iterations of the retrieval sample set;

[0147] A sample updating unit is used to randomly extract a first target question sample from the query data pool and a second target question sample from the initial retrieval sample set in each round of iteration, based on the user preference distance index and document answer probability corresponding to the first target question sample and the second target question sample respectively, call the preset sample to determine the sample passing probability corresponding to the first target question sample through the probability calculation formula, and when the first random probability is less than the sample passing probability, replace the second target question sample in the initial retrieval sample set with the first target question sample, and update the number of iterations of the retrieval sample set; wherein the first random probability is a probability value randomly generated by a random number generator;

[0148] The retrieval sample set determination unit is used to use the current initial retrieval sample set as the retrieval sample set when the number of iterations of the retrieval sample set reaches a maximum preset number of iterations.

[0149] Further, based on the above-mentioned embodiment of the invention, the search result generating module 34 includes:

[0150] An answer extraction unit is used to extract the answer corresponding to each question sample in the retrieval sample set;

[0151] The search result generation unit is used to splice the user's questions and the answers corresponding to each question sample into the final search result according to a preset question-answer splicing format.

[0152] Furthermore, based on the above-mentioned embodiment of the invention, the data retrieval device further includes:

[0153] A user preference matrix generation module to be updated is used to update the element values ​​of a third number of elements in the user preference matrix of the user preference information according to a preset matrix element update range threshold after generating the final search result, so as to obtain the user preference matrix to be updated;

[0154] The matrix update module is used to call the preset matrix to determine the matrix passing probability corresponding to the user preference matrix to be updated through the probability calculation formula based on the user preference matrix, the user preference matrix to be updated, the retrieval sample set and the query data pool, and when the second random probability is less than the matrix passing probability, replace the user preference matrix corresponding to the questioning user with the user preference matrix to be updated.

[0155] The data retrieval device provided in the embodiment of the present invention can execute the data retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0156] Embodiment 4

[0157] Figure 4A schematic diagram of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0158] like Figure 4 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41 in communication, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 53 are connected to each other through the bus 44. The input / output (I / O) interface 45 is also connected to the bus 44.

[0159] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a data retrieval method.

[0161] In some embodiments, the data retrieval method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the data retrieval method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the data retrieval method in any other suitable manner (e.g., by means of firmware).

[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] In some embodiments, the data retrieval method may be implemented as a computer program, which is invisibly included in a computer program product, which implements the data retrieval method of the present invention when executed by a processor, and the computer program product may be understood as a software product that implements its solution mainly through a computer program. The computer program for implementing the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0166] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0167] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0168] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data retrieval method, characterized in that: The method comprises: Build a query data pool based on the user classification of the user asking the question; Determining a user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determining an initial retrieval sample set in the query data pool based on the user preference distance index; Based on the user preference distance index and answer outlier situation of each question sample, the initial retrieval sample set is updated using the question samples in the query data pool to obtain a retrieval sample set; Generate a final search result according to the search sample set and the user question; The step of determining the user preference distance index of each question sample in the query data pool according to the user question and the user preference information of the questioning user, and determining the initial retrieval sample set in the query data pool based on the user preference distance index, includes: Obtaining a user preference matrix corresponding to the questioning user, and extracting a question vector of each question sample in the query data pool; Calling a preset vectorization model to convert the user question into a user question vector; Based on the user preference matrix, the user question vector and the question vector of each question sample, calling a preset user preference distance index calculation formula to determine the user preference distance index of each question sample; Based on each of the user preference distance indicators, a preset retrieval document extraction probability calculation formula is called to determine the extraction probability of each of the question samples; randomly extracting a second number of question samples from the query data pool according to each of the extraction probabilities to construct the initial retrieval sample set; The method of updating the initial search sample set based on the user preference distance index and answer outlier status of each question sample and using the question samples in the query data pool to obtain a search sample set includes: Based on the answer vector of each of the question samples, a preset answer outlier deviation calculation formula is called to determine the answer outlier deviation of each of the question samples; Based on the outlier deviation of each answer, a preset document answer probability calculation formula is called to determine the document answer probability of each question sample; Initialize the number of iterations of the retrieval sample set; In each round of iteration, a first target question sample is randomly selected from the query data pool and a second target question sample is randomly selected from the initial retrieval sample set. Based on the user preference distance index and document answer probability corresponding to the first target question sample and the second target question sample, respectively, the preset sample is called to determine the sample passing probability corresponding to the first target question sample through the probability calculation formula, and when the first random probability is less than the sample passing probability, the second target question sample in the initial retrieval sample set is replaced by the first target question sample, and the number of iterations of the retrieval sample set is updated; wherein, the first random probability is a probability value randomly generated by a random number generator; When the number of iterations of the retrieval sample set reaches a maximum preset number of iterations, the current initial retrieval sample set is used as the retrieval sample set.

2. The method according to claim 1, characterized in that The step of constructing a query data pool according to the user classification of the user asking the question includes: Obtaining the user classification of the questioning user, and searching for a corresponding target historical database in a preset historical database set according to the user classification; Randomly extracting a first number of question samples from each other historical database in the preset historical database set that does not match the user classification; The query data pool is constructed based on all question samples in the target historical database and question samples extracted from each of the other historical databases.

3. The method according to claim 1, characterized in that Generating the final search result according to the search sample set and the user question includes: Extracting answers corresponding to each of the question samples in the retrieval sample set; The user's question and the answer corresponding to each of the question samples are spliced ​​into the final search result according to a preset question-answer splicing format.

4. The method according to claim 1, characterized in that: After generating the final search results, the method further includes: According to a preset matrix element update range threshold, updating element values ​​of a third number of elements in the user preference matrix of the user preference information to obtain a user preference matrix to be updated; Based on the user preference matrix, the user preference matrix to be updated, the retrieval sample set and the query data pool, a preset matrix is ​​called to determine the matrix passing probability corresponding to the user preference matrix to be updated through a probability calculation formula, and when the second random probability is less than the matrix passing probability, the user preference matrix corresponding to the questioning user is replaced with the user preference matrix to be updated.

5. A data retrieval device, characterized in that: The device comprises: A query data pool building module is used to build a query data pool according to the user classification of the user asking the question; A retrieval sample set initialization module, used to determine a user preference distance index of each question sample in the query data pool according to the user question and user preference information of the questioning user, and determine an initial retrieval sample set in the query data pool based on the user preference distance index; A retrieval sample set updating module, configured to update the initial retrieval sample set using the question samples in the query data pool based on the user preference distance index and answer outlier status of each question sample to obtain a retrieval sample set; A retrieval result generating module, used to generate a final retrieval result according to the retrieval sample set and the user question; The retrieval sample set initialization module includes: A user preference matrix and question vector acquisition unit, used to acquire the user preference matrix corresponding to the questioning user, and extract the question vector of each question sample in the query data pool; A user question conversion unit, used to call a preset vectorization model to convert the user question into a user question vector; A user preference distance index determination unit, configured to determine a user preference distance index for each of the question samples by calling a preset user preference distance index calculation formula based on the user preference matrix, the user question vector, and the question vector of each of the question samples; A sample extraction probability determination unit, used to determine the extraction probability of each of the question samples by calling a preset retrieval document extraction probability calculation formula based on each of the user preference distance indicators; An initial retrieval sample set construction unit, configured to randomly select a second number of question samples from the query data pool according to each of the extraction probabilities to construct the initial retrieval sample set; The retrieval sample set updating module comprises: An answer outlier deviation determination unit is used to determine the answer outlier deviation of each of the question samples by calling a preset answer outlier deviation calculation formula based on the answer vector of each of the question samples; A document answer probability determination unit, configured to determine the document answer probability of each of the question samples by calling a preset document answer probability calculation formula based on the outlier deviation of each of the answers; An iteration number initialization unit, used to initialize the number of iterations of the retrieval sample set; A sample updating unit is used to randomly extract a first target question sample from the query data pool and a second target question sample from the initial retrieval sample set in each round of iteration, based on the user preference distance index and document answer probability corresponding to the first target question sample and the second target question sample respectively, call a preset sample to determine the sample passing probability corresponding to the first target question sample through a probability calculation formula, and when the first random probability is less than the sample passing probability, replace the second target question sample in the initial retrieval sample set with the first target question sample, and update the number of iterations of the retrieval sample set; wherein the first random probability is a probability value randomly generated by a random number generator; The retrieval sample set determining unit is configured to use the current initial retrieval sample set as the retrieval sample set when the number of iterations of the retrieval sample set reaches a maximum preset number of iterations.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data retrieval method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data retrieval method according to any one of claims 1 to 4 when executed.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data retrieval method according to any one of claims 1 to 4.

Images