Data processing method, device and computer program product

Abstract

The invention relates to a data processing method and device and a computer program product. The method comprises the following steps: acquiring a text unit, and performing vector conversion on the text unit to obtain a text vector of the text unit; constructing a target database based on the text units, the text vectors and the association relationship set of the text units; and constructing a target corpus according to the target database. According to the embodiment of the invention, the data volume in the target corpus can be enriched, the comprehensiveness of the data in the target corpus is improved, and the accuracy of the query result can be improved when the data query is performed on the target corpus.

Description

Technical Field

[0001] The present disclosure relates to but is not limited to the field of information technology, and in particular to a data processing method, device, and computer program product. Background Art

[0002] Data query refers to querying and analyzing the data in the database based on the query statement input by the user, and then obtaining the query results. For example, you can query the answer from the pre-built knowledge base based on the query statement, and feed back the query answer to the user.

[0003] However, the data in the pre-built knowledge base in the related technology is relatively simple and incomplete, resulting in inaccurate query results. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, device and computer program product, which can enrich the amount of data in the target corpus, increase the comprehensiveness of the data in the target corpus, and improve the accuracy of the query results when performing data queries on the target corpus.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] Acquire a text unit, perform vector conversion on the text unit, and obtain a text vector of the text unit;

[0007] Building a target database based on the text unit, the text vector and the associated relationship set of the text unit;

[0008] A target corpus is constructed according to the target database.

[0009] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, including:

[0010] A conversion module is configured to obtain a text unit, perform vector conversion on the text unit, and obtain a text vector of the text unit;

[0011] A first construction module is configured to construct a target database based on the text unit, the text vector and the association relationship set of the text unit;

[0012] The second construction module is configured to construct a target corpus according to the target database.

[0013] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0014] processor;

[0015] Memory for storing computer programs or instructions;

[0016] The processor executes the computer program or instructions to implement the steps of any one of the methods in the first aspect above.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the storage medium stores a computer program or instructions. When the computer program or instructions in the storage medium are executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0018] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of any one of the methods described in the first aspect.

[0019] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0020] The technical solution provided by the embodiments of the present disclosure can determine the association relationship between various text units, obtain an association relationship set, perform vector conversion on the text unit to obtain the text vector of the text unit, and build a target database based on the text unit, the text vector and the association relationship set of the text unit; and build a target corpus based on the target database.

[0021] The target corpus is constructed based on text units, text vectors, and the association relationship sets between text units. The text units are utilized from multiple dimensions, which can enrich the amount of data in the target corpus, increase the comprehensiveness of the data in the target corpus, and improve the accuracy of the query results when performing data queries on the target corpus.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] Figure 1 is a flowchart of a data processing method according to an exemplary embodiment. Figure 1 .

[0025] Figure 2 is a flowchart of a data processing method according to an exemplary embodiment. Figure 2 .

[0026] Figure 3is a flowchart of a data processing method according to an exemplary embodiment. Figure 3 .

[0027] Figure 4 It is a block diagram of a data processing device according to an exemplary embodiment.

[0028] Figure 5 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] Figure 1 A data processing method according to an exemplary embodiment is shown in FIG. Figure 1 .like Figure 1 As shown, the method mainly includes the following steps:

[0031] In step 101, a text unit is obtained, and a vector conversion is performed on the text unit to obtain a text vector of the text unit;

[0032] In step 102, a target database is constructed based on text units, text vectors, and association relationship sets of text units;

[0033] In step 103, a target corpus is constructed according to the target database.

[0034] It should be noted that the data processing method proposed in the present disclosure can be applied to electronic devices or servers. Here, the electronic device may include: terminal devices, for example, mobile terminals or fixed terminals. Among them, mobile terminals may include: mobile phones, tablet computers, laptop computers and other devices. Fixed terminals may include: desktop computers or smart TVs, etc. In the embodiments of the present disclosure, the data processing method may be configured in a data processing device, which may be set in a server or in an electronic device, and the embodiments of the present disclosure do not limit this.

[0035] It should be noted that the execution subject of the embodiment of the present disclosure may be, for example, a central processing unit (CPU) in a server or electronic device in terms of hardware, or may be, for example, a related background service in a server or electronic device in terms of software, without limitation.

[0036] In some embodiments, the acquired raw data may be analyzed to obtain the initial text, wherein the raw data may include at least one of the following: text, table, picture, audio, and video.

[0037] Taking the original data as an example, after obtaining the picture, image recognition technology and / or (Artificial Intelligence, AI) model can be called to recognize the picture to determine the text content in the picture, that is, the initial text. Exemplarily, the initial text can be in doc format, html format, pdf format, etc.

[0038] In some embodiments, the text in the original data may also be recognized by using Optical Character Recognition (OCR) technology to obtain the initial text.

[0039] In some instances, the raw data is input into an artificial intelligence model (e.g., a deep learning model) to obtain an initial text output by the artificial intelligence model. Exemplarily, the deep learning model may include, but is not limited to, various neural networks, such as convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0040] In some embodiments, when there are at least two formats of data in the original data, the original data can be processed using a multimodal model to obtain an initial text. Exemplarily, for data including tables, pictures, and audio and video formats, a multimodal model can be used to obtain text data corresponding to the table, text data corresponding to the picture, and text data corresponding to the audio and video, respectively, and the initial text is obtained based on the text data corresponding to the table, the text data corresponding to the picture, and the text data corresponding to the audio and video.

[0041] In other embodiments, after obtaining the initial text, key information in the initial text can also be extracted. The extracted key information will be used as knowledge to be retrieved, wherein the key information may include but is not limited to: file name, time, content, Uniform Resource Locator (URL), etc. During the retrieval, the key information can be returned to the large model. For example, for the data from the web page source, it is necessary to extract the [text] therein, and it is also necessary to extract information such as [formulating authority] and [implementation date].

[0042] After obtaining the initial text, the initial text may be segmented. Segmenting the initial text refers to dividing the parsed text content into multiple blocks. In some embodiments, the initial text may be segmented by a pre-trained text processing model to obtain at least one text unit. Exemplarily, the initial text may be segmented based on an embedding model to obtain at least one text unit to ensure the semantic integrity of the initial text.

[0043] In some embodiments, the initial text is segmented in at least one of the following ways:

[0044] Structural segmentation: For the initial text with a document directory structure, the document structure can be extracted and the initial text can be segmented in the form of the smallest directory;

[0045] Sentence segmentation: Based on segmentation characters, the initial text is segmented at the granularity of "sentences" to retain the complete semantics of a sentence. For example, segmentation characters include: period, exclamation mark, question mark, line break, etc.

[0046] Fixed-length segmentation: According to the length limit of the minimum text unit (token) of the text processing model, the initial text is segmented into text units of fixed length. For example, the initial text can be segmented into 256 text units, or the initial text can be segmented into 512 text units.

[0047] Custom segmentation: Users can segment the initial text according to its structural characteristics.

[0048] In some other embodiments, the initial text may be segmented based on a set segmentation algorithm to obtain text units. The set segmentation algorithm includes at least one of the following: a segmentation algorithm based on string matching, a segmentation algorithm based on understanding, and a segmentation algorithm based on statistics.

[0049] Here, the segmentation algorithm based on string matching refers to matching the string to be segmented with the words in the set machine dictionary according to the set strategy, finding the words corresponding to the substring of the string in the dictionary, and then obtaining the segmentation result. For example, the initial text contains "Today's weather is really good", if the set machine dictionary contains the words "today", "weather" and "really good", then the result of segmenting the initial text, that is, the text units are "today", "weather" and "really good" respectively.

[0050] The segmentation algorithm based on understanding achieves the segmentation effect by simulating human understanding of sentences through computers, that is, syntactic and semantic analysis is performed while segmenting, and ambiguity is resolved using syntactic and semantic information. For example, a segmentation algorithm based on understanding is obtained by model training based on a large amount of collected data, and then the initial text to be segmented is input into the segmentation algorithm based on understanding to obtain and output text units.

[0051] The statistical segmentation algorithm refers to the use of the probability of co-occurrence between characters and between words as the basis for segmentation. The statistical segmentation algorithm belongs to dictionary-free segmentation. It is necessary to count the various word groups in the initial text, calculate the probability of adjacent occurrence of each word, and compare the probability of adjacent occurrence of each word with the set probability threshold. When the probability of adjacent occurrence of each word is greater than the set probability threshold, it is determined that this word group can constitute a text unit.

[0052] In some embodiments, the text unit may be in the form of a set. Exemplarily, the text unit includes at least one of a word in the initial text, a sentence in the initial text, a symbol in the initial text, and a chapter in the initial text.

[0053] In some other embodiments, the text unit may be directly obtained. For example, the obtained raw data may be directly analyzed to obtain the text unit. The raw data may include at least one of the following: text, table, picture, audio and video. The text unit includes at least one of the following: word, sentence, symbol, chapter.

[0054] After obtaining at least one text unit, vector conversion may be performed on the at least one text unit to obtain a text vector of the text unit. In some embodiments, vector conversion processing may be performed on the text unit based on a vector conversion model to obtain a text vector of the text unit.

[0055] After obtaining the text units and the text vectors of the text units, a target database may be constructed based on the text units, the text vectors and the association relationship sets of the text units, wherein the association relationship set may include the association relationships between the various text units.

[0056] Exemplarily, a text database can be constructed based on text units, a vector database can be constructed based on text vectors, and a relational database can be constructed based on the associated relationship sets of each text unit, and then a target database can be constructed based on the text database, the vector database, and the relational database. Exemplarily, the text database is used to retrieve text related to the query word, the vector database is used to retrieve vectors related to the query word, and the relational database is used to query text related to the query results from the text database and the vector database, and to expand and retrieve texts close to the separated text units. When searching, the logic of the text database is similar to that of the vector database, except that the search of the text database uses query statement search instead of the vector of the query statement.

[0057] like Figure 2 As shown, in the data preparation stage, after the initial text 201 is segmented, the text unit 202 can be obtained, and then the text unit 202 and the identifier of the text unit 202 are stored in the text database 203, and the text unit 202 is vectorized to obtain a text vector, and the identifier of the text vector and the text vector are stored in the vector database 204. In the data retrieval stage, the query statement 205 can be vectorized, the vector database 204 can be queried based on the vector of the query statement, and the text database 203 can be queried based on the query statement, and then the query results of the vector database 204 and the query results of the text database 203 are merged and output to the large speech model 206.

[0058] The technical solution provided by the embodiments of the present disclosure performs segmentation processing on the initial text to obtain at least one text unit, determines the association relationship between each text unit, performs vector conversion on at least one text unit to obtain a text vector of the text unit, builds a text database based on the text unit, also called a sparse database; builds a vector database based on the text vector, also called a dense database; builds a relational database according to the association relationship between each text unit, and then builds a target corpus based on the text database, the vector database and the relational database.

[0059] On the one hand, by segmenting the initial text to obtain at least one text unit, the vocabulary in the target corpus can be enriched; on the other hand, the target corpus can be constructed based on text units, text vectors and the association between text units, and each text unit can be utilized from multiple dimensions, which can enrich the amount of data in the target corpus and increase the comprehensiveness of the data in the target corpus. When performing data queries on the target corpus, the accuracy of the query results can be improved.

[0060] In some embodiments, constructing a target database based on text units, text vectors, and association relationship sets of text units includes:

[0061] Building a text database based on the identification of the text unit and the key information of the text unit;

[0062] Based on the identifier of the text vector and the text vector, a vector database is constructed; wherein the identifier of the text vector and the identifier of the text unit have a corresponding relationship;

[0063] Construct a relational database based on the identifiers and associated relationship sets of text units;

[0064] Construct the target database based on text database, vector database and relational database.

[0065] It should be noted that when the initial text is segmented, the identifier of the text unit and the key information of the text unit can be determined based on the context information of each text unit, and a text database can be constructed based on the identifier of the text unit and the key information of the text unit. Exemplarily, a mapping relationship between the identifier of the text unit and the key information of the text unit can be established, and the identifier of the text unit, the key information of the text unit and the mapping relationship can be stored in the text database.

[0066] Exemplarily, the segmented text unit, the id (unique identifier) ​​of the text unit, and the key information of the text unit can be stored in a text database, wherein the key information of the text unit can be the meta-information of the text unit. Exemplarily, the meta-information can include: the author, time, place, etc. of the original text in which the text unit is located.

[0067] After the text unit is converted into a vector, the identifier of the text vector can be determined, wherein the identifier of the text vector and the identifier of the text unit have a corresponding relationship. After a certain text unit is retrieved, the text vector corresponding to the text unit can be determined based on the identifier of the text unit and the corresponding relationship. Exemplarily, the identifier of the text unit and the identifier of the text vector of the text unit can be the same.

[0068] After obtaining the identifier of the text vector, a vector database can be constructed based on the identifier of the text vector and the text vector. Exemplarily, a mapping relationship between the identifier of the text vector and the text vector can be established, and the identifier of the text vector, the text vector and the mapping relationship can be stored in the vector database.

[0069] Exemplarily, the text vector of the segmented text unit and the id (unique identifier) ​​of the text vector can be stored in a vector database. As a unique identifier, the id can reduce the storage cost of the vector and improve the efficiency of vector query. Each database has an id field, and the data in each database can be associated by the id. Exemplarily, the text unit and the text vector can be associated by the id.

[0070] Since the text database stores text units, when searching, you can directly use a query statement to search the text database, where the query statement can be a query word. For example, the text database stores [Today's weather is cloudy]. When searching, if the query word is [Today's weather], the preceding text will be retrieved.

[0071] Since the vector database stores text vectors, during retrieval, the vector database can be retrieved using the vector of the query statement. For example, the vector database stores the vector of [Today's weather is cloudy]. During retrieval, if the query word is [Today's weather], the query word will be converted into a vector, and then the vector database will be queried based on the converted vector to obtain other vectors closest to the converted vector, so that the closest text can be retrieved. In some embodiments, there is a one-to-one correspondence between the text units in the text database and the text vectors in the vector database.

[0072] In the disclosed embodiments, a relational database may also be constructed based on the identification of the text unit and the association relationship set between each text unit. In some embodiments, context analysis may be performed on each text unit based on a semantic analysis model to determine the association relationship between each text unit, thereby forming an association relationship set, and then a relational database may be constructed based on the identification of the text unit and the association relationship between each text unit in the association relationship set, so as to determine the adjacent text units through the relational database. Exemplarily, a knowledge graph may be constructed based on the identification of the text unit and the association relationship between each text unit.

[0073] In the disclosed embodiments, a text database can be constructed based on the identification of text units and key information of text units, and text units can be retrieved through the text database to realize keyword retrieval; a vector database can be constructed based on the identification of text vectors and the text vectors; wherein the identification of the text vector and the identification of the text unit have a corresponding relationship; the text vector can be queried through the vector data path to realize the retrieval of similar text units; a relational database can be constructed based on the identification and association of text units, so that related text units can be retrieved, and the comprehensiveness of the retrieval results can be improved through the constructed target database.

[0074] In some embodiments, a relational database is constructed based on the identifiers and associated relationship sets of the text units, including:

[0075] Conduct semantic analysis on text units to determine hierarchical relationships and peer relationships in text units;

[0076] A relational database is constructed based on hierarchical relationships, peer relationships, and identification of text units.

[0077] It should be noted that after obtaining the text unit, the text unit can be input into a pre-trained text processing model, and the text processing model will perform structural processing on the text unit according to the semantics of the text unit, generate a relationship graph between each text unit, and store the generated relationship graph in a relational database. Among them, the relationship graph may include the hierarchical relationship (parent-child relationship) between each text unit, and may also include the same-level relationship (brother relationship) between each text unit. The relationship graph may also be called a parent-child relationship graph.

[0078] After determining the hierarchical relationship and the same-level relationship between each text unit, a relational database can be constructed based on the hierarchical relationship, the same-level relationship and the identification of the text unit. By constructing a relational database, after retrieving the results from the text database and the vector database, the retrieved nodes can be used to retrieve the parent, child and sibling nodes of the corresponding nodes through the relational database to expand the final search results, thereby improving the comprehensiveness and accuracy of the final search results.

[0079] In some embodiments, the method further comprises:

[0080] Querying the target corpus based on at least one first query statement in the data set to obtain a first query result of the first query statement; wherein the first query statement has a reference result;

[0081] Determine a target evaluation value of the target corpus based on the first query statement, the first query result, and the reference result;

[0082] When the target evaluation value is greater than the actual evaluation value of the historical corpus, the historical corpus is adjusted to the target corpus.

[0083] It should be noted that the evaluation value is positively correlated with the retrieval quality (precision) of the corpus. The larger the evaluation value, the higher the retrieval quality and precision of the corpus, and the smaller the evaluation value, the lower the retrieval quality and precision of the corpus.

[0084] Here, the target corpus can also be referred to as a knowledge base. After obtaining the target corpus, the results can be retrieved from the target corpus through questions (query statements), and the retrieval results can be evaluated to determine the retrieval quality of the target corpus.

[0085] In some embodiments, a data set may be set, which may also be referred to as an evaluation set. The data set may include at least one first query statement, and each first query statement may have at least one reference result. For example, if the first query statement is a question, the reference result may be a standard answer to the first query statement.

[0086] In the disclosed embodiment, the target corpus can be queried based on each first query statement to obtain the first query result of each first query statement, and the target evaluation value of the target corpus can be determined based on the first query statement, the first query result and the reference result.

[0087] In some embodiments, the difference values ​​between the first query result and the reference result of each first query statement may be determined respectively, and the target evaluation value may be obtained based on the difference values ​​corresponding to each first query statement.

[0088] Exemplarily, the first query result of the first query statement includes multiple answers, namely A1, A2 and A3, and the standard answers to the first query statement are B1 and B2, respectively. The difference value between A1 and B1 and B2 can be determined to obtain two difference values ​​corresponding to A1, the difference value between A2 and B1 and B2 can be determined to obtain two difference values ​​corresponding to A2, the difference value between A3 and B1 and B2 can be determined to obtain two difference values ​​corresponding to A3, and then based on the two difference values ​​corresponding to A1, the two difference values ​​corresponding to A2, and the two difference values ​​corresponding to A, the difference value corresponding to the first query statement is determined.

[0089] After obtaining the difference values ​​corresponding to each first query statement in the above manner, the target evaluation value of the target corpus can be obtained based on the difference values ​​corresponding to each first query statement. Exemplarily, the target evaluation value is determined based on the average value of the difference values ​​corresponding to each first query statement. Among them, the difference value is negatively correlated with the target evaluation value, and the target evaluation value is positively correlated with the precision of the target corpus, that is, the greater the difference between the first query result and the reference result, the lower the target evaluation value, the worse the retrieval quality of the target corpus, that is, the lower the precision of the target corpus; the smaller the difference between the first query result and the reference result, the higher the target evaluation value, the higher the retrieval quality of the target corpus, that is, the higher the precision of the target corpus.

[0090] In some embodiments, after determining the target evaluation value, the target evaluation value can be compared with the actual evaluation value of the historical corpus. When the target evaluation value is greater than the actual evaluation value of the historical corpus, it is determined that the accuracy of the target corpus is higher than the accuracy of the historical corpus. At this time, the historical corpus can be adjusted to the target corpus. The historical corpus can be a corpus generated before the current moment, and can be, for example, a corpus currently in use.

[0091] In some embodiments, the actual evaluation value of the historical corpus can be determined based on the actual query statement and the actual query result in the process of actually using the historical corpus for retrieval. The actual evaluation value is used to indicate the precision of the historical corpus, that is, the accuracy of the retrieval results obtained by searching the historical corpus. The actual evaluation value is positively correlated with the precision of the historical corpus. The lower the actual evaluation value, the worse the retrieval quality of the historical corpus, that is, the lower the precision of the historical corpus; the higher the actual evaluation value, the higher the retrieval quality of the historical corpus, that is, the higher the precision of the historical corpus.

[0092] In some embodiments, in response to detecting an adjustment instruction, when it is determined based on the target evaluation value that the accuracy of the target corpus is higher than the accuracy of the historical corpus, the historical corpus is adjusted to the target corpus. Wherein, the adjustment instruction is used to indicate that the version of the corpus currently in use is adjusted. In some embodiments, the adjustment instruction can be triggered based on the user's input operation, and in other embodiments, the electronic device can also automatically initiate the adjustment instruction. Exemplarily, when the usage time of the historical corpus reaches a preset time, the adjustment instruction is triggered, and the adjustment for the historical corpus is performed.

[0093] In the disclosed embodiment, when it is determined based on the target evaluation value that the accuracy of the target corpus is higher than that of the historical corpus, the historical corpus is adjusted to the target corpus, which can ensure the accuracy of the corpus actually used, thereby improving the retrieval quality. When the version of the corpus is adjusted, the above-mentioned evaluation index (target evaluation value) is used to measure the version effect (accuracy) of the target corpus. When the accuracy of the target corpus is better than the previous version (historical corpus), the target corpus is put into use, which can ensure that the accuracy of the adjusted corpus is better.

[0094] In some embodiments, determining a target evaluation value of a target corpus based on the first query statement, the first query result, and the reference result includes:

[0095] Determining a first evaluation value based on the number of first target results and the number of first query results in the first query result; wherein the similarity between the first target result and the reference result of the same first query statement is greater than a first threshold;

[0096] determining a second evaluation value based on the number of first target results and the number of reference results;

[0097] Determining a third evaluation value based on the relevance between the first query result and the first query statement, and the relevance between the first query result and the reference result;

[0098] Based on at least one of the first evaluation value, the second evaluation value, and the third evaluation value, a target evaluation value is determined.

[0099] It should be noted that after retrieving the target corpus based on the first query statement, at least one first query result can be obtained. In the disclosed embodiment, the first target result can be determined from the at least one first query result. In some embodiments, the similarity between each first query result and the reference result of the first query statement can be determined respectively, and the first query result whose similarity is greater than the first threshold value can be determined as the first target result. Exemplarily, the first query result that is the same as the reference result can be determined as the first target result. Among them, the first target result can be understood as a query result with relatively high accuracy, and can also be called a correct result.

[0100] In some embodiments, the first threshold may be obtained based on experience or experiments, and when the similarity between the first query result and the reference result is greater than the first threshold, it indicates that the first query result is relatively close to the reference result, or is the same as the reference result.

[0101] Exemplarily, the data set (evaluation set) consists of a first query statement (question) and a reference result (standard answer) corresponding to the first query statement, wherein there may be multiple reference results, exemplarily, A1, A2 and A3. For each first query statement in the data set, the target corpus is used for retrieval, and the first query result (retrieval result) is compared with the reference result of the first query statement in the data set, wherein there may be multiple first query results, exemplarily, A1, A2 and A4. If the first query result is in the reference result, the first query result is determined to be correct, exemplarily, A1 and A2 are correct, that is, A1 and A2 are the first target results, and the number of first target results is 2.

[0102] In an embodiment of the present disclosure, after determining the number of first target results, a first evaluation value may be determined based on the number of first target results and the number of first query results, wherein the first evaluation value may also be referred to as an accuracy value, which is used to indicate query accuracy.

[0103] In some embodiments, determining the first evaluation value based on the number of first target results in the first query result and the number of first query results includes:

[0104] Determining a first to-be-processed value of the first query statement based on a ratio of the number of first target results of the first query statement to the number of first query results;

[0105] A first evaluation value is determined based on an average value of first to-be-processed values ​​of each first query statement in the data set.

[0106] Exemplarily, for each first query statement in the data set, the calculation formula of the first to-be-processed value p is as follows:

[0107] p = m / n (1);

[0108] In formula (1), p represents the first value to be processed; m represents the number of first target results; and n represents the number of first query results. If the first target result does not exist in the first query result, p=0.

[0109] After obtaining the first processed value of each first query statement, the first evaluation value can be determined based on the average value of the first processed value of each first query statement in the data set, that is, the p of all the first query statements in the data set are averaged to obtain the average value, and the first evaluation value is determined based on the average value. The first evaluation value is the average accuracy.

[0110] For example, the average value of the first value to be processed can be directly determined as the first evaluation value, or the average value of the first value to be processed can be weighted based on a preset first weight value to obtain the first evaluation value. The first weight value is used to indicate the proportion of the first evaluation value in the target evaluation value, which can be determined based on needs.

[0111] In the disclosed embodiment, the average precision can be determined according to the number of correct results and the number of all retrieved query results, thereby improving the accuracy of the quality assessment of the target corpus.

[0112] In some embodiments, a second evaluation value may also be determined based on the number of first target results and the number of reference results. Exemplarily, the second evaluation value may be a recall rate, which is used to indicate the accuracy of the retrieved first query result.

[0113] In some embodiments, determining the second evaluation value based on the number of the first target results and the number of the reference results includes:

[0114] Determine a second to-be-processed value of the first query statement based on a ratio of the number of first target results of the first query statement to the number of reference results;

[0115] A second evaluation value is determined based on an average value of the second to-be-processed values ​​of each first query statement in the data set.

[0116] Exemplarily, for each first query statement in the data set, the calculation formula of the second to-be-processed value r is as follows:

[0117] r = m / k (2);

[0118] In formula (2), r represents the second to-be-processed value; m represents the number of first target results; and k represents the number of reference results. If the first target result does not exist in the first query result, then r=0.

[0119] After obtaining the second pending value of each first query statement, the second evaluation value can be determined based on the average value of the second pending value of each first query statement in the data set, that is, the r of all the first query statements in the data set is averaged to obtain the average value, and the second evaluation value is determined based on the average value. The second evaluation value is the average recall rate.

[0120] For example, the average value of the obtained second value to be processed can be directly determined as the second evaluation value, or the average value of the second value to be processed can be weighted based on a preset second weight value to obtain the second evaluation value. The second weight value is used to indicate the proportion of the second evaluation value in the target evaluation value, which can be determined based on needs.

[0121] In the disclosed embodiment, the average recall rate can be determined according to the number of correct results and the number of reference results, thereby improving the accuracy of the quality assessment of the target corpus.

[0122] In some embodiments, a third evaluation value can be determined based on the correlation between the first query result and the first query statement, and the correlation between the first query result and the reference result, wherein the third evaluation value is used to indicate contextual relevance, that is, the third evaluation value can be determined based on the correlation between the first query result and the first query statement, and the correlation between the first query result and the reference result.

[0123] In some embodiments, a third evaluation value may be determined based on the correlation between the first query result and the first query statement; a third evaluation value may be determined based on the correlation between the first query result and the reference result, that is, two third evaluation values ​​may be determined, and each third evaluation value is used to indicate the degree of correlation between the first query result and the first query statement, and the degree of correlation between the first query result and the reference result. In other embodiments, a third evaluation value may also be determined based on the correlation between the first query result and the first query statement and the correlation between the first query result and the reference result, that is, the third evaluation value can indicate both the degree of correlation between the first query result and the first query statement and the degree of correlation between the first query result and the reference result.

[0124] In some embodiments, the relevance between the first query result and the first query statement and the relevance between the first query result and the reference result may be determined based on a pre-trained relevance determination model.

[0125] Exemplarily, for each first query statement, a first query result is retrieved from the target corpus, and the first query results are A, B, and C. The first query statement and the first query result can be input into a pre-trained relevance determination model together, and the relevance between the first query statement and the first query result can be output based on the relevance determination model. For example, the relevance between the first query statement and the first query result can be scored based on the relevance determination model. Exemplarily, it can be determined that the relevance between the first query statement and the first query result is 90 points. In some embodiments, the score of the relevance can be limited to 0 to 100.

[0126] In some embodiments, before the first query statement and the first query result are input into the relevance determination model, the first query statement and the first query result can be concatenated to obtain a first statement to be processed, so that the relevance determination model can perform context analysis on the first statement to be processed and improve the accuracy of the obtained relevance.

[0127] Exemplarily, for each first query statement, a first query result is retrieved from the target corpus, and the reference result and the first query result can be input into a pre-trained relevance determination model together, and the relevance between the reference result and the first query result can be output based on the relevance determination model. For example, the relevance between the reference result and the first query result can be scored based on the relevance determination model. Exemplarily, it can be determined that the relevance between the reference result and the first query result is 80 points. In some embodiments, the score of the relevance can be limited to 0 to 100.

[0128] In some embodiments, before inputting the reference result and the first query result into the relevance determination model, the reference result and the first query result can be concatenated to obtain a second statement to be processed, so that the relevance determination model can perform context analysis on the second statement to be processed and improve the accuracy of the obtained relevance.

[0129] In the disclosed embodiment, a first evaluation value can be determined based on the number of first target results and the number of first query results in the first query result, a second evaluation value can be determined based on the number of first target results and the number of reference results, a third evaluation value can be determined based on the relevance between the first query result and the first query statement, and the relevance between the first query result and the reference result, and then a target evaluation value can be determined based on at least one of the first evaluation value, the second evaluation value, and the third evaluation value. By comprehensively considering the precision, recall rate, and relevance of the first query result, the accuracy of the obtained target evaluation value can be improved, thereby accurately evaluating the retrieval quality of the target corpus.

[0130] In some embodiments, the method further comprises:

[0131] In response to detecting the second query statement, querying the text database based on the second query statement to obtain a second query result;

[0132] Querying the vector database based on the vector of the second query statement to obtain a third query result;

[0133] Determine a second target result from the relational database based on the second query result and the third query result;

[0134] Output the second target result.

[0135] It should be noted that after the target corpus is obtained, the target corpus can be used. Exemplarily, the target corpus can be retrieved based on the second query statement.

[0136] In some embodiments, after the second query statement is detected, the second query statement may be preprocessed. For example, at least one of the following preprocessing may be performed on the second query statement: word expansion processing on the second query statement; word segmentation processing on the second query statement; named entity recognition (NER) on the second query statement; intent understanding on the second query statement; weighted processing (term-weight) on each word in the second query statement

[0137] In some embodiments, the pre-trained large model can be used to perform word expansion processing on the second query statement to obtain multiple query statements, and the multiple query statements are used to query the target corpus to retrieve better results. The second query statement is subjected to named entity recognition, the second query statement is subjected to intent understanding, and each word in the second query statement is subjected to weighted processing, mainly to assist in recalling and re-ranking the search results.

[0138] It should be noted that recall refers to finding qualified text units from the target corpus. There are two recall methods: vector recall and keyword recall. Vector recall is to convert the query statement into a vector, and then use the vector to retrieve similar data from the vector database. Keyword recall is to retrieve similar data from the text database based on the query statement. Rearrangement refers to reordering the recall results, putting the query results with higher relevance in front. Filtering refers to filtering out the results based on similarity scores, keywords, metadata, etc., and then obtaining the final query results.

[0139] Among them, querying the text database based on the second query statement may be the above-mentioned keyword recall process, and querying the vector database based on the vector of the second query statement may be the above-mentioned vector recall process.

[0140] In the disclosed embodiment, after detecting the second query statement, the text database can be queried based on the second query statement to obtain a second query result, the vector database can be queried based on the vector of the second query statement to obtain a third query result, and the second target result can be determined from the relational database based on the second query result and the third query result, and the second target result can be output. The accuracy and comprehensiveness of the second target result can be improved from multiple query tasks that execute the second query statement.

[0141] In some embodiments, based on the second query result and the third query result, determining a second target result from the relational database includes:

[0142] Based on the identifier of the first text unit in the second query result, determining a text unit associated with the first text unit from a relational database;

[0143] Based on the identifier of the first text vector in the third query result, determining from the relational database a second text unit corresponding to the first text vector and a text unit associated with the second text unit;

[0144] A second target result is determined based on the first text unit, the second text unit, a text unit associated with the first text unit, and a text unit associated with the second text unit.

[0145] Here, when the first text unit is obtained by querying from a text database, the text unit associated with the first text unit can be determined from the relational database based on the first text unit. When the first text vector is obtained by querying from a vector database, the second text unit corresponding to the first text vector and the text unit associated with the second text unit can be determined from the relational database.

[0146] For example, Figure 3 As shown, the second text vector queried from the vector database 301 is child node 1.2 and child node 2.1, then the parent node 1, leaf 1.2.1, leaf 1.2.2, sibling nodes (child node 1.1 and child node 1.3) of child node 1.2 can be determined from the relational database 302, as well as the parent node 2 and sibling node 2.2 of child node 2.1.

[0147] The data in each database are associated through the identification of text units and text vectors. After the first text unit and the first text vector are determined, the related (similar) text units can be determined from the relational database through the identification of the first text unit and the identification of the first text vector, which can improve the richness and accuracy of the second target result.

[0148] In some embodiments, the method further comprises:

[0149] When the total number of text units in the second target result is less than a preset number, and / or the relevance between the text units in the second target result is less than a second threshold, generating a third query statement based on the second target result and the second query statement;

[0150] Querying the target corpus based on the third query statement to obtain a third target result;

[0151] Output the third target result.

[0152] It should be noted that, when the total number of text units in the second target result is less than the preset number, it is determined that the query results are very few; when the correlations between the various text units in the second target result are less than the second threshold, it is determined that the correlations of the results obtained by the query are very low, that is, the quality of the second target result is poor.

[0153] The preset number and the second threshold may be determined as required, for example, may be obtained based on empirical values ​​or experiments.

[0154] In the disclosed embodiment, when the quality of the second target result is poor, the second target result will not be directly output, but a secondary search will be performed. When a secondary search is required, a third query statement can be generated based on the second target result and the second query statement. For example, a new query statement (third query statement) can be regenerated using a large model, and a complete search can be performed again to obtain the third target result, which can improve the accuracy of the query result finally obtained.

[0155] The knowledge base (corpus) retrieval system in the present disclosure includes a knowledge base preparation module, a knowledge base retrieval module and a knowledge base management module.

[0156] The knowledge base preparation module can extract the initial text from the original data by data extraction, perform text segmentation on the initial text to obtain text units, then perform vector conversion on the text units to obtain text vectors, and perform storage processing on the text units and text vectors, that is, store the text units and text vectors in the text database and vector database respectively. The knowledge base retrieval module can query and understand the second query statement, and recall based on the understanding result to obtain the initial target result, and reorder and filter the initial target result to obtain the second target result. When the total number of text units in the second target result is less than the preset number, and / or the correlation between each text unit in the second target result is less than the second threshold, a secondary query (retrieval) is performed to obtain the third target result. The knowledge base management module can perform version authority management of the knowledge base group, version authority management of the knowledge base, and retrieval quality evaluation of the knowledge base.

[0157] In some embodiments, each corpus (knowledge base) can be grouped based on different query services to obtain different corpus groups, that is, different types of query services can correspond to different corpus groups. Exemplarily, there are multiple corpora, namely corpus A, corpus B and corpus C, then the first type of service can use corpus A and corpus B, corpus A and corpus B constitute the corpus group of the first type of service; the second type of service can use corpus B and corpus C, corpus B and corpus C constitute the corpus group of the second type of service.

[0158] When the second query statement is detected, the type of query service indicated by the second query statement can be determined first, and then the target corpus corresponding to the second query statement is determined based on the type of query service, and then the target corpus is queried based on the second query statement to obtain the second target result.

[0159] In some embodiments, when a target corpus is constructed based on any of the above methods, the target corpus group corresponding to the target corpus can be determined based on the type of query service corresponding to the target corpus, and the target corpus can be adjusted to the target corpus group, that is, only the relevant corpus group can be maintained. By grouping the corpus, when a corpus needs to be adjusted, it is sufficient to maintain a corpus group of a type of query service, which can reduce the maintenance cost of the corpus. In some embodiments, the target corpus can form a corpus group alone, or it can form a corpus group together with other corpora.

[0160] In other embodiments, when constructing a corpus, the corpus group can be divided into versions, and the version of the corpus group can be associated with the type of query business. Then, when there is a query business, the version corresponding to the query business can be determined, and different corpus groups can be selected according to different versions, which facilitates iterative upgrades of the corpus.

[0161] Figure 4 FIG. 1 is a block diagram of a data processing device according to an exemplary embodiment. Figure 4 As shown, the device 400 mainly includes:

[0162] The conversion module 401 is configured to obtain a text unit, perform vector conversion on the text unit, and obtain a text vector of the text unit;

[0163] A first construction module 402 is configured to construct a target database based on the text unit, the text vector and the association relationship set of the text unit;

[0164] The second construction module 403 is configured to construct a target corpus according to the target database.

[0165] In some embodiments, the first building module 402 is configured to:

[0166] Building a text database based on the identifier of the text unit and the key information of the text unit;

[0167] Based on the identifier of the text vector and the text vector, construct a vector database; based on the identifier of the text unit and the association relationship set, construct a relational database;

[0168] The target database is constructed based on the text database, the vector database and the relational database.

[0169] In some embodiments, the first building module 402 is configured to:

[0170] Performing semantic analysis on the text unit to determine the hierarchical relationship and the same-level relationship in the text unit;

[0171] The relational database is constructed based on the hierarchical relationship, the peer relationship and the identifier of the text unit.

[0172] In some embodiments, the apparatus 400 further includes

[0173] A first query module is configured to query the target corpus based on at least one first query statement in the data set to obtain a first query result of the first query statement; wherein the first query statement has a reference result;

[0174] A first determination module is configured to determine a target evaluation value of the target corpus based on the first query statement, the first query result, and the reference result;

[0175] The adjustment module is configured to adjust the historical corpus to the target corpus when the target evaluation value is greater than the actual evaluation value of the historical corpus.

[0176] In some embodiments, the first determining module is configured to:

[0177] Determining a first evaluation value based on the number of first target results in the first query result and the number of the first query result; wherein the similarity between the first target result and the reference result of the same first query statement is greater than a first threshold;

[0178] determining a second evaluation value based on the number of the first target results and the number of the reference results;

[0179] Determining a third evaluation value based on the relevance between the first query result and the first query statement, and the relevance between the first query result and the reference result;

[0180] The target evaluation value is determined based on at least one of the first evaluation value, the second evaluation value, and the third evaluation value.

[0181] In some embodiments, the first determining module is configured to:

[0182] determining a first to-be-processed value of the first query statement based on a ratio of the number of the first target results of the first query statement to the number of the first query results;

[0183] The first evaluation value is determined based on an average value of the first to-be-processed values ​​of each of the first query statements in the data set.

[0184] In some embodiments, the first determining module is configured to:

[0185] Determining a second to-be-processed value of the first query statement based on a ratio of the number of the first target results of the first query statement to the number of the reference results;

[0186] The second evaluation value is determined based on an average value of the second to-be-processed values ​​of each of the first query statements in the data set.

[0187] In some embodiments, the apparatus 400 further includes:

[0188] A second query module is configured to query the text database based on the second query statement in response to detecting the second query statement to obtain a second query result;

[0189] A third query module is configured to query the vector database based on the vector of the second query statement to obtain a third query result;

[0190] A second determination module is configured to determine a second target result from the relational database based on the second query result and the third query result;

[0191] The first output module is configured to output the second target result.

[0192] In some embodiments, the second determining module is configured to:

[0193] Based on the identifier of the first text unit in the second query result, determining a text unit associated with the first text unit from the relational database;

[0194] Based on the identifier of the first text vector in the third query result, determining from the relational database a second text unit corresponding to the first text vector and a text unit associated with the second text unit;

[0195] The second target result is determined based on the first text unit, the second text unit, a text unit associated with the first text unit, and a text unit associated with the second text unit.

[0196] In some embodiments, the apparatus 400 further includes:

[0197] a generating module configured to generate a third query statement based on the second target result and the second query statement when the total number of text units in the second target result is less than a preset number and / or the relevance between the text units in the second target result is less than a second threshold;

[0198] a fourth query module, configured to query the target corpus based on the third query statement to obtain a third target result;

[0199] The second output module is configured to output the third target result.

[0200] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0201] Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, which may be a computer or a terminal in one or more of the above embodiments. Figure 5 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device 500 adopts general computer hardware and includes a processor 501 , a memory 502 , a bus 503 , an input device 504 , and an output device 505 .

[0202] In some possible implementations, the memory 502 may include computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory and / or random access memory. The memory 502 may store an operating system, application programs, other program modules, executable code, program data, user data, etc.

[0203] Input device 504 can be used to input commands and information to the electronic device, such as a keyboard or a pointing device, such as a mouse, trackball, touchpad, microphone, joystick, game pad, satellite TV antenna, scanner or similar device. Input device 504 can be connected to processor 501 via bus 503.

[0204] The output device 505 can be used to output information to the electronic device 500 . In addition to the monitor, the output device 505 can also be other peripheral output devices, such as a speaker and / or a printing device. The output device 505 can also be connected to the processor 501 through the bus 503 .

[0205] The electronic device 500 may be connected to a network, such as a local area network (LAN), via an antenna 506. In a networked environment, executable instructions may be stored in a remote storage device, rather than being limited to local storage.

[0206] When the processor 501 in the electronic device 500 executes the executable code or application stored in the memory 502, the electronic device 500 can implement the image processing method in the above embodiments. The specific execution process refers to the above embodiments and will not be repeated here.

[0207] The memory 502 may store information for implementing Figure 4 The executable instructions of the functions of the conversion module 401, the first construction module 402 and the second construction module 403 in the embodiment of the present invention. Figure 4 The functions / implementation processes of the conversion module 401, the first construction module 402 and the second construction module 403 can be realized by Figure 5 The processor 501 in the embodiment calls the executable instructions stored in the memory 502 to implement the above-mentioned implementation process and functions.

[0208] Based on the same inventive concept, the embodiment of the present disclosure further provides a storage medium. The storage medium stores instructions. When the instructions are executed on a computer, the instructions are used to execute the data processing method in one or more of the above embodiments.

[0209] Based on the same inventive concept, the embodiments of the present disclosure further provide a computer program or a computer program product. When the computer program product is executed on a computer, the computer implements the data processing method in one or more of the above embodiments.

[0210] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0211] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Acquire a text unit, perform vector conversion on the text unit, and obtain a text vector of the text unit; Building a target database based on the text unit, the text vector and the associated relationship set of the text unit; A target corpus is constructed according to the target database.

2. The method according to claim 1, characterized in that: The step of constructing a target database based on the text unit, the text vector, and the associated relationship set of the text unit includes: Building a text database based on the identifier of the text unit and the key information of the text unit; Building a vector database based on the identifier of the text vector and the text vector; Constructing a relational database according to the identifier of the text unit and the set of associated relationships; The target database is constructed based on the text database, the vector database and the relational database.

3. The method according to claim 2, characterized in that The step of constructing a relational database according to the identifier of the text unit and the associated relationship set includes: Performing semantic analysis on the text unit to determine the hierarchical relationship and the same-level relationship in the text unit; The relational database is constructed based on the hierarchical relationship, the peer relationship and the identifier of the text unit.

4. The method according to claim 1, characterized in that The method further comprises: Querying the target corpus based on at least one first query statement in the data set to obtain a first query result of the first query statement; wherein the first query statement has a reference result; Determining a target evaluation value of the target corpus based on the first query statement, the first query result, and the reference result; In a case where the target evaluation value is greater than the actual evaluation value of the historical corpus, the historical corpus is adjusted to be the target corpus.

5. The method according to claim 4, characterized in that The determining, based on the first query statement, the first query result, and the reference result, a target evaluation value of the target corpus includes: Determining a first evaluation value based on the number of first target results in the first query result and the number of the first query result; determining a second evaluation value based on the number of the first target results and the number of the reference results; Determining a third evaluation value based on the relevance between the first query result and the first query statement, and the relevance between the first query result and the reference result; The target evaluation value is determined based on at least one of the first evaluation value, the second evaluation value, and the third evaluation value.

6. The method according to claim 5, characterized in that The determining a first evaluation value based on the number of first target results in the first query result and the number of the first query result includes: determining a first to-be-processed value of the first query statement based on a ratio of the number of the first target results of the first query statement to the number of the first query results; The first evaluation value is determined based on an average value of the first to-be-processed values ​​of each of the first query statements in the data set.

7. The method according to claim 5, characterized in that The determining of a second evaluation value based on the number of the first target results and the number of the reference results includes: Determining a second to-be-processed value of the first query statement based on a ratio of the number of the first target results of the first query statement to the number of the reference results; The second evaluation value is determined based on an average value of the second to-be-processed values ​​of each of the first query statements in the data set.

8. The method according to claim 2, characterized in that: The method further comprises: In response to detecting a second query statement, querying the text database based on the second query statement to obtain a second query result; Querying the vector database based on the vector of the second query statement to obtain a third query result; Based on the second query result and the third query result, determining a second target result from the relational database; The second target result is outputted.

9. The method according to claim 8, characterized in that The determining a second target result from the relational database based on the second query result and the third query result includes: Based on the identifier of the first text unit in the second query result, determining a text unit associated with the first text unit from the relational database; Based on the identifier of the first text vector in the third query result, determining from the relational database a second text unit corresponding to the first text vector and a text unit associated with the second text unit; The second target result is determined based on the first text unit, the second text unit, a text unit associated with the first text unit, and a text unit associated with the second text unit.

10. The method according to claim 9, characterized in that The method further comprises: If the total number of text units in the second target result is less than a preset number, and / or the relevance between the text units in the second target result is less than a second threshold, generating a third query statement based on the second target result and the second query statement; Querying the target corpus based on the third query statement to obtain a third target result; The third target result is outputted.

11. A data processing device, characterized in that: include: A conversion module is configured to obtain a text unit, perform vector conversion on the text unit, and obtain a text vector of the text unit; A first construction module is configured to construct a target database based on the text unit, the text vector and the association relationship set of the text unit; The second construction module is configured to construct a target corpus according to the target database.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images