A data completion method and device, electronic equipment and medium

Abstract

This application discloses a data completion method, apparatus, electronic device, and medium. The method includes: acquiring a user-inputted completion request text; converting the completion request text into a query vector; indexing a target embedding vector in a pre-built vector database whose similarity to the query vector meets preset conditions; extracting target metadata corresponding to the target embedding vector from the vector database; and returning the target metadata as target completion data to the client. Thus, based on embedding vector technology, target embedding vectors that satisfy the specified region can be quickly searched in a pre-built vector database, thereby extracting highly reliable target metadata as completion data. Furthermore, even if the data is encrypted or uses abbreviations, the vector similarity indexing method provided in this application can still achieve efficient and accurate data extraction and completion, thereby improving data integrity and usability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data completion method, apparatus, electronic device, and medium. Background Technology

[0002] With the development of the information age, all industries generate massive amounts of data every day, which is usually stored in a structured format. Some columns are used to store core data content (e.g., event descriptions), while other columns record metadata related to that data content (e.g., time, location, people involved, etc.).

[0003] In the actual process of collecting, organizing, and storing user data, metadata is often missing. Currently, large language models can be used to complete the collected data, thus ensuring data integrity. However, during the completion process, some data may be hidden or encrypted for confidentiality purposes, and some data may be represented using abbreviations, which can prevent the large model from accurately understanding and completing the semantics, reducing data integrity and usability.

[0004] Therefore, how to efficiently and accurately complete incomplete data information to improve data integrity and usability is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, one aspect of this application provides a data completion method, the method comprising:

[0006] Get the user-inputted text to be completed;

[0007] Convert the request text to be completed into a query vector;

[0008] In a pre-built vector database, the index is the target embedding vector whose similarity to the query vector meets a preset condition;

[0009] Extract the target metadata corresponding to the target embedding vector from the vector database;

[0010] The target metadata is used as target completion data and returned to the client.

[0011] Optionally, the target embedding vector whose similarity to the index and the query vector meets a preset condition includes:

[0012] Extract the target similarity that is greater than a threshold to obtain the candidate embedding vector corresponding to the target similarity;

[0013] When the target similarity is one, the candidate embedding vector is used as the target embedding vector;

[0014] When there are multiple target similarities, the candidate metadata corresponding to different data attributes of the candidate embedding vector is scored; and the candidate embedding vector corresponding to the candidate metadata with the largest score is taken as the target embedding vector.

[0015] Optionally, extracting the target metadata corresponding to the target embedding vector from the vector database includes:

[0016] When the target similarity is one, all metadata corresponding to the target embedding vector is used as the target metadata;

[0017] When there are multiple target similarities, the metadata corresponding to the one with the largest score is used as the target metadata.

[0018] Optionally, scoring the candidate metadata for different data attributes corresponding to the candidate embedding vector includes:

[0019] Based on the target similarity, determine the Boltzmann score corresponding to each candidate embedding vector;

[0020] Determine whether identical candidate metadata exists within the same data attribute;

[0021] If it does not exist, the Boltzmann score is used as the score value of the corresponding candidate metadata;

[0022] If it exists, perform the following steps:

[0023] The score value of the target candidate metadata is determined to be the corresponding Boltzmann score; the target candidate metadata refers to other metadata in the same data attribute besides the same candidate metadata.

[0024] The Boltzmann scores corresponding to the same candidate metadata are aggregated and calculated; and the aggregated calculation result is used as the score value of the same candidate metadata.

[0025] Optionally, scoring the candidate metadata for different data attributes corresponding to the candidate embedding vector includes:

[0026] The specified large model is invoked to score the correlation between the request text to be completed and each of the candidate metadata.

[0027] Optionally, in the absence of the target similarity, the data completion method includes:

[0028] The target completed data is marked as a NaN value; and the NaN value is returned to the client.

[0029] Determine whether the manually completed data input by the user has been obtained within the preset time period;

[0030] If so, the request text to be completed is completed using the manually completed data;

[0031] If not, after each preset period, proceed to the step of indexing the target embedding vector in the pre-built vector database, where the similarity between the target embedding vector and the query vector meets the preset conditions, and then execute the subsequent steps.

[0032] Optionally, the step of constructing the vector database includes:

[0033] Obtain the dataset to be processed; the dataset to be processed includes core data content and metadata of different data attributes related to the core data content;

[0034] The core data content in the dataset to be processed is converted into embedded vectors in batches.

[0035] Based on the FAISS vector index, an index is constructed for the embedded vector to obtain the index number;

[0036] Establish a mapping relationship between the embedded vector, the index number, and the metadata of the different data attributes to construct the vector database.

[0037] Another aspect of this application provides a data completion apparatus, the apparatus comprising:

[0038] The request retrieval module is used to retrieve the user-inputted request text to be completed;

[0039] The query vector acquisition module is used to convert the request text to be completed into a query vector;

[0040] The target embedding vector indexing module is used to index target embedding vectors in a pre-built vector database that have a similarity to the query vector that meets a preset condition.

[0041] The target metadata extraction module is used to extract the target metadata corresponding to the target embedding vector from the vector database;

[0042] The data output module is used to return the target metadata as target completion data to the client.

[0043] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the data completion method.

[0044] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data completion method.

[0045] The data completion method, apparatus, electronic device, and medium provided in this application have the following beneficial effects: Based on embedding vector technology, target embedding vectors that satisfy the region can be quickly searched in a pre-built vector database, thereby extracting highly reliable target metadata as completion data. Furthermore, even if the data is encrypted or uses abbreviations, the vector similarity indexing method provided in this application can still achieve efficient and accurate data extraction and completion, thereby improving data integrity and usability. Attached Figure Description

[0046] Figure 1 A flowchart illustrating a data completion method provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram illustrating the principle of a data completion method provided in an embodiment of this application;

[0048] Figure 3 A schematic diagram illustrating the principle of a data completion method provided in another embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a data completion device provided in an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0051] The attached diagram is labeled as follows: 40 is the request acquisition module, 41 is the query vector acquisition module, 42 is the target embedding vector index module, 43 is the target metadata extraction module, 44 is the data output module, 50 is the memory, 51 is the processor, 55 is the display screen, 53 is the input / output interface, 54 is the communication interface, 55 is the power supply, 56 is the communication bus, 501 is the computer program, 505 is the operating system, and 503 is the data. Detailed Implementation

[0052] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0053] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same attribute from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0054] Figure 1 This is a flowchart illustrating a data completion method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0055] S10: Obtain the text to be completed by the user;

[0056] S11: Convert the request text to be completed into a query vector;

[0057] Figure 2 This is a schematic diagram illustrating the principle of a data completion method provided in an embodiment of this application. In a specific embodiment, as shown... Figure 2 As shown, when a user needs data completion, the user inputs the request text to be completed from the client. To achieve fast data completion, the server converts the request text into a query vector upon receiving it.

[0058] It should be noted that, in this specific embodiment, the request text to be completed refers to the object that the user needs to complete, and is descriptive text representing the core content of the data. The query vector refers to the encoded vector used to initiate a search.

[0059] In one alternative embodiment, in order to achieve fast conversion between text and vectors, the request text to be completed can be converted into a query vector by a pre-trained target large language model. The target large language model includes, but is not limited to, ChatGPT, GPT-4, Claude, and Tongyi Qianwen. This application does not limit the specific target large language model.

[0060] In addition, the conversion can be achieved through pre-trained encoders, distributed static word vectors, and vector models. This application does not limit the specific method for converting the request text to be completed into a query vector.

[0061] S12: In a pre-built vector database, the target embedding vector whose similarity to the index and query vectors meets the preset conditions;

[0062] It is understandable that if the request text to be completed is encrypted or uses abbreviations, and a large model is used to perform semantic parsing on the request text to generate the target completion data, the encryption and abbreviations may prevent the large model from accurately understanding the semantics, resulting in low data completion accuracy.

[0063] Therefore, in order to solve this technical problem, the data completion method provided in this application includes, for example... Figure 2 As shown, a vector database is pre-built. When the server generates query vectors, they are indexed in the vector database. It should be noted that the vector database is used to store complete structured data, specifically including embedded vectors and metadata of each data attribute. The embedded vectors refer to fixed-dimensional vectors obtained by encoding the indexed objects.

[0064] It is understood that, in specific embodiments, indexing in the vector database refers to matching the query vector with the embedding vector, and using the embedding vector whose similarity meets a preset condition as the target embedding vector after indexing. In an optional embodiment, since the user input request is a text completion request, the embedding vector refers to the vector encoded from the core data, and correspondingly, the metadata refers to relevant data information about the core data. In an optional embodiment, the query vector and the embedding vector can be understood as event descriptions, while the metadata refers to relevant information about the event (e.g., time, location, etc.).

[0065] It is worth noting that this application does not limit the storage method of the vector database. In one optional embodiment, it can be stored using a row-column structure, where each row represents a complete data record, specifically including the embedded vector, metadata under different data attributes, and the index number of the embedded vector. The first row of the row-column structure stores descriptive text of the data type, while each column in the list structure represents specific metadata corresponding to a data attribute. Of course, in row-column storage, columns can also be used to store complete data, and rows can be used to store specific metadata under different data attributes; this application does not limit this approach.

[0066] Table 1 is a data storage diagram of a vector database provided in an embodiment of this application. For ease of understanding, the following explanation will be based on Table 1 and an example of an intelligence analysis application scenario.

[0067] Table 1. Schematic diagram of data storage for a vector database.

[0068] Index number Embedded vector Main party object side time …… Place 001 E1 A1 B1 T1 …… D1 002 E2 A2 B2 T2 …… D2 003 E3 A2 B3 T3 …… D3 004 E4 A3 B4 T3 …… D4 005 E5 A4 B5 T3 …… D5 …… …… …… …… …… …… …… 00n En Ai Bj Tx …… Dt

[0069] As shown in Table 1, in one optional embodiment, the vector database stores complete intelligence information. The first row stores data attributes, which can also be understood as data types. Each row below stores a complete piece of intelligence data. Each column stores specific metadata information under different data attributes.

[0070] It should be noted that the embedded vector refers to the vector encoded from the intelligence description text in each complete intelligence data set, while the information such as subject, object, time and location refers to the relevant metadata information of the corresponding complete intelligence data set, that is, the metadata information of the intelligence description.

[0071] In one alternative embodiment, such as Figure 2 As shown, the user inputs the request text to be completed from the client. Specifically, the input content is "Intelligence X1 = Client R and Client P signed...". After the client sends Intelligence X1 to the server, the server converts Intelligence X1 into a query vector C. Further, an index is performed in the vector database to retrieve target embedding vectors whose similarity to query vector C meets preset conditions. It can be understood that the similarity between query vector C and target embedding vectors can, in fact, be understood as the similarity between two corresponding texts. In an optional embodiment, similarity can be calculated using cosine similarity or a large language model; this application does not limit the method of similarity calculation.

[0072] S13: Extract the target metadata corresponding to the target embedding vector from the vector database;

[0073] S14: Return the target metadata as target completion data to the client.

[0074] Furthermore, such as Figure 2As shown, after retrieving the target embedding vector from the vector database, the target metadata corresponding to the target embedding vector is used as target completion data and returned to the client. It is worth noting that in specific embodiments, after determining the target embedding vector, the entire row of data containing the target embedding vector can be returned to the client, or only the target metadata value can be returned to the client; this application does not limit this. For example, in Table 1, if the indexed target embedding vector is embedding vector E1, the entire row of data containing embedding vector E1 can be returned to the client, or only the information such as subject A1, object B1, time T1, and location D1 corresponding to embedding vector E1 can be returned to the client.

[0075] It should be noted that, in one optional embodiment, the target embedding vector can be one or more. If there is only one target embedding vector, all corresponding target metadata can be directly returned to the client. When there are multiple target embedding vectors, all target metadata corresponding to the one with the highest similarity can be returned, or all target metadata corresponding to all target embedding vectors can be returned, allowing the user to choose the final completed data.

[0076] Of course, in another alternative embodiment, when there are multiple target embedding vectors, the data with the highest frequency can be selected from each data attribute as the target metadata. If the frequencies are the same, the metadata with the highest similarity can be selected as the target metadata. For ease of understanding, an example will be given below.

[0077] For example, in Table 1, if the index results include embedding vectors E1 to E5, where embedding vector E5 corresponds to the highest similarity, when extracting target metadata, for the subject data attribute, since subject A2 appears twice, subject A2 is returned as the target metadata. For the object, the objects corresponding to embedding vectors E1 to E5 are all different; in this case, object B5, corresponding to the embedding vector E5 with the highest similarity, is returned to the client as the target metadata. The same applies to other data attributes, which will not be elaborated here.

[0078] Therefore, the data completion method provided in this application, based on embedding vector technology, can quickly search for target embedding vectors that satisfy the region in a pre-built vector database, thereby extracting highly reliable target metadata as completion data. Furthermore, even if the data is encrypted or uses abbreviations, the vector similarity indexing method provided in this application can still achieve efficient and accurate data extraction and completion, thereby improving data integrity and usability.

[0079] Figure 3This is a schematic diagram illustrating the principle of a data completion method provided in another embodiment of this application. In an optional embodiment, the target embedding vector whose similarity between the index and the query vector meets a preset condition includes:

[0080] Extract the target similarity with a similarity greater than a threshold to obtain the candidate embedding vector corresponding to the target similarity;

[0081] When the target similarity is one, the candidate embedding vector is used as the target embedding vector;

[0082] When there are multiple target similarities, the candidate metadata corresponding to different data attributes of the candidate embedding vector is scored; and the candidate embedding vector corresponding to the candidate metadata with the largest score is used as the target embedding vector.

[0083] Based on the above embodiments, after calculating the similarity between the query vector and each embedded vector in the vector database, further, target similarities with similarities greater than a threshold are extracted. The higher the similarity, the more similar the query vector and the embedded vector are in terms of association, semantics, etc. When the similarity is greater than the threshold, it can be understood that the metadata of the corresponding embedded vector has a high confidence level as target completion data. It is understood that after extracting the target similarity, candidate embedded vectors can be obtained, where there is a one-to-one correspondence between the candidate embedded vectors and the target similarities.

[0084] In one alternative embodiment, such as Figure 3 As shown, when the target similarity is one, that is, there is only one similarity greater than the threshold, there is also only one candidate embedding vector, which can be used as the target embedding vector.

[0085] When there are multiple target similarities, there are also multiple candidate embedding vectors, and the number of both is the same. In this case, to achieve high-precision data completion, in one optional embodiment, the candidate metadata under different data attributes corresponding to each candidate embedding vector is scored. Based on the scored data, the candidate embedding vector corresponding to the highest score is used as the target embedding vector. For ease of understanding, examples from Table 1 and intelligence analysis scenarios will be provided below.

[0086] For example, with a similarity comparison threshold of 0.5, if the user inputs a request "Intelligence X1 = Client R and Client P signed a contract...", the client sends Intelligence X1 to the server, and the server converts Intelligence X1 into query vector C. In Table 1, after indexing and similarity calculation, the candidate embedding vectors obtained by query vector C include embedding vectors E1 to E5, and the calculated similarity between query vector C and each candidate embedding vector is greater than 0.5, that is, the target similarity is [0.83, 0.72, 0.65, 0.92, 0.66], among which embedding vector E4 has the highest similarity.

[0087] Therefore, in this example, there are multiple target similarities and multiple candidate embedding vectors. In this case, it is necessary to score the candidate metadata for different data attributes corresponding to embedding vectors E1 to E5. For example, for the subject, it is necessary to score subject A1, subject A2, subject A3, and subject A4 respectively.

[0088] It should be noted that there may be identical candidate metadata. For example, there may be two subjects A2. Scoring can be done separately for the same subject A2, or only one subject can be scored. This application does not limit this.

[0089] After scoring, if subject A1 has the highest score, then the embedding vector E1 corresponding to subject A1 is taken as the target embedding vector. Metadata for other data attributes is scored and calculated similarly to determine the target embedding vector. It is understandable that after... Figure 3 The judgment between similarity and threshold, as shown, and the judgment of the number of target similarities, may result in one or more target embedding vectors.

[0090] Therefore, the data completion method provided in this application improves the accuracy of data completion when determining the target embedding vector. When there are multiple candidate embedding vectors, the candidate metadata of different data attributes are scored respectively, and the target embedding vector is determined at a smaller granularity.

[0091] Based on the above embodiments, as an optional embodiment, target metadata corresponding to the target embedding vector is extracted from the vector database, including:

[0092] When the target similarity is one, all metadata corresponding to the target embedding vector is used as the target metadata;

[0093] When there are multiple target similarities, the metadata corresponding to the one with the highest score will be used as the target metadata.

[0094] It is understandable that, such as Figure 3As shown, when there is only one target similarity, there is also only one corresponding target embedding vector. In this case, when extracting target metadata, all target metadata corresponding to the target embedding vector is returned to the client as target completion data. Of course, the entire row of data containing the target embedding vector can also be returned to the client.

[0095] In another alternative embodiment, such as Figure 3 As mentioned above, when there are multiple target similarities, since the candidate metadata under different data attributes are scored separately, and the target embedding vector is determined based on the score value, the corresponding target embedding vector may be one or multiple.

[0096] When there is only one target embedding vector, the scores of the candidate metadata representing different data attributes are all at their maximum values. In this case, all metadata corresponding to the target embedding vector can be used as the target metadata. For example, in Table 1 of the above embodiment, if the candidate embedding vectors include embedding vectors E1 to E5, and the target similarity is [0.83, 0.72, 0.65, 0.92, 0.66], then for each data attribute from subject to location, the scores corresponding to subject A2, object B3, time T3, and location D3 corresponding to embedding vector E3 are all the highest. In this case, there is only one target embedding vector, E3. When extracting the target metadata, all metadata corresponding to embedding vector E3 is used as the target metadata.

[0097] When there are multiple target embedding vectors, when extracting target metadata, the metadata with the highest score under different data attributes is extracted as the target metadata. For example, in Table 1 of the above embodiment, if the candidate embedding vectors include embedding vectors E1 to E5, and the target similarity is [0.83, 0.72, 0.65, 0.92, 0.66].

[0098] After scoring the candidate metadata for different data attributes of embedding vectors E1 to E5, subject A4 is the metadata with the highest score for the subject data attribute, object B5 is the metadata with the highest score for the object data attribute, time T1 is the metadata with the highest score for the time data attribute, and location D1 is the metadata with the highest score for the location data attribute. Correspondingly, the target embedding vectors include embedding vector E1 and embedding vector E5. Therefore, when extracting the target metadata, subject A4, object B5, time T1, and location D1 are combined to form the target metadata, which is then returned to the client as target completion data.

[0099] In other words, in this embodiment of the application, when there are multiple target embedding vectors, the extraction of target metadata does not involve taking all the metadata corresponding to the target embedding vector as the target metadata. Instead, a portion of metadata is extracted from different target embedding vectors and combined to obtain a complete and non-repeating target metadata.

[0100] Therefore, the data completion method provided in this application takes the metadata corresponding to the largest score as the target metadata, and can extract the target completion data with the highest confidence to complete the text to be completed, thereby ensuring the accuracy of data completion.

[0101] In one optional embodiment, scoring is performed on candidate metadata for different data attributes corresponding to candidate embedding vectors, including:

[0102] Based on the target similarity, determine the Boltzmann score corresponding to each candidate embedding vector;

[0103] Determine whether identical candidate metadata exists within the same data attribute;

[0104] If it does not exist, the Boltzmann score will be used as the score value for the corresponding candidate metadata.

[0105] If it exists, perform the following steps:

[0106] The score value of the target candidate metadata is determined to be the corresponding Boltzmann score; the target candidate metadata is other metadata in the same data attribute besides the same candidate metadata.

[0107] The Boltzmann scores corresponding to the same candidate metadata are aggregated and calculated; and the aggregated calculation result is used as the score value of the same candidate metadata.

[0108] It is understandable that, based on the above embodiments, the reliability of scoring candidate metadata is crucial to ensuring data completion accuracy. Therefore, in an optional embodiment, based on the selected target similarity, the Boltzmann score corresponding to each different candidate embedding vector is calculated. Specifically, the calculation formula can be found in formula (1):

[0109]

[0110] Among them, Score m Let S be the Boltzmann score of the m-th candidate embedding vector. m Let T be the target similarity of the m-th candidate embedding vector, and let T be the temperature parameter. The temperature parameter T controls the degree of emphasis in the similarity assessment; a smaller T indicates a greater emphasis on the maximum value, while a larger T indicates a greater emphasis on the uniform value.

[0111] In an alternative embodiment, as the temperature parameter T approaches 0, the output is more biased towards the maximum value; that is, the larger the value, the higher the confidence level. In this case, the Boltzmann score... m The higher the value, the higher the confidence level of the corresponding candidate metadata. As the temperature parameter T approaches infinity, the output becomes more uniformly distributed; in this case, the more frequently a value appears, the higher the confidence level. Therefore, for the same Boltzmann score... m The more candidate metadata there are, the higher the confidence level of the corresponding candidate metadata. In a specific embodiment, the temperature parameter T can be selected according to actual needs, and the initial default value can be set to 1.

[0112] To further improve the reliability of target metadata extraction, the Boltzmann score is calculated. m Next, determine whether identical candidate metadata exists under the same data attribute. If not, then calculate the Boltzmann score. m This serves as the score for the corresponding candidate metadata. For ease of understanding, the following explanation will refer to Table 1 of the above embodiment.

[0113] For example, if the candidate embedding vectors include embedding vectors E1 to E5, and the target similarity is [0.83, 0.72, 0.65, 0.92, 0.66]. Under the object data attributes, the candidate metadata includes object B1 to object B5. Obviously, each candidate metadata is different. Therefore, Boltzmann scores are calculated for object B1 to object B5 respectively. m Where m is 5. The calculated Boltzmann scores are then used as the corresponding scores for object B1 to object B5.

[0114] In another optional embodiment, if identical candidate metadata exists under the same data attribute, then the Boltzmann scores corresponding to the identical candidate metadata need to be aggregated, and the result of the aggregation calculation is used as the score value of the identical metadata. It should be noted that the aggregation calculation may include, but is not limited to, summation and averaging.

[0115] Furthermore, since there are no other metadata with the same candidate metadata, the Boltzmann score can be directly used as the score value for the corresponding candidate metadata. For ease of understanding, Table 1 provided in the above embodiment will be used as an example for illustration.

[0116] For example, if the candidate embedding vectors include embedding vectors E1 to E5, and the target similarity is [0.83, 0.72, 0.65, 0.92, 0.66]. If the temperature parameter T is 1, the Boltzmann scores corresponding to embedding vectors E1 to E5 are calculated to be [0.21, 0.19, 0.18, 0.23, 0.18].

[0117] For the subject-side data attribute, the candidate metadata for embedding vectors E2 and E3 is the same: subject-side A2. In this case, the Boltzmann scores of 0.19 and 0.18 corresponding to subject-side A2 need to be summed to obtain an aggregation result of 0.37. Therefore, the final score of subject-side A2 is 0.37. Clearly, the final score of subject-side A2 is higher than the other scores. When extracting target metadata, subject-side A2 is used as the target completion data for the subject-side data attribute. Similarly, the same processing is performed on the candidate metadata of other data attributes, which will not be elaborated here.

[0118] To achieve efficient data completion, in another optional embodiment, the candidate metadata for different data attributes corresponding to the candidate embedding vector is scored, including:

[0119] Call the specified large model and score the relevance of the text to be completed to each candidate metadata.

[0120] Specifically, a large model can be specified to score the relevance between the request text and each candidate metadata, and the one with the highest score can be used as the target metadata. Similarly, the specified large model can include, but is not limited to, ChatGPT, GPT-4, Claude, and Tongyi 1000 Questions, and this application does not limit it.

[0121] In an optional embodiment, when no target similarity exists, the data completion method provided in this application includes:

[0122] Mark the target completion data as NaN value; and return the NaN value to the client;

[0123] Determine whether the manually completed data input by the user has been obtained within the preset time period;

[0124] If so, complete the data manually by filling in the text of the request for completion;

[0125] If not, after each preset period, proceed to the step of indexing the target embedded vector in the pre-built vector database, where the similarity between the indexed and queried vectors meets the preset conditions, and then execute the subsequent steps.

[0126] It is understandable that, such as Figure 3As shown, in a specific embodiment, there may be target embedding vectors that do not meet the preset conditions, that is, the similarity is not greater than the target similarity threshold. In this case, if the metadata corresponding to the embedding vector with low similarity is returned, the confidence level is too low, and the accuracy of the completed data does not meet the user's needs.

[0127] Therefore, to solve the aforementioned technical problems, the target completion data can be marked as NaN (NaN Imputation) values, which can be understood as null values. In other words, if the current recommendation results have too low credibility, the user can either complete the data themselves, or the system can wait until the vector database is richer before attempting completion again.

[0128] Specifically, after returning the NaN value to the client, if the user's manually completed data can be obtained within a preset time period, indicating that the user has chosen to complete the data themselves, then the requested text will be completed using the manually completed data.

[0129] If no manually completed data is received within the preset time period, it indicates that the user has given up on completing the data themselves. In this case, at preset intervals, such as one week, the process can be restarted to retrieve the target embedding vectors from the pre-built vector database whose similarity to the query vectors meets preset conditions, and then proceed with subsequent steps. In other words, at preset intervals, the indexing process is restarted to check if the vector database already contains data that meets the preset similarity conditions.

[0130] It is understood that, in specific embodiments, the vector database can be dynamically updated. If a request for completion is missing, the query vector and the corresponding text to be completed can be stored first, and an indexing attempt can be made once every preset period.

[0131] Of course, it should be noted that, to conserve storage resources, if the number of attempts reaches a preset number, the corresponding query vector and the corresponding text to be completed will be deleted, and a deletion signal will be returned to the client. Alternatively, a signal can be sent to the client, allowing the user to decide whether to delete. In another optional embodiment, when there is insufficient remaining space to store the query vector and the corresponding text to be completed, data that has consistently failed to be deleted will be removed based on the principle of first-in, first-out storage.

[0132] In one alternative embodiment, the step of constructing the vector database includes:

[0133] Obtain the dataset to be processed; the dataset to be processed includes the core data content, as well as metadata of different data attributes related to the core data content;

[0134] Batch convert the core data content of the dataset to be processed into embedded vectors;

[0135] Based on the FAISS vector index, an index is constructed for the embedded vector to obtain the index number;

[0136] Establish mapping relationships between embedded vectors, index numbers, and metadata of different data attributes to construct a vector database.

[0137] In a specific embodiment, a dataset to be processed is obtained. If the dataset is determined to be complete, it can be processed and stored in a vector database. The dataset to be processed includes multiple complete data entries, each containing core data content and metadata related to different data attributes.

[0138] To ensure data integrity, after obtaining the dataset to be processed, it undergoes preprocessing, specifically deduplication and missing data removal. Following this preprocessing, a specified large model (which may include, but is not limited to, GPT, BERT, and FastText) is used to batch convert the core data content of the dataset into embedding vectors. For example, the GPT large model can be used to batch generate embedding vectors from the core data content of the dataset, generating fixed-dimensional vectors of 1536.

[0139] Meanwhile, to accelerate subsequent indexing, in one optional embodiment, an IVF (Inverted File) index is constructed based on the FAISS vector index to obtain the index number. It is understood that the IVF index can reduce memory usage and employs a clustering partitioning algorithm, making it particularly suitable for fast and efficient query scenarios.

[0140] Furthermore, a mapping relationship is established between embedded vectors, index numbers, and metadata of different data attributes, thereby obtaining a vector database as shown in Table 1 of the above embodiment.

[0141] The data completion method provided in this application achieves a matching accuracy of over 83%, compared to previous string matching methods that required manual experience and had lower actual matching accuracy. Furthermore, previous string matching calculations were time-consuming; in application scenarios with 300,000 request texts to be completed, the query time was long and inefficient. This application, however, can achieve rapid indexing based on pre-constructed vector data using batch processing, thus improving data completion efficiency.

[0142] In the above embodiments, the data completion method has been described in detail. This application also provides an embodiment of a data completion device.

[0143] Figure 4This is a schematic diagram of the structure of a data completion device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes:

[0144] The request retrieval module 40 is used to retrieve the user-inputted request text to be completed;

[0145] The query vector acquisition module 41 is used to convert the request text to be completed into a query vector.

[0146] The target embedding vector indexing module 42 is used to index target embedding vectors whose similarity to the query vectors meets preset conditions in a pre-built vector database.

[0147] The target metadata extraction module 43 is used to extract the target metadata corresponding to the target embedding vector from the vector database;

[0148] The data output module 44 is used to complete the target metadata as target data and return it to the client.

[0149] Furthermore, the data completion device provided in this application embodiment also includes:

[0150] The candidate embedding vector determination module is used to extract target similarities with similarity greater than a threshold in order to obtain the candidate embedding vectors corresponding to the target similarities.

[0151] The target embedding vector determination module is used to use the candidate embedding vector as the target embedding vector when there is only one target similarity; when there are multiple target similarities, it scores the candidate metadata corresponding to different data attributes of the candidate embedding vector; and uses the candidate embedding vector corresponding to the candidate metadata with the largest score as the target embedding vector.

[0152] The target metadata determination module is used to determine all metadata corresponding to the target embedding vector as target metadata when there is only one target similarity; when there are multiple target similarities, the metadata corresponding to the one with the largest score is used as target metadata.

[0153] The Boltzmann score determination module is used to determine the Boltzmann score corresponding to each candidate embedding vector based on the target similarity.

[0154] The score determination module is used to determine whether there are identical candidate metadata in the same data attribute. If not, the Boltzmann score is used as the score of the corresponding candidate metadata. If it exists, the following steps are performed: determine the score of the target candidate metadata as the corresponding Boltzmann score; the target candidate metadata is other metadata in the same data attribute besides the identical candidate metadata; aggregate the Boltzmann scores corresponding to the identical candidate metadata; and use the aggregate calculation result as the score of the identical candidate metadata.

[0155] The model invocation module is used to invoke a specified large model and score the relevance of the text to be completed to each candidate metadata.

[0156] The tagging module is used to mark the target completion data as NaN values ​​and return the NaN values ​​to the client.

[0157] The processing module is used to determine whether the user-inputted manual completion data has been obtained within a preset time period; if so, the manual completion data is used to complete the text to be completed; if not, after a preset period of time, the module enters the step of indexing the target embedding vector in the pre-built vector database where the similarity between the index and query vectors meets the preset conditions, and then executes the subsequent steps.

[0158] The dataset acquisition module is used to acquire the dataset to be processed; the dataset to be processed includes the core data content and metadata of different data attributes related to the core data content.

[0159] The embedding vector generation module is used to convert the core data content in the dataset to be processed into embedding vectors in batches.

[0160] The index building module is used to build an index for the embedded vector based on the FAISS vector index, and obtain the index number;

[0161] The mapping relationship establishment module is used to establish mapping relationships between embedded vectors, index numbers, and metadata of different data attributes in order to build a vector database.

[0162] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: a memory 50 for storing computer programs;

[0163] The processor 51 is configured to implement the steps of the data completion method as described in the above embodiments when executing a computer program.

[0164] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.

[0165] The processor 51 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 51 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 51 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 51 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 51 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0166] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 50 is used to store at least the following computer program 501, which, after being loaded and executed by the processor 51, is capable of implementing the relevant steps of the data completion method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, and the storage method may be temporary or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include, but is not limited to, the relevant data involved in the data completion method.

[0167] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54, a power supply 55, and a communication bus 56.

[0168] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0169] The electronic device provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the data completion method described in the above embodiments.

[0170] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A method of completing data, characterized by, The method includes: Get the user-inputted text to be completed; Convert the request text to be completed into a query vector; In a pre-built vector database, the index is the target embedding vector whose similarity to the query vector meets a preset condition; Extract the target metadata corresponding to the target embedding vector from the vector database; The target metadata is returned to the client as target completion data; The target embedding vectors whose similarity to the index and the query vector meets preset conditions include: Extract the target similarity that is greater than a threshold to obtain the candidate embedding vector corresponding to the target similarity; When the target similarity is one, the candidate embedding vector is used as the target embedding vector; When there are multiple target similarities, the candidate metadata corresponding to different data attributes of the candidate embedding vector is scored; and the candidate embedding vector corresponding to the candidate metadata with the largest score is taken as the target embedding vector.

2. The data completion method of claim 1, wherein, The step of extracting the target metadata corresponding to the target embedding vector from the vector database includes: When the target similarity is one, all metadata corresponding to the target embedding vector is used as the target metadata; When there are multiple target similarities, the metadata corresponding to the one with the largest score is used as the target metadata.

3. The data completion method of claim 1, wherein, The scoring of candidate metadata for different data attributes corresponding to the candidate embedding vector includes: Based on the target similarity, determine the Boltzmann score corresponding to each candidate embedding vector; Determine whether identical candidate metadata exists within the same data attribute; If it does not exist, the Boltzmann score is used as the score value of the corresponding candidate metadata; If it exists, perform the following steps: The score value of the target candidate metadata is determined to be the corresponding Boltzmann score; the target candidate metadata refers to other metadata in the same data attribute besides the same candidate metadata. The Boltzmann scores corresponding to the same candidate metadata are aggregated and calculated; and the aggregated calculation result is used as the score value of the same candidate metadata.

4. The data completion method as described in claim 1, characterized in that, The scoring of candidate metadata for different data attributes corresponding to the candidate embedding vector includes: The specified large model is invoked to score the correlation between the request text to be completed and each of the candidate metadata.

5. The data completion method as described in claim 1, characterized in that, In the absence of the target similarity, the method includes: The target completed data is marked as a NaN value; and the NaN value is returned to the client. Determine whether the manually completed data input by the user has been obtained within the preset time period; If so, the request text to be completed is completed using the manually completed data; If not, after each preset period, proceed to the step of indexing the target embedding vector in the pre-built vector database, where the similarity between the target embedding vector and the query vector meets the preset conditions, and then execute the subsequent steps.

6. The data completion method as described in claim 1, characterized in that, The steps for constructing the vector database include: Obtain the dataset to be processed; the dataset to be processed includes core data content and metadata of different data attributes related to the core data content; The core data content in the dataset to be processed is converted into embedded vectors in batches. Based on the FAISS vector index, an index is constructed for the embedded vector to obtain the index number; Establish a mapping relationship between the embedded vector, the index number, and the metadata of the different data attributes to construct the vector database.

7. A data completion device, characterized in that, The device includes: The request retrieval module is used to retrieve the user-inputted request text to be completed; The query vector acquisition module is used to convert the request text to be completed into a query vector; The target embedding vector indexing module is used to index target embedding vectors in a pre-built vector database that have a similarity to the query vector that meets a preset condition. The target metadata extraction module is used to extract the target metadata corresponding to the target embedding vector from the vector database; The data output module is used to return the target metadata as target completion data to the client; A candidate embedding vector determination module is used to extract the target similarity with a similarity greater than a threshold to obtain the candidate embedding vector corresponding to the target similarity. The target embedding vector determination module is used to: when the target similarity is one, use the candidate embedding vector as the target embedding vector; when the target similarity is multiple, score the candidate metadata corresponding to different data attributes of the candidate embedding vector; and use the candidate embedding vector corresponding to the candidate metadata with the largest score as the target embedding vector.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data completion method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the data completion method according to any one of claims 1 to 6.

Images