Vectorization model determination method and device, medium and product

Abstract

The invention discloses a vectorization model determination method and device, a medium and a product, and relates to the field of data retrieval, and the method comprises the steps: carrying out local vectorization based on n vectorization models and j segmentation lengths, and obtaining j * n local vector libraries; acquiring m pieces of test data; searching the m test data by utilizing j * n local vector libraries, and determining a corresponding return result; respectively carrying out Boolean vector assignment according to a return result to obtain Boolean vectors; determining a retrieval score according to the Boolean vector; and plug-in of the knowledge base is carried out according to the local vector library corresponding to the Boolean vector with the highest retrieval score. According to the method, the appropriate vector model and the text segmentation granularity can be rapidly and accurately determined at the same time, and the optimal retrieval performance of the knowledge base is achieved.

Description

Technical Field

[0001] The present application relates to the field of data retrieval, and in particular to a vectorized model determination method, device, medium and product. Background Art

[0002] The plug-in knowledge base is a unique data storage and management tool. Its main function is to provide knowledge information in real time to expand or limit the knowledge scope of the language model. Unlike the basic model, the plug-in knowledge base does not rely on language data training, but works by organizing and storing structured and unstructured knowledge. The plug-in knowledge base can provide more comprehensive and accurate knowledge support to enhance the capabilities of the language model. When using the plug-in knowledge base, you first need to select the appropriate vector model and the granularity of text segmentation to achieve the best knowledge base retrieval performance.

[0003] Computers are good at processing numerical data, while non-numerical data, such as text and images, are more difficult to process. By vectorizing, they are converted into numerical vectors, making it easier for computers to process and analyze using their powerful computing power. Vectorization can express data features and relationships in numerical form. For example, in word vectors, similar words are close in the vector space, while different words are far apart. The semantic similarity can be determined by calculating the vector distance. Compared with raw non-numerical data, vectorized data is more efficient in computation. When large language models perform matrix operations, vectorized data can be quickly calculated using an efficient matrix operation library.

[0004] The choice of vectorization model has an important impact on the accuracy and performance of large language model retrieval enhancement. At the same time, in the vectorization implementation step, the granularity of file segmentation has an important impact on large model retrieval enhancement. Text segmentation refers to the process of dividing long texts into shorter text fragments. Since long texts usually contain a lot of information and details, directly processing these long texts may lead to insufficient technical resources and low computing efficiency. Therefore, by dividing long texts into small pieces, text processing and analysis can be performed more efficiently. Since long texts usually contain a lot of information and details, directly processing these long texts may lead to insufficient computing resources and low computing efficiency. By dividing long texts into small pieces, text processing and analysis can be performed more efficiently.

[0005] How to quickly and accurately determine the appropriate vector model and the granularity of text segmentation is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of this application is to provide a vectorization model determination method, device, medium and product, which can quickly and accurately determine the appropriate vector model and the granularity of text segmentation at the same time, so as to achieve optimal knowledge base retrieval performance.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for determining a vectorized model, the method comprising:

[0009] Based on n vectorization models and j segmentation lengths, local vectorization is performed to obtain j*n local vector libraries;

[0010] Get m test data;

[0011] Use j*n local vector libraries to search for m test data and determine the corresponding return results;

[0012] According to the returned results, Boolean vectors are assigned respectively to obtain Boolean vectors;

[0013] Determine a retrieval score based on the Boolean vector;

[0014] The external knowledge base is constructed based on the local vector library corresponding to the Boolean vector with the highest retrieval score.

[0015] 2. The method for determining a vectorized model according to claim 1, wherein the step of obtaining m test data comprises:

[0016] Get m test articles;

[0017] A fragment of a set number of words in the test article is used as the test data.

[0018] Optionally, the using j*n local vector libraries to search for m test data and determine corresponding return results specifically includes:

[0019] Take m test data as search terms;

[0020] According to the k-th search term, j*n local vector libraries are used for searching, and j*n groups of results are returned.

[0021] Optionally, Boolean vectors are assigned values ​​according to the returned results to obtain Boolean vectors, specifically including:

[0022] The k-th search term uses the i-th group of local vector libraries to get the returned results;

[0023] According to the returned results, a Boolean matrix is ​​obtained;

[0024] Get a Boolean vector from a Boolean matrix.

[0025] Optionally, determining the search score according to the Boolean vector specifically includes:

[0026] Using the formula Determine the retrieval score B of the i-th group of local vector libraries i ;

[0027] Among them, A j0 The Boolean vector obtained by using the i-th set of local vector libraries for the j-th test data.

[0028] In a second aspect, the present application provides a vectorization model determination device, the vectorization model determination device comprising:

[0029] A local vector library determination module is used to perform local vectorization based on n vectorization models and j segmentation lengths to obtain j*n local vector libraries;

[0030] A test data acquisition module is used to acquire m test data;

[0031] A return result determination module is used to retrieve m test data using j*n local vector libraries and determine the corresponding return results;

[0032] A Boolean vector determination module is used to assign Boolean vectors according to the returned results to obtain Boolean vectors;

[0033] An optimal local vector library determination module, used for determining a retrieval score according to a Boolean vector;

[0034] The plug-in knowledge base retrieval module is used to search the plug-in knowledge base according to the local vector library corresponding to the Boolean vector with the highest retrieval score.

[0035] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the vectorization model determination method.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the vectorization model determination method when executed by a processor.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the vectorization model determination method when executed by a processor.

[0038] According to the specific embodiments provided in this application, this application has the following technical effects:

[0039] The present application provides a vectorization model determination method, device, medium and product. Based on n vectorization models and j segmentation lengths, local vectorization is performed to obtain j*n local vector libraries, that is, the vectorization model is combined with different segmentation lengths, and then different test data are retrieved through the combined local vector library. The local vector library retrieval obtains 10 return results. The higher the result ranking, the more relevant it is to the search content, and then the search score is determined according to the Boolean vector; the optimal combination of the vectorization model and the segmentation length is quickly and accurately determined through the search score, and the optimal knowledge base retrieval performance is further achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a schematic diagram of a process flow of a vectorized model determination method in one embodiment of the present application;

[0042] Figure 2 A schematic diagram of the principle of the method for vectorization model determination;

[0043] Figure 3 Schematic diagram of the process of determining the local vector library. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0046] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a vectorization model determination method is provided, which includes the following S101 to S106.

[0047] S101, based on n vectorization models and j segmentation lengths, local vectorization is performed to obtain j*n local vector libraries, namely V0, V1, V2, ..., Vn-1, Vn, ..., Vj*n-1. The specific vectorization path is as follows: Figure 3 As shown; the lengths of j segments are S0, S1, ..., Sv ..., Sj-1;

[0048] S102, obtaining m test data, namely Q0, Q1, Q2, ..., Qm-1;

[0049] S102 specifically includes:

[0050] S21, obtain m test articles, namely T1, T2, T3, ...Tm-1; the strongly related text retrieved by Q0 is T0, the strongly related text retrieved by Q1 is T1, and so on;

[0051] S22, taking a segment of a set number of characters in the test article as test data. The set number of characters is about 200-1000 characters;

[0052] S103, using j*n local vector libraries to search for m test data and determine corresponding return results;

[0053] The standardized format of the returned results is: {"[Source:1](a.txt)\n\nIntelligent normal operation focuses on the stable support of key core technologies and equipment manufacturing.","[Source:2](b.txt)\n\nThe operating environment of AI enterprises is taking shape","[Source:3](c.txt)\n\nArtificial intelligence will change the lives of many people in the future. I think some people are afraid of this, but technology is always neutral. What matters is how you use it to achieve both.","[Source:4](d.txt)\n\nFrom mechanized replacement to automated reduction of manpower, and then to intelligent unmanned, the intelligent construction of mines is considered to be an important way to improve the inherent safety level of mines and promote the transformation of mine safety governance mode to pre-prevention.","[Source:5](e.txt)\n\nIntel held the "Intel AI Summit" event for AI developers in Tokyo on the 6th. Strengthen cooperation with artificial intelligence hardware / software developers to build a strong ecosystem.","[Source:6](f.txt )\n\nBy 2026, the proportion of intelligent production capacity in coal mines nationwide will be no less than 60%, the normal operation rate of intelligent working faces will be no less than 80%, and a group of intelligent single-shift operators will not exceed 50 people will be created.","[Source:7](g.txt)\n\nEnsure normal operation\nAt present, the intelligentization of mines has basically solved the problem from scratch, but it is still in its infancy as a whole. ","[Source:8](h.txt)\n\nThe AIxiv column is a column where the heart of the machine publishes academic and technical content. In the past few years, the AIxiv column of Machine Heart has received more than 2,000 reports,","[Source:9](i.txt)\n\nMr. Gondor pointed out the four characteristics of artificial intelligence computers: "speed", "cost", "safety" and "personalization". ","[Source:10](i.txt)\n\nThe Robotaxi "bridgehead" of Carrot Run was attacked. \nRecently, many complaints about the "Carrot Run" Robotaxi service have emerged on the public message board in Wuhan. "}.

[0054] Among them, [Source: 1] indicates the serial number of the returned result; (a.txt) indicates that the search result comes from the original document a.txt; a paragraph is the relevant paragraph retrieved according to the search term, that is, the paragraph segmented during vectorization.

[0055] For m (Q0, Q1, Q2, Q3, ... Qm-1) test data as search terms, the kth search term (Qk) can return j*n groups of results by searching from j*n vector libraries (V0, V1, V2... Vj*n-1). (The result returned by each search term searching each vector library is: 1*10 vector text; each search term Qk returns j*n groups of results from the above j*n vector libraries, and each result is 1*10 vector text).

[0056] Suppose the search return vector text of the k-th search term in the i-th group (i-th vector library) vector library is (Aki0, Aki1, ... Aki9);

[0057] S103 specifically includes:

[0058] S31, taking m test data as search terms;

[0059] S32, according to the kth search word, use j*n local vector libraries to perform search, and obtain j*n groups of returned results.

[0060] S104, performing Boolean vector assignments according to the returned results to obtain Boolean vectors;

[0061] S104 specifically includes:

[0062] S41, the kth search term uses the ith group of local vector libraries to obtain the returned results;

[0063] S42, obtaining a Boolean matrix according to the returned result;

[0064] S43, obtaining a Boolean vector according to the Boolean matrix.

[0065] The kth search term appears in the returned vector text (Aki0, Aki1, ... Aki9). According to the search term selection method, when the text Tk and the search term Qk appear in the result Aki0, the current search result is considered accurate. Therefore, if the text Tk and the search term Qk appear in the result position t (the value range of t is 0-9), the corresponding result Boolean vector (size 1*10) is (0, 0, ... 1, ... 0), the t position is 1, and the other positions are 0.

[0066] When t = 0, it is considered that the retrieval accuracy for this vectorized model is high and the retrieval performance is good. It can be concluded that the Boolean matrix returned by the retrieval of m test data in the i-group vector library is set as A m*10 ;

[0067] S105, determining a search score according to the Boolean vector;

[0068] Using the formula Determine the retrieval score B of the i-th group of local vector libraries i ; B i The higher it is, the better the retrieval effect is;

[0069] Among them, A j0 The Boolean vector obtained by using the i-th set of local vector libraries for the j-th test data.

[0070] As a specific is reasonable, in the result subscript i, B i The value of is the largest, the u-th vectorization model in the local vector library VSu is selected, and the text segmentation length selected for text segmentation is Sv.

[0071] S106, searching for an external knowledge base based on the local vector library corresponding to the Boolean vector with the highest search score.

[0072] Based on the same inventive concept, the embodiment of the present application also provides a vector model determination device for implementing the vector model determination method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more vector model determination device embodiments provided below can refer to the limitations of the vector model determination method above, and will not be repeated here.

[0073] In an exemplary embodiment, a vectorization model determination device is provided, comprising:

[0074] A local vector library determination module is used to perform local vectorization based on n vectorization models and j segmentation lengths to obtain j*n local vector libraries;

[0075] A test data acquisition module is used to acquire m test data;

[0076] A return result determination module is used to retrieve m test data using j*n local vector libraries and determine the corresponding return results;

[0077] A Boolean vector determination module is used to assign Boolean vectors according to the returned results to obtain Boolean vectors;

[0078] An optimal local vector library determination module, used for determining a retrieval score according to a Boolean vector;

[0079] The plug-in knowledge base retrieval module is used to search the plug-in knowledge base according to the local vector library corresponding to the Boolean vector with the highest retrieval score.

[0080] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a vectorized model determination method is implemented.

[0081] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0082] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0084] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0085] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0086] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0087] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for determining a vectorized model, characterized in that: The vectorized model determination method comprises: Based on n vectorization models and j segmentation lengths, local vectorization is performed to obtain j*n local vector libraries; Get m test data; Use j*n local vector libraries to search for m test data and determine the corresponding return results; According to the returned results, Boolean vectors are assigned respectively to obtain Boolean vectors; Determine a retrieval score based on the Boolean vector; The external knowledge base is constructed based on the local vector library corresponding to the Boolean vector with the highest retrieval score.

2. The method for determining a vectorized model according to claim 1, characterized in that: The obtaining of m test data specifically includes: Get m test articles; A fragment of a set number of words in the test article is used as the test data.

3. The method for determining a vectorized model according to claim 1, characterized in that: The method of using j*n local vector libraries to search for m test data and determine corresponding return results specifically includes: Take m test data as search terms; According to the k-th search term, j*n local vector libraries are used for searching, and j*n groups of results are returned.

4. The method for determining a vectorized model according to claim 3, characterized in that: According to the returned results, Boolean vectors are assigned respectively to obtain Boolean vectors, including: The k-th search term uses the i-th group of local vector libraries to get the returned results; According to the returned results, a Boolean matrix is ​​obtained; Get a Boolean vector from a Boolean matrix.

5. The method for determining a vectorized model according to claim 1, characterized in that: Determining the search score according to the Boolean vector specifically includes: Using the formula Determine the retrieval score B of the i-th group of local vector libraries i ; Among them, A j0 The Boolean vector obtained by using the i-th set of local vector libraries for the j-th test data.

6. A vector model determination device, characterized in that: The vectorized model determination device comprises: A local vector library determination module is used to perform local vectorization based on n vectorization models and j segmentation lengths to obtain j*n local vector libraries; A test data acquisition module is used to acquire m test data; A return result determination module is used to retrieve m test data using j*n local vector libraries and determine the corresponding return results; A Boolean vector determination module is used to assign Boolean vectors according to the returned results to obtain Boolean vectors; An optimal local vector library determination module, used for determining a retrieval score according to a Boolean vector; The plug-in knowledge base retrieval module is used to search the plug-in knowledge base according to the local vector library corresponding to the Boolean vector with the highest retrieval score.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the vectorization model determination method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vectorization model determination method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vectorization model determination method according to any one of claims 1 to 5 is implemented.

Images