Information processing method and device, computing equipment cluster and storage medium

Abstract

The invention provides an information processing method, an information processing device, a computing device cluster and a computer readable storage medium. The method comprises the following steps: acquiring a plurality of to-be-stored texts, wherein the plurality of to-be-stored texts comprise a first text and a second text; processing the first text by adopting a first embedding model to obtain a first result; a second embedding model is adopted to process the second text, a second result is obtained, and the dimension number of the first embedding model is different from that of the second embedding model; and storing the first result and the second result in a knowledge base. According to the method, storage resources can be saved in the knowledge base construction process, and the situation that the number of vector dimensions generated by the embedded model is too high is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an information processing method, an information processing device, a computing device cluster and a computer readable storage medium. BACKGROUND

[0002] In the art, there is a technology of converting a segment of text containing certain knowledge information into a vector by embedding a model, so as to construct a knowledge base by the vector. However, in the prior art, the same embedding model is used to embed all segments of all texts in the same way, and vectors of the same dimension are obtained. Such an embedding method is prone to cause the problem of embedding some segments by using an embedding model with too high dimension, so as to obtain vectors with too high dimension, thereby causing waste of storage resources for storing the knowledge base. Therefore, there is an urgent need in the art for a knowledge base construction technology capable of saving storage resources and avoiding too high dimension of vectors generated by an embedding model. SUMMARY

[0003] To this end, the present application is committed to providing an information processing method, an information processing device, a computing device cluster and a computer readable storage medium, which can save storage resources and avoid too high dimension of vectors generated by an embedding model in the process of constructing a knowledge base.

[0004] In one aspect, the present application provides an information processing method, comprising: obtaining a plurality of to-be-stored texts, the plurality of to-be-stored texts comprising a first text and a second text; processing the first text by using a first embedding model to obtain a first result; processing the second text by using a second embedding model to obtain a second result, the dimension number of the first embedding model and the second embedding model being different; and storing the first result and the second result to a knowledge base.

[0005] According to the present aspect, different texts are processed by embedding models with different dimension numbers, which is beneficial to selecting an embedding model with a suitable dimension number according to the characteristics of the text itself to perform embedding operation, so as to obtain a vector with a suitable dimension number for constructing a knowledge base. In this way, it is beneficial to avoid selecting an embedding model with too large dimension number to perform embedding operation on a relatively short text or a text that does not need to be represented by a large number of dimensions, so as to cause the dimension number of the obtained vector to be too large, thereby causing waste of computing and storage resources.

[0006] In a possible implementation of the present application, the first text is processed by using the first embedding model to obtain a first result, including: dividing the first text into multiple segments according to the characteristics of the first text; embedding the multiple segments by using the first embedding model to obtain multiple first vectors as the first result. Wherein, the second text is processed by using the second embedding model to obtain a second result, including: dividing the second text into multiple segments according to the characteristics of the second text; embedding the multiple segments by using the second embedding model to obtain multiple second vectors as the second result.

[0007] According to the present implementation, different ways are used to divide and embed the segments according to different characteristics of the text, so that the vectors obtained by embedding have different dimensions, which can make the number of dimensions of the vectors obtained by embedding match the characteristics of the text and the length of the segments, avoiding the problem of waste of storage resources caused by using the same embedding model to embed the segments of all texts.

[0008] In a possible implementation of the present application, the characteristics of the first text or the characteristics of the second text include one or more of the following: white paper type, product manual type, and question and answer type.

[0009] According to the present implementation, white papers, product manuals, and questions and answers are common texts that usually contain a lot of knowledge information, dividing the knowledge texts into these three types is conducive to further processing (including segment division and embedding) according to different characteristics of the text, so that the constructed knowledge base has more knowledge information and the vectors therein have appropriate dimensions.

[0010] In a possible implementation of the present application, the first text is divided into multiple segments according to the characteristics of the first text, including: if the first text belongs to the white paper type, the first text is divided into multiple segments by using a first method, and the first method includes dividing according to a first fixed length.

[0011] According to the present implementation, since the paragraphs of the text of the white paper type are long and often have long narrative and explanation sentences, segmenting the text of the white paper type according to a fixed length can make appropriate division according to the characteristics of the text of the white paper type, thereby obtaining vectors with appropriate dimensions.

[0012] In a possible implementation of the present application, the first text is divided into multiple segments according to the characteristics of the second text, including: if the second text belongs to the product manual type, the second text is divided into multiple segments by using a second method, and the second method includes dividing according to a second fixed length smaller than the first fixed length.

[0013] According to the present implementation, since the product specification type text paragraph is shorter than the white paper type text paragraph, the product specification type text is divided according to the second fixed length smaller than the first fixed length, which is suitable for the characteristics of the type of text, so as to obtain a vector with a suitable dimension.

[0014] In a possible implementation of the present application, the first text is divided into multiple segments according to the characteristics of the first text, including: if the first text belongs to the question and answer type, the first text is divided into multiple segments by using a first mode, wherein the first mode includes dividing according to the paragraph.

[0015] According to the present implementation, since the paragraph of the question and answer type text is short and usually consists of short questions and answers, the question and answer type text can be divided according to the paragraph, so as to divide the segments with a suitable length according to the characteristics of the question and answer type text, thereby obtaining a vector with a suitable dimension.

[0016] In a possible implementation of the present application, the method further includes: embedding the inquiry sentence of the user by using a first embedding model to obtain a first retrieval vector; embedding the inquiry sentence by using a second embedding model to obtain a second retrieval vector; and retrieving in the knowledge base through the first retrieval vector and the second retrieval vector to obtain multiple retrieval results.

[0017] According to the present implementation, when retrieving in the knowledge base constructed based on the above-mentioned mode, the inquiry sentence is embedded in different dimensions by using different embedding models used when constructing the knowledge base to obtain retrieval vectors in different dimensions, so as to retrieve in the vector database in different dimensions by using the retrieval vectors in different dimensions, which can fully utilize the knowledge contained in different types of vectors, perform comprehensive retrieval, fully extract the knowledge information contained in the vectors in different dimensions, and obtain more optimized retrieval results, while saving the storage resources required for storing the knowledge base.

[0018] In a possible implementation of the present application, the retrieving in the knowledge base through the first retrieval vector and the second retrieval vector to obtain multiple retrieval results includes: retrieving in the knowledge base through the first retrieval vector to obtain X first retrieval results; retrieving in the knowledge base through the second retrieval vector to obtain X second retrieval results; rearranging the X first retrieval results and the X second retrieval results according to the similarity; and selecting the first Y retrieval results from high to low as the multiple retrieval results.

[0019] According to the present implementation, the search results obtained from the knowledge base are rearranged according to the similarity, and the several search results with the highest similarity are selected as the final results, so that the search results as accurate as possible and meeting the user's requirements can be obtained. The search results meeting the requirements most can be filtered out by rearranging according to the unified similarity standard, and efficient matching is realized.

[0020] In a possible implementation of the present application, the method further comprises: constructing a prompt word according to the query statement and the plurality of search results; inputting the query statement and the prompt word into the large language model to obtain an answer to the query statement.

[0021] According to the present implementation, by searching and constructing the prompt word in the above manner, the quality of the answer can be ensured, while the vector dimension of the short text segment is reduced, so that the database storage space required by the question and answer method is smaller. In addition, by constructing the prompt word for the plurality of search results ranked at the top, and inputting the prompt word and the query statement into the large language model to obtain the answer, important and key information can be obtained through sufficient background knowledge, so that the quality and accuracy of the answer are improved.

[0022] In a possible implementation of the present application, the constructing of the prompt word according to the query statement and the plurality of search results comprises: decomposing the query statement into a plurality of questions, each question corresponding to one search result of the plurality of search results; combining each question with the corresponding search result; and taking the plurality of combinations of the plurality of questions and the plurality of search results as the prompt word.

[0023] According to the present implementation, by decomposing the query statement into a plurality of questions, and combining the questions and the answers into a plurality of combinations, and taking the plurality of combinations as the prompt word, the user can be prompted from different aspects and directions about the answer content that the user wants to obtain, and the most comprehensive and accurate answer result can be obtained.

[0024] In another aspect, the present application also provides an information processing apparatus, comprising: an acquisition module configured to acquire a plurality of to-be-stored texts, the plurality of to-be-stored texts comprising a first text and a second text; a first processing module configured to process the first text by using a first embedding model to obtain a first result; a second processing module configured to process the second text by using a second embedding model to obtain a second result, the number of dimensions of the first embedding model and the second embedding model being different; and a storage module configured to store the first result and the second result to a knowledge base.

[0025] In a possible implementation of the present application, the first processing module is further configured to: divide the first text into a plurality of segments according to a characteristic of the first text; and embed the plurality of segments respectively by using the first embedding model to obtain a plurality of first vectors as the first result. The second processing module is further configured to: divide the second text into a plurality of segments according to a characteristic of the second text; and embed the plurality of segments respectively by using the second embedding model to obtain a plurality of second vectors as the second result.

[0026] In a possible implementation of the present application, the characteristic of the first text or the characteristic of the second text comprises one or more of the following: a white paper type, a product manual type, and a question and answer type.

[0027] In a possible implementation of the present application, the first processing module is further configured to: if the first text belongs to the white paper type, divide the first text into a plurality of segments by using a first manner, and the first manner comprises dividing according to a first fixed length.

[0028] In a possible implementation of the present application, the second processing module is further configured to: if the second text belongs to the product manual type, divide the second text into a plurality of segments by using a second manner, and the second manner comprises dividing according to a second fixed length smaller than the first fixed length.

[0029] In a possible implementation of the present application, the first processing module is further configured to: if the first text belongs to the question and answer type, divide the first text into a plurality of segments by using a first manner, and the first manner comprises dividing according to a paragraph.

[0030] In a possible implementation of the present application, the apparatus further comprises: a first embedding module configured to embed the inquiry sentence of the user by using the first embedding model to obtain a first retrieval vector; a second embedding module configured to embed the inquiry sentence by using the second embedding model to obtain a second retrieval vector; and a retrieval module configured to perform retrieval in the knowledge base by using the first retrieval vector and the second retrieval vector to obtain a plurality of retrieval results.

[0031] In a possible implementation of the present application, the retrieval module is further configured to: perform retrieval in the knowledge base by using the first retrieval vector to obtain X first retrieval results; perform retrieval in the knowledge base by using the second retrieval vector to obtain X second retrieval results; rearrange the X first retrieval results and the X second retrieval results according to a similarity; and select a first Y retrieval results with a highest similarity to a lowest similarity as the plurality of retrieval results.

[0032] In one possible implementation of this application, the apparatus further includes: a construction module for constructing prompt words based on the query statement and multiple search results; and an input module for inputting the query statement and prompt words into a large language model to obtain an answer for the query statement.

[0033] In one possible implementation of this application, the construction module is further configured to: decompose the query statement into multiple questions, each question corresponding to one of the multiple search results; combine each question with the corresponding search result; and use the multiple combinations of multiple questions and multiple search results as prompt words.

[0034] On the other hand, this application also provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the above-described information processing method.

[0035] In another aspect, this application provides a computer-readable storage medium storing a computer program for performing the above-described information processing method.

[0036] On the other hand, this application provides a computer program product, including program code, which, when a computer runs the computer program product, causes the computer to implement the above-described information processing method.

[0037] Any of the information processing devices, computing device clusters, computer-readable storage media, or computer program products provided above are used to execute the knowledge base construction method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding schemes in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0038] The specific embodiments of this application are described in detail below with reference to the accompanying drawings, wherein:

[0039] Figure 1 A schematic diagram of the architecture of an information processing method according to an embodiment of this application is shown;

[0040] Figure 2 A flowchart illustrating an information processing method according to an embodiment of this application is shown;

[0041] Figure 3 A flowchart illustrating an information processing method according to another embodiment of this application is shown;

[0042] Figure 4 A flowchart illustrating an information processing method according to another embodiment of this application is shown;

[0043] Figure 5 Showing according to Figure 4 A schematic diagram of the prompt words in the information processing method of the embodiment; Figure 6 This diagram shows a schematic representation of the structure of an information processing apparatus according to an embodiment of the present application;

[0044] Figure 7 This diagram illustrates the structure of a computing device according to an embodiment of the present application.

[0045] Figure 8 This diagram illustrates the structure of a computing device cluster according to an embodiment of the present application.

[0046] Figure 9 A schematic diagram of the structure of a computing device cluster according to another embodiment of this application is shown. Detailed Implementation

[0047] To enable those skilled in the art to more clearly understand the concepts and ideas of this application, the application is described in detail below with reference to specific embodiments. It should be understood that the embodiments given herein are only a part of all possible embodiments of this application. After reading the specification of this application, those skilled in the art are capable of making improvements, modifications, or substitutions to parts or the entirety of the following embodiments, and such improvements, modifications, or substitutions are also included within the scope of protection claimed in this application.

[0048] In this document, the terms "one," "an," and other similar words are not intended to indicate that only one of the described things exists, but rather that the description refers only to one of the described things, which may have one or more. In this document, the terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" means that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meanings of the terms "comprising," "including," and other similar words should be considered open-ended, not closed-ended. For example, "A includes B" means that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0049] In this document, the terms "first," "second," and other similar terms are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. In this document, the terms "embodiment," "this embodiment," "an embodiment," or "an example" do not indicate that the description applies only to one specific embodiment, but rather that such description may also be applicable to one or more other embodiments. Those skilled in the art will understand that any description made herein with respect to one embodiment can be substituted, combined, or otherwise combined with the descriptions in one or more other embodiments, and the new embodiments resulting from such substitutions, combinations, or other combinations are readily conceived by those skilled in the art and fall within the scope of protection of this application.

[0050] Retrieval-augmented generation (RAG) refers to optimizing the output of large language models to reference authoritative knowledge bases beyond the training data sources before generating responses. Large Language Models (LLMs) are trained on massive amounts of data, using billions of parameters to generate raw outputs for tasks such as answering questions, translating languages, and completing sentences. Building on the already powerful capabilities of LLMs, RAG extends them to access domain-specific or organizational internal knowledge bases, all without requiring model retraining. This is a cost-effective way to improve LLM outputs, ensuring they remain relevant, accurate, and usable across various contexts.

[0051] Embedding refers to the mapping of high-dimensional discrete features to a relatively low-dimensional continuous vector space. In the RAG system, document embedding is performed to convert text into computable vectors, facilitating subsequent similarity calculations and retrieval. Text embedding relies on text embedding models. Embedding models are specifically designed to learn how to transform raw data (such as text, images, or other categories of data) into dense vectors (i.e., embeddings). This process involves learning the underlying structure and features of the data, enabling the representation of this raw data in a form more suitable for machine learning models. In other words, text embedding models can preserve the semantic features of text, which may include vocabulary, grammar, semantics, sentiment, emotion, topic, context, etc., thus allowing for the finding of semantically similar text fragments during retrieval. For example, embedding the sentence "Your text string goes here" using the text-embedding-ada-002 model will generate a 1536-dimensional vector, resulting in: "-0.006929283495992422,-0.005336422007530928,……-4547132266452536e-05,-0.024047505110502243", which is an array of length 1536. This vector contains all the features of the sentence, including vocabulary and grammar, and can be stored in a vector database for subsequent semantic searches.

[0052] In some techniques in this field, low-dimensional embedding models do not perform well for embedding long texts. Therefore, in order to achieve good embedding results, high-dimensional embedding models are used regardless of the size of the document fragment, resulting in high-dimensional vectors after embedding, large storage space, large amount of similarity calculation during retrieval, and slow retrieval speed.

[0053] Therefore, in some embodiments of this application, multiple different embedding models are selected, and the appropriate text fragment length range is calculated based on their dimensionality data. When embedding text fragments in the RAG system, the lowest-dimensional embedding model suitable for the text length is selected for embedding, reducing the embedding dimension while ensuring the representational capability of the embedding model. Shorter texts have fewer features and do not require a high embedding dimension. High-dimensional embedding is used for long texts (large descriptions in white paper documents), while low-dimensional embedding is used for short texts (such as question-answer pairs in most FAQ documents). By minimizing the embedding dimension of each text fragment, the consumption of storage resources is significantly reduced, and the retrieval speed of medium and low-dimensional vector libraries is greatly improved.

[0054] Figure 1 This diagram illustrates the architecture of a knowledge base construction, information retrieval, and question-answering method according to an embodiment of this application.

[0055] like Figure 1 As shown, after loading a document, the RAG system selects the corresponding segmentation method based on the document type identifier. FAQ (frequently-asked questions) are segmented by question paragraphs, meaning each question and answer is segmented into a fragment. Other documents are segmented by text token length. After segmentation, the corresponding embedding model is selected based on the text fragment length. After embedding, a vector of the corresponding dimension is generated and then stored in the vector database. In other words, long documents are segmented into smaller text fragments, such as segmenting by a fixed number of tokens or by chapter, using different segmentation strategies for different document types. Then, based on the token length of the text fragment to be embedded, the appropriate model with the lowest dimension is selected from multiple configured embedding models for embedding, generating a vector and storing it in the corresponding vector database. As shown in the figure, there are three types of vectors: vectors of the first dimension, vectors of the second dimension, and vectors of the third dimension, corresponding to three vector databases. However, those skilled in the art should understand that the types of vectors and vector databases in this application can have any suitable number, and this application does not limit this. Users can determine the selection and number of embedding models based on the characteristics of the document to be embedded, with each embedding model corresponding to a vector database. When a user makes a query request, the system receives the user's query statement and embeds it using the configured M (e.g., 3) embedding models, forming multiple corresponding vectors. It then retrieves the top K relevant text fragments from the corresponding dimension's vector database. From these K×M fragments, the top N most similar fragments are selected as background knowledge through reranking. Finally, these top N retrieved fragments, along with the query statement, are used to construct prompt words. In other words, when a user asks a question in the RAG system, the query statement is first embedded using embedding models of different dimensions. Then, relevant fragments are retrieved from the vector database based on the embedded vectors. All fragments retrieved from different vector databases are merged and highly sorted. Prompt words are constructed using the top few reranked fragments and the query statement. Finally, the results are sent to a larger model for summary and answering.

[0056] Figure 2 A flowchart illustrating an information processing method according to an embodiment of this application is shown.

[0057] According to this embodiment, the information processing method includes steps S210 to S240, and each step is described in detail below.

[0058] S210. Obtain multiple texts to be stored, including a first text and a second text.

[0059] In this embodiment, text can refer to textual content or a collection of texts containing knowledge and information. For example, text can be documents, articles, papers, Q&A, white papers, instructions, manuals, books, etc. In this embodiment, the text to be stored can refer to text waiting to be stored in a knowledge base. Generally, the text to be stored is the original text obtained, with diverse content and formats, which cannot be directly used to build a knowledge base and needs to be processed before being stored in the knowledge base for querying or retrieval operations. In this embodiment, the first and second texts can refer to any two texts with different content from the text to be stored. Although only the first and second texts are given in this embodiment, those skilled in the art should understand that the technical solution of this embodiment can also process third, fourth, or even more texts; the first and second texts here are merely illustrative.

[0060] S220. The first text is processed using the first embedding model to obtain the first result.

[0061] S230. The second text is processed using the second embedding model to obtain the second result. The first embedding model and the second embedding model have different numbers of dimensions.

[0062] In this embodiment, embedding refers to the way high-dimensional discrete features are mapped to a relatively low-dimensional continuous vector space. In the RAG system, embedding document fragments is used to convert text into computable vectors, facilitating subsequent similarity calculations and retrieval. Text embedding relies on a text embedding model. An embedding model is specifically designed to learn how to convert raw data (such as text, images, or other categories of data) into dense vectors (i.e., embeddings). This process involves learning the underlying structure and features of the data, enabling the raw data to be represented in a form more suitable for machine learning models. Text embedding models can preserve the semantic features of the text, which may include vocabulary, grammar, semantics, sentiment, emotion, topic, context, etc., thus enabling the retrieval of semantically similar text fragments.

[0063] In this embodiment, processing text using an embedding model to obtain the result can refer to the process of inputting text or fragments of text into an embedding model and converting it into a corresponding vector through the model's calculations. The result obtained by the embedding model can be a vector directly obtained after the embedding calculation, or it can be the result of further calculations or processing based on the vector.

[0064] In this embodiment, the different number of dimensions of the first and second embedding models can refer to the different number of dimensions of the vectors they generate, thus resulting in different vectors and different results. In some embodiments, the vector obtained by the first embedding model is one processing result, the vector obtained by the second embedding model is another processing result, and so on. Each processing result can correspond to a vector database, which can store vectors with a specific number of dimensions. These vector databases together constitute a knowledge base.

[0065] As an example, to process the first text using the first embedding model to obtain the first result, the first text can first be divided into multiple segments based on its characteristics; then, the first embedding model is used to embed each segment separately to obtain multiple first vectors, which serve as the first result. Similarly, to process the second text using the second embedding model to obtain the second result, the second text can first be divided into multiple segments based on its characteristics; then, the second embedding model is used to embed each segment separately to obtain multiple second vectors, which serve as the second result.

[0066] In this example, the characteristics of text can refer to the information structure or writing style. For example, based on length, texts can be categorized as long texts, Chinese texts, and short texts; based on language, they can be categorized as Chinese texts and English texts; and based on content, they can be categorized as introductory texts, descriptive texts, and expository texts. For instance, a text can be a question-and-answer (Q&A) text consisting of multiple questions and answers, a long text (e.g., a book) consisting of multiple relatively long paragraphs, or a short text (e.g., an online article) consisting of multiple shorter paragraphs. In this example, a fragment can refer to a longer text divided into several shorter paragraphs according to certain rules.

[0067] In this example, dividing text into multiple segments based on its characteristics can mean determining a segmentation method based on the text's characteristics, and then dividing the text into multiple segments according to that method. For example, longer paragraphs are segmented into longer segments, and shorter paragraphs are segmented into shorter segments.

[0068] In this example, an embedding model is used to embed multiple segments separately to obtain multiple vectors. This can refer to inputting multiple segments of a text into an embedding model corresponding to that text, and the embedding model calculates multiple vectors of a corresponding number of dimensions. In this embodiment, the number of dimensions of the embedding model is adapted to the type of text and the length of its segments. For shorter text segments, the number of dimensions of the corresponding embedding model is usually less, while for longer text segments, the number of dimensions of the corresponding embedding model is usually more.

[0069] As an example, the characteristics of the first text or the second text include one or more of the following: white paper type, product manual type, and question and answer type.

[0070] In this example, white paper-type texts have relatively long paragraphs, product instruction manual-type texts have relatively short paragraphs, and question-and-answer type texts consist of questions and answers, with the shortest paragraphs. Different text segmentation methods can be used based on these characteristics.

[0071] As an example, to divide the first text into multiple segments based on its characteristics, when the first text is a white paper type, a first method can be used to divide the first text into multiple segments, the first method including dividing according to a first fixed length. As an example, to divide the first text into multiple segments based on its characteristics, when the second text is a product manual type, a second method can be used to divide the second text into multiple segments, the second method including dividing according to a second fixed length less than the first fixed length. As an example, to divide the first text into multiple segments based on its characteristics, when the first text is a question-and-answer type, a first method can be used to divide the first text into multiple segments, wherein the first method includes dividing according to paragraphs.

[0072] In this example, since white paper-type text has the longest paragraphs, the first method of segmentation can be used, which is to segment according to the longest fixed length. Since product instruction manual-type text has centered paragraphs, the second method of segmentation can be used, which is to segment according to the shortest fixed length. Since question-and-answer type text has the shortest paragraphs, the third method of segmentation can be used.

[0073] Specifically, three segmentation strategies can be adopted: 1) Segment by text length, with each segment consisting of 1000 tokens and an overlap of 200 tokens. This is suitable for documents with long paragraphs, many of which exceed 500 words, such as white papers. 2) Segment by text length, with each segment consisting of 500 tokens and an overlap of 100 tokens. This is suitable for documents with moderately long semantic descriptions, where most paragraphs are fully explained within 500 words, such as product manuals, where each feature is described in a short paragraph. 3) Segment by paragraph. This is suitable for FAQ-type documents, where the answer to each question is generally short, mostly within 200 words. The determination of the three document types is made manually during document import according to the classification criteria.

[0074] As an example, each embedding model can have a different number of dimensions. Taking three embedding models as an example, the first embedding model can have 1 to 1000 dimensions, the second embedding model can have 1000 to 3000 dimensions, and the third embedding model can have 3000 to 5000 dimensions.

[0075] In this example, a first embedding model with 1 to 1000 dimensions can refer to a vector whose dimensions are between 1 and 1000 after embedding; a second embedding model with 1000 to 3000 dimensions can refer to a vector whose dimensions are between 1000 and 3000 after embedding; and a third embedding model with 3000 to 5000 dimensions can refer to a vector whose dimensions are between 3000 and 5000 after embedding.

[0076] Specifically, three different embedding models can be selected: BAAI / bge-base-zh (768 dimensions), BAAI / bge-m3 (1024 dimensions), and openai / text-embedding-3-large (3072 dimensions). These models embed text fragments of less than 200 tokens, 200-500 tokens, and 500-1000 tokens respectively, based on their token length, and store the embeddings in the corresponding dimension vector database. Therefore, the text fragment embedding process is fully automatic, requiring no manual selection of the embedding model. In this way, compared to sampling all high-dimensional (3072-dimensional) embedding models, after embedding according to text fragment length, if each of the three interval ranges accounts for 1 / 3 of the text fragments, the storage space after vectorization can be saved by (1 / 3)×(3072-1024) / 3072+(1 / 3)×(3072-768) / 3072=47%. The computational cost of vector similarity calculation during the retrieval process will also be reduced by 47%, thereby improving retrieval efficiency and question response speed.

[0077] S240. Store the first and second results in the knowledge base.

[0078] In this embodiment, a knowledge base can refer to a collection of interconnected knowledge fragments stored, organized, managed, and used in computer memory using one or more knowledge representation methods to meet the needs of solving problems in one or more domains. These knowledge fragments include domain-related theoretical knowledge, factual data, heuristic knowledge derived from expert experience, such as definitions, theorems, and operational rules related to a certain domain, as well as common sense knowledge. Knowledge base construction can refer to the process of creating and improving a knowledge base, including collecting knowledge from a specific domain, organizing and storing it according to certain rules, thereby constructing a knowledge base that can be accessed or retrieved.

[0079] In this embodiment, storing the first and second results in the knowledge base can mean storing the vectors obtained from the first embedding model and the vectors obtained from the second embedding model in different vector databases, with the knowledge base consisting of multiple vector databases. In this embodiment, a vector database can refer to a database that stores vectors, where all the data stored in such a database is a vector of a certain type.

[0080] Figure 3 A flowchart illustrating an information processing method according to an embodiment of this application is shown.

[0081] According to this embodiment, the information processing method includes steps S310 to S370, and each step is described in detail below.

[0082] S310. Obtain multiple texts to be stored, including a first text and a second text.

[0083] S320. The first text is processed using the first embedding model to obtain the first result.

[0084] S330. The second text is processed using the second embedding model to obtain the second result. The first embedding model and the second embedding model have different numbers of dimensions.

[0085] S340. Store the first and second results in the knowledge base.

[0086] For details regarding S310 to S340, please refer to the above section. Figure 2 The detailed descriptions of S210 to S240 in the embodiments will not be repeated here.

[0087] S350. The first embedding model is used to embed the user's query statement to obtain the first retrieval vector.

[0088] In this embodiment, the query statement can refer to a retrieval field entered by the user for querying information, such as keywords or questions. For example, a query statement can refer to a statement entered by the user indicating what aspects of information they want to ask or what aspects of answers they want to receive. For example, if a user wants to know about "three-layer networking," their query statement could be "What is three-layer networking?" or "What are the advantages of three-layer networking?" In this embodiment, the first retrieval vector can refer to a vector that can be queried in a knowledge base (which includes a vector database corresponding to the first retrieval vector). It is transformed from the user's query statement and can be processed by the vector database to obtain relevant knowledge information.

[0089] In this embodiment, the first retrieval vector is obtained by embedding the user's query statement using a first embedding model. This can mean embedding the query statement using a first embedding model with a specific number of dimensions to obtain a first retrieval vector with a specific number of dimensions. The number of dimensions of the retrieval vector corresponds to the number of dimensions of the embedding model, and thus corresponds to the number of dimensions of the vectors in the vector database in the knowledge base obtained by embedding the corresponding embedding model.

[0090] S360. The query statement is embedded using the second embedding model to obtain the second retrieval vector.

[0091] In this embodiment, the second retrieval vector can refer to a vector with a different number of dimensions than the first retrieval vector. The knowledge base contains a vector database corresponding to the second retrieval vector, whose number of dimensions is the same as the second retrieval vector, and therefore the same as the number of dimensions of the second embedding model.

[0092] S370. Search the knowledge base using the first search vector and the second search vector to obtain multiple search results.

[0093] In this embodiment, the search results can refer to the vectors most similar to the search vector obtained by searching and matching in the knowledge base using the search vector. In this embodiment, obtaining multiple search results by searching the knowledge base using the first search vector and the second search vector can mean searching in the vector database corresponding to the first search vector in the knowledge base, and then searching in the vector database corresponding to the second search vector in the knowledge base, thus obtaining multiple search results. These multiple search results can be obtained directly from the search vector, or they can be obtained by further calculations or processing based on the search results obtained directly from the search vector.

[0094] As an example, in order to obtain multiple search results by searching the knowledge base using the first search vector and the second search vector, we can first search the knowledge base using the first search vector to obtain X first search results; then, search the knowledge base using the second search vector to obtain X second search results; then, rearrange the X first search results and X second search results according to similarity; finally, select the top Y search results with the highest to lowest similarity as multiple search results.

[0095] In this example, the knowledge base contains vector databases corresponding to the first and second search vectors, respectively. Searching within these databases using either the first or second search vector allows for retrieval of information from vector databases with the same number of dimensions, based on the dimension of the first or second search vector. Rearranging the X first and X second search results based on similarity in this example can mean arranging all search results (i.e., the retrieved vectors) in all vector databases according to their similarity to the search vectors, with the most similar search results at the top and the least similar at the bottom. Alternatively, selecting the top Y search results (from highest to lowest similarity) as multiple search results can mean that after arranging all search results by similarity, the top Y search results are the most similar Y search results, and these are used as the final search results.

[0096] Specifically, the user's query can be embedded using three different embedding models, and the top 5 text fragments can be obtained by searching the corresponding vector database. These fragments can then be merged to obtain 3×5=15 fragments. Finally, a rearrangement model (e.g., bge-reranker-base) can be used to select the top 3 fragments from these 15 fragments as reference knowledge.

[0097] Figure 4 A flowchart illustrating an information processing method according to an embodiment of this application is shown.

[0098] According to this embodiment, the information processing method includes steps S410 to S490, and each step is described in detail below.

[0099] S410. Obtain multiple texts to be stored, including a first text and a second text.

[0100] S420. The first text is processed using the first embedding model to obtain the first result.

[0101] S430. The second text is processed using the second embedding model to obtain the second result. The first embedding model and the second embedding model have different numbers of dimensions.

[0102] S440. Store the first and second results in the knowledge base.

[0103] For details regarding S410 to S440, please refer to the above section. Figure 2 The detailed descriptions of S210 to S240 in the embodiments will not be repeated here.

[0104] S450. The user's query statement is embedded using the first embedding model to obtain the first retrieval vector.

[0105] S460. The query statement is embedded using the second embedding model to obtain the second retrieval vector.

[0106] S470. Search the knowledge base using the first search vector and the second search vector to obtain multiple search results.

[0107] For details regarding S450 to S470, please refer to the above section. Figure 3 The detailed descriptions of S350 to S370 in the embodiments will not be repeated here. S480: Construct prompt words based on the query statement and multiple search results.

[0108] In this embodiment, prompt words can refer to a special type of language input used to guide the AI ​​model to perform specific tasks, such as generating text, images, or performing other types of creative work. The design and selection of prompt words have a crucial impact on the output of the AI ​​model, as they directly determine how the AI ​​understands and responds to user instructions. In this embodiment, constructing prompt words based on the query statement and multiple search results can mean combining the query statement and multiple search results to form prompt words that indicate to the AI ​​model what kind of answer is needed, thereby helping the user obtain more accurate and comprehensive answers.

[0109] S490. Input the query statement and prompt words into the large language model to obtain the answer to the query statement.

[0110] In this embodiment, a large language model can refer to a deep learning model trained using a large amount of text data, capable of generating natural language text or understanding the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, large language models adopt a similar architecture and pre-training objectives to small models, differing only in increased model size, training data, and computational resources. In this embodiment, inputting the query statement and prompt words into the large language model to obtain an answer to the query statement can mean inputting the query statement along with prompt words previously constructed from the query statement and search results into the large language model as input data. The large language model then calculates and processes the data to obtain an answer corresponding to the user's query statement. Compared to the answer obtained by simply inputting the query statement, such an answer is more accurate and comprehensive, better answering the user's questions and improving the efficiency of information retrieval for the user.

[0111] As an example, to construct prompt words based on a query and multiple search results, the query can first be broken down into multiple questions, each question corresponding to one of the multiple search results; then, each question and its corresponding search result can be combined; finally, the multiple combinations of multiple questions and multiple search results can be used as prompt words.

[0112] In this example, breaking down a query into multiple questions can mean that, based on the content of each search result, a question relating to one aspect of the knowledge information involved in the query is taken as a sub-question and used to form a combination of sub-questions and sub-answers with the search results. Each search result constitutes a combination, and multiple search results form multiple combinations of sub-questions and sub-answers, which then serve as prompts.

[0113] Specifically, based on the top 3 reference knowledge and the user's query, a structure similar to... Figure 5 The prompt word is shown, and then the constructed prompt word is sent to the large model interface (the interface of your own open source large model or a commercial large model).

[0114] Based on the foregoing Figure 2 The present application also provides an information processing device, the structural schematic diagram of which is shown in the embodiment of the method described above. Figure 6 As shown. This device is used to perform the aforementioned... Figure 2 Each step in the process.

[0115] According to this embodiment, the information processing device includes an acquisition module 610, a first processing module 620, a second processing module 630, and a storage module 640. The acquisition module 610 acquires multiple texts to be stored, including a first text and a second text. The first processing module 620 processes the first text using a first embedding model to obtain a first result. The second processing module 630 processes the second text using a second embedding model to obtain a second result. The first and second embedding models have different numbers of dimensions. The storage module 640 stores the first and second results in a knowledge base.

[0116] It should be noted that, Figure 6 The information processing apparatus 600 provided in the illustrated embodiment, when executing the information processing method, is only illustrated by the division of the above-described functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the information processing apparatus 600 provided in the above embodiment and... Figure 2 The information processing method embodiments shown belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0117] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0118] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 702 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 702 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0119] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0120] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0121] The memory 706 stores executable program code, and the processor 704 executes this executable program code to implement the functions of the aforementioned acquisition module, first processing module, second processing module, and storage module, thereby realizing the information processing method. That is, the memory 706 stores instructions for executing the information processing method.

[0122] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0123] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0124] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for performing information processing methods.

[0125] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing information processing methods. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing information processing methods.

[0126] It should be noted that the memories 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the information processing device. That is, the instructions stored in the memories 706 of different computing devices 700 can implement the functions of one or more modules among the acquisition module, the first processing module, the second processing module, and the storage module.

[0127] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for executing the functions of the acquisition module and the first processing module. Simultaneously, the memory 706 in computing device 700B stores instructions for executing the functions of the second processing module and the storage module.

[0128] Figure 9 The connection method between the computing device clusters shown can be such that, considering the information processing method provided in this application requires a large amount of data storage, the functions implemented by the second processing module and the storage module are delegated to the computing device 700B for execution.

[0129] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0130] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different in that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing information processing methods.

[0131] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing information processing methods. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing information processing methods.

[0132] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform an information processing method.

[0133] Embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the knowledge base construction methods according to various embodiments of this application described above.

[0134] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0135] The concepts, principles, and ideas of this application have been described in detail above with reference to specific embodiments (including examples and instances). Those skilled in the art should understand that the embodiments of this application are not limited to those given above. After reading this application, those skilled in the art can make any possible improvements, substitutions, and equivalents to the steps, methods, apparatus, and components in the above embodiments, and such improvements, substitutions, and equivalents should be considered to fall within the scope of this application. The scope of protection of this application is limited to the claims.

Claims

1. An information processing method, characterized in that, The method includes: Acquire multiple texts to be stored, including a first text and a second text; The first text is processed using the first embedding model to obtain the first result; The second text is processed using a second embedding model to obtain a second result. The first embedding model and the second embedding model have different numbers of dimensions. Store the first result and the second result in the knowledge base.

2. The method according to claim 1, characterized in that, The first step of processing the first text using a first embedding model to obtain a first result includes: Based on the characteristics of the first text, the first text is divided into multiple segments; The first embedding model is used to embed the multiple segments respectively to obtain multiple first vectors, which are used as the first result; The step of processing the second text using a second embedding model to obtain a second result includes: Based on the characteristics of the second text, the second text is divided into multiple segments; The second embedding model is used to embed the multiple segments respectively to obtain multiple second vectors, which are used as the second result.

3. The method according to claim 2, characterized in that, The characteristics of the first text or the characteristics of the second text include one or more of the following: White paper type, product manual type, Q&A type.

4. The method according to claim 3, characterized in that, The step of dividing the first text into multiple segments based on its characteristics includes: If the first text is a white paper, the first text is divided into multiple segments using a first method, which includes dividing according to a first fixed length.

5. The method according to claim 4, characterized in that, The step of dividing the first text into multiple segments based on the characteristics of the second text includes: If the second text is a product instruction manual type, the second text is divided into multiple segments using a second method, which includes dividing it according to a second fixed length that is less than the first fixed length.

6. The method according to claim 3, characterized in that, The step of dividing the first text into multiple segments based on its characteristics includes: If the first text is a question-and-answer type, the first text is divided into multiple segments using a first method, wherein the first method includes segmentation by paragraph.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The user's query statement is embedded using the first embedding model to obtain the first retrieval vector; The query statement is embedded using the second embedding model to obtain a second retrieval vector; The knowledge base is searched using the first and second search vectors to obtain multiple search results.

8. The method according to claim 7, characterized in that, The search is performed in the knowledge base using the first retrieval vector and the second retrieval vector to obtain multiple search results, including: The knowledge base is searched using the first search vector to obtain X first search results; The second retrieval vector is used to search the knowledge base to obtain X second retrieval results; The X first search results and the X second search results are rearranged according to similarity. The top Y search results with the highest to lowest similarity are selected as the multiple search results.

9. The method according to claim 7, characterized in that, The method further includes: Construct prompt words based on the query statement and the multiple search results; The query statement and the prompt word are input into the large language model to obtain the answer to the query statement.

10. The method according to claim 9, characterized in that, The step of constructing prompt words based on the query statement and the multiple search results includes: The query statement is broken down into multiple questions, and each question corresponds to one of the multiple search results; Each question is combined with the corresponding search result; Multiple combinations of the multiple questions and the multiple search results are used as prompt words.

11. An information processing device, characterized in that, The device includes: The acquisition module is used to acquire multiple texts to be stored, including a first text and a second text. The first processing module is used to process the first text using a first embedding model to obtain a first result; The second processing module is used to process the second text using a second embedding model to obtain a second result, wherein the first embedding model and the second embedding model have different numbers of dimensions; A storage module is used to store the first result and the second result in a knowledge base.

12. The apparatus according to claim 11, characterized in that, The first processing module is further configured to: Based on the characteristics of the first text, the first text is divided into multiple segments; The first embedding model is used to embed the multiple segments respectively to obtain multiple first vectors, which are used as the first result; The second processing module is further configured to: Based on the characteristics of the second text, the second text is divided into multiple segments; The second embedding model is used to embed the multiple segments respectively to obtain multiple second vectors, which are used as the second result.

13. The apparatus according to claim 12, characterized in that, The characteristics of the first text or the characteristics of the second text include one or more of the following: White paper type, product manual type, Q&A type.

14. The apparatus according to claim 13, characterized in that, The first processing module is further configured to: If the first text is a white paper, the first text is divided into multiple segments using a first method, which includes dividing according to a first fixed length.

15. The apparatus according to claim 14, characterized in that, The second processing module is further configured to: If the second text is a product instruction manual type, the second text is divided into multiple segments using a second method, which includes dividing it according to a second fixed length that is less than the first fixed length.

16. The apparatus according to claim 13, characterized in that, The first processing module is further configured to: If the first text is a question-and-answer type, the first text is divided into multiple segments using a first method, wherein the first method includes segmentation by paragraph.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The device further includes: The first embedding module is used to embed the user's query statement using the first embedding model to obtain a first retrieval vector; The second embedding module is used to embed the query statement using the second embedding model to obtain a second retrieval vector; The retrieval module is used to perform retrieval in the knowledge base using the first retrieval vector and the second retrieval vector to obtain multiple retrieval results.

18. The apparatus according to claim 17, characterized in that, The retrieval module is further configured to: The knowledge base is searched using the first search vector to obtain X first search results; The second retrieval vector is used to search the knowledge base to obtain X second retrieval results; The X first search results and the X second search results are rearranged according to similarity. The top Y search results with the highest to lowest similarity are selected as the multiple search results.

19. The apparatus according to claim 17, characterized in that, The device further includes: The construction module is used to construct prompt words based on the query statement and the multiple search results; The input module is used to input the query statement and the prompt words into the large language model to obtain the answer to the query statement.

20. The apparatus according to claim 19, characterized in that, The construction module is further configured to: The query statement is broken down into multiple questions, and each question corresponds to one of the multiple search results; Each question is combined with the corresponding search result; Multiple combinations of the multiple questions and the multiple search results are used as prompt words.

21. A computing device cluster, characterized in that, The system includes at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the information processing method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for performing the information processing method according to any one of claims 1 to 10.

Images