Rail transit RAG application data retrieval method and device, equipment and medium

Abstract

The invention provides a rail transit RAG application data retrieval method and device, equipment and a medium. The method comprises the steps that retrieval keywords are obtained; according to the retrieval keyword, performing data retrieval through an RAG system to obtain the similarity between the candidate metadata and the vector embedded by each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; obtaining a matching score of each candidate metadata through a DRMM model; determining the weighted sum of the matching score of each candidate metadata and the similarity of the embedded vector as a retrieval value of the candidate metadata; and obtaining metadata obtained by retrieval according to the retrieval value, and feeding back the metadata obtained by retrieval. According to the method, multi-level information retrieval of the data is achieved through the RAG system and the DRMM model, and the retrieval efficiency and accuracy of the rail transit data can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of rail transit technology, and in particular to a method, device, equipment, and medium for retrieving rail transit RAG application data. Background Art

[0002] With the rapid development of rail transit technology, the amount of data generated in the industry continues to increase, covering a variety of data types from equipment operating status, fault reports, maintenance records to technical manuals and operation reports. These data not only have mixed characteristics of structured and unstructured, but also contain a large number of domain-specific terms and complex technical descriptions, which poses great challenges to efficient retrieval and intelligent generation of information.

[0003] Traditional keyword index-based retrieval methods are difficult to meet the complex information needs in the rail transit field, while emerging large models and embedding technologies have advantages in processing complex semantic relationships, and can more effectively capture the semantic connections between data and provide more accurate retrieval results.

[0004] However, in the process of applying embedding models in the field of rail transit, due to the complex semantic characteristics of rail transit data, simple embedding and vector storage methods are difficult to effectively improve retrieval efficiency. Summary of the invention

[0005] In order to solve one of the above-mentioned technical defects, the present application provides a rail transit RAG application data retrieval method, device, equipment, and medium.

[0006] In a first aspect, the present application provides a method for retrieving rail transit RAG application data, the method comprising:

[0007] Get search keywords;

[0008] According to the search keywords, data is searched through the RAG system to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding;

[0009] The matching score of each candidate metadata is obtained through the DRMM model;

[0010] The weighted sum of the matching scores of each candidate metadata and the similarity of the embedded vector is determined as the search value of the candidate metadata;

[0011] The retrieved metadata is obtained according to the search value, and the retrieved metadata is fed back.

[0012] Optionally, the RAG system includes a BGE model and a Milvus vector database;

[0013] Before obtaining the search keywords, it also includes:

[0014] Segment the text paragraphs and tables in the rail transit metadata to obtain segmented data;

[0015] Perform vector embedding processing on the segmented data through the BGE model;

[0016] The vector embedding processed data is stored in the Milvus vector database.

[0017] Optionally, the text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data, including:

[0018] The text paragraphs of the rail transit metadata are segmented according to logical subsections to obtain segmented data;

[0019] For the table of rail transit metadata, it is directly used as the segmented data, or segmented according to rows or columns to obtain the segmented data.

[0020] Optionally, after segmenting the text paragraphs and tables in the rail transit metadata to obtain segmented data, the following steps are further included:

[0021] Restore the split data in the original order.

[0022] Optionally, vector embedding processing is performed on the segmented data through a BGE model, including:

[0023] Through the semantic understanding ability of the BGE model, the segmented data is vector embedded and converted into a high-dimensional vector representation.

[0024] Optionally, the vector embedding processed data is stored in the Milvus vector database, including:

[0025] The vector embedded processed data is imported into the Milvus vector database, so that the Milvus vector database can use its efficient data storage mechanism to quickly persist the vector embedded processed data and establish an efficient index structure for the embedded vectors in the vector embedded processed data.

[0026] Optionally, data retrieval is performed through the RAG system according to the search keywords to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata, including:

[0027] Based on the BM25 algorithm, the initial metadata containing the search keywords in the RAG system is determined;

[0028] Convert search keywords into search vectors, and convert each initial metadata into an initial vector;

[0029] Determine the similarity between the search vector and each initial vector, and determine the similarity as the similarity of the vector embedded in the corresponding initial metadata;

[0030] The candidate metadata is determined according to the similarity between the search vector and each initial vector.

[0031] In a second aspect of the present application, a rail transit RAG application data retrieval device is provided, the device comprising:

[0032] An acquisition module is used to obtain search keywords;

[0033] A retrieval module is used to perform data retrieval through the RAG system according to the retrieval keywords obtained by the acquisition module, and obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding;

[0034] A calculation module, used for obtaining the matching score of each candidate metadata obtained by the retrieval module through the DRMM model;

[0035] A determination module, configured to determine a weighted sum of the matching scores of the candidate metadata obtained by the calculation module and the similarities of the embedded vectors obtained by the retrieval module as a retrieval value of the candidate metadata;

[0036] The feedback module is used to obtain the retrieved metadata according to the retrieval value obtained by the determination module, and to feed back the retrieved metadata.

[0037] In a third aspect of the present application, an electronic device is provided, including:

[0038] Memory;

[0039] Processor; and

[0040] Computer programs;

[0041] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect above.

[0042] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement the method described in the first aspect above.

[0043] The present application provides a method, device, equipment, and medium for retrieval of rail transit RAG application data, the method comprising: obtaining retrieval keywords; performing data retrieval through the RAG system according to the retrieval keywords to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; obtaining the matching score of each candidate metadata through the DRMM model; determining the weighted sum between the matching score of each candidate metadata and the similarity of the embedded vector as the retrieval value of the candidate metadata; obtaining the retrieved metadata according to the retrieval value, and feeding back the retrieved metadata. The method of the present application realizes multi-level information retrieval of data through the RAG system and the DRMM model, which can effectively improve the retrieval efficiency and accuracy of rail transit data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A schematic diagram of a flow chart of a rail transit RAG application data retrieval method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the architecture of a DRMM provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the architecture of a rail transit RAG application data retrieval method provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a rail transit RAG application data retrieval device provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0051] In the process of implementing this application, the inventors found that the traditional keyword index-based retrieval method is difficult to meet the complex information needs in the field of rail transit, while the emerging large model and embedding technology have advantages in processing complex semantic relationships, and can more effectively capture the semantic connections between data and provide more accurate retrieval results. However, in the process of applying the embedded model in the field of rail transit, due to the complex semantic characteristics of rail transit data, simple embedding and vector storage methods are difficult to effectively improve retrieval efficiency.

[0052] In response to the above problems, the present application provides a method, device, equipment, and medium for retrieval of rail transit RAG application data, the method comprising: obtaining retrieval keywords; performing data retrieval through the RAG system according to the retrieval keywords to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; obtaining the matching score of each candidate metadata through the DRMM model; determining the weighted sum between the matching score of each candidate metadata and the similarity of the embedded vector as the retrieval value of the candidate metadata; obtaining the retrieved metadata according to the retrieval value, and feeding back the retrieved metadata. The method of the present application realizes multi-level information retrieval of data through the RAG system and the DRMM model, which can effectively improve the retrieval efficiency and accuracy of rail transit data.

[0053] See also Figure 1 This embodiment provides a method for retrieving rail transit RAG application data. The implementation process of the method is as follows:

[0054] 101, obtain search keywords.

[0055] For example, when a user needs to perform data retrieval, he or she will enter a search keyword, and this step is to obtain the search keyword.

[0056] For another example, when a user needs to perform data retrieval, he or she will enter a retrieval description, and this step will extract retrieval keywords from the retrieval description.

[0057] 102. According to the search keywords, data is searched through a RAG (Retrieval-Augmented Generation) system to obtain similarities between candidate metadata and vectors embedded in each candidate metadata.

[0058] The candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding.

[0059] Among them, the RAG system includes the BGE model and the Milvus vector database.

[0060] The rail transit metadata will be stored in the RAG system before executing step 101. The storage process is as follows:

[0061] 201, segment the text paragraphs and tables in the rail transit metadata to obtain segmented data.

[0062] Since rail transit technical documents are highly technical and logical, it is not appropriate to segment them according to the traditional unified chunksize, and it is easy to lose context information. In order to meet the requirements of large models for context integrity and details in the retrieval and generation stages, it is necessary to perform targeted segmentation on text and table data. Step 201 is the rail transit metadata segmentation process.

[0063] The execution process of step 201 is a content type-based segmentation strategy for the complex structure of rail transit metadata, which can ensure that both text paragraphs and table information can be effectively processed and loaded into the system while maintaining the original document structure.

[0064] Among them, the text paragraph is the main text of the rail transit metadata, which mainly includes paragraphs, sections and technical specifications. For example, the text paragraphs of the rail transit metadata are segmented according to logical sections to obtain segmented data. This segmentation method allows the context of each section to be input into the RAG system and DRMM model relatively independently, and the order between its original paragraphs can be maintained. In addition, the main logic of the document can be separated through paragraph-level segmentation to avoid semantic drift during subsequent retrieval and generation. At the same time, this paragraph-level segmentation can also effectively avoid excessive input that exceeds the token limit of the RAG system and DRMM model, ensuring that large sections of text can be processed in a refined manner, thereby improving the accuracy and semantic consistency of information retrieval.

[0065] For the table of rail transit metadata, it is directly used as the segmented data, or it is segmented by rows or columns to obtain the segmented data. This segmentation method can fully consider the characteristics of highly structured data contained in the table information. Tables often contain large-scale numerical information, key technical parameters, equipment operation status and other key information in metadata, so it is particularly important to retain their original structure by segmenting them as a whole. For smaller tables, they can be loaded directly into the system without further segmentation. However, when the number of tokens in a table exceeds the limit of the RAG system and the DRMM model, it needs to be segmented into several small tables to maintain the coherence and hierarchy of the information. At this time, it will be segmented by rows or columns to ensure that the segmented table subsets still have the integrity and meaning of the original data, avoiding the loss of important context due to segmentation. At the same time, there may be logical associations between different tables. When segmenting, the associated table items need to be kept in the same subtable so that the structure that conforms to the logical order can be restored later.

[0066] Step 201 is the rail transit metadata segmentation process, which can be used to uniformly segment the metadata of technical documents in the rail transit field. By executing step 201, the paragraph structure and semantic consistency can be maintained. Through the paragraph and subsection-based text segmentation strategy and the dynamic table segmentation strategy, the precise analysis and structured storage of the diverse information in the rail transit metadata are finally achieved, providing a solid data foundation for subsequent retrieval and generation.

[0067] In addition, after executing step 201 to segment the text paragraphs and tables in the rail transit metadata and obtaining the segmented data, the segmented data will be restored in the original order to preserve the structure of the original document as much as possible. This processing method can restore the article logic to the greatest extent when the RAG system performs retrieval and generation, avoid information loss caused by excessive segmentation, and ensure that the RAG system and the DRMM model can generate accurate answers under global semantic consistency.

[0068] 202, perform vector embedding processing on the segmented data through the BGE model.

[0069] Among them, the process of vector embedding processing is vectorization processing, which generates a multi-dimensional semantic embedding process.

[0070] When executing step 202, the segmented data can be vector-embedded and converted into a high-dimensional vector representation through the semantic understanding capability of the BGE model.

[0071] For example, the segmented data is input into the BGE model one by one for vector embedding generation to form a high-dimensional vector representation. These vectors capture the deep semantic features of the text content, such as the working parameters of the equipment, the description of the operating status, the fault log record, and other information, and map the semantically similar text to the similar position in the vector space.

[0072] Since rail transit metadata (i.e., text data in rail transit, such as operation reports, technical manuals, and standard specifications) has a large number of specialized terms and specific knowledge structures, this requires the use of an embedding model that can capture fine-grained semantics. The BGE model is a bidirectional generative model that is suitable for processing various text types including long text paragraphs, technical instructions, and operation records. The BGE model converts the segmented data into a high-dimensional vector representation through pre-trained semantic understanding capabilities, so that semantically similar content shows a close vector distribution in the embedding space. For specific technical content or information such as rail transit equipment parameters, the BGE model can well convert it into an embedding vector, thereby achieving efficient and accurate retrieval.

[0073] 203, storing the vector embedding processed data into the Milvus vector database.

[0074] After executing step 202, the vector embedded processed data needs to be efficiently stored and managed to facilitate fast retrieval and high concurrent processing in the RAG system. Therefore, it is necessary to execute step 203 to achieve efficient storage and management of the vector embedded processed data. When executing step 203, the vector embedded processed data can be imported into the Milvus vector database, so that the Milvus vector database uses its efficient data storage mechanism to quickly persist the vector embedded processed data, and establish an efficient index structure for the embedded vectors in the vector embedded processed data. For example, the vector embedded processed data is imported into the Milvus vector database. The Milvus vector database can use its efficient data storage mechanism to quickly persist the vector data and support ultra-large-scale rail transit data management. At the same time, the Milvus vector database provides indexing and quantization algorithms, which can establish an efficient index structure for embedded vectors, ensure retrieval speed, and quickly return relevant results even with extremely high data volumes.

[0075] The Milvus vector database supports large-scale, high-dimensional vector data storage and management, and can well adapt to the characteristics of large rail transit data volume and frequent data updates. By combining with the BGE model, the Milvus vector database provides a convenient vector query API that can achieve fast similarity search, thereby directly loading the high-dimensional vectors embedded in the BGE model into the database, and directly performing efficient nearest neighbor search when retrieval is required.

[0076] Through steps 202 and 203, the rail transit metadata can be stored in the RAG system. Through steps 202 and 203, the semantic understanding and efficient retrieval requirements of the rail transit RAG application data retrieval method provided in this embodiment in the retrieval enhancement generation process can be met, such as combining the BGE model and the Milvus vector database to realize vector embedding processing. Through steps 202 and 203, not only can the proprietary terms and complex semantic relationships in the rail transit industry be captured, but also the embedded data can be efficiently managed and called with the support of the Milvus vector database, providing high-quality context support for subsequent information retrieval and generation tasks.

[0077] The Milvus vector database in the RAG system obtained through steps 201 to 203 can quickly retrieve the most similar vector set through the API provided by it, thereby achieving efficient retrieval. For example, after the user enters the fault problem description, the RAG system can call the BGE model through the Milvus vector database, embed similar technical records or solution documents in the vector database, and provide users with accurate technical support information.

[0078] With the help of BGE model and Milvus vector database, RAG system realizes the complete process from efficient vector embedding, storage to retrieval. With this architecture, RAG system can provide more intelligent retrieval results through semantic matching when facing professional content and complex queries of rail transit, which improves the efficiency and accuracy of users when querying rail transit knowledge base, querying technical manuals and troubleshooting information.

[0079] In addition, it is recommended that rail transit data be updated at any time, especially equipment operation data and maintenance records. The RAG system will also periodically regenerate updated embedded vectors through the BGE model to replace the old vectors in the Milvus vector database. At the same time, the Milvus vector database can automatically manage the updated index to ensure that the system is always in an efficient and up-to-date state.

[0080] Based on the above RAG system, the implementation process of step 102 is as follows:

[0081] 102-1, based on the BM25 algorithm, determine the initial metadata containing the search keyword in the RAG system.

[0082] For example, the BM25 algorithm can be used for quick initial screening to select initial metadata containing search keywords and reduce the size of the data.

[0083] 102-2, converting the search keyword into a search vector, and converting each initial metadata into an initial vector.

[0084] 102-3, determine the similarity between the search vector and each initial vector, and determine the similarity as the similarity of the vector embedded in the corresponding initial metadata.

[0085] 102-4, determining candidate metadata according to the similarity between the search vector and each initial vector.

[0086] Through steps 102 - 2 to 102 - 4 , dense vector retrieval can be implemented, that is, the search keywords and initial metadata are converted into vector representations, and the initial metadata with semantics similar to the search keywords is screened using vector similarity to obtain the top N initial metadata as candidate metadata.

[0087] In step 102, hybrid retrieval can be implemented to obtain preliminary selected candidate metadata. In hybrid retrieval, the BM25 algorithm is used to quickly screen through vector similarity and keyword matching to generate candidate metadata. Then, the candidate metadata is handed over to the DRMM model for deep semantic matching processing in step 103.

[0088] 103, obtain the matching score of each candidate metadata through the DRMM (Deep Relevance Matching Model) model.

[0089] Among them, the DRMM model architecture is as follows Figure 2 As shown. The DRMM model is a retrieval model based on deep learning, which is particularly suitable for processing complex semantic matching tasks. It can capture the semantic relationship between the query and the document by calculating the word vector similarity histogram between the query and the candidate document and using the neural network model for detailed matching. Therefore, the DRMM model can be used to perform deep semantic matching on the candidate metadata obtained by the RAG system in step 102, and the candidate metadata after preliminary screening can be matched with higher semantic accuracy.

[0090] In the specific implementation of step 103, local interactions can be first established based on each pair of terms in the search keyword and the candidate metadata. For each search keyword, the variable-length local interactions are converted into a fixed-length matching histogram. Based on the fixed-length matching histogram, a feedforward neural network is used to learn the hierarchical matching pattern and generate a matching score for each search keyword. Finally, the scores from each single search keyword are aggregated by calculating the aggregation weight to generate the matching score of each candidate metadata.

[0091] Through step 102 and step 103, a double-layer retrieval is realized. In step 102, the initial screening capability of the hybrid retrieval is used to control the data volume within a certain range, thereby maximizing the efficiency of the double-layer retrieval. In complex and domain-specific scenarios, the double-layer structure of the RAG system superimposed on the DRMM model has obvious advantages:

[0092] 1) Possess higher semantic matching capabilities: The DRMM model handles complex semantic matching problems and captures the deep semantic connection between search keywords and rail transit metadata through a deep learning model, which is suitable for professional needs.

[0093] 2) Effectively optimize retrieval efficiency: Hybrid retrieval effectively eliminates irrelevant content in the initial screening, allowing the DRMM model to be executed on a small-scale candidate set, thereby improving the overall response speed.

[0094] 3) Applicable to both open and closed domains: Particularly suitable for open domain tasks (such as knowledge question-answering systems) and closed domains (such as technical document retrieval and industry-specific data queries) that require precise semantic understanding.

[0095] 104 , determining a weighted sum of the matching scores of each candidate metadata and the similarity of the embedded vector as a search value of the candidate metadata.

[0096] For example, the search value of any candidate metadata is a weighted sum of the matching score of the candidate metadata and the similarity of the vector embedded in the candidate metadata.

[0097] 105 , obtaining retrieved metadata according to the search value, and feeding back the retrieved metadata.

[0098] For example, K candidate metadata with the highest search values ​​are determined as the retrieved metadata.

[0099] The rail transit RAG application data retrieval method provided in this embodiment is an efficient metadata segmentation, vectorization embedding and retrieval method in the rail transit field, which makes full use of the semantic understanding ability of the embedding model and the retrieval efficiency of the double-layer retrieval, thereby improving the retrieval effect of big data in the rail transit field while maintaining data integrity. It can solve the problem of low retrieval accuracy and insufficient result accuracy caused by the huge amount of rail transit data and complex semantic information.

[0100] The rail transit RAG application data retrieval method provided in this embodiment realizes multi-level information retrieval through the RAG system and DRMM model, thereby improving the retrieval efficiency and accuracy of rail transit data. The rail transit RAG application data retrieval method provided in this embodiment processes rail transit metadata in a targeted manner under big data conditions, and can efficiently support business needs such as information query, intelligent generation and technical support.

[0101] For specific implementation, see Figure 3 Before executing the rail transit RAG application data retrieval method provided in this embodiment, the rail transit metadata (i.e. Figure 3 The text paragraphs and table contents in the document) are segmented (the segmentation of the text paragraphs is Figure 3 The text segmentation in the table content segmentation process is Figure 3 The text paragraphs are segmented into sections to maintain the paragraph structure and semantic consistency. Then, the segmented data is vectorized using the BGE model to generate a multi-dimensional semantic embedding (i.e. Figure 3 The vector embedding in the vector is stored in the Milvus vector database. Finally, the RAG system performs a hybrid search (i.e. Figure 3 BM25 search and dense vector search in ), and then search based on DRMM model (i.e. Figure 3 The DRMM deep learning algorithm in the calculation of similarity) obtains the retrieved metadata (i.e. Figure 3 in the generated answer).

[0102] The rail transit RAG application data retrieval method provided in this embodiment is a large-scale metadata processing method suitable for the rail transit field. Through the multi-stage processing of segmentation, vectorization and double-layer retrieval, the storage and retrieval of rail transit data are more structured and intelligent. The rail transit RAG application data retrieval method provided in this embodiment divides the rail transit metadata independently according to text paragraphs and tables for subsequent vectorization processing and retrieval. The BGE model is embedded in Milvus as a base. A two-layer retrieval system combining the RAG system and the DRMM model was constructed. The first layer is a hybrid retrieval layer, which combines the BM25 algorithm with vectorized embedded retrieval for rapid screening and preliminary matching, which can improve the document hit rate for the knowledge base system built on the RAG system, and can effectively improve the retrieval accuracy of the RAG system.

[0103] The present embodiment provides a method for retrieval of rail transit RAG application data, which obtains retrieval keywords; based on the retrieval keywords, performs data retrieval through the RAG system to obtain the similarity between candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; obtains the matching score of each candidate metadata through the DRMM model; determines the weighted sum between the matching score of each candidate metadata and the similarity of the embedded vector as the retrieval value of the candidate metadata; obtains the retrieved metadata based on the retrieval value, and feeds back the retrieved metadata. The method of the present embodiment realizes multi-level information retrieval of data through the RAG system and the DRMM model, which can effectively improve the retrieval efficiency and accuracy of rail transit data.

[0104] Based on the same inventive concept of the rail transit RAG application data retrieval method, this embodiment provides a rail transit RAG application data retrieval device, see Figure 4 , the device comprises:

[0105] The acquisition module 401 is used to acquire search keywords.

[0106] The retrieval module 402 is used to perform data retrieval through the RAG system according to the retrieval keywords obtained by the acquisition module 401, and obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata. The candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding.

[0107] The calculation module 403 is used to obtain the matching score of each candidate metadata obtained by the retrieval module 402 through the DRMM model.

[0108] The determination module 404 is used to determine the weighted sum of the matching scores of the candidate metadata obtained by the calculation module 403 and the similarities of the embedded vectors obtained by the retrieval module 402 as the retrieval value of the candidate metadata.

[0109] The feedback module 405 is used to obtain the retrieved metadata according to the search value obtained by the determination module 404, and to feed back the retrieved metadata.

[0110] Among them, the RAG system includes the BGE model and the Milvus vector database.

[0111] The device also includes:

[0112] The processing module is used to segment the text paragraphs and tables in the rail transit metadata to obtain segmented data. The segmented data is vector-embedded through the BGE model. The vector-embedded data is stored in the Milvus vector database.

[0113] The text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data, including:

[0114] The text paragraphs of the rail transit metadata are segmented according to logical sections to obtain segmented data.

[0115] For the table of rail transit metadata, it is directly used as the segmented data, or segmented according to rows or columns to obtain the segmented data.

[0116] Among them, after segmenting the text paragraphs and tables in the rail transit metadata and obtaining the segmented data, it also includes:

[0117] Restore the split data in the original order.

[0118] Among them, the segmented data is vector-embedded through the BGE model, including:

[0119] Through the semantic understanding ability of the BGE model, the segmented data is vector embedded and converted into a high-dimensional vector representation.

[0120] The vector embedding and processed data is stored in the Milvus vector database, including:

[0121] The vector embedded processed data is imported into the Milvus vector database, so that the Milvus vector database can use its efficient data storage mechanism to quickly persist the vector embedded processed data and establish an efficient index structure for the embedded vectors in the vector embedded processed data.

[0122] The retrieval module 402 is used to determine the initial metadata containing the retrieval keyword in the RAG system based on the BM25 algorithm. The retrieval keyword is converted into a retrieval vector, and each initial metadata is converted into an initial vector. The similarity between the retrieval vector and each initial vector is determined, and the similarity is determined as the similarity of the vector embedded in the corresponding initial metadata. The candidate metadata is determined according to the similarity between the retrieval vector and each initial vector.

[0123] The device provided in this embodiment realizes multi-level information retrieval of data through the RAG system and the DRMM model, which can effectively improve the retrieval efficiency and accuracy of rail transit data.

[0124] Based on the same inventive concept of the rail transit RAG application data retrieval method, this embodiment provides an electronic device, such as Figure 5 As shown, it includes: a memory 501, a processor 502, and a computer program.

[0125] The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the above-mentioned rail transit RAG application data retrieval method.

[0126] Specifically,

[0127] Get the search keywords.

[0128] According to the search keywords, data is searched through the RAG system to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata. The candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding.

[0129] The matching score of each candidate metadata is obtained through the DRMM model.

[0130] The weighted sum of the matching score of each candidate metadata and the similarity of the embedded vector is determined as the search value of the candidate metadata.

[0131] The retrieved metadata is obtained according to the search value, and the retrieved metadata is fed back.

[0132] Optionally, the RAG system includes a BGE model and a Milvus vector database.

[0133] Before obtaining the search keywords, it also includes:

[0134] The text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data.

[0135] The segmented data is then vector-embedded using the BGE model.

[0136] The vector embedding processed data is stored in the Milvus vector database.

[0137] Optionally, the text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data, including:

[0138] The text paragraphs of the rail transit metadata are segmented according to logical sections to obtain segmented data.

[0139] For the table of rail transit metadata, it is directly used as the segmented data, or segmented according to rows or columns to obtain the segmented data.

[0140] Optionally, after segmenting the text paragraphs and tables in the rail transit metadata to obtain segmented data, the following steps are further included:

[0141] Restore the split data in the original order.

[0142] Optionally, vector embedding processing is performed on the segmented data through a BGE model, including:

[0143] Through the semantic understanding ability of the BGE model, the segmented data is vector embedded and converted into a high-dimensional vector representation.

[0144] Optionally, the vector embedding processed data is stored in the Milvus vector database, including:

[0145] The vector embedded processed data is imported into the Milvus vector database, so that the Milvus vector database can use its efficient data storage mechanism to quickly persist the vector embedded processed data and establish an efficient index structure for the embedded vectors in the vector embedded processed data.

[0146] Optionally, data retrieval is performed through the RAG system according to the search keywords to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata, including:

[0147] Based on the BM25 algorithm, the initial metadata containing the search keywords in the RAG system is determined.

[0148] The search keywords are converted into search vectors, and each initial metadata is converted into an initial vector.

[0149] The similarity between the search vector and each initial vector is determined, and the similarity is determined as the similarity of the vector embedded in the corresponding initial metadata.

[0150] The candidate metadata is determined according to the similarity between the search vector and each initial vector.

[0151] The electronic device provided in this embodiment, on which the computer program is executed by the processor to realize multi-level information retrieval of data through the RAG system and the DRMM model, can effectively improve the retrieval efficiency and accuracy of rail transit data.

[0152] Based on the same inventive concept of the rail transit RAG application data retrieval method, this embodiment provides a computer-readable storage medium, and a computer program is stored thereon. The computer program is executed by a processor to implement the rail transit RAG application data retrieval method.

[0153] Specifically,

[0154] Get the search keywords.

[0155] According to the search keywords, data is searched through the RAG system to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata. The candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding.

[0156] The matching score of each candidate metadata is obtained through the DRMM model.

[0157] The weighted sum of the matching score of each candidate metadata and the similarity of the embedded vector is determined as the search value of the candidate metadata.

[0158] The retrieved metadata is obtained according to the search value, and the retrieved metadata is fed back.

[0159] Optionally, the RAG system includes a BGE model and a Milvus vector database.

[0160] Before obtaining the search keywords, it also includes:

[0161] The text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data.

[0162] The segmented data is then vector-embedded using the BGE model.

[0163] The vector embedding processed data is stored in the Milvus vector database.

[0164] Optionally, the text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data, including:

[0165] The text paragraphs of the rail transit metadata are segmented according to logical sections to obtain segmented data.

[0166] For the table of rail transit metadata, it is directly used as the segmented data, or segmented according to rows or columns to obtain the segmented data.

[0167] Optionally, after segmenting the text paragraphs and tables in the rail transit metadata to obtain segmented data, the following steps are further included:

[0168] Restore the split data in the original order.

[0169] Optionally, vector embedding processing is performed on the segmented data through a BGE model, including:

[0170] Through the semantic understanding ability of the BGE model, the segmented data is vector embedded and converted into a high-dimensional vector representation.

[0171] Optionally, the vector embedding processed data is stored in the Milvus vector database, including:

[0172] The vector embedded processed data is imported into the Milvus vector database, so that the Milvus vector database can use its efficient data storage mechanism to quickly persist the vector embedded processed data and establish an efficient index structure for the embedded vectors in the vector embedded processed data.

[0173] Optionally, data retrieval is performed through the RAG system according to the search keywords to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata, including:

[0174] Based on the BM25 algorithm, the initial metadata containing the search keywords in the RAG system is determined.

[0175] The search keywords are converted into search vectors, and each initial metadata is converted into an initial vector.

[0176] The similarity between the search vector and each initial vector is determined, and the similarity is determined as the similarity of the vector embedded in the corresponding initial metadata.

[0177] The candidate metadata is determined according to the similarity between the search vector and each initial vector.

[0178] The computer-readable storage medium provided in this embodiment has a computer program thereon which is executed by a processor to implement multi-level information retrieval of data through the RAG system and the DRMM model, which can effectively improve the retrieval efficiency and accuracy of rail transit data.

[0179] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0180] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0181] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0183] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0184] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A rail transit RAG application data retrieval method, characterized in that: The method comprises: Get search keywords; According to the search keyword, data is searched through the RAG system to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; The matching score of each candidate metadata is obtained through the DRMM model; The weighted sum of the matching scores of each candidate metadata and the similarity of the embedded vector is determined as the search value of the candidate metadata; The retrieved metadata is obtained according to the search value, and the retrieved metadata is fed back.

2. The method according to claim 1, characterized in that The RAG system includes a BGE model and a Milvus vector database; Before obtaining the search keyword, the method further includes: Segment the text paragraphs and tables in the rail transit metadata to obtain segmented data; Performing vector embedding processing on the segmented data through a BGE model; The vector embedding processed data is stored in the Milvus vector database.

3. The method according to claim 2, characterized in that The text paragraphs and tables in the rail transit metadata are segmented to obtain segmented data, including: The text paragraphs of the rail transit metadata are segmented according to logical subsections to obtain segmented data; For the table of rail transit metadata, it is directly used as the segmented data, or segmented according to rows or columns to obtain the segmented data.

4. The method according to claim 2, characterized in that: After segmenting the text paragraphs and tables in the rail transit metadata to obtain segmented data, the method further includes: Restore the split data in the original order.

5. The method according to claim 2, characterized in that: The step of performing vector embedding processing on the segmented data by using the BGE model includes: Through the semantic understanding ability of the BGE model, the segmented data is vector-embedded and converted into a high-dimensional vector representation.

6. The method according to claim 2, characterized in that The step of storing the vector embedded processed data into the Milvus vector database comprises: The vector-embedded processed data is imported into the Milvus vector database, so that the Milvus vector database uses its efficient data storage mechanism to quickly persist the vector-embedded processed data, and establishes an efficient index structure for the embedded vectors in the vector-embedded processed data.

7. The method according to claim 1, characterized in that The step of performing data retrieval through the RAG system according to the search keyword to obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata includes: Based on the BM25 algorithm, determine the initial metadata containing the search keyword in the RAG system; Convert search keywords into search vectors, and convert each initial metadata into an initial vector; Determine the similarity between the search vector and each initial vector, and determine the similarity as the similarity of the vector embedded in the corresponding initial metadata; The candidate metadata is determined according to the similarity between the search vector and each initial vector.

8. A rail transit RAG application data retrieval device, characterized in that: The device comprises: An acquisition module is used to obtain search keywords; A retrieval module, used to perform data retrieval through the RAG system according to the retrieval keywords obtained by the acquisition module, and obtain the similarity between the candidate metadata and the vectors embedded in each candidate metadata; wherein the candidate metadata is one or more of the rail transit metadata, and the rail transit metadata is stored in the RAG system after segmentation and vector embedding; A calculation module, used for obtaining a matching score of each candidate metadata obtained by the retrieval module through a DRMM model; A determination module, configured to determine a weighted sum of the matching scores of the candidate metadata obtained by the calculation module and the similarities of the embedded vectors obtained by the retrieval module as a retrieval value of the candidate metadata; The feedback module is used to obtain the retrieved metadata according to the search value obtained by the determination module, and to feed back the retrieved metadata.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images