Data processing method, data retrieval method, device and storage medium
By constructing data nodes and description information, generating semantic descriptions and sharding nonlinear structure data, the problem of inability to effectively retrieve graph and tree structure data in the prior art is solved, and precise retrieval and semantic sharding of these data are realized.
Patent Information
- Application Number
- CN202411934599.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-08
- Estimated Expiration
- 2044-12-26
AI Technical Summary
The prior art cannot effectively search graph and tree structure data, especially in semantic sharding and vectorization, which makes it difficult to locate information during the search process.
By constructing data nodes and description information, and generating semantic descriptions for data nodes in parent-child relationships, non-linear structure data are sharded, sharded vectors are generated, and stored in vector databases to build indexes to realize semantic sharding and vectorization processing of tree structure and graph structure data.
It realizes accurate retrieval of nonlinear structural data, provides a data foundation, and improves the accuracy and efficiency of retrieval.
Smart Images
Figure CN119357312B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of document retrieval, and in particular to a data processing method, a data retrieval method, a device and a storage medium. Background Art
[0002] In the evolution of artificial intelligence and machine learning, natural language processing (NLP) has always been a key area. As a powerful tool in NLP, large language models (LLMs) can generate high-quality natural language texts through training on large-scale corpora. However, with the increase in data volume and the diversification of user needs, LLMs have encountered bottlenecks in answering knowledge in certain professional fields. To solve this problem, retrieval augmented generation (RAG) technology came into being. RAG combines information retrieval and language generation, and improves the accuracy and relevance of answers by retrieving relevant documents and integrating them into the generation model. However, the parsed data in RAG technology is unstructured data, and there is no parsing for graph and tree structured data. Naive RAG and advanced RAG cannot perform semantic segmentation and vectorization well for the above types of data. It is difficult to locate information during the retrieval process, and thus it is impossible to use the information well to complete the corresponding retrieval work. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a data processing method, a data retrieval method, a device and a storage medium to solve the problem that graph and tree structure data cannot be retrieved in the prior art.
[0004] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a data processing method, comprising:
[0005] Constructing a plurality of data nodes corresponding to the nonlinear structure data according to the hierarchical relationship of the acquired nonlinear structure data;
[0006] Determine the description information corresponding to each data node;
[0007] Generate semantic descriptions corresponding to data nodes with hierarchical relationships that are parent-child relationships;
[0008] The nonlinear structure data is segmented according to each data node, the description information corresponding to each data node, and each semantic description to obtain corresponding fragments;
[0009] Determine the slice vector corresponding to each slice;
[0010] All shard vectors and hierarchical relationships are stored in the vector database, and an index corresponding to each shard vector is constructed.
[0011] In an embodiment of the present application, the nonlinear structure data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments, including: converting all data nodes and the descriptive information corresponding to all data nodes into text data in a target format according to a hierarchical relationship; the text data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments.
[0012] In an embodiment of the present application, converting all data nodes and the descriptive information corresponding to all data nodes into text data in a target format according to a hierarchical relationship includes: determining the node level corresponding to each data node according to the hierarchical relationship; for each data node, determining the indentation amount corresponding to the data node according to the node level; for each data node, converting the data node and the descriptive information corresponding to the data node into text data in a target format according to the indentation amount.
[0013] In an embodiment of the present application, the text data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments, including: segmenting the text data according to the first target characters included in the text data to obtain fragments corresponding to each data node; segmenting the text data according to the second target characters included in the text data to obtain fragments corresponding to the descriptive information corresponding to each data node; segmenting the text data according to each semantic description to obtain fragments corresponding to each semantic description.
[0014] In an embodiment of the present application, the target format is markdown format.
[0015] A second aspect of the present application provides a data retrieval method, comprising:
[0016] Acquire search data input by a user when searching for nonlinear structural data;
[0017] Searching for a target shard vector matching the retrieved data in a vector database, where the vector database is constructed using the above data processing method;
[0018] Determine the target shard to be recalled according to the target shard vector;
[0019] The search result corresponding to the search data is determined according to the target shard.
[0020] In an embodiment of the present application, determining the target shard to be recalled according to the target shard vector includes: when the target shard vector is a shard corresponding to a data node, determining all child data nodes corresponding to the data node corresponding to the target shard vector; determining the shards corresponding to all child data nodes as the target shards, and merging the target shards for recall.
[0021] In an embodiment of the present application, determining the retrieval result corresponding to the retrieval data according to the target segment includes: inputting the target segment and the retrieval data into a prompt project to generate corresponding prompt words through the prompt project; inputting the prompt words into a large language model to determine the output result of the large language model as the retrieval result corresponding to the retrieval data.
[0022] A third aspect of the present application provides a data processing device, including:
[0023] a memory configured to store instructions;
[0024] The processor is configured to call instructions from the memory and can implement the above-mentioned data processing method or the above-mentioned data retrieval method when executing the instructions.
[0025] A fourth aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data processing method or the above-mentioned data retrieval method.
[0026] Through the above technical solution, data nodes and description information are constructed for nonlinear structure data, and semantic descriptions are constructed for data nodes of parent-child relationship. Then, the nonlinear structure data is segmented based on the data nodes, description information and semantic description, and corresponding segmentation vectors are generated. Semantic segmentation and vectorization optimization processing can be performed on nonlinear structure data with tree structure and graph structure, which provides a data basis for the retrieval of nonlinear structure data and realizes accurate retrieval of nonlinear structure data.
[0027] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:
[0029] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present application is shown;
[0030] Figure 2A schematic diagram schematically shows a directory structure according to an embodiment of the present application;
[0031] Figure 3 A schematic diagram of a data tree structure RAG architecture according to an embodiment of the present application is schematically shown;
[0032] Figure 4 A schematic diagram of a process flow of a data retrieval method according to an embodiment of the present application is schematically shown;
[0033] Figure 5 The structure block diagram of a data processing device according to an embodiment of the present application is schematically shown;
[0034] Figure 6 The schematic diagram shows a structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0036] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indications will also change accordingly.
[0037] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0038] Figure 1 The following schematically shows a flow chart of a data processing method according to an embodiment of the present application. Figure 1As shown, an embodiment of the present application provides a data processing method, which may include the following steps.
[0039] S102: construct a plurality of data nodes corresponding to the nonlinear structure data according to the hierarchical relationship of the acquired nonlinear structure data.
[0040] It can be understood that nonlinear structured data includes tree structured data and graph structured data. Tree structured data is a hierarchical structure defined by branch relationships, which can represent a one-to-many relationship between data elements. Graph structured data uses diagrams to illustrate the hierarchical structure between data. Figure 2 As shown, Figure 2 The directory structure is schematically shown. Both the fault diagnosis tree and the directory structure belong to nonlinear structure data. The data in the nonlinear structure data has a hierarchical relationship. For this type of data, the corresponding data nodes can be constructed according to their hierarchical relationship. Among them, each node of the tree structure data has zero or more child nodes, and the node without a parent node is called the root node. Each non-root node has one and only one parent node. Except for the root node, each child node can be divided into multiple non-intersecting subtrees.
[0041] S104: Determine description information corresponding to each data node.
[0042] It can be understood that the data node with description or name in the original nonlinear structure data can be directly used as the description information corresponding to the data node. The data node without description or name in the original nonlinear structure data can be filled with corresponding description information by automatic generation or manual addition.
[0043] S106, generating semantic descriptions corresponding to data nodes whose hierarchical relationships are parent-child relationships.
[0044] It can be understood that the data nodes with a parent-child relationship are the parent node and its corresponding child node. For data nodes with a parent-child relationship, the corresponding semantic description can be generated through the large model according to the scenario of the data itself, which can explain the relationship between nodes and improve the parsing effect of nonlinear structure data for the subsequent data retrieval process.
[0045] S108, segmenting the nonlinear structure data according to each data node, the description information corresponding to each data node, and each semantic description to obtain corresponding fragments.
[0046] Specifically, each data node is segmented from the nonlinear structure data to obtain a shard corresponding to each data node. The semantic description corresponding to the data nodes with a hierarchical relationship of parent-child relationship is segmented from the nonlinear structure data to obtain a shard corresponding to each semantic description. The description information corresponding to each data node is further segmented to segment the description information into multiple shards. Furthermore, a corresponding recursive index can be constructed for each shard according to the position of each shard in the nonlinear structure data, so that the shard can be found in the nonlinear structure data according to the recursive index.
[0047] S110, determining a slice vector corresponding to each slice.
[0048] It can be understood that the slices can be mapped to real vectors through natural language processing technology, and the vocabulary can be converted from discrete, high-dimensional one-hot encoding to continuous, low-dimensional vector representation through mathematical methods, so that the relationship between words can be measured by the similarity between vectors. Specifically, nonlinear structure data is divided into chunks, and these chunks are encoded into slice vectors. These slice vectors can represent the semantic information of the text and are a form of expression of text data in mathematical space. The dimension of the slice vector represents the feature space of the text. The dimension and value of the vector can reflect the similarity and difference between texts. Specifically, the word2vec model or the BERT pre-training model can be used to determine the slice vector corresponding to each slice.
[0049] S112, store all the slice vectors and hierarchical relationships in the vector database, and construct an index corresponding to each slice vector. Specifically, the hierarchical relationship between the slice vectors can be recorded in metadata, and then all the slice vectors and metadata are stored in the vector database.
[0050] Through the above technical solution, data nodes and description information are constructed for nonlinear structure data, and semantic descriptions are constructed for data nodes of parent-child relationship. Then, the nonlinear structure data is segmented based on the data nodes, description information and semantic description, and corresponding segmentation vectors are generated. Semantic segmentation and vectorization optimization processing can be performed on nonlinear structure data with tree structure and graph structure, which provides a data basis for the retrieval of nonlinear structure data and realizes accurate retrieval of nonlinear structure data.
[0051] In an embodiment of the present application, the nonlinear structure data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments, including: converting all data nodes and the descriptive information corresponding to all data nodes into text data in a target format according to a hierarchical relationship; the text data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments.
[0052] Specifically, the target format is markdown format, which is a lightweight markup language. All data nodes corresponding to nonlinear structure data and the description information corresponding to all data nodes are converted into text data in markdown format that is easier to be understood and analyzed by large language models according to the hierarchical relationship, thereby enhancing the parsing effect of nonlinear structure data.
[0053] In an embodiment of the present application, converting all data nodes and the descriptive information corresponding to all data nodes into text data in a target format according to a hierarchical relationship includes: determining the node level corresponding to each data node according to the hierarchical relationship; for each data node, determining the indentation amount corresponding to the data node according to the node level; for each data node, converting the data node and the descriptive information corresponding to the data node into text data in a target format according to the indentation amount.
[0054] It can be understood that the node level of each data node refers to the hierarchical position of the data node in the nonlinear structure data. The indentation amount refers to the corresponding indentation amount when the nonlinear structure data is converted into the target format. The indentation amount can reflect the hierarchical relationship of the data nodes. The indentation amount of data nodes at the same level is the same.
[0055] In an embodiment of the present application, the text data is segmented according to each data node, the descriptive information corresponding to each data node, and each semantic description to obtain corresponding fragments, including: segmenting the text data according to the first target characters included in the text data to obtain fragments corresponding to each data node; segmenting the text data according to the second target characters included in the text data to obtain fragments corresponding to the descriptive information corresponding to each data node; segmenting the text data according to each semantic description to obtain fragments corresponding to each semantic description.
[0056] It can be understood that the data nodes can be distinguished by line breaks, and the first target character can be a line break. The description information can be separated into different description contents by periods and / or commas, and the second target character can be periods and / or commas. Each parent-child data node has a corresponding semantic description, and each semantic description can be separated separately. In this way, the characteristics of text data in markdown format can be used to accurately segment nonlinear structure data.
[0057] Figure 3 The following is a schematic diagram of a data tree structure RAG architecture according to an embodiment of the present application. Figure 3 As shown, the embodiment of the present application provides a data tree structure RAG architecture. Specifically, it includes the following steps.
[0058] S1. Preprocessing based on nonlinear structure data. Specifically, nonlinear structure data such as file directories and fault diagnosis trees usually have root nodes, child nodes, etc. For this type of data, we add description information to all nodes (which can be automatically generated or manually added), or the original data has descriptions or names, and construct corresponding data nodes for this type of data.
[0059] Specifically, refer to Figure 2 , design data nodes for the target structure (using Python code as an example):
[0060] class Node:
[0061] def __init__(self, description):
[0062] self.description = description
[0063] self.children = []
[0064] Furthermore, construct data nodes and description information:
[0065] root_node = Node("Web development project code directory")
[0066] node1 = Node("README.md (project introduction)")
[0067] node2 = Node("package.json build (npm package configuration file)")
[0068] node3 = Node("build (build script directory)")
[0069] S2. Convert the nonlinear structure data into text data in markdown format.
[0070] Specifically, the conversion code logic (taking Python code as an example):
[0071] def tree_to_markdown(node, level=0):
[0072] # Output node content, with indentation in front to indicate the level
[0073] with open("markdown.md", "a", encoding="utf-8") as f:
[0074] f.write(" " * (level * 4) + "- " + node.content + "\n")
[0075] # Recursively process child nodes
[0076] for child in node.children:
[0077] tree_to_markdown(child, level + 1)
[0078] but, Figure 2 The conversion effect of the directory structure is as follows:
[0079] - Web Development Project Code Directory
[0080] - README.md (project introduction)
[0081] - package.json build (npm package configuration file)
[0082] - build (build script directory)
[0083] - build.js (production environment build script)
[0084] - utils.js (build related tool methods)
[0085] - webpack.base.cont.js (wabpack basic configuration)
[0086] - webpack.base.cont.js (wabpack development environment configuration)
[0087] - webpack.prod.conf.js (wabpack production environment configuration)
[0088] - config (project configuration)
[0089] - node_modules (dependency package download directory)
[0090] - src (source code directory)
[0091] - main.js (entry js file)
[0092] - app.vue (root component)
[0093] - components (public component directory)
[0094] - assets (resource directory, where the resources will be built by wabpack)
[0095] - index.js
[0096] - routes (front-end routing)
[0097] - views (page directory)
[0098] - hello.vue
[0099] - notfound.vue
[0100] - static (pure static resources, will not be built by wabpack)
[0101] S3: Segment the text data in markdown format based on the hierarchical relationship of non-linear structure data. This step needs to ensure the integrity of each tree (data node) in the markdown file, as well as the semantic relationship between the parent node and the child node.
[0102] Specifically, S31, each tree (data node) is retained as a complete shard.
[0103] S32, double-layer recursive sharding. The first layer refers to building a recursive index for each tree (data node), and its segmentation method is line breaks, that is, segmentation is performed on each data node. For example: "assets (resource directory, where resources will be built by wabpack)" is a shard. The second layer: further build a recursive search for each data node, and its segmentation method is punctuation marks, such as periods / commas, etc. For example: "assets (resource directory, where resources will be built by wabpack)" is segmented into "Assets", "(resource directory", "where resources will be built by wabpack)".
[0104] S33, construct hierarchical relationship semantics for the first-level recursive index, and construct a semantically described index for the parent and child class nodes through the large model according to the scenario of the data itself. For example:
[0105] - src (source code directory)
[0106] - main.js (entry js file)
[0107] The semantic relationship between the parent and child nodes above: The "src" directory is the source code directory of the project, and "main.js" is an entry JavaScript file in the directory. "main.js" is usually used to initialize the project and load other necessary modules. In addition, a recursive index is built for the semantic description.
[0108] S34. The above shards can be vectorized using the vector method in the simple and advanced RAG technology, and the hierarchical relationship can be recorded in the metadata, and the shard vectors and metadata can be stored in the vector database.
[0109] Figure 4 The following schematically shows a flow chart of a data retrieval method according to an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a data retrieval method, which may include the following steps.
[0110] S402, obtaining search data input by the user when searching for nonlinear structural data.
[0111] S404, searching a vector database for a target shard vector that matches the search data, the vector database being constructed using the above data processing method.
[0112] S406: Determine the target fragment to be recalled according to the target fragment vector.
[0113] S408: Determine the search result corresponding to the search data according to the target slice.
[0114] The principle of the vectorization model in RAG technology (retrieval-augmented generation) mainly involves converting text data into mathematical vectors and retrieving relevant information by calculating the distance or similarity between these vectors, thereby enhancing the ability of the generation model. Specifically, the shard vectors are stored in a vector database for fast retrieval based on the user's query. The vector database stores each data node, the description information corresponding to each data node, the shard vector corresponding to each semantic description, and the hierarchical relationship between the shard vectors. During the retrieval process, the retrieval data entered by the user can be vectorized to obtain the query vector, and then calculated based on the distance between the vectors. For example, by calculating the distance between the query vector and the shard vector stored in the vector database, such as the Euclidean distance or cosine similarity, the shard vector most similar to the query is found, that is, the target shard vector. The target shard corresponding to the recall is determined according to the target shard vector, and the retrieval result corresponding to the retrieval data is determined according to the target shard vector. In this way, the vector database obtained by the above scheme provides a data basis for the retrieval of nonlinear structure data and realizes the accurate retrieval of nonlinear structure data.
[0115] In an embodiment of the present application, determining the target shard to be recalled based on the target shard vector includes: when the target shard vector is a shard corresponding to a data node, determining all sub-data nodes corresponding to the data node corresponding to the target shard vector; determining the shard corresponding to all sub-data nodes as the target shard, and merging the target shards for recall. It can be understood that merging and recalling means merging and recalling all sub-data nodes corresponding to the data node corresponding to the target shard vector. In this way, the information of the upper and lower layers of the data nodes of the nonlinear structure data can also be combined for analysis during retrieval to improve the retrieval accuracy.
[0116] In an embodiment of the present application, determining the retrieval result corresponding to the retrieval data according to the target segment includes: inputting the target segment and the retrieval data into a prompt project to generate corresponding prompt words through the prompt project; inputting the prompt words into a large language model to determine the output result of the large language model as the retrieval result corresponding to the retrieval data.
[0117] Specifically, the retrieved target segments are input into the prompt project together with the retrieval data of the original query to generate prompt words, and then the prompt words are input into the large language model. The large language model generates the final retrieval results based on this information and its own language generation capabilities, which can improve the retrieval quality and relevance of nonlinear structure data.
[0118] like Figure 3 As shown, the embodiment of the present application provides a tree structure RAG architecture, which also includes the following steps:
[0119] S35. Obtaining search data input by the user when searching for nonlinear structural data. Usually, the search data input by the user is a question sentence.
[0120] S36. Searching for a target shard vector matching the retrieved data in a vector database, where the vector database is constructed using the above data processing method, and determining the target shard to be recalled based on the target shard vector.
[0121] S37. The retrieved target fragments are input into the prompt project prompt together with the retrieval data of the original query to generate prompt words, and then the prompt words are input into the LLM (Large Language Model). LLM generates the final retrieval results based on this information and its own language generation capabilities.
[0122] In a specific embodiment, the data retrieval method may be to read the following markdown formatted data, answer user questions, and follow the following instructions:
[0123] 1. If the user asks for a directory, it will be returned as a file directory. For example:
[0124] Directory examples:
[0125] C: / Users / user / 2projects / graph_rag / segment_model.py
[0126] D: / Users / 00759075 / AppData / Local / Continuum / anaconda2 / envs /
[0127] 2. If the answer is a directory, do not return anything other than the directory.
[0128] Among them, the actual markdown data is:
[0129] - Web Development Project Code Directory
[0130] - README.md (project introduction)
[0131] - package.json build (npm package configuration file)
[0132] - build (build script directory)
[0133] - build.js (production environment build script)
[0134] - utils.js (build related tool methods)
[0135] - webpack.base.cont.js (wabpack basic configuration)
[0136] - webpack.dev.conf.js (wabpack development environment configuration)
[0137] - webpack.prod.conf.js (wabpack production environment configuration)
[0138] - config (project configuration)
[0139] - node_modules (dependency package download directory)
[0140] - src (source code directory)
[0141] - main.js (entry js file)
[0142] - app.vue (root component)
[0143] - components (public component directory)
[0144] - assets (resource directory, where the resources will be built by wabpack)
[0145] - index.js
[0146] - routes (front-end routing)
[0147] - views (page directory)
[0148] - hello.vue
[0149] - notfound.vue
[0150] - static (pure static resources, will not be built by wabpack)
[0151] #######
[0152] User question: {query}
[0153] Output answer:
[0154] Then, the above data retrieval method finally obtains the following effect:
[0155] 1. Path generation:
[0156] (1) Query = "Configuration file location of wabpack development environment"
[0157] Build result = build / webpack.dev.conf.js
[0158] (2) Query = "Where is the index.js file in the web development project code?"
[0159] Result = src / assets / index.js
[0160] 2. Semantic relationship generation:
[0161] (1) Query = "What is the relationship between the src directory and main.js?"
[0162] Result = `src` directory is the source code directory of the project, and `main.js` is an entry JavaScript file in the directory. `main.js` is usually used to initialize the project and load other necessary modules.
[0163] Figure 1 FIG. 1 is a flow chart of a data processing method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0164] Figure 5 The structure block diagram of a data processing device according to an embodiment of the present application is schematically shown. Figure 5 As shown, an embodiment of the present application provides a data processing device, which may include:
[0165] a memory configured to store instructions; and
[0166] The processor is configured to call instructions from the memory and implement the above-mentioned data processing method or data retrieval method when executing the instructions.
[0167] Specifically, in the embodiment of the present application, the processor may be configured to:
[0168] According to the hierarchical relationship of the acquired nonlinear structure data, multiple data nodes corresponding to the nonlinear structure data are constructed; the descriptive information corresponding to each data node is determined; the semantic description corresponding to the data nodes whose hierarchical relationship is a parent-child relationship is generated; the nonlinear structure data is divided according to each data node, the descriptive information corresponding to each data node and each semantic description to obtain corresponding slices; the slice vector corresponding to each slice is determined; all the slice vectors and the hierarchical relationship are stored in a vector database, and an index corresponding to each slice vector is constructed.
[0169] In an embodiment of the present application, the processor may also be configured to:
[0170] All data nodes and the description information corresponding to all data nodes are converted into text data in a target format according to a hierarchical relationship; the text data is segmented according to each data node, the description information corresponding to each data node, and each semantic description to obtain corresponding fragments.
[0171] In an embodiment of the present application, the processor may also be configured to:
[0172] The node level corresponding to each data node is determined according to the hierarchical relationship; for each data node, the indentation amount corresponding to the data node is determined according to the node level; for each data node, the data node and the description information corresponding to the data node are converted into text data in a target format according to the indentation amount.
[0173] In an embodiment of the present application, the processor may also be configured to:
[0174] The text data is segmented according to the first target characters included in the text data to obtain a segment corresponding to each data node; the text data is segmented according to the second target characters included in the text data to obtain a segment corresponding to the description information corresponding to each data node; the text data is segmented according to each semantic description to obtain a segment corresponding to each semantic description.
[0175] In an embodiment of the present application, the processor may also be configured to:
[0176] The target format is markdown.
[0177] In an embodiment of the present application, the processor may also be configured to:
[0178] Acquire the search data input by the user when searching for nonlinear structural data; search for the target slice vector matching the search data in the vector database, the vector database is constructed using the above data processing method; determine the target slice to be recalled according to the target slice vector; determine the search result corresponding to the search data according to the target slice.
[0179] In an embodiment of the present application, the processor may also be configured to:
[0180] In the case where the target shard vector is a shard corresponding to a data node, all child data nodes corresponding to the data node corresponding to the target shard vector are determined; the shards corresponding to all child data nodes are determined as target shards, and the target shards are merged and recalled.
[0181] In an embodiment of the present application, the processor may also be configured to:
[0182] According to the target segment and the search data, the prompt project is input to generate corresponding prompt words through the prompt project; the prompt words are input to the large language model to determine the output result of the large language model as the search result corresponding to the search data.
[0183] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned data processing method or the above-mentioned data retrieval method.
[0184] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store data processing data or data retrieval data. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a data processing method or a data retrieval method is implemented.
[0185] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0186] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0187] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0188] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0190] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0191] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0192] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0193] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0194] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A data processing method, characterized in that: include: Constructing a plurality of data nodes corresponding to the nonlinear structure data according to the hierarchical relationship of the acquired nonlinear structure data; Determine the description information corresponding to each data node; Generate semantic descriptions corresponding to the data nodes whose hierarchical relationships are parent-child relationships; Convert all data nodes, description information and semantic descriptions corresponding to all data nodes into text data in a target format according to the hierarchical relationship; Segmenting the text data according to the first target character included in the text data to obtain a slice corresponding to each data node; Segmenting the text data according to the second target characters included in the text data to obtain segments corresponding to the description information corresponding to each data node; Segmenting the text data according to each semantic description to obtain segments corresponding to each semantic description; Determine the slice vector corresponding to each slice; All the sharding vectors and the hierarchical relationships are stored in a vector database, and an index corresponding to each sharding vector is constructed.
2. The data processing method according to claim 1, characterized in that: The step of converting all data nodes, description information and semantic descriptions corresponding to all data nodes into text data in a target format according to the hierarchical relationship includes: Determine the node level corresponding to each data node according to the hierarchical relationship; For each data node, determining an indentation amount corresponding to the data node according to the node level; For each data node, the data node, the description information corresponding to the data node, and the semantic description are converted into text data in a target format according to the indentation amount.
3. The data processing method according to claim 1, characterized in that: The target format is markdown format.
4. A data retrieval method, characterized in that: include: Acquire search data input by a user when searching for nonlinear structural data; Searching for a target sharding vector matching the search data in a vector database, wherein the vector database is constructed using the data processing method according to any one of claims 1 to 3; Determine the target slice to be recalled according to the target slice vector; A search result corresponding to the search data is determined according to the target slice.
5. The data retrieval method according to claim 4, characterized in that: Determining the target slice to be recalled according to the target slice vector includes: In a case where the target shard vector is a shard corresponding to a data node, determining all child data nodes corresponding to the data node corresponding to the target shard vector; The shards corresponding to all child data nodes are determined as the target shards, and the target shards are merged and recalled.
6. The data retrieval method according to claim 4, characterized in that: Determining the search result corresponding to the search data according to the target slice includes: Inputting the target segment and the search data into a prompting project to generate corresponding prompt words through the prompting project; The prompt word is input into a large language model to determine an output result of the large language model as a search result corresponding to the search data.
7. A data processing device, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and implement the data processing method according to any one of claims 1 to 3 or the data retrieval method according to any one of claims 4 to 6 when executing the instructions.
8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing a machine to execute the data processing method according to any one of claims 1 to 3 or the data retrieval method according to any one of claims 4 to 6.
Citation Information
Patent Citations
KR20240157591A