Data storage method, data query method, electronic equipment and computer medium

By storing file data, result data and content data in a distributed object storage system, a distributed data analysis engine and a memory-based key-value pair database, combined with multiple storage systems, the problem of LLM massive data storage and retrieval is solved. It realizes efficient and flexible data management and query, and improves the performance and user experience of the big data model.

CN119003690AInactive Publication Date: 2024-11-22中科先进(深圳)集成技术有限公司
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411503638.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-25
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing data storage methods are difficult to meet the needs of efficient storage and rapid retrieval of massive data of large language models (LLM), resulting in low flexibility in data storage and unable to further improve the performance of the LLM model.

Method used

By storing the file data of the file to be stored in a distributed object storage system, the result data is stored in a distributed data analysis engine, and the content data is stored in a memory-based high-performance key-value pair database, using multiple storage systems The combination of data management and query.

Benefits of technology

It realizes efficient storage and rapid retrieval of data, reduces system deployment and maintenance costs, improves the flexibility and performance of data storage, is suitable for enterprise-level applications of all sizes, and improves the performance and user experience of big data models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003690A_ABST
    Figure CN119003690A_ABST
Patent Text Reader

Abstract

The invention discloses a data storage method, a data query method, electronic equipment and a computer medium, and the data storage method comprises the following steps: obtaining a to-be-stored file, and storing file data of the to-be-stored file in a first storage system; storing result data of the to-be-stored file in a second storage system; and storing the content data of the to-be-stored file in a third storage system. According to the data storage method, the original file data and the result data subjected to model processing can be separately stored, lightweight storage is achieved, and the requirement of system storage for performance is lowered; in addition, the content data of the to-be-stored file is stored in the third storage system, and the file data and / or the result data are / is managed through the content data in the third storage system, so that the data can be quickly retrieved and accessed subsequently through the content data, and the flexibility of data storage is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data storage method, a data query method, an electronic device and a computer medium. Background Art

[0002] With the widespread application of large language models (LLMs), efficient storage and fast retrieval of massive data have become urgent issues to be solved. Existing data storage methods use common object storage and search engines to achieve data storage and retrieval. However, using a single technology for data storage is difficult to meet the diverse needs of LLM data processing, and the data flexibility is low, making it difficult to further improve the overall performance of the LLM model. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a data storage method, a data query method, an electronic device and a computer medium.

[0004] To solve the above problems, the present application provides a first technical solution: providing a data storage method, comprising: obtaining a file to be stored, the file to be stored comprising original file data, result data after being processed by a corresponding model, and content data; storing the file data in a first storage system; storing the result data in a second storage system; storing the content data in a third storage system, so as to manage the file data and / or the result data through the content data in the third storage system.

[0005] Optionally, before the step of storing the content data in the third storage system so as to manage the file data and / or the result data through the content data in the third storage system, the data storage method includes: parsing the file data to obtain the tag data and / or category data of the file data; obtaining the path data of the file data in the first storage system; the step of storing the content data in the third storage system so as to manage the file data and / or the result data through the content data in the third storage system includes: storing the path data, the tag data and / or the category data in the third storage system.

[0006] Optionally, after the step of storing the above-mentioned content data in a third storage system to manage the above-mentioned file data and / or the above-mentioned result data through the content data in the above-mentioned third storage system, the above-mentioned data storage method includes: slicing the above-mentioned label data to obtain label slices; and storing the above-mentioned label slices in the above-mentioned third storage system.

[0007] Optionally, storing the above-mentioned result data in the second storage system includes: decomposing the above-mentioned result data according to the label data and / or category data of the above-mentioned file data; and storing the decomposed result data in an index corresponding to the above-mentioned second storage system.

[0008] Optionally, after the step of storing the above-mentioned content data in the third storage system to manage the above-mentioned file data and / or the above-mentioned result data through the content data in the above-mentioned third storage system, the above-mentioned data storage method also includes: creating a first data node according to the above-mentioned category data, and establishing a first association relationship of the above-mentioned first data node, and / or, creating a second data node according to the above-mentioned label data, and establishing a second association relationship of the above-mentioned second data node; storing the first association relationship of the above-mentioned first data node and / or the second association relationship of the above-mentioned second data node in a fourth storage system.

[0009] Optionally, after the step of storing the first association relationship of the first data node and / or the second association relationship of the second data node in the fourth storage system, the data storage method further includes: establishing a first knowledge graph of the first data node based on the first association relationship of the first data node, and storing the first knowledge graph in the fourth storage system, and / or establishing a second knowledge graph of the second data node based on the second association relationship of the second data node, and storing the second knowledge graph in the fourth storage system.

[0010] To solve the above problems, the present application provides a second technical solution: providing a data query method, the above data query method is used to query the data stored by the above data storage method, the above data query method comprises: receiving a retrieval instruction, the above retrieval instruction includes the content to be queried; based on the above content to be queried, querying at least one of the first storage system, the second storage system and the third storage system to obtain return data; outputting the above return data.

[0011] Optionally, the above-mentioned querying in at least one of the first storage system, the second storage system and the third storage system based on the above-mentioned content to be queried to obtain return data also includes: querying in at least one of the first storage system, the second storage system and the third storage system according to the above-mentioned content to be queried to obtain first query data; when the above-mentioned first query data does not meet the preset requirements, obtaining the associated data of the above-mentioned content to be queried through the association relationship of the fourth storage system; querying in at least one of the first storage system, the second storage system and the third storage system based on the above-mentioned associated data to obtain second query data; and generating the above-mentioned return data based on the above-mentioned second query data.

[0012] To solve the above problems, the present application provides a third technical solution: an electronic device is provided, comprising a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement any of the methods described above.

[0013] To solve the above problems, the present application provides a fourth technical solution: providing a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement any of the methods described above.

[0014] The present application provides a data storage method, a data query method, an electronic device and a computer medium. The data storage method acquires the file to be stored, stores the file data of the file to be stored in a first storage system; stores the result data of the file to be stored in a second storage system; and stores the content data of the file to be stored in a third storage system, so as to manage the file data and / or result data through the content data in the third storage system. In the above manner, the data storage method of the present application can store the original file data and the result data after model processing separately, realize lightweight storage, and reduce the performance requirements of system storage; and the present application also manages the file data and / or result data through the content data in the third storage system by storing the content data of the file to be stored in the third storage system, so as to enable the subsequent rapid retrieval and access of data through the content data, thereby improving the flexibility of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 This is a flowchart of a first embodiment of a data storage method provided by the present application; Figure 2 is a flow chart of a second embodiment of the data storage method provided by the present application; Figure 3 is a flowchart of a third embodiment of the data storage method provided by the present application; Figure 4 It is a flowchart of the first embodiment of the data query method provided by the present application; Figure 5 is a flow chart of a second embodiment of the data query method provided by the present application; Figure 6 is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0018] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0019] Existing LLM models usually use distributed databases or distributed file systems to store data. These systems can manage and store large-scale data and provide certain scalability and fault tolerance. For example, massive data can be stored and retrieved through the Hadoop distributed file system, distributed NoSQL database system, object storage service, etc., to achieve storage management of large-scale data. Alternatively, the LLM model can also use the in-memory database for fast access and caching of data; or, the LLM model can also use the search engine for efficient indexing and retrieval of data.

[0020] When using any of the above solutions for data storage, for example, when using a distributed database or distributed file system, the deployment and operation and maintenance costs of the distributed database and distributed file system are high and the technical requirements are high. For data that is frequently read and written, the performance of the distributed system is poor and cannot meet the needs of efficient storage and access, resulting in low flexibility in data storage. For another example, when using an in-memory database for data storage, due to memory limitations, the in-memory database is usually not able to store large-scale data and cannot meet the needs of massive data. In addition, the persistence problem of the in-memory database architecture also poses a risk of data loss, resulting in low data storage performance and security performance. For another example, when using a search engine for data storage and retrieval, a large amount of central processing unit (CPU) and memory resources are usually consumed, resulting in an increase in system load and affecting the overall performance of the LLM model.

[0021] In view of this, an embodiment of the present application proposes a data storage method, which comprehensively utilizes multiple databases for data storage, so as to reduce deployment costs and further improve the flexibility of data storage while achieving high performance such as efficient data storage and management, fast data search and positioning, and efficient retrieval.

[0022] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the data storage method provided by the present application. Figure 1 As shown, in this embodiment, the data storage method includes the following steps: Step S11: Acquire the file to be stored, which includes original file data, result data processed by the corresponding model, and content data.

[0023] Specifically, the file types of the files to be stored include, but are not limited to, document files, audio and video files, and picture files to be stored. The data storage method of this embodiment is applied to big data models with a large number of parameters and large data requirements. Big data models include, but are not limited to, large language models, visual models, and the like. Correspondingly, the files to be stored are related data files of the big data model, and the files to be stored can generally include file data, result data, and content data. File data is the original file data used by the big data model when performing data processing; result data is the minimum set of data obtained after the big data model processes the original file data. After receiving the content input by the user, the big data model performs data query and processing based on the input content and obtains result data; content data corresponds to file data, and content data is used to represent the content of file data. For example, the corresponding content data can be obtained by obtaining the directory, label, category, and other data of the file data.

[0024] Step S12: storing the file data in the first storage system.

[0025] After acquiring the file data of the file to be stored, the file data is stored in the first storage system. The first storage system is a distributed object storage system, and the first storage system includes multiple storage nodes to realize parallel storage of data and increase the data storage capacity. In a possible implementation mode, the first storage system can protect data through erasure codes and checksums to ensure that data can be restored in the event of hardware failure or data corruption, thereby improving the data security of the original file data. Specifically, the first storage system divides the data to be stored into multiple parts, one part of which is a data block and the other part is a parity block. Even if part of the hard disk is damaged, as long as the number of remaining data blocks reaches a certain proportion, all the data can be restored. In other implementation modes, the first storage system includes but is not limited to data storage systems such as MinIO, Ceph, and OSS.

[0026] Step S13: Store the result data in the second storage system.

[0027] After obtaining the result data of the file to be stored, the result data is stored in the second storage system. The second storage system is a distributed data analysis engine, and the second storage system includes but is not limited to a storage system constructed by a full-text search engine such as Elasticsearch, RedisSearch, OpenSearch, ManticoreSearch, etc. After the result data is submitted to the second storage system, the second storage system stores the weight and word segmentation of the result data in the data.

[0028] In a possible implementation, the second storage system can store result data through an index, multiple result data are stored in the index, and an inverted index structure is used, each index is composed of a list of all non-repeating words, and each word includes corresponding result data. Through the above index structure, after the result data is stored in the second storage system, the model can search by inputting the content into the second storage system. The second storage system ranks and scores the results according to the weight, and then presents the returned results to the user to quickly search in a large amount of data. In addition, efficient full-text search is achieved through the second storage system, resource optimization is achieved, and CPU and memory consumption are reduced.

[0029] Step S14: storing the content data in the third storage system, so as to manage the file data and / or result data through the content data in the third storage system.

[0030] After obtaining the content data of the file to be stored, the content data is stored in the third storage system, which is a high-performance key-value pair database or database cluster based on memory, and the content data corresponding to the file data is stored in the memory of the third storage system to improve the reading and writing speed of the content data.

[0031] Content data includes but is not limited to the directory, label, category and other data of the file data. Since the content data is used to represent the content of the file data, and the content of the file data is related to the result data of the second storage system in terms of content, the data storage method of this embodiment stores the content data in the third storage system, so that the directory label of the file data can be obtained through the fast reading and writing operation of the third storage system, so as to realize the fast query and update of the document content, and then the file data and / or result data can be managed by calling / querying the content data in the third storage system, which greatly improves the efficiency of content management.

[0032] In an embodiment of the present application, the data storage method acquires the file to be stored, stores the file data of the file to be stored in the first storage system; stores the result data of the file to be stored in the second storage system; and stores the content data of the file to be stored in the third storage system, so as to manage the file data and / or result data through the content data in the third storage system. In the above manner, the original file data and the result data after model processing can be stored separately, so as to achieve lightweight storage and reduce the performance requirements of system storage; and the present application also manages the file data and / or result data through the content data in the third storage system by storing the content data of the file to be stored in the third storage system, so as to enable the rapid retrieval and access of data through the content data in the third storage system in the future, thereby improving the flexibility of data storage.

[0033] In one embodiment, see Figure 2 , Figure 2 1 is a flow chart of the second embodiment of the data storage method provided by the present application. Figure 2 As shown, in this embodiment, the content data includes tag data and / or category data of the file data, and path data of the file data. The data storage method includes the following steps: Step S21: Acquire the file to be stored, which includes original file data, result data processed by the corresponding model, and content data.

[0034] Step S22: storing the file data in the first storage system.

[0035] Step S23: Store the result data in the second storage system.

[0036] Specifically, step S21 to step S23 are similar to the above-mentioned step S11 to step S13, and will not be repeated here.

[0037] Step S24: Parse the file data to obtain label data and / or category data of the file data.

[0038] Specifically, the label data of the file data is used to represent the specific attributes or characteristics of the file data. The file data can be classified or marked with multiple label data; the category data of the file data is used to represent a larger group or category to which the file data belongs, which is used for organization and archiving. In a possible implementation, there can be multiple subcategories or label data under one category data to form a hierarchical structure of data; when parsing the file data, the directory, hierarchical results, attribute classification, etc. of the file data can be obtained through at least one of the label data and category data of the file data.

[0039] After acquiring the file data, the file data may be parsed. The file data may be processed by algorithms or software such as classifiers to parse the file data to obtain the label data and / or category data of the file data; the label data and / or category data of the file data may also be obtained by manual labeling, which is not specifically limited here.

[0040] Step S25: Obtain path data of the file data in the first storage system.

[0041] Specifically, the content data also includes path data, and the path data includes a storage path of the file data in the first storage system. The path data may be presented in the form of a character string, or the path data may be stored in a text in the form of a configuration file.

[0042] Step S26: storing the path data, label data and / or category data in a third storage system.

[0043] In one embodiment, the path data, label data, and category data are stored in a third storage system; in another embodiment, the path data and label data are stored in a third storage system, or the path data and category data are stored in a third storage system.

[0044] In the implementation of the present application, the data storage method can locate the location of the original file data through the path data in the third storage system; improve the organization, management and retrieval capabilities of the files to be classified through the label data and / or category data in the third storage system, and each label data and category data has corresponding path data, so as to find the corresponding original file data according to the label and / or classification, thereby improving the efficiency of data query.

[0045] Optionally, after step S14, the data storage method includes: slicing the label data to obtain label slices; and storing the label slices in a third storage system.

[0046] Specifically, after parsing the file data and obtaining the label data, the label data can also be sliced ​​to obtain label slices; and the label slices are stored in a third storage system. Among them, slicing the label data can divide the label data into small data subsets for easy processing and analysis; or, extract a data subset with specific features from the label data to process and understand it according to the extracted specific features; or, further process the sliced ​​data subset, such as cleaning, conversion, or use for model training and evaluation. In the above manner, the data storage method of this embodiment can further slice the label data of the third storage system to improve data processing and management capabilities, so that the label slices can be used in subsequent big data model training or analysis processes.

[0047] Optionally, in the data storage method of this embodiment, step S13 further includes: decomposing the result data according to the label data and / or category data of the file data; and storing the decomposed result data in the index corresponding to the second storage system.

[0048] Specifically, since the file data and the result data have a corresponding relationship in content, after parsing the file data and obtaining the corresponding label data and / or category data, the result data can be decomposed by at least one of the labels of the label data and the categories of the category data to store the decomposed result data in the index corresponding to the second storage system to constitute multiple indexes of the second storage system.

[0049] The model can input the content to be queried into the second storage system, and the second storage system searches according to the tags and / or categories corresponding to the content to be queried, and ranks and scores the results according to the weights of the searched result data, and then stores the returned result data in the index of the second storage system and presents it to the user, so as to quickly search in large amounts of data.

[0050] Through the above method, the result data of this embodiment can realize high-speed search while realizing distributed storage, and has powerful full-text search and data analysis capabilities while being more flexible in performance and query. The big data model can also perform flexibly data management through label data and / or category data of file data, thereby improving the flexibility of data storage.

[0051] Optionally, see Figure 3 , Figure 3 FIG. 1 is a flow chart of a third embodiment of the data storage method provided by the present application. Figure 3 As shown, after step S14 or step S26, the data storage method of this embodiment further includes: Step S31: creating a first data node according to the category data and establishing a first association relationship of the first data node, and / or creating a second data node according to the tag data and establishing a second association relationship of the second data node.

[0052] Specifically, after obtaining the category data, a first data node is constructed according to the category data, and a first association relationship of a category related to the first data node is established through a preset first query logic. Similarly, a second data node can also be created according to the obtained label data, and a second association relationship of the second data node can be established through a preset second query logic. The first query logic and the second query logic can be obtained through data conversion, aggregation sorting, index optimization query, etc., or the first query logic and the second query logic can be written in an artificially defined manner, which is not specifically limited here.

[0053] Step S32: storing the first association relationship of the first data node and / or the second association relationship of the second data node in the fourth storage system.

[0054] After creating data nodes and establishing association relationships between the data nodes, the first association relationship of the first data node and / or the second association relationship of the second data node can be stored in a fourth storage system so that the model can further expand the data source based on the association relationship between the data nodes and improve the ability of data association and analysis, so that the big data model can flexibly retrieve the corresponding data source according to different scenarios, thereby improving the performance of the big data model.

[0055] Furthermore, after step S32, the data storage method of this embodiment also includes: establishing a first knowledge graph of the first data node based on the first association relationship of the first data node, and storing the first knowledge graph in a fourth storage system, and / or, establishing a second knowledge graph of the second data node based on the second association relationship of the second data node, and storing the second knowledge graph in the fourth storage system.

[0056] Specifically, after constructing the data nodes and their associations, a first knowledge graph of the first data node can be established, and / or a second knowledge graph of the second data node can be established. The first knowledge graph is used to display the association feature relationship of the first data node under its corresponding category data in a structured graph manner, and the second knowledge graph is used to display the association feature relationship of the second data node under its corresponding label data in a structured graph manner. The first knowledge graph and the second knowledge graph can be displayed through a unified graph structure, or through different graph structures.

[0057] The fourth storage system may be a graph database, and the fourth storage system is used to store the first knowledge graph and the second knowledge graph, so as to store the corresponding first association relationship in the fourth storage system through the first knowledge graph, and / or store the corresponding second association relationship in the fourth storage system through the second knowledge graph. The fourth storage system includes but is not limited to graph databases such as Neo4j, Memgraph, TigerGraph, and JanusGraph.

[0058] In this embodiment, by storing the first knowledge graph and / or the second knowledge graph in the fourth storage system, the subsequent big data model can integrate information of related data through the knowledge graph in the fourth storage system, and apply the related information to the data processing process, so that the big data model can flexibly retrieve the corresponding data source according to different scenarios, thereby improving the performance of the big data model.

[0059] The present application also provides a data query method, which is used to query the data stored by the data storage method of any of the above embodiments. Figure 4 , Figure 4 1 is a flow chart of the first embodiment of the data query method provided by the present application. Figure 4 As shown, the data query method of this embodiment includes the following steps: Step S41: receiving a search instruction, wherein the search instruction includes content to be searched.

[0060] Specifically, the search instruction may include an instruction generated by triggering a preset control button, for example, the control button may be triggered by single-clicking, double-clicking, dragging, sliding, etc. The control button may be a physical hardware button or a control area in a web page design; or, the search instruction may include an instruction generated by recognizing a preset voice message. Thus, the search instruction may be input in different ways, making the entire interactive process convenient and quick.

[0061] Step S42: query at least one of the first storage system, the second storage system and the third storage system based on the content to be queried to obtain return data.

[0062] Specifically, after obtaining the content to be queried, a query is performed based on the content to be queried in at least one of the first storage system, the second storage system, the third storage system, and the fourth storage system to obtain return data. It can be understood that when performing data query, it can be performed only in the original file data in the first storage system; or the result data in the second storage system can be retrieved; or the content data in the third storage system can be retrieved and then further searched in the first storage system or the second storage system according to the label data, classification data, or path data of the content data, so as to output return data in combination with the contents of multiple storage systems; or the fourth storage system can be searched to obtain the associated data of the data node corresponding to the content to be queried, and the associated data can be retrieved in at least one of the first storage system, the second storage system, and the third storage system according to the label, classification, etc. of the associated data.

[0063] After querying in at least one of the above methods, the returned data output after processing based on the query content is obtained. In a possible implementation, after obtaining the returned data, the returned data can be further stored as new result data in the second storage system to facilitate calling the query content in subsequent queries.

[0064] Step S43: Output return data.

[0065] After obtaining the return data, the return data is output so that the big data model can further perform data processing, natural language processing and other operations based on the return data to improve the processing flexibility of the big data model, or the return data is output to the user end for the user to view.

[0066] In an embodiment of the present application, the data query method can obtain return data by querying the content to be queried in at least one of the first storage system, the second storage system, and the third storage system, so as to be able to integrate the file data, result data and / or content data in multiple storage systems to output the return data, so that the big data model can flexibly retrieve the corresponding data source according to different scenarios, thereby improving the performance of the big data model.

[0067] In one embodiment, see Figure 5 , Figure 5 1 is a flow chart of the second embodiment of the data query method provided by the present application. Figure 5 As shown, in this embodiment, step S42 further includes the following steps: Step S51: query at least one of the first storage system, the second storage system and the third storage system according to the content to be queried to obtain first query data.

[0068] Specifically, the first query data can be obtained by querying at least one of the first storage system, the second storage system and the third storage system based on the content to be queried. The first query data is the relevant data content queried in at least one of the first storage system, the second storage system and the third storage system. The first query data can be stored in text or in the form of a string.

[0069] Step S52: when the first query data does not meet the preset requirement, the associated data of the content to be queried is obtained through the association relationship of the fourth storage system.

[0070] After obtaining the first query data, determine whether the first query data meets the preset requirements, which include but are not limited to the first query data is not empty (i.e., the corresponding data is found), the relevance of the first query data is lower than the preset threshold, etc. When the first query data does not meet the preset requirements, that is, the corresponding data cannot be found in at least one of the first storage system, the second storage system, and the third storage system, or the data found in at least one of the first storage system, the second storage system, and the third storage system has poor relevance, at this time, the data query method of this embodiment obtains the associated data of the content to be queried through the association relationship of the fourth storage system. Among them, the knowledge graph of the data nodes related to the content to be queried can be queried in the knowledge graph stored in the fourth storage system, and then the associated data related to the content to be queried can be obtained according to the association relationship of the knowledge graph.

[0071] Step S53: query at least one of the first storage system, the second storage system and the third storage system based on the associated data to obtain second query data.

[0072] Based on the acquired associated data, a query is further performed in at least one of the first storage system, the second storage system, and the third storage system to obtain second query data corresponding to the associated data.

[0073] Step S54: Generate return data based on the second query data.

[0074] After the second query data is acquired, return data is generated according to the second query data.

[0075] In an embodiment of the present application, the data query method queries at least one of the first storage system, the second storage system, and the third storage system according to the content to be queried to obtain the first query data. When the first query data does not meet the preset requirements, the associated data of the content to be queried is obtained through the association relationship of the fourth storage system, and based on the associated data, at least one of the first storage system, the second storage system, and the third storage system is queried to obtain the second query data, and the return data is generated based on the second query data. Through the above method, the data query method of this embodiment can not only generate corresponding return data under the data dimension of the content to be queried, but also can further query and generate corresponding return data from the dimension of the associated data of the content to be queried when the data amount of the content to be queried is insufficient, thereby improving the association and analysis capabilities in the data query process, thereby improving the accuracy and comprehensiveness of the query results, so that the second query data of the associated data can provide data support for the big data model as supplementary content, and further improve the application performance of the big data model.

[0076] Different from the prior art, the data storage method of the embodiment of the present application reduces the storage volume of the big data model by storing the file data of the file to be stored in the first storage system; storing the result data of the file to be stored in the second storage system; and storing the content data of the file to be stored in the third storage system, and effectively reduces the deployment and maintenance costs of the system through lightweight design, while improving the performance and manageability of data storage, bringing users a better user experience; due to its simple configuration and superior performance, the data storage method of the embodiment of the present application can be applied to enterprise-level applications of various sizes, improve the flexibility of data storage, and thus improve the performance of the big data model. The data query method of the embodiment of the present application has powerful full-text search and data analysis capabilities, and can provide more relevant information from different angles, improve the accuracy and comprehensiveness of search results. Compared with traditional data query methods, the data query method of the embodiment of the present application has more advantages in performance and query flexibility, and supports real-time search and distributed architecture to provide powerful data support for big data models, further improving the performance of big data models.

[0077] See also Figure 6 , Figure 6 Schematic diagram of the structure of an embodiment of the electronic device provided by the present application. Figure 6 As shown, the electronic device 50 of this embodiment includes a memory 52 and a processor 51, and the processor 51 is connected to the memory 52. ​​The memory 52 is used to store program instructions for implementing the method described in any of the above embodiments. The processor 51 is used to execute the program instructions stored in the memory 52.

[0078] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0079] The memory 52 can be a memory stick, a TF card, etc., which can store all the information in the electronic device 50, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the string matching prediction device has a memory function and can ensure normal operation. The memory of the string matching prediction device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to the purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application.

[0084] See also Figure 7 , Figure 7 Schematic diagram of the structure of an embodiment of the computer-readable storage medium provided by the present application. Figure 7 As shown, the computer-readable storage medium of the present application stores program instructions 61 that can implement all the above methods, wherein the program instructions 61 can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage device includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or electronic devices such as a computer, a server, a mobile phone, and a tablet.

[0085] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data storage method, characterized in that: include: Acquire a file to be stored, wherein the file to be stored includes original file data, result data processed by a corresponding model, and content data; Storing the file data in a first storage system; storing the result data in a second storage system; The content data is stored in a third storage system, so that the file data and / or the result data are managed through the content data in the third storage system.

2. The data storage method according to claim 1, characterized in that: Before the step of storing the content data in a third storage system to manage the file data and / or the result data through the content data in the third storage system, the data storage method includes: Parsing the file data to obtain label data and / or category data of the file data; Acquire path data of the file data in the first storage system; The storing the content data in a third storage system so as to manage the file data and / or the result data through the content data in the third storage system includes: The path data, the label data and / or the category data are stored in the third storage system.

3. The data storage method according to claim 2, characterized in that: After the step of storing the content data in a third storage system to manage the file data and / or the result data through the content data in the third storage system, the data storage method includes: Slicing the label data to obtain label slices; The label slice is stored in the third storage system.

4. The data storage method according to claim 2, characterized in that: The storing the result data in a second storage system comprises: Decomposing the result data according to the label data and / or category data of the file data; The decomposed result data is stored in the index corresponding to the second storage system.

5. The data storage method according to claim 2, characterized in that: After the step of storing the content data in a third storage system to manage the file data and / or the result data through the content data in the third storage system, the data storage method further includes: Creating a first data node according to the category data and establishing a first association relationship of the first data node, and / or creating a second data node according to the tag data and establishing a second association relationship of the second data node; The first association relationship of the first data node and / or the second association relationship of the second data node are stored in a fourth storage system.

6. The data storage method according to claim 5, characterized in that: After the step of storing the first association relationship of the first data node and / or the second association relationship of the second data node in the fourth storage system, the data storage method further includes: Based on the first association relationship of the first data node, establish a first knowledge graph of the first data node, and store the first knowledge graph in the fourth storage system, and / or, Based on the second association relationship of the second data node, a second knowledge graph of the second data node is established, and the second knowledge graph is stored in the fourth storage system.

7. A data query method, characterized in that: The data query method is used to query the data stored by the data storage method according to any one of claims 1 to 6, and the data query method comprises: Receiving a search instruction, wherein the search instruction includes content to be searched; Performing a query based on the to-be-queried content in at least one of the first storage system, the second storage system, and the third storage system to obtain return data; The returned data is output.

8. The data query method according to claim 7, characterized in that: The querying at least one of the first storage system, the second storage system, and the third storage system based on the content to be queried to obtain return data further includes: Performing a query in at least one of the first storage system, the second storage system, and the third storage system according to the content to be queried to obtain first query data; When the first query data does not meet the preset requirement, obtaining the associated data of the content to be queried through the association relationship of the fourth storage system; Performing a query in at least one of the first storage system, the second storage system, and the third storage system based on the associated data to obtain second query data; The return data is generated based on the second query data.

9. An electronic device, characterized in that: The method comprises a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Intelligent search method and system based on knowledge graph

    CN112148885A

  • Simulation data storage method and device, electronic equipment and storage medium

    CN113407533A

  • Data storage and query method and device, equipment and readable storage medium

    CN114139040A

  • High-efficiency fine-tuning evaluation method and system for large language model

    CN118535887A