Data storage method, data retrieval method and related devices

Abstract

The embodiment of the invention provides a data storage method, a data retrieval method and related devices. The method comprises the following steps: storing multiple paths of storage data in a first database, and generating an index according to a storage position of the storage data in the first database; generating integrated data according to the storage data and the index; storing the integrated data into a second database to realize data storage; the method comprises the steps of extracting first data corresponding to storage data from to-be-retrieved data; according to the matching degree of the data in the first database and the first data, searching in the first database to obtain second data, and determining a target index corresponding to the second data; searching in a second database according to the target index to obtain target data so as to realize data retrieval; by combining the first database and the second database, all data contents can be quickly accessed, and the memory consumption of the system is reduced; quick vector data retrieval matching can be carried out, so that the retrieval speed is improved under the condition of mass data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data storage method, a data retrieval method and related devices. Background Art

[0002] With the development of computer technology, network technology and artificial intelligence technology, a large amount of data needs to be stored and retrieved. Currently, data management is mainly achieved through databases. Some databases, such as the milvus database, are prone to a sharp increase in memory consumption when storing large amounts of data, which also affects the efficiency of data access and retrieval. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a data storage method, a data retrieval method and related devices, which can improve the data retrieval speed.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a data storage method, including:

[0005] Acquire a data set, wherein the data set includes multi-channel storage data;

[0006] Storing the plurality of storage data in a first database, and generating an index according to the storage location of the storage data in the first database;

[0007] Generate integrated data according to the multiple channels of stored data and the index;

[0008] The integrated data is stored in a second database, the second database being different from the first database.

[0009] In some embodiments, the first database includes a plurality of sub-databases; storing the plurality of storage data in the first database, and generating an index at a storage location of the first database according to the storage data, comprises:

[0010] The storage data of different paths are stored in different sub-databases respectively, and indexes are generated according to the storage positions of the storage data in the sub-databases.

[0011] In some embodiments, the index includes a plurality of sub-indexes; storing the storage data of different paths in different sub-databases respectively, and generating an index according to the storage location of the storage data in the sub-database, comprises:

[0012] The storage data of the previous path is stored in a first sub-database, and a sub-index corresponding to the storage data of the previous path is generated according to the storage position of the storage data of the previous path in the first sub-database, wherein the first sub-database is any sub-database;

[0013] The sub-index corresponding to the storage data of the previous path and the storage data of the next path are stored in a second sub-database, and the sub-index corresponding to the storage data of the next path is generated at the storage position of the second sub-database according to the storage data of the next path, and the second sub-database is a sub-database different from the first sub-database.

[0014] In some embodiments, the same sub-database of the first database stores data of different data sets.

[0015] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data retrieval method, wherein data is stored according to the data storage method of the first aspect of an embodiment of the present application, and the data retrieval method comprises:

[0016] Get the data to be retrieved;

[0017] Extracting first data corresponding to the stored data from the data to be retrieved;

[0018] According to the matching degree between the data in the first database and the first data, searching in the first database to obtain a plurality of second data, and determining a target index corresponding to the second data;

[0019] The target data is obtained by searching in the second database according to the target index.

[0020] In some embodiments, searching the first database for a plurality of second data according to the matching degree between the data in the first database and the first data, and determining the target index corresponding to the second data includes:

[0021] According to the matching degree between the data in the first sub-database of the first database and the previous first data, searching in the first sub-database to obtain the second data of the plurality of first sub-databases, and determining the sub-index corresponding to the second data of the first sub-database;

[0022] According to the sub-index of the second data corresponding to the first sub-database and the matching degree between the data in the second sub-database of the first database and the latter first data, a plurality of second data of the second sub-database are searched in the second sub-database to obtain the second data, and the sub-index corresponding to the second data of the second sub-database is determined as the target index.

[0023] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a data storage device, including:

[0024] A first input module is used to obtain a data set, wherein the data set includes multi-channel storage data;

[0025] A first storage module, used for storing the plurality of storage data in a first database, and generating an index according to a storage location of the storage data in the first database;

[0026] An integration module, used for generating integrated data according to the multiple channels of stored data and the index;

[0027] The second storage module is used to store the integrated data in a second database, where the second database is different from the first database.

[0028] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application provides a data retrieval device, comprising:

[0029] The second input module is used to obtain the data to be retrieved;

[0030] A data extraction module, used for extracting first data corresponding to the stored data from the data to be retrieved;

[0031] A first retrieval module, configured to search the first database for a plurality of second data according to a matching degree between the data in the first database and the first data, and determine a target index corresponding to the second data;

[0032] The second retrieval module is used to search for target data in the second database according to the target index.

[0033] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the data storage method described in the first aspect of the embodiment of the present application and the data retrieval method described in the second aspect of the embodiment of the present application.

[0034] To achieve the above-mentioned purpose, the sixth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the data storage method described in the first aspect of the embodiment of the present application and the data retrieval method described in the second aspect of the embodiment of the present application.

[0035] The data storage method, data retrieval method and related devices proposed in the present application store multi-channel storage data in a first database, generate an index according to the storage position of the storage data in the first database; generate integrated data according to the multi-channel storage data and the index; store the integrated data in a second database to realize data storage; extract the first data corresponding to the storage data from the data to be retrieved; according to the matching degree between the data in the first database and the first data, find and obtain multiple second data in the first database, and determine the target index corresponding to the second data; find and obtain the target data in the second database according to the target index to realize data retrieval; store data by combining the first database and the second database, so that all data contents can be quickly accessed and the memory consumption of the system can be reduced; and fast vector data retrieval and matching can be performed to improve the retrieval speed in the case of large amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a step diagram of a data storage method provided by an embodiment of the present application;

[0037] Figure 2 It is a sub-step diagram of step S200 provided in an embodiment of the present application;

[0038] Figure 3 It is a sub-step diagram of step S210 provided in an embodiment of the present application;

[0039] Figure 4 is a step diagram of a data retrieval method provided in an embodiment of the present application;

[0040] Figure 5 It is a sub-step diagram of step S700 provided in an embodiment of the present application;

[0041] Figure 6 is a structural diagram of a data storage device provided in an embodiment of the present application;

[0042] Figure 7 is a structural diagram of a data retrieval device provided in an embodiment of the present application;

[0043] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0047] With the development of computer technology, network technology and artificial intelligence technology, a large amount of data needs to be stored and retrieved. Currently, data management is mainly achieved through databases. Some databases, such as the milvus database, are prone to a sharp increase in memory consumption when storing large amounts of data, which also affects the efficiency of data access and retrieval.

[0048] In order to solve the above problems, the embodiments of the present application provide a data storage method, a data retrieval method and related devices, which can improve the data retrieval speed.

[0049] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0050] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0051] The data storage method, data retrieval method and related devices provided in the embodiments of the present application relate to the field of artificial intelligence technology. The data storage method and data retrieval method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements data storage methods and data retrieval methods, etc., but is not limited to the above forms.

[0052] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0053] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant regulations. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0054] The data storage method, data retrieval method and related devices provided in the embodiments of the present application are specifically described through the following embodiments. First, the data storage method in the embodiments of the present application is described.

[0055] Reference Figure 1 , the embodiment of the present application provides a data storage method, comprising the following steps:

[0056] Step S100, obtaining a data set;

[0057] Step S200, storing the multi-channel storage data in a first database, and generating an index according to the storage location of the storage data in the first database;

[0058] Step S300, generating integrated data according to the multi-channel storage data and index;

[0059] Step S400: storing the integrated data in the second database.

[0060] For step S100, a data set is input through an input device, and a data set input from the input device is received. Alternatively, a data set is read from an external storage device. Alternatively, a data set is read from an external database.

[0061] The data set includes multiple storage data. For example, the data format of the data set is as follows: {"A": [one-dimensional array with a length of a], "B": [one-dimensional array with a length of b], "C": [one-dimensional array with a length of c], "D": string, ...}. Of course, in other embodiments, the data type of the storage data of the data set is not limited to array type and string type, and the data type of the storage data of the data set can be other types, such as character type, floating point type, etc.

[0062] The lengths of the one-dimensional arrays of different paths are different. For example, the storage data A is a one-dimensional array with a length of 1000, the storage data B is a one-dimensional array with a length of 2000, and the storage data C is a one-dimensional array with a length of 3000. Of course, in other embodiments, the lengths of the storage data of different paths may be the same, for example, the storage data A, the storage data B, and the storage data C are all one-dimensional arrays with a length of 1000; or the storage data of some paths are the same, for example, the storage data A is a one-dimensional array with a length of 1000, the storage data B is a one-dimensional array with a length of 2000, and the storage data C is a one-dimensional array with a length of 1000.

[0063] Financial data is stored through one-dimensional data, such as account information, transaction information, and credit scores in personal financial information.

[0064] When image data needs to be stored, encode the image data and convert it into a one-dimensional array. Convert the image to a grayscale image and extract the pixel values ​​of the pixels of the image data. The pixel values ​​are usually represented in the form of a two-dimensional array, where each row represents a row of pixels in the image and each pixel consists of its color value. Flatten the two-dimensional array of pixel values ​​to convert the two-dimensional pixel array into a one-dimensional array. This can be achieved through a simple list derivation or loop, connecting the values ​​of all pixels to form a long list. In addition, the pixel values ​​can be normalized to the range of 0 to 1, which is usually achieved by dividing each pixel value by 255. A one-dimensional array that stores image data.

[0065] When audio data needs to be stored, encode the audio data and extract sample data from the waveform format data of the audio file. The sample data is represented by a two-dimensional array, where each row represents a channel and each column represents a sample point. Merge all channels into a single one-dimensional array. This can be achieved by simply concatenating the sample data of all channels. In addition, the values ​​of the one-dimensional data can be normalized to the range of 0 to 1. A one-dimensional array that stores audio data.

[0066] Reference Figure 2 For step S200, storing the multi-channel storage data in the first database, and generating an index according to the storage location of the storage data in the first database, comprises the following steps:

[0067] Step S210: storing the storage data of different paths in different sub-databases respectively, and generating an index according to the storage location of the storage data in the sub-database.

[0068] Reference Figure 3 , further, step S211, storing the storage data of the previous path into the first sub-database, generating a sub-index corresponding to the storage data of the previous path in the storage position of the first sub-database according to the storage data of the previous path; step S212, storing the sub-index corresponding to the storage data of the previous path and the storage data of the next path into the second sub-database, generating a sub-index corresponding to the storage data of the next path in the storage position of the second sub-database according to the storage data of the next path. Wherein, the first sub-database is any sub-database, and the second sub-database is a sub-database different from the first sub-database.

[0069] The first database is the milvus database. The milvus database has the following advantages: high performance, ease of use, elastic scalability, reliability, hybrid search, unified architecture, etc. It can process trillion-scale vector data sets in milliseconds and provide an average search latency of milliseconds. It provides a rich API to simplify the management of unstructured data and supports client libraries in multiple languages, including Python, Java, Go, C++, and Node.js. It supports cloud-native architecture design and has elastic scalability at the component level, which can automatically expand or reduce resources on demand. It has built-in replication and failover / recovery functions to ensure that data and applications can maintain business continuity in the event of failures. It introduces multi-vector support and a hybrid search framework, allowing multiple vector fields to be introduced in a single collection. These fields can represent different aspects of the data, come from different embedding models or undergo different processing methods. It combines stream processing and batch processing to balance timeliness and efficiency, and provides a unified interface to make vector similarity search simpler and more convenient. However, the milvus database has poor memory management capabilities, which makes it easy for the memory consumption of the milvus database to increase sharply under large amounts of data.

[0070] The first database includes multiple sub-databases, such as database milvus A, database milvus B, database milvus C, database milvus D, and so on.

[0071] The storage data of different paths are stored in different sub-databases respectively; for example, the storage data A is stored in the database milvus A, the storage data B is stored in the database milvus B, the storage data C is stored in the database milvus C, and the storage data D is stored in the database milvus D.

[0072] In addition, the same sub-database of the first database stores storage data of different data sets. For example, the second data set includes storage data E, storage data F, and storage data G. The storage data E, storage data F, and storage data G of the second data set are respectively stored in different sub-databases of the first database. The database milvus A storing storage data A continues to store storage data E, the database milvus B storing storage data B continues to store storage data F, and the database milvus C storing storage data C continues to store storage data G.

[0073] If the storage data A is stored first, and then the storage data B is stored, the storage data A is the storage data of the previous path, and the storage data B is the storage data of the next path. Similarly, if the storage data B is stored first, and then the storage data C is stored, the storage data B is the storage data of the previous path, and the storage data C is the storage data of the next path. Similarly, if the storage data C is stored first, and then the storage data D is stored, the storage data C is the storage data of the previous path, and the storage data D is the storage data of the next path.

[0074] The storage data of the previous path is stored in the first sub-database, and a sub-index corresponding to the storage data of the previous path is generated according to the storage position of the storage data of the previous path in the first sub-database, that is, the storage data A is stored in the database milvus A, and then the sub-index 1 corresponding to the storage data A is generated according to the storage position of the storage data A in the database milvus A.

[0075] The sub-index corresponding to the storage data of the previous path and the storage data of the next path are stored in the second sub-database, and the sub-index corresponding to the storage data of the next path is generated according to the storage position of the storage data of the next path in the second sub-database, that is, the sub-index 1 and the storage data B are stored in the database milvus B, and then the sub-index 2 corresponding to the storage data B is generated according to the storage position of the storage data B in the database milvusB.

[0076] When there are other stored data to be stored, the steps of storing the sub-index corresponding to the previous stored data and the next stored data in the second sub-database, and generating the sub-index corresponding to the next stored data in the storage position of the second sub-database according to the next stored data. For example, sub-index 2 and stored data C are stored in database milvus C, and then sub-index 3 corresponding to stored data C is generated according to the storage position of stored data C in database milvus C; sub-index 3 and stored data D are stored in database milvus D, and then sub-index 4 corresponding to stored data D is generated according to the storage position of stored data D in database milvus D, until all stored data are stored in the sub-databases respectively.

[0077] It should be noted that the sub-index corresponding to the previous path of stored data may be stored only in the second sub-database, or may be stored in the first sub-database and the second sub-database.

[0078] On the one hand, the sub-index corresponding to the storage data of the previous path is only stored in the second sub-database; for example, the sub-index 1 corresponding to the storage data A is only stored in the database milvus B. If the sub-index 1 in the database milvus B can meet all query requirements, and the data synchronization mechanism between the database milvus A and the database milvus B can ensure the consistency of the index, then the sub-index 1 corresponding to the storage data A is only stored in the database milvus B. If the database milvusA is mainly used for write operations, and the database milvus B is mainly used for read operations, then retaining the index in the database milvus B can improve the performance of the read operation.

[0079] On the other hand, the sub-index corresponding to the storage data of the previous path is stored in the first sub-database and the second sub-database; for example, the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B. If the query often needs to directly access the storage data A in the database milvus A, then the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B, which can improve the query efficiency. If the storage data A is frequently updated in milvus A, and the sub-index 1 in the database milvus B is not updated synchronously, data inconsistency may occur; in this case, the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B, which helps maintain data consistency. In addition, for security or compliance reasons, it is necessary to store data and indexes separately, and the sub-index 1 corresponding to the storage data A needs to be stored in the database milvus A and the database milvus B. If the database milvus B fails, the sub-index 1 retained in the database milvus A can be used as a backup to ensure the availability of the storage data A.

[0080] In summary, whether to retain the sub-index corresponding to the previous storage data in the first database needs to be comprehensively considered based on specific business needs, system architecture, performance requirements, maintenance costs and other factors. In some cases, in order to improve query performance and data consistency, it may be necessary to retain the index in both databases. In other cases, if query efficiency and data consistency can be guaranteed by other mechanisms, then the first database may no longer need to store the sub-index corresponding to the previous storage data.

[0081] For step S300, integrated data is generated based on multiple storage data and indexes. For example, an index composed of multiple sub-indexes such as sub-index 1, sub-index 2, sub-index 3, and sub-index 4 and multiple storage data such as storage data A, storage data B, storage data C, and storage data D are used to form integrated data.

[0082] For step S400, the integrated data is stored in a second database. For example, the second database is a mysql database. The index composed of multiple sub-indexes such as sub-index 1, sub-index 2, sub-index 3, and sub-index 4 and the integrated data composed of multiple storage data such as storage data A, storage data B, storage data C, and storage data D are stored in the mysql database.

[0083] MySQL database is a database based on Structured Query Language (SQL). MySQL database has the following advantages: lightweight, cross-platform, high performance, reliability, scalability, flexibility, ease of use, standardization and security. MySQL database can run on a variety of operating systems, including various Unix and Linux variants, Windows, macOS, etc. MySQL database has fast processing speed and efficient data storage capabilities when processing large amounts of data. MySQL database provides a variety of data integrity and security features, including transaction processing, backup and recovery functions. MySQL database can be expanded by increasing hardware resources or through clustering and partitioning technology. MySQL database supports multiple storage engines, allowing users to choose different storage engines according to application requirements. MySQL database provides a wealth of client tools and APIs, supports multiple programming languages, and makes it easy to interact. MySQL database provides a variety of security features, including user permission management, encrypted connections, and audit logs.

[0084] In order to solve the problem of a sharp increase in memory consumption when storing data in a large amount of data only through the milvus database, the milvus database and the mysql database are combined to store data, so that all data contents can be accessed quickly and the system's memory consumption can be reduced.

[0085] Reference Figure 4 , the present application embodiment provides a data retrieval method, comprising the following steps:

[0086] Step S500, obtaining the data to be retrieved;

[0087] Step S600, extracting first data corresponding to the stored data from the data to be retrieved;

[0088] Step S700, searching the first database for multiple second data based on the matching degree between the data in the first database and the first data, and determining the target index corresponding to the second data;

[0089] Step S800: searching the second database for target data according to the target index.

[0090] For step S500, the data to be retrieved is input through an input device, and the data to be retrieved input from the input device is received, or the data to be retrieved is read from an external storage device.

[0091] For step S600, first data corresponding to the storage data is extracted from the data to be retrieved. For example, the relevant content of storage data A, the relevant content of storage data B, and the relevant content of storage data D are extracted from the data to be retrieved, and the relevant content of storage data A, the relevant content of storage data B, and the relevant content of storage data D are the first data.

[0092] Reference Figure 5 For step S700, according to the matching degree between the data in the first database and the first data, multiple second data are searched in the first database, and the target index corresponding to the second data is determined, including the following steps:

[0093] Step S710, searching in the first sub-database for a plurality of second data of the first sub-database according to the matching degree between the data in the first sub-database of the first database and the previous first data, and determining the sub-index corresponding to the second data of the first sub-database;

[0094] Step S720, according to the sub-index of the second data corresponding to the first sub-database and the matching degree between the data in the second sub-database of the first database and the latter first data, the second data of multiple second sub-databases are searched in the second sub-database, and the sub-index corresponding to the second data of the second sub-database is determined as the target index.

[0095] Exemplarily, the first database is a milvus database, the first data stores the relevant content of data A, the first sub-database is a database milvus A, the second data stores the relevant content of data B, and the second sub-database is a database milvus B.

[0096] According to the matching degree between the data in the first sub-database of the first database and the previous first data, the second data of multiple first sub-databases are searched in the first sub-database to determine the sub-index corresponding to the second data of the first sub-database, then the matching degree between each data of the database milvus A and the relevant content of the stored data A is calculated, and the milvus vector search function is used to search in the database milvus A to obtain the first n data with the highest matching degree with the stored data A, and the first n data with the highest matching degree with the stored data A are used as the second data of the database milvus A; and the sub-index corresponding to the first n data is obtained.

[0097] According to the sub-index of the second data of the first sub-database corresponding to the first sub-database and the matching degree of the data in the second sub-database of the first database with the latter first data, the second data of multiple second sub-databases are searched in the second sub-database, and the sub-index corresponding to the second data of the second sub-database is determined as the target index. Then, the matching degree of each data of the database milvus B and the relevant content of the storage data B is calculated, and the milvus vector search function is used to search the database milvus B according to the index 1 and the first n data to obtain the first m data with the highest matching degree with the storage data B, where m is less than n. The sub-index corresponding to the first m data is obtained.

[0098] By analogy with the above method, until the data with the highest matching degree with the last stored data is found in the last sub-database, the indexes of these data are obtained as the target indexes, namely, the index corresponding to the stored data A, the index corresponding to the stored data B, and the index corresponding to the stored data D.

[0099] For step S800, the target data is searched in the second database according to the target index. For example, storage data A, storage data B and storage data D are searched in the mysql database according to the target index as the target data.

[0100] It is capable of performing fast vector data retrieval and matching, thereby improving retrieval speed in the case of large amounts of data.

[0101] The embodiment of the present application further provides a data storage device, which applies the above data storage method.

[0102] Reference Figure 6 , a data storage device includes: a first input module 710, a first storage module 720, an integration module 730 and a second storage module 740.

[0103] Among them, the first input module 710 is used to obtain a data set, which includes multi-channel storage data; the first storage module 720 is used to store the multi-channel storage data in a first database, and generate an index according to the storage location of the storage data in the first database; the integration module 730 is used to generate integrated data according to the multi-channel storage data and the index; the second storage module 740 is used to store the integrated data in a second database, which is different from the first database.

[0104] The data set is input through an input device, and the first input module 710 receives the data set input from the input device; or, the first input module 710 reads the data set from an external storage device; or, the first input module 710 reads the data set from an external database.

[0105] The data set includes multiple storage data. For example, the data format of the data set is as follows: {"A": [one-dimensional array with a length of a], "B": [one-dimensional array with a length of b], "C": [one-dimensional array with a length of c], "D": string, ...}. Of course, in other embodiments, the data type of the storage data of the data set is not limited to array type and string type, and the data type of the storage data of the data set can be other types, such as character type, floating point type, etc.

[0106] The lengths of the one-dimensional arrays of different paths are different. For example, the storage data A is a one-dimensional array with a length of 1000, the storage data B is a one-dimensional array with a length of 2000, and the storage data C is a one-dimensional array with a length of 3000. Of course, in other embodiments, the lengths of the storage data of different paths may be the same, for example, the storage data A, the storage data B, and the storage data C are all one-dimensional arrays with a length of 1000; or the storage data of some paths are the same, for example, the storage data A is a one-dimensional array with a length of 1000, the storage data B is a one-dimensional array with a length of 2000, and the storage data C is a one-dimensional array with a length of 1000.

[0107] Financial data is stored through one-dimensional data, such as account information, transaction information, and credit scores in personal financial information.

[0108] When image data needs to be stored, encode the image data and convert it into a one-dimensional array. Convert the image to a grayscale image and extract the pixel values ​​of the pixels of the image data. The pixel values ​​are usually represented in the form of a two-dimensional array, where each row represents a row of pixels in the image and each pixel consists of its color value. Flatten the two-dimensional array of pixel values ​​to convert the two-dimensional pixel array into a one-dimensional array. This can be achieved through a simple list derivation or loop, connecting the values ​​of all pixels to form a long list. In addition, the pixel values ​​can be normalized to the range of 0 to 1, which is usually achieved by dividing each pixel value by 255. A one-dimensional array that stores image data.

[0109] When audio data needs to be stored, encode the audio data and extract sample data from the waveform format data of the audio file. The sample data is represented by a two-dimensional array, where each row represents a channel and each column represents a sample point. Merge all channels into a single one-dimensional array. This can be achieved by simply concatenating the sample data of all channels. In addition, the values ​​of the one-dimensional data can be normalized to the range of 0 to 1. A one-dimensional array that stores audio data.

[0110] The first storage module 720 stores the storage data of different paths in different sub-databases respectively, and generates an index according to the storage location of the storage data in the sub-database.

[0111] Further, the storage data of the previous path is stored in the first sub-database, and a sub-index corresponding to the storage data of the previous path is generated at the storage position of the first sub-database according to the storage data of the previous path, and the first sub-database is any sub-database; the sub-index corresponding to the storage data of the previous path and the storage data of the next path are stored in the second sub-database, and a sub-index corresponding to the storage data of the next path is generated at the storage position of the second sub-database according to the storage data of the next path, and the second sub-database is a sub-database different from the first sub-database.

[0112] The first database is a milvus database, which includes a plurality of sub-databases, such as a database milvusA, a database milvus B, a database milvus C, and a database milvus D.

[0113] The storage data of different paths are stored in different sub-databases respectively; for example, the storage data A is stored in the database milvus A, the storage data B is stored in the database milvus B, the storage data C is stored in the database milvus C, and the storage data D is stored in the database milvus D.

[0114] In addition, the same sub-database of the first database stores storage data of different data sets. For example, the second data set includes storage data E, storage data F, and storage data G. The storage data E, storage data F, and storage data G of the second data set are respectively stored in different sub-databases of the first database. The database milvus A storing storage data A continues to store storage data E, the database milvus B storing storage data B continues to store storage data F, and the database milvus C storing storage data C continues to store storage data G.

[0115] If the storage data A is stored first, and then the storage data B is stored, the storage data A is the storage data of the previous path, and the storage data B is the storage data of the next path. Similarly, if the storage data B is stored first, and then the storage data C is stored, the storage data B is the storage data of the previous path, and the storage data C is the storage data of the next path. Similarly, if the storage data C is stored first, and then the storage data D is stored, the storage data C is the storage data of the previous path, and the storage data D is the storage data of the next path.

[0116] The storage data of the previous path is stored in the first sub-database, and a sub-index corresponding to the storage data of the previous path is generated according to the storage position of the storage data of the previous path in the first sub-database, that is, the storage data A is stored in the database milvus A, and then the sub-index 1 corresponding to the storage data A is generated according to the storage position of the storage data A in the database milvus A.

[0117] The sub-index corresponding to the storage data of the previous path and the storage data of the next path are stored in the second sub-database, and the sub-index corresponding to the storage data of the next path is generated according to the storage position of the storage data of the next path in the second sub-database, that is, the sub-index 1 and the storage data B are stored in the database milvus B, and then the sub-index 2 corresponding to the storage data B is generated according to the storage position of the storage data B in the database milvusB.

[0118] When there are other stored data to be stored, the steps of storing the sub-index corresponding to the previous stored data and the next stored data in the second sub-database, and generating the sub-index corresponding to the next stored data in the storage position of the second sub-database according to the next stored data. For example, sub-index 2 and stored data C are stored in database milvus C, and then sub-index 3 corresponding to stored data C is generated according to the storage position of stored data C in database milvus C; sub-index 3 and stored data D are stored in database milvus D, and then sub-index 4 corresponding to stored data D is generated according to the storage position of stored data D in database milvus D, until all stored data are stored in the sub-databases respectively.

[0119] It should be noted that the sub-index corresponding to the previous path of stored data may be stored only in the second sub-database, or may be stored in the first sub-database and the second sub-database.

[0120] On the one hand, the sub-index corresponding to the storage data of the previous path is only stored in the second sub-database; for example, the sub-index 1 corresponding to the storage data A is only stored in the database milvus B. If the sub-index 1 in the database milvus B can meet all query requirements, and the data synchronization mechanism between the database milvus A and the database milvus B can ensure the consistency of the index, then the sub-index 1 corresponding to the storage data A is only stored in the database milvus B. If the database milvusA is mainly used for write operations, and the database milvus B is mainly used for read operations, then retaining the index in the database milvus B can improve the performance of the read operation.

[0121] On the other hand, the sub-index corresponding to the storage data of the previous path is stored in the first sub-database and the second sub-database; for example, the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B. If the query often needs to directly access the storage data A in the database milvus A, then the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B, which can improve the query efficiency. If the storage data A is frequently updated in milvus A, and the sub-index 1 in the database milvus B is not updated synchronously, data inconsistency may occur; in this case, the sub-index 1 corresponding to the storage data A is stored in the database milvus A and the database milvus B, which helps maintain data consistency. In addition, for security or compliance reasons, it is necessary to store data and indexes separately, and the sub-index 1 corresponding to the storage data A needs to be stored in the database milvus A and the database milvus B. If the database milvus B fails, the sub-index 1 retained in the database milvus A can be used as a backup to ensure the availability of the storage data A.

[0122] In summary, whether to retain the sub-index corresponding to the previous storage data in the first database needs to be comprehensively considered based on specific business needs, system architecture, performance requirements, maintenance costs and other factors. In some cases, in order to improve query performance and data consistency, it may be necessary to retain the index in both databases. In other cases, if query efficiency and data consistency can be guaranteed by other mechanisms, then the first database may no longer need to store the sub-index corresponding to the previous storage data.

[0123] The integration module 730 generates integrated data according to the multiple storage data and indexes. For example, an index composed of multiple sub-indexes such as sub-index 1, sub-index 2, sub-index 3, and sub-index 4 and multiple storage data such as storage data A, storage data B, storage data C, and storage data D form integrated data.

[0124] The second storage module 740 stores the integrated data in a second database. For example, the second database is a mysql database. The integrated data composed of multiple sub-indexes such as sub-index 1, sub-index 2, sub-index 3, and sub-index 4 and multiple storage data such as storage data A, storage data B, storage data C, and storage data D are stored in the mysql database.

[0125] In order to solve the problem of a sharp increase in memory consumption when storing data in a large amount of data only through the milvus database, the milvus database and the mysql database are combined to store data, so that all data contents can be accessed quickly and the system's memory consumption can be reduced.

[0126] The embodiment of the present application further provides a data retrieval device, which applies the above data retrieval method.

[0127] Reference Figure 7 The data retrieval device includes: a second input module 810, a data extraction module 820, a first retrieval module 830 and a second retrieval module 840.

[0128] Among them, the second input module 810 is used to obtain the data to be retrieved; the data extraction module is used to extract the first data corresponding to the stored data from the data to be retrieved; the first retrieval module is used to search for multiple second data in the first database according to the matching degree between the data in the first database and the first data, and determine the target index corresponding to the second data; the second retrieval module is used to search for the target data in the second database according to the target index.

[0129] The data to be retrieved is inputted through an input device, and the second input module 810 receives the data to be retrieved inputted from the input device, or reads the data to be retrieved from an external storage device.

[0130] The data extraction module 820 extracts first data corresponding to the storage data from the data to be retrieved. For example, the relevant content of storage data A, the relevant content of storage data B, and the relevant content of storage data D are extracted from the data to be retrieved, and the relevant content of storage data A, the relevant content of storage data B, and the relevant content of storage data D are the first data.

[0131] The first retrieval module 830 searches in the first sub-database for multiple second data of the first sub-database according to the degree of match between the data in the first sub-database of the first database and the previous first data, and determines the sub-index corresponding to the second data of the first sub-database; searches in the second sub-database for multiple second data of the second sub-database according to the sub-index corresponding to the second data of the first sub-database and the degree of match between the data in the second sub-database of the first database and the subsequent first data, and determines the sub-index corresponding to the second data of the second sub-database as the target index.

[0132] Exemplarily, the first database is a milvus database, the first data stores the relevant content of data A, the first sub-database is a database milvus A, the second data stores the relevant content of data B, and the second sub-database is a database milvus B.

[0133] According to the matching degree between the data in the first sub-database of the first database and the previous first data, the second data of multiple first sub-databases are searched in the first sub-database to determine the sub-index corresponding to the second data of the first sub-database, then the matching degree between each data of the database milvus A and the relevant content of the stored data A is calculated, and the milvus vector search function is used to search in the database milvus A to obtain the first n data with the highest matching degree with the stored data A, and the first n data with the highest matching degree with the stored data A are used as the second data of the database milvus A; and the sub-index corresponding to the first n data is obtained.

[0134] According to the sub-index of the second data of the first sub-database corresponding to the first sub-database and the matching degree of the data in the second sub-database of the first database with the latter first data, the second data of multiple second sub-databases are searched in the second sub-database, and the sub-index corresponding to the second data of the second sub-database is determined as the target index. Then, the matching degree of each data of the database milvus B and the relevant content of the storage data B is calculated, and the milvus vector search function is used to search the database milvus B according to the index 1 and the first n data to obtain the first m data with the highest matching degree with the storage data B, where m is less than n. The sub-index corresponding to the first m data is obtained.

[0135] By analogy, until the data with the highest matching degree with the last stored data is found in the last sub-database, the indexes of these data are obtained as the target indexes, namely, the index corresponding to the stored data A, the index corresponding to the stored data B, and the index corresponding to the stored data D.

[0136] The second search module 840 searches the second database for target data according to the target index. For example, storage data A, storage data B and storage data D are searched in the mysql database according to the target index as target data.

[0137] It is capable of performing fast vector data retrieval and matching, thereby improving retrieval speed in the case of large amounts of data.

[0138] The present application also provides an electronic device. Figure 8 The electronic device includes a memory 902 and a processor 901. The memory 902 stores a computer program. When the processor 901 executes the computer program, the above data storage method and data retrieval method are implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0139] The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; the memory 902 can be implemented in the form of a read-only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device, or a random access memory (Random Access Memory, RAM). The memory 902 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 is used to call and execute the data storage method and data retrieval method of the embodiments of the present application.

[0140] The input / output interface 903 is used to realize information input and output; the communication interface 904 is used to realize communication interaction between this device and other devices, and communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.); the bus 905 transmits information between the various components of the device (such as the processor 901, the memory 902, the input / output interface 903 and the communication interface 904); wherein the processor 901, the memory 902, the input / output interface 903 and the communication interface 904 realize communication connection with each other within the device through the bus 905.

[0141] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data storage method and data retrieval method are implemented.

[0142] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0144] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0145] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0147] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0148] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0149] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0153] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data storage method, characterized in that: include: Acquire a data set, wherein the data set includes multi-channel storage data; Storing the plurality of storage data in a first database, and generating an index according to the storage location of the storage data in the first database; Generate integrated data according to the multiple channels of stored data and the index; The integrated data is stored in a second database, the second database being different from the first database.

2. The data storage method according to claim 1, characterized in that: The first database includes a plurality of sub-databases; storing the plurality of storage data in the first database, and generating an index at a storage location of the first database according to the storage data, comprises: The storage data of different paths are stored in different sub-databases respectively, and indexes are generated according to the storage positions of the storage data in the sub-databases.

3. The data storage method according to claim 2, characterized in that: The index includes a plurality of sub-indexes; storing the storage data of different paths in different sub-databases respectively, and generating an index at a storage location of the sub-database according to the storage data, including: The storage data of the previous path is stored in a first sub-database, and a sub-index corresponding to the storage data of the previous path is generated according to the storage position of the storage data of the previous path in the first sub-database, wherein the first sub-database is any sub-database; The sub-index corresponding to the storage data of the previous path and the storage data of the next path are stored in a second sub-database, and the sub-index corresponding to the storage data of the next path is generated at the storage position of the second sub-database according to the storage data of the next path, and the second sub-database is a sub-database different from the first sub-database.

4. The data storage method according to claim 2, characterized in that: The same sub-database of the first database stores data of different data sets.

5. A data retrieval method, characterized in that: The data is stored according to the data storage method according to any one of claims 1 to 5, and the data retrieval method comprises: Get the data to be retrieved; Extracting first data corresponding to the stored data from the data to be retrieved; According to the matching degree between the data in the first database and the first data, searching in the first database to obtain a plurality of second data, and determining a target index corresponding to the second data; The target data is obtained by searching in the second database according to the target index.

6. The data retrieval method according to claim 5, characterized in that: The step of searching the first database for a plurality of second data according to the matching degree between the data in the first database and the first data, and determining a target index corresponding to the second data includes: According to the matching degree between the data in the first sub-database of the first database and the previous first data, searching in the first sub-database to obtain the second data of the plurality of first sub-databases, and determining the sub-index corresponding to the second data of the first sub-database; According to the sub-index of the second data corresponding to the first sub-database and the matching degree between the data in the second sub-database of the first database and the latter first data, a plurality of second data of the second sub-database are searched in the second sub-database to obtain the second data, and the sub-index corresponding to the second data of the second sub-database is determined as the target index.

7. A data storage device, characterized in that: include: A first input module is used to obtain a data set, wherein the data set includes multi-channel storage data; A first storage module, used for storing the plurality of storage data in a first database, and generating an index according to a storage location of the storage data in the first database; An integration module, used for generating integrated data according to the multiple channels of stored data and the index; The second storage module is used to store the integrated data in a second database, where the second database is different from the first database.

8. A data retrieval device, characterized in that: include: The second input module is used to obtain the data to be retrieved; A data extraction module, used for extracting first data corresponding to the stored data from the data to be retrieved; A first retrieval module, configured to search the first database for a plurality of second data according to a matching degree between the data in the first database and the first data, and determine a target index corresponding to the second data; The second retrieval module is used to search for target data in the second database according to the target index.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the data storage method according to any one of claims 1 to 4 and the data retrieval method according to any one of claims 5 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 4 and the data retrieval method according to any one of claims 5 to 6 are implemented.

Images