Data storage method, retrieval method, system, device and storage medium
By performing high-dimensional and low-dimensional conversion of data and storing it at different levels in the database, the inefficiency of retrieval caused by the complexity of different types of data storage is solved, and efficient storage and rapid retrieval of data are achieved.
Patent Information
- Application Number
- CN202311182707.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-09-12
AI Technical Summary
In the prior art, different types of data are stored in different databases, resulting in increased complexity during data retrieval, affecting efficiency, and matching data contents need to be analyzed one by one.
By obtaining the target data set to be stored, extracting its data type, and performing high-dimensional conversion and low-dimensional conversion, the high-dimensional vector data set and low-dimensional vector data set are obtained, and they are stored separately according to the pre-set index structure. It realizes efficient retrieval at different levels of the database.
Efficient storage and rapid retrieval of target data are achieved, avoiding the search of original data features one by one, and narrowing the search range through high-dimensional and low-dimensional vector features, improving the search speed.
Smart Images

Figure CN117390013B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a data storage method, a retrieval method, a system, a device and a storage medium. Background Art
[0002] With the continuous development and popularization of Internet information technology, traditional paper documents are gradually replaced by digital documents, traditional businesses are gradually turning to e-commerce platforms, and more and more users choose to publish digital content on social media platforms and online forums. These have led to the generation of a large amount of data, including different types of data such as text, images, and audio. Therefore, the storage and management of these data is very necessary.
[0003] At present, different types of data are stored in different databases. When a certain type of data needs to be found, the corresponding database must be found first, and then the data must be retrieved based on the key information in the original data. Using multiple databases to store different types of data increases the complexity of data retrieval. In addition, retrieval based on the key information of the original data requires analyzing the data content one by one and matching them. It can be seen that this way of storing and retrieving data seriously affects the efficiency of data retrieval. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to provide a data storage method, a retrieval method, a system, a device and a storage medium, which can achieve efficient storage and rapid retrieval of target data.
[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data storage method, the method comprising: obtaining a target data set to be stored, and extracting a data type of the target data set, wherein the target data set includes original data features; performing a high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; performing a low-dimensional conversion on the high-dimensional vector data set according to the data type to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; according to a pre-set index structure, storing the low-dimensional vector data set, the high-dimensional vector data set and the target data set in a first vector layer, a second vector layer and an original data layer of a database respectively, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other.
[0006] In some embodiments, the target data set is subjected to high-dimensional conversion according to the data type to obtain a high-dimensional vector data set, including: when the data type is of the first type, performing a word segmentation operation on the target data set and obtaining a word segmentation result, and performing a high-dimensional vector mapping operation on the word segmentation result to obtain a high-dimensional vector data set; when the data type is of the second type, inputting the target data set into a pre-trained conversion model, performing a feature extraction operation on the target data set according to the conversion model to obtain high-dimensional vector features, and performing a high-dimensional vectorization operation on the high-dimensional vector features to obtain a high-dimensional vector data set.
[0007] In some embodiments, the target data set is subjected to high-dimensional conversion according to the data type to obtain a high-dimensional vector data set, and further includes: when the target data set to be stored includes at least two interrelated sub-target data sets, each of the sub-target data sets is subjected to high-dimensional conversion according to the data type of each of the sub-target data sets to obtain a plurality of sub-high-dimensional vector data sets containing sub-high-dimensional vector features; and the plurality of sub-high-dimensional vector data sets are subjected to vector fusion to obtain a high-dimensional vector data set including a plurality of sub-high-dimensional vector features.
[0008] In some embodiments, the high-dimensional vector data set is converted to a low-dimensional data set according to the data type to obtain a low-dimensional vector data set, including: when the data type is a linear type, multiple linear vector features are extracted from the high-dimensional vector data set, and a low-dimensional vector mapping operation is performed on the high-dimensional vector data set according to the linear vector features to obtain a low-dimensional vector data set; when the data type is a non-linear type, the similarity value between each non-linear vector feature in the high-dimensional vector data set is calculated, and a low-dimensional vector mapping operation is performed on the non-linear vector features according to the similarity value to obtain a low-dimensional vector data set.
[0009] In some embodiments, according to a pre-set index structure, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset are respectively stored in the first vector layer, the second vector layer and the original data layer of the database, including: extracting index information of the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset, and determining the index structures corresponding to the high-dimensional vector dataset, the low-dimensional vector dataset and the target dataset respectively according to the index information; dividing the high-dimensional vector dataset, the low-dimensional vector dataset and the target dataset into a plurality of feature data groups, wherein the feature data group includes a plurality of feature data; performing dimensional variance calculation on the feature data of each feature data group to obtain a maximum dimensional variance value, and calculating an average value or a mode value of the vector data, and obtaining a dimensional partition value according to the average value or the mode value; obtaining storage partitions of each feature data group, and storing the high-dimensional vector dataset, the low-dimensional vector dataset and the target dataset in the first vector layer, the second vector layer and the original data layer of the database according to the corresponding index structure according to the maximum variance value, the dimensional partition value and the storage partition.
[0010] In some embodiments, after storing the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset in the first vector layer, the second vector layer and the original data layer of the database respectively according to a pre-set index structure, it also includes: recording the number of retrievals of each of the high-dimensional vector dataset, the low-dimensional vector dataset and the target dataset; when the number of retrievals exceeds a preset retrieval threshold, adjusting the high-dimensional vector dataset / the low-dimensional vector dataset / the target dataset to a priority node of the corresponding index structure, wherein the priority node is used to represent a priority retrieval position of the data.
[0011] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes a data retrieval method, which includes: obtaining a target retrieval feature of a target data set to be retrieved; if the target retrieval feature is a low-dimensional retrieval feature, determining a low-dimensional vector data set in a first vector layer of a database and a low-dimensional vector feature corresponding to the low-dimensional vector data set according to the low-dimensional retrieval feature; determining a high-dimensional vector data set and a high-dimensional vector feature corresponding to the high-dimensional vector data set in a second vector layer of the database according to the low-dimensional vector feature after high-dimensional conversion; determining a target data set corresponding to the high-dimensional vector feature from an original data layer of the database according to the high-dimensional vector feature after low-dimensional conversion, wherein the retrieved target data set includes original data features, the original data features match the target retrieval feature, and the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other.
[0012] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a data storage system, the system comprising: an acquisition module, used to acquire a target data set to be stored and extract the data type of the target data set, wherein the target data set includes original data features; a high-dimensional conversion module, used to perform high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; a low-dimensional conversion module, used to perform low-dimensional conversion on the high-dimensional vector data set according to the data type to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; a storage module, used to store the low-dimensional vector data set, the high-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database according to a pre-set index structure, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other.
[0013] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method described in the first aspect embodiment or the second aspect embodiment.
[0014] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect embodiment or the method described in the second aspect embodiment.
[0015] The embodiments of the present application propose a data storage method, a retrieval method, a system, a device and a storage medium, which first obtain a target data set to be stored and extract the data type of the target data set, wherein the target data set includes original data features; then, according to the data type, the target data set is subjected to high-dimensional transformation to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; then, according to the data type, the high-dimensional vector data set is subjected to low-dimensional transformation to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; thereafter, according to a pre-set index structure, the low-dimensional vector data set, the high-dimensional vector data set and the target data set are respectively stored in the first vector layer, the second vector layer and the original data layer of the database, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are related to each other.
[0016] That is, the target data set in the present application and the high-dimensional vector data set and the low-dimensional vector data set corresponding to the target data set can be stored in the same database, without being stored in multiple databases respectively, thereby achieving efficient storage of multiple data modes of the target data; and, through such a data storage method, when searching the target data set, it is not necessary to search the original data features of the target data set one by one, but it is possible to search through the high-dimensional vector features and low-dimensional vector features associated with the original data features, so as to narrow the search scope of the target data set through the low-dimensional vector data and the high-dimensional vector data. Since the retrieval speed of the low-dimensional vector data is very fast, the target data can be quickly retrieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a schematic diagram of an application scenario of a data storage system provided in an embodiment of the present application;
[0018] Figure 2 This is an optional flowchart of the data storage method provided in the embodiment of the present application;
[0019] Figure 3 yes Figure 2 An implementation flow chart of step S102;
[0020] Figure 4 yes Figure 2 Another implementation flow chart of step S102;
[0021] Figure 5 yes Figure 2 An implementation flow chart of step S103;
[0022] Figure 6 yes Figure 2 An implementation flow chart of step S104;
[0023] Figure 7 yes Figure 2 An implementation flow chart after step S104;
[0024] Figure 8 is an optional flow chart of the data retrieval method provided in the embodiment of the present application;
[0025] Fig. 9 This is a schematic diagram of a functional module for data storage provided by an embodiment of the present application;
[0026] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0028] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0030] First, some nouns involved in this application are analyzed:
[0031] High-dimensional vector data refers to data represented in a multi-dimensional space, where each data point consists of multiple features or dimensions, and these features can be different types of data such as numerical values, categories, text, etc. In an embodiment of the present application, a high-dimensional vector data set includes one or more high-dimensional vector data.
[0032] Low-dimensional vector data refers to data represented in a lower dimensional space, where each data point consists of a small number of features or dimensions, and low-dimensional vector data has fewer dimensions than high-dimensional vector data. In an embodiment of the present application, a low-dimensional vector data set includes one or more low-dimensional vector data.
[0033] With the continuous development and popularization of Internet information technology, traditional paper documents are gradually replaced by digital documents, traditional businesses are gradually turning to e-commerce platforms, and more and more users choose to publish digital content on social media platforms and online forums. These have led to the generation of a large amount of data, including different types of data such as text, images, and audio. Therefore, the storage and management of these data is very necessary.
[0034] At present, different types of data are stored in different databases. When a certain type of data needs to be found, the corresponding database must be found first, and then the data must be retrieved based on the key information in the original data. Using multiple databases to store different types of data increases the complexity of data retrieval. In addition, retrieval based on the key information of the original data requires analyzing the data content one by one and matching them. It can be seen that this way of storing and retrieving data seriously affects the efficiency of data retrieval.
[0035] Based on this, an embodiment of the present application proposes a data storage method, a retrieval method, a system, a device and a storage medium, which first obtains a target data set to be stored and extracts the data type of the target data set, wherein the target data set includes original data features; then, according to the data type, the target data set is converted into a high-dimensionality to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; then, according to the data type, the high-dimensional vector data set is converted into a low-dimensionality to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; thereafter, according to a pre-set index structure, the low-dimensional vector data set, the high-dimensional vector data set and the target data set are respectively stored in the first vector layer, the second vector layer and the original data layer of the database, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are related to each other.
[0036] That is, the target data set in the present application and the high-dimensional vector data set and the low-dimensional vector data set corresponding to the target data set can be stored in the same database, without being stored in multiple databases respectively, thereby achieving efficient storage of multiple data modes of the target data; and, through such a data storage method, when searching the target data set, it is not necessary to search the original data features of the target data set one by one, but it is possible to search through the high-dimensional vector features and low-dimensional vector features associated with the original data features, so as to narrow the search scope of the target data set through the low-dimensional vector data and the high-dimensional vector data. Since the retrieval speed of the low-dimensional vector data is very fast, the target data can be quickly retrieved.
[0037] The data storage method, retrieval method, system, device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data storage system in the embodiments of the present application is described.
[0038] For example, Figure 1 As shown, Figure 1This is a schematic diagram of the application scenario of the data storage system provided in the embodiment of the present application. In the data storage system (hereinafter referred to as the "system"), a user terminal 11 and a server terminal 12 are included, wherein the server terminal 12 includes a database, which can store high-dimensional vector data sets, low-dimensional vector data sets and target data sets in different data layers; the user can perform a search operation on the user terminal 11, and when the user enters the target search feature on the user terminal 11, the server terminal 12 can search the database according to the target search feature to obtain the target data set. It should be noted that the server terminal 12 can also be connected to multiple user terminals 11 to implement data retrieval operations for multiple users, which is specifically set according to the actual needs of the operator, and the embodiment of the present application does not make specific restrictions here.
[0039] The data storage method in the embodiments of the present application can be illustrated by the following embodiments.
[0040] In some embodiments, the embodiments of the present application can also acquire and process relevant data based on artificial intelligence technology. For example, the target data set can be converted into high dimensions and low dimensions through artificial intelligence to be stored in a database on the server side. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0041] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0042] The data storage method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The data storage method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data storage method, etc., but is not limited to the above forms.
[0043] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0044] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.
[0045] Based on this, the multi-channel dynamic file direct transmission method in the embodiment of the present application can be explained through the following embodiment.
[0046] Figure 2 is an optional flowchart of the data storage method provided in an embodiment of the present application. Figure 2 The method may include but is not limited to steps S101 to S104.
[0047] Step S101, obtaining a target data set to be stored, and extracting the data type of the target data set, wherein the target data set includes original data features;
[0048] In some embodiments, the target data set may be obtained by user input, or acquired from a database that pre-stores the target data set.
[0049] In some embodiments, the target data set may be in a variety of data forms such as text, audio, image or video, and each target data set may include one or more target data, and the data features of the target data are the original data features of the target data set.
[0050] For example, a target data set is student information, which includes the height and weight of all students. Then each student corresponds to a target data, and the height information and weight information are the original data features of the target data set.
[0051] In some embodiments, the data type of the target data set can be obtained by an extraction operation, and when the target data set is stored in a specific storage format, the data type of the target data set can be extracted. For example, when the target data set is stored in a txt file, it can be determined that the data type of the target data set is a text type.
[0052] In some embodiments, the data type of the target data set can also be determined by the transmission channel. If the data is transmitted via HTTP or HTTPS protocol, it usually contains a Content-Type field in the request header (HTTP Header), which indicates the data type of the request or response body. Exemplarily, when Content-Type is "text / plain", its data type is determined to be a text type, and when Content-Type is "image / jpeg", "image / png", or "image / gif", its data type is determined to be an image type.
[0053] Step S102, performing high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features;
[0054] In some embodiments, different high-dimensional conversion operations can be performed on the target data set for different data types of the target data set, and a high-dimensional vector data set corresponding to the target data set is generated, wherein the high-dimensional vector data set includes one or more high-dimensional vector data, and each high-dimensional vector data includes a high-dimensional vector feature.
[0055] In some embodiments, after the target data set is converted into a high-dimensional data set through high-dimensional transformation, various machine learning algorithms can be used more easily to perform processing operations such as classification, clustering or regression on the high-dimensional vector data to more accurately reflect the differences and similarities between the target data sets. Since the high-dimensional vector data set is associated with the target data set, the associated storage of the high-dimensional vector data set and the target data set can narrow the search scope of the target data set by retrieving the high-dimensional vector features to obtain the corresponding high-dimensional vector data set, thereby achieving rapid retrieval of the target data set.
[0056] Step S103, performing low-dimensional conversion on the low-dimensional vector data set according to the data type to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features;
[0057] In some embodiments, different low-dimensional conversion operations can be performed on a high-dimensional vector dataset for different data types of a target dataset, and a low-dimensional vector dataset corresponding to the high-dimensional vector dataset is generated, wherein the low-dimensional vector dataset includes one or more low-dimensional vector data, and each low-dimensional vector data includes a low-dimensional vector feature.
[0058] In some embodiments, after a high-dimensional vector data set is converted into a low-dimensional data set through low-dimensional transformation, various machine learning algorithms can be more easily used to perform classification, clustering or regression processing operations on the low-dimensional vector data to more accurately reflect the differences and similarities between the target data sets. Since the low-dimensional vector data set is associated with the target data set, through the associated storage of the low-dimensional vector data set and the target data set, the search scope of the target data set can be narrowed by retrieving the low-dimensional vector features to obtain the corresponding low-dimensional vector data set during retrieval, thereby achieving rapid retrieval of the target data set.
[0059] Step S104, according to a pre-set index structure, the low-dimensional vector data set, the high-dimensional vector data set and the target data set are respectively stored in the first vector layer, the second vector layer and the original data layer of the database, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are correlated with each other.
[0060] In some embodiments, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset are respectively stored in different data layers of the database. For example, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset can be respectively stored in the first vector layer, the second vector layer and the original data layer.
[0061] In some embodiments, multiple index structures may be preset, and relevant personnel may select one or more index structures to store data sets according to specific needs.
[0062] In some embodiments, commonly used index structures include B-Tree index, Hash index, Full-Text index, R-Tree index, etc.
[0063] In some embodiments, low-dimensional vector data sets, high-dimensional vector data sets, and target data sets (hereinafter referred to as "data sets") can be stored according to different index structures, wherein the index structures corresponding to each data set can be different, and each storage layer can include multiple different index structures for storing each data set in a variety of different storage methods, so that when retrieving data, different index structures can be used for retrieval according to different requirements. Exemplarily, the original data layer uses a B-tree index structure, while the first vector layer and the second vector layer use two index structures, KD (K-Dimensional Tree, KD) tree and LSH (Locality Sensitive Hashing, LSH) tree, to store data sets.
[0064] In some embodiments, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset may be stored in different data layers of the database, respectively, thereby achieving efficient data storage.
[0065] In some embodiments, more data storage layers may be set in the database according to actual needs to store other transformed forms of the target data set to achieve more efficient data management. For example, a metadata layer may be set, where the metadata layer is used to store metadata related to the target data set, such as the source of the data, creation time, extraction or dimensionality reduction method used, etc., which can be used for data management and auditing.
[0066] In some embodiments, since the low-dimensional vector dataset is obtained by low-dimensional transformation of the high-dimensional vector dataset, and the high-dimensional vector dataset is obtained by high-dimensional transformation of the target dataset, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset are associated with each other.
[0067] In some embodiments, since each data layer has its own index structure, the interaction and association between index structures at different levels are usually realized through some mapping relationships, thereby realizing the association between different data layers. Specifically, each index entry in the upper index structure may contain a pointer to an index entry in the index structure of the lower index, so that the lower index entry can be quickly found from the upper index entry. For example, an index entry in the original data layer may contain a pointer to the corresponding index entry in the second vector layer.
[0068] It can be understood that by performing high-dimensional conversion and low-dimensional conversion on the target data set, the multiple modal data of the target data set are stored in different data layers of the database, thereby avoiding the inefficient storage caused by storing different modal data in different databases. At the same time, such a storage method can improve the retrieval efficiency of the target data set.
[0069] like Figure 3 As shown, Figure 3 yes Figure 2 An implementation flow chart of step S102. In some embodiments, step S101 may also include steps S201 to S202:
[0070] Step S201, when the data type is the first type, performing a word segmentation operation on the target data set and obtaining a word segmentation result, performing a high-dimensional vector mapping operation on the word segmentation result to obtain a high-dimensional vector data set;
[0071] In some embodiments, the first type of data includes text type. When the target data set is text type data, the target data set can be firstly segmented to obtain a segmentation result. Then, a high-dimensional vector mapping operation is performed on the segmentation result to obtain a high-dimensional vector data set.
[0072] In some embodiments, for text data, word embeddings or sentence embeddings technology can be used to convert text into a high-dimensional vector.
[0073] In some embodiments, the word vector algorithm Word2Vec or the word vector algorithm GloVe may be used, both of which map words into a vector space so that semantically similar words are also close in distance in the vector space.
[0074] In some embodiments, a sentence vectorization algorithm BERT or Sentence-BERT may be used, both of which map the entire sentence into a high-dimensional vector space to capture the semantic information of the sentence.
[0075] In some embodiments, before the target data set is subjected to high-dimensional conversion, the high-dimensional vector data set may be preprocessed, wherein the preprocessing may include data cleaning, format conversion, feature extraction and other related operations.
[0076] Step S202, when the data type is the second type, the target data set is input into a pre-trained conversion model, a feature extraction operation is performed on the target data set according to the conversion model to obtain high-dimensional vector features, and a high-dimensional vectorization operation is performed on the high-dimensional vector features to obtain a high-dimensional vector data set.
[0077] In some embodiments, the second type of data includes data such as images, audio, and video that need to be processed using a data model.
[0078] In some embodiments, for image data, a convolutional neural network (CNN) can be used to convert the image into a high-dimensional vector. During the convolutional neural network training process, the feature representation learned by the CNN can capture various visual attributes of the image. Usually, we can extract feature vectors from the fully connected layer (or other appropriate layer) of the CNN and use them as high-dimensional vector representations of the image.
[0079] In some embodiments, for audio data, deep learning methods such as recurrent neural networks (RNN) or one-dimensional convolutional neural networks (1D CNN) can be used to convert audio signals into high-dimensional vectors. Exemplarily, the audio signal is first preprocessed, such as extracting Mel-Frequency Cepstral Coefficients (MFCC) or other features, and then, RNN or 1D CNN is used to learn the vector representation of audio features.
[0080] In some embodiments, for video data, a three-dimensional convolutional neural network (3D CNN) or a two-stream convolutional neural network (Two-Stream CNN) can be used to convert a video frame sequence into a high-dimensional vector. Among them, 3D CNN can capture the spatial and temporal information of the video, while the two-stream CNN learns the spatial information (using RGB frames) and temporal information (using optical flow) of the video respectively, and then, feature vectors can be extracted from the appropriate layers of these networks and used as high-dimensional vector representations of the video.
[0081] In some embodiments, high-dimensional vector data sets are usually represented by floating-point numbers, which may require a large storage space. In order to reduce the storage requirements, vector encoding techniques such as product quantization (PQ) or SimHash can be used to encode the original floating-point vector into a more compact binary or integer vector. This encoding method can greatly reduce the storage space requirements while maintaining the approximate calculation of the similarity between vectors.
[0082] In some embodiments, various compression algorithms such as LZ77 (Lempel-Ziv77), LZ78 (Lempel-Ziv78) or Brotli can be used to compress high-dimensional vector data sets to further reduce storage space requirements. The compressed data can be stored in the database with less storage space.
[0083] like Figure 4 As shown, Figure 4 yes Figure 2Another implementation flow chart of step S102. In some embodiments, step S102 may further include steps S301 to S302:
[0084] Step S301, when the target data set to be stored includes at least two mutually related sub-target data sets, high-dimensional conversion is performed on each sub-target data set according to the data type of each sub-target data set to obtain multiple sub-high-dimensional vector data sets containing sub-high-dimensional vector features;
[0085] In some embodiments, the target data set may contain sub-target data sets of more than one data type, and there are associations between the multiple sub-target data sets. To achieve faster data retrieval, the multiple associated sub-target data sets can be converted to high dimensions to obtain each sub-high-dimensional vector data set containing sub-high-dimensional vector features.
[0086] In some embodiments, the interrelated sub-target data sets may be multiple representations of one data, such as text type data and image type data of city weather; or, they may be two different data that are related, such as text type data of city weather and city introduction video type data.
[0087] In some embodiments, the target data set may include multiple sub-target data sets, some of which are related to each other, while others are not. In this regard, high-dimensional transformation can be performed on the related sub-target data sets.
[0088] Step S302: vector fusion is performed on the multiple sub-high-dimensional vector data sets to obtain a high-dimensional vector data set including features of the multiple sub-high-dimensional vector data sets.
[0089] In some embodiments, vector fusion can be performed on multiple sub-high-dimensional vector data sets containing sub-high-dimensional vector features obtained after high-dimensional conversion. The high-dimensional vector data set obtained after vector fusion includes sub-high-dimensional vector features of multiple sub-target data sets participating in the vector fusion. In this way, a high-dimensional vector data set that is fused with multiple different modalities can utilize information from different perception channels and better understand and express the characteristics of multimodal objects, so that when retrieving the target data set, the target data set can be retrieved more accurately and quickly.
[0090] In some embodiments, the vector fusion method may include a variety of vector fusion methods such as a vector concatenation method, a weighted average method, a feature cross method, a linear transformation method, or a model fusion method.
[0091] Exemplarily, the target dataset includes a sub-target dataset A of a text data type and a sub-target dataset B of an image data type associated with the data. First, word embedding (such as Word2Vec or GloVe) is used to convert the sub-target dataset A into a sub-high-dimensional vector dataset A. Assuming that the word embedding function is E, then for the sub-target dataset A, we can get its vector representation as E(A); similarly, a pre-trained convolutional neural network (such as VGG or ResNet) can be used to convert the sub-target dataset B into a sub-high-dimensional vector dataset B. Assuming that our convolutional neural network is C, then for the sub-target dataset B, its vector representation can be C(B). Then, the sub-high-dimensional vector dataset A and the sub-high-dimensional vector dataset B can be vector fused to form a new vector. For example, when the vector concatenation method is used for vector fusion, the fused high-dimensional vector dataset can be obtained as [E(A), C(B)].
[0092] Exemplarily, after obtaining the fused high-dimensional vector data set, the high-dimensional vector data set can be further subjected to vector data feature enhancement processing operations. For example, the fused high-dimensional vector data set [E(A), C(B)] can be input into a fully connected neural network F (Fully Connected Network, FCN), and the data result after vector data feature enhancement can be obtained as F([E(A), C(B)]).
[0093] like Figure 5 As shown, Figure 5 yes Figure 2 An implementation flow chart of step S103. In some embodiments, step S103 may include steps S401 to S402:
[0094] Step S401, when the data type is a linear type, extract multiple linear vector features from the high-dimensional vector data set, and perform a low-dimensional vector mapping operation on the high-dimensional vector data set according to the linear vector features to obtain a low-dimensional vector data set;
[0095] In some embodiments, after storing the high-dimensional vector dataset corresponding to the target dataset in the database, the high-dimensional vector dataset needs to be converted into a low-dimensional dataset to obtain a low-dimensional vector dataset, wherein the low-dimensional vector dataset contains low-dimensional vector features, and the retrieval speed of the low-dimensional vector features is very fast.
[0096] In some embodiments, the low-dimensional vector mapping operation is a dimensionality reduction process, which is a process of mapping high-dimensional data to a low-dimensional space, aiming to preserve the key features and structure of the data. It is understandable that when processing high-dimensional vectors, obtaining a low-dimensional vector data set through dimensionality reduction techniques can help reduce computational complexity, reduce storage requirements, and improve visualization effects.
[0097] In some embodiments, when the data type of the target data set also includes a linear type and a nonlinear type, when the target data set is of a linear type, multiple linear vector features can be extracted from the high-dimensional vector data set, and then, a low-dimensional vector mapping operation is performed on the high-dimensional vector data set according to the linear vector features to obtain a low-dimensional vector data set.
[0098] In some embodiments, principal component analysis (PCA) is a linear dimensionality reduction technique that achieves dimensionality reduction by finding the main variance direction of the data and projecting the data into a low-dimensional space composed of principal components. PCA is usually used to process continuous data, i.e., linear data, and assumes that the data has a linear structure in the low-dimensional space.
[0099] In some embodiments, Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction method that attempts to find the direction that maximizes the distance between categories when projecting data into a low-dimensional space. LDA is typically used for classification problems, and its goal is to maximize the separability of categories in the reduced-dimensional space.
[0100] Step S402, when the data type is a nonlinear type, calculate the similarity values between the nonlinear vector features in the high-dimensional vector data set, perform low-dimensional vector mapping operations on the nonlinear vector features according to the similarity values, and obtain a low-dimensional vector data set.
[0101] In some embodiments, when the data type of the target data set is a nonlinear type, a nonlinear dimensionality reduction method needs to be used accordingly.
[0102] In some embodiments, t-Distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction method that maps high-dimensional data to a low-dimensional space while trying to preserve the similarity structure in the original data. t-SNE achieves dimensionality reduction by minimizing the KL (Kullback-Leibler) divergence between similarity distributions in high-dimensional space and low-dimensional space. t-SNE is particularly suitable for visualizing high-dimensional data such as images, text, and bioinformatics data.
[0103] like Figure 6 As shown, Figure 6 yes Figure 2 An implementation flow chart of step S104. In some embodiments, step S104 may include steps S501 to S504:
[0104] Step S501, respectively extracting index information of the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset, and determining index structures corresponding to the high-dimensional vector dataset, the low-dimensional vector dataset and the target dataset according to the index information;
[0105] In some embodiments, the index information may be information such as key attributes or key fields, and the index structures of the low-dimensional vector data sets, high-dimensional vector data sets, and target data sets are determined one by one based on the index information. Exemplarily, the index information of the target data set is a data sequence number, and each data sequence number is unique, so a B+ tree can be used as the index structure for storage.
[0106] In some embodiments, in order to achieve efficient vector retrieval, it is necessary to use a suitable data structure (i.e., index structure) to store high-dimensional vectors. Commonly used data structures include KD tree, ball tree (Ball-Tree) and VP tree (Vantage-PointTree). These tree structures can be organized according to the distance relationship between vectors to reduce unnecessary distance calculations during retrieval.
[0107] In some embodiments, to further improve the retrieval efficiency, the database can be partitioned using indexing technology. For example, similar vectors can be grouped together using inverted indexing technology to reduce the search space during the retrieval process. Partitioning technology such as Locality Sensitive Hashing (LSH) or k-means clustering can be used to divide the vector database into smaller subsets, thereby achieving distributed storage and parallel retrieval.
[0108] Step S502, dividing the high-dimensional vector data set, the low-dimensional vector data set and the target data set into a plurality of feature data groups, wherein the feature data group includes a plurality of vector data;
[0109] In some embodiments, after determining the corresponding index structure, it is necessary to further determine a partitioning strategy to better store the data set.
[0110] In some embodiments, the high-dimensional vector data set, the low-dimensional vector data set and the target data set are first divided into a plurality of feature data groups, respectively. The feature data groups are divided in order to determine the maximum variance value and the dimension division value of the target dimension from the plurality of groups.
[0111] Step S503, performing dimension variance calculation on the feature data of each feature data group to obtain the maximum dimension variance value, and calculating the average value or mode value of the feature data, and obtaining the dimension division value according to the average value or mode value;
[0112] In some embodiments, related technologies usually use the median of the target data as the partition value. These simple strategies may not work well when processing high-dimensional data or non-uniformly distributed data. In this regard, in the embodiments of the present application, the dimension of the maximum variance of the target data is used as the partition dimension to maximize the difference in the subspace.
[0113] For example, assume that the feature data group is an n-dimensional data set D, and our goal is to select a partition dimension d. Then, the variance of each feature data group can be calculated, and then the dimension with the largest variance, i.e., the dimension with the largest variance value, is selected as the partition dimension. The specific calculation formula of the partition dimension d is shown in the following formula (1):
[0114] d=argmax_{i=1}^n*var(D_i) (1)
[0115] Among them, Di represents the data in the i-th dimension of the data set D, var(Di) represents the variance of Di, and argmax means finding the dimension that maximizes the variance.
[0116] In some embodiments, the dimension division value represents the corresponding dimension division value required after determining the maximum variance value of the division dimension, wherein the dimension division value is used to assign the high-dimensional vector data set, the low-dimensional vector data set and the target data set to corresponding sub-nodes or regions. The dimension division value determines the location and method of division, and plays an important role in the storage and query of the data set.
[0117] Step S504, obtaining the storage partition of each feature data group, according to the maximum variance value, the dimension partition value and the storage partition, storing the high-dimensional vector data set, the low-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database according to the corresponding index structure.
[0118] In some embodiments, storage partitions are used to represent different storage areas, and the first vector layer, the second vector layer, and the original data layer each correspond to one or more storage partitions.
[0119] In some embodiments, after determining the maximum variance value, dimension division value and storage partition, the high-dimensional vector dataset, low-dimensional vector dataset and target dataset can be stored in the first vector layer, the second vector layer and the original data layer of the database according to the corresponding index structures at different layers.
[0120] In some embodiments, different data layers of the database may be further divided into multiple storage partitions, where different storage partitions may correspond to different index structures. The same data set may be stored in different storage partitions using different index structures, or different data may be stored in different storage partitions using the same index structure.
[0121] like Figure 7 As shown, Figure 7 yes Figure 2 An implementation flow chart after step S104. In some embodiments, step S104 may further include steps S601 to S602:
[0122] Step S601, recording the number of retrievals of each high-dimensional vector data set, low-dimensional vector data set and target data set;
[0123] In some embodiments, the number of retrieval times refers to data access frequency or query load.
[0124] In some embodiments, in each data storage layer of the database, the retrieval times of each high-dimensional vector data set, low-dimensional vector data set and target data set can also be recorded and saved, and the storage of the data set can be adjusted according to the retrieval times to make the storage of the data set more efficient and reasonable, thereby improving the efficiency of data set retrieval.
[0125] Step S602, when the number of retrievals exceeds a preset retrieval threshold, the high-dimensional vector dataset / low-dimensional vector dataset / target dataset is adjusted to a priority node of a corresponding index structure, wherein the priority node is used to represent a priority retrieval position of the data.
[0126] In some embodiments, a retrieval threshold may be preset, and the retrieval threshold is used for comparison with the number of retrieval times. When the number of retrieval times is greater than the retrieval threshold, the storage structure may be optimized.
[0127] In some embodiments, when the number of retrievals exceeds a preset retrieval threshold, the high-dimensional vector data set / low-dimensional vector data set / target data set can be adjusted to a priority node of a corresponding index structure. Exemplarily, the index structure of the first vector layer includes three node layers. When performing data retrieval, the first node layer is usually prioritized for retrieval. Then, when the number of retrievals of a certain high-dimensional vector data set exceeds a preset retrieval threshold, the high-dimensional vector data set can be adjusted to the first node layer of the index structure corresponding to the first vector layer, thereby completing the dynamic adjustment of the storage of the data set and further improving the retrieval efficiency of the data set.
[0128] In some embodiments, storage resources may also be dynamically adjusted according to load conditions, which may specifically include dynamic adjustment of storage resources, dynamic adjustment of computing resources, and dynamic adjustment of network resources.
[0129] In some embodiments, the dynamic adjustment of storage resources includes dynamic migration of data and dynamic adjustment of replicas. For example, when the storage space of a storage node of the index structure is insufficient, part of the data of the storage node can be migrated to other storage nodes; or when the access frequency of a data set increases, the number of replicas of the data set can be increased to improve the access speed of the data.
[0130] In some embodiments, the dynamic adjustment of computing resources includes dynamic scheduling and dynamic expansion of storage nodes. For example, when the computing load of a storage node is too high, a portion of the storage data set of the storage node can be scheduled to other nodes; or when the overall storage demand of the database increases, new storage nodes can be added to improve the overall storage capacity of the data.
[0131] In some embodiments, the dynamic adjustment of network resources includes the dynamic adjustment of network topology. For example, when the network bandwidth is insufficient, since the same data set may be stored using different index structures, when the network delay is too high, another index structure may be selected for retrieval to optimize and improve the retrieval speed.
[0132] Figure 8 is an optional flow chart of the data retrieval method provided in the embodiment of the present application. Figure 8 The method may include but is not limited to steps S701 to S704.
[0133] Step S701, obtaining target retrieval features of the target data set to be retrieved;
[0134] In some embodiments, the database storing the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset obtained by the data storage method provided by the embodiments of the present application can be used to retrieve the dataset.
[0135] In some embodiments, the target retrieval feature may be key feature data of a high-dimensional vector dataset / a low-dimensional vector dataset / a target dataset.
[0136] Step S702: if the target search feature is a low-dimensional search feature, determine a low-dimensional vector data set and a low-dimensional vector feature corresponding to the low-dimensional vector data set in the first vector layer of the database according to the low-dimensional search feature;
[0137] In some embodiments, when the target retrieval feature is a low-dimensional retrieval feature, a low-dimensional vector data set corresponding to the low-dimensional retrieval feature can be retrieved in the first vector layer of the database, and the low-dimensional vector data set includes the low-dimensional vector feature.
[0138] Step S703, determining a high-dimensional vector data set and high-dimensional vector features corresponding to the high-dimensional vector data set in the second vector layer of the database according to the low-dimensional vector features after high-dimensional conversion;
[0139] In some embodiments, since the low-dimensional vector dataset is converted from a high-dimensional vector dataset, after obtaining the low-dimensional vector dataset, the low-dimensional vector features corresponding to the low-dimensional vector dataset can also be converted into high dimensions. The high-dimensional vector features obtained by the high-dimensional conversion can be used to perform retrieval in the second vector layer, and the high-dimensional vector dataset corresponding to the retrieved low-dimensional vector dataset can be determined, and the high-dimensional vector dataset includes the high-dimensional vector features.
[0140] Step S704, based on the high-dimensional vector features after low-dimensional conversion, determine the target data set corresponding to the high-dimensional vector features from the original data layer of the database, wherein the retrieved target data set includes the original data features, the original data features match the target retrieval features, and the high-dimensional vector features, the low-dimensional vector features and the original data features are correlated with each other.
[0141] In some embodiments, since the high-dimensional vector dataset is converted from the target dataset, after obtaining the high-dimensional vector dataset, the high-dimensional vector features corresponding to the high-dimensional vector dataset can be converted into low dimensions, and the original data features of the target dataset obtained by the low-dimensional conversion are used to perform retrieval at the original data layer, thereby ultimately determining the target dataset corresponding to the retrieved high-dimensional vector dataset.
[0142] In some embodiments, by first performing a fast similarity search on a low-dimensional vector data set in a first vector layer, then performing an accurate similarity match on a corresponding high-dimensional vector data set in a second vector layer, and finally returning the final target data set retrieval result in the original data layer, it can be understood that through the hierarchical retrieval strategy of the database, the advantages of the multi-level storage structure can be fully utilized to achieve efficient retrieval of the target data set.
[0143] It should be noted that since the low-dimensional vector features contain a small amount of data and have low retrieval complexity, the low-dimensional vector data set corresponding to the target data set that is finally retrieved can be quickly determined through the low-dimensional retrieval features. In this way, the retrieval range can be narrowed in advance, and then the high-dimensional vector data set can be further determined by using the low-dimensional vector features after high-dimensional conversion. The retrieval range can be accurately determined to finally determine the target data set, avoiding the low efficiency of data set retrieval caused by searching the target data sets one by one.
[0144] In some embodiments, the target retrieval feature may also be a high-dimensional vector feature. In this case, the high-dimensional vector data set may be directly determined from the second vector layer of the database, and the target data set corresponding to the high-dimensional vector feature may be determined from the original data layer of the database based on the high-dimensional vector feature after low-dimensional conversion. Alternatively, the target retrieval feature may also be an original data feature. In this case, the corresponding target data set may be directly determined from the original data layer of the database.
[0145] like Fig. 9 As shown, Fig. 9 : is a functional module diagram of data storage provided in an embodiment of the present application. An embodiment of the present application also provides a data storage system that can implement the above data storage method. The data storage system includes:
[0146] An acquisition module 801 is used to acquire a target data set to be stored and extract a data type of the target data set, wherein the target data set includes original data features;
[0147] A high-dimensional conversion module 802 is used to perform high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features;
[0148] A low-dimensional conversion module 803 is used to perform low-dimensional conversion on the high-dimensional vector data set according to the data type to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features;
[0149] The storage module 804 is used to store the low-dimensional vector data set, the high-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database respectively according to a pre-set index structure, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are correlated with each other.
[0150] In some embodiments, the target data set may be obtained by user input, or acquired from a database that pre-stores the target data set.
[0151] In some embodiments,
[0152] In some embodiments, the target data set may be in a variety of data forms such as text, audio, image or video, and each target data set may include one or more target data, and the data features of the target data are the original data features of the target data set.
[0153] For example, a target data set is student information, which includes the height and weight of all students. Then each student corresponds to a target data, and the height information and weight information are the original data features of the target data set.
[0154] In some embodiments, the data type of the target data set can be obtained by an extraction operation, and when the target data set is stored in a specific storage format, the data type of the target data set can be extracted. For example, when the target data set is stored in a txt file, it can be determined that the data type of the target data set is a text type.
[0155] In some embodiments, the data type of the target data set can also be determined by the transmission channel. If the data is transmitted via HTTP or HTTPS protocol, it usually contains a Content-Type field in the request header (HTTP Header), which indicates the data type of the request or response body. Exemplarily, when Content-Type is "text / plain", its data type is determined to be a text type, and when Content-Type is "image / jpeg", "image / png", or "image / gif", its data type is determined to be an image type.
[0156] In some embodiments, different high-dimensional conversion operations can be performed on the target data set for different data types of the target data set, and a high-dimensional vector data set corresponding to the target data set is generated, wherein the high-dimensional vector data set includes one or high-dimensional vector data, and each high-dimensional vector data includes a high-dimensional vector feature.
[0157] In some embodiments, after the target data set is converted into a high-dimensional data set through high-dimensional transformation, various machine learning algorithms can be used more easily to perform processing operations such as classification, clustering or regression on the high-dimensional vector data to more accurately reflect the differences and similarities between the target data sets. Since the high-dimensional vector data set is associated with the target data set, the associated storage of the high-dimensional vector data set and the target data set can narrow the search scope of the target data set by retrieving the high-dimensional vector features to obtain the corresponding high-dimensional vector data set, thereby achieving rapid retrieval of the target data set.
[0158] In some embodiments, different low-dimensional conversion operations can be performed on a high-dimensional vector dataset for different data types of a target dataset, and a low-dimensional vector dataset corresponding to the high-dimensional vector dataset is generated, wherein the low-dimensional vector dataset includes one or low-dimensional vector data, and each low-dimensional vector data includes a low-dimensional vector feature.
[0159] In some embodiments, after a high-dimensional vector data set is converted into a low-dimensional data set through low-dimensional transformation, various machine learning algorithms can be more easily used to perform classification, clustering or regression processing operations on the low-dimensional vector data to more accurately reflect the differences and similarities between the target data sets. Since the low-dimensional vector data set is associated with the target data set, through the associated storage of the low-dimensional vector data set and the target data set, the search scope of the target data set can be narrowed by retrieving the low-dimensional vector features to obtain the corresponding low-dimensional vector data set during retrieval, thereby achieving rapid retrieval of the target data set.
[0160] In some embodiments, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset are respectively stored in different data layers of the database. For example, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset can be respectively stored in the first vector layer, the second vector layer and the original data layer.
[0161] In some embodiments, multiple index structures may be preset, and relevant personnel may select one or more index structures to store data sets according to specific needs.
[0162] In some embodiments, commonly used index structures include B-Tree index, Hash index, Full-Text index, R-Tree index, etc.
[0163] In some embodiments, low-dimensional vector data sets, high-dimensional vector data sets, and target data sets (hereinafter referred to as "data sets") can be stored according to different index structures, wherein the index structures corresponding to each data set can be different, and each storage layer can include different index structures for storing each data set in a variety of different storage methods, so that when retrieving data, different index structures can be used for retrieval according to different requirements. Exemplarily, the original data layer uses a B-tree index structure, while the first vector layer and the second vector layer use two index structures, KD (K-Dimensional Tree, KD) tree and LSH (Locality Sensitive Hashing, LSH) tree, to store data sets.
[0164] In some embodiments, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset may be stored in different data layers of the database, respectively, thereby achieving efficient data storage.
[0165] In some embodiments, more data storage layers may be set in the database according to actual needs to store other transformed forms of the target data set to achieve more efficient data management. For example, a metadata layer may be set, where the metadata layer is used to store metadata related to the target data set, such as the source of the data, creation time, extraction or dimensionality reduction method used, etc., which can be used for data management and auditing.
[0166] In some embodiments, since the low-dimensional vector dataset is obtained by low-dimensional transformation of the high-dimensional vector dataset, and the high-dimensional vector dataset is obtained by high-dimensional transformation of the target dataset, the low-dimensional vector dataset, the high-dimensional vector dataset and the target dataset are associated with each other.
[0167] In some embodiments, since each data layer has its own index structure, the interaction and association between index structures at different levels are usually realized through some mapping relationships, thereby realizing the association between different data layers. Specifically, each index entry in the upper index structure may contain a pointer to an index entry in the index structure of the lower index, so that the lower index entry can be quickly found from the upper index entry. For example, an index entry in the original data layer may contain a pointer to the corresponding index entry in the second vector layer.
[0168] In some embodiments, for the convenience of users, the data storage system can provide a unified API interface and query language to support transparent access to the multi-level storage structure. In addition, the data storage system can also provide visualization tools and monitoring functions to help users understand the status of data storage and retrieval, and make adjustments and optimizations as needed.
[0169] It can be understood that by performing high-dimensional conversion and low-dimensional conversion on the target data set, the multiple modal data of the target data set are stored in different data layers of the database, thereby avoiding the inefficient storage caused by storing different modal data in different databases. At the same time, such a storage method can improve the retrieval efficiency of the target data set.
[0170] The specific implementation of the data storage system is basically the same as the specific implementation of the data storage method described above, and will not be repeated here. On the premise of meeting the requirements of the embodiments of the present application, the data storage system can also be provided with other functional modules to implement the data storage method in the above embodiments.
[0171] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data storage method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.
[0172] like Fig.10 As shown, Fig.10 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:
[0173] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0174] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data storage method of the embodiments of this application;
[0175] Input / output interface 903, used to implement information input and output;
[0176] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0177] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0178] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0179] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data storage method is implemented.
[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0181] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0182] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0183] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0184] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0185] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0186] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0187] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0188] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0191] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A data storage method, characterized in that: The method comprises: Acquire a target data set to be stored, and extract a data type of the target data set, wherein the target data set includes original data features; According to the data type, performing high-dimensional conversion on the target data set to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; According to the data type, the high-dimensional vector data set is converted into a low-dimensional data set to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; According to a pre-set index structure, the low-dimensional vector data set, the high-dimensional vector data set and the target data set are respectively stored in the first vector layer, the second vector layer and the original data layer of the database, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other, the first vector layer, the second vector layer and the original data layer are all provided with corresponding index structures, there is a mapping association relationship between adjacent index structures, and each upper-level index structure contains a pointer to the lower-level index structure.
2. The data storage method according to claim 1, characterized in that: The step of performing high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set includes: When the data type is the first type, performing a word segmentation operation on the target data set to obtain a word segmentation result, and performing a high-dimensional vector mapping operation on the word segmentation result to obtain a high-dimensional vector data set; When the data type is the second type, the target data set is input into a pre-trained conversion model, a feature extraction operation is performed on the target data set according to the conversion model to obtain high-dimensional vector features, and a high-dimensional vectorization operation is performed on the high-dimensional vector features to obtain a high-dimensional vector data set.
3. The data storage method according to claim 1, characterized in that: The step of performing high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set further includes: When the target data set to be stored includes at least two mutually related sub-target data sets, high-dimensional conversion is performed on each of the sub-target data sets according to the data type of each of the sub-target data sets to obtain a plurality of sub-high-dimensional vector data sets containing sub-high-dimensional vector features; Vector fusion is performed on the plurality of sub-high-dimensional vector data sets to obtain a high-dimensional vector data set including features of the plurality of sub-high-dimensional vector data sets.
4. The data storage method according to claim 1, characterized in that: The step of converting the high-dimensional vector data set into a low-dimensional data set according to the data type to obtain a low-dimensional vector data set includes: When the data type is a linear type, extracting multiple linear vector features from the high-dimensional vector data set, and performing a low-dimensional vector mapping operation on the high-dimensional vector data set according to the linear vector features to obtain a low-dimensional vector data set; When the data type is a nonlinear type, similarity values between nonlinear vector features in the high-dimensional vector data set are calculated, and low-dimensional vector mapping operations are performed on the nonlinear vector features according to the similarity values to obtain a low-dimensional vector data set.
5. The data storage method according to claim 1, characterized in that: The method of storing the low-dimensional vector data set, the high-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database respectively according to the preset index structure includes: Respectively extracting index information of the low-dimensional vector dataset, the high-dimensional vector dataset, and the target dataset, and determining index structures corresponding to the high-dimensional vector dataset, the low-dimensional vector dataset, and the target dataset according to the index information; Dividing the high-dimensional vector data set, the low-dimensional vector data set and the target data set into a plurality of feature data groups respectively, wherein the feature data groups include a plurality of feature data; Performing dimension variance calculation on the feature data of each feature data group to obtain a maximum dimension variance value, and calculating an average value or a mode value of the vector data, and obtaining a dimension division value according to the average value or the mode value; Obtain storage partitions for each of the feature data groups, and according to the maximum variance value, the dimensionality division value and the storage partitions, store the high-dimensional vector data set, the low-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database respectively according to the corresponding index structure.
6. The data storage method according to claim 1, characterized in that: After storing the low-dimensional vector data set, the high-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database respectively according to the preset index structure, the method further includes: Recording the number of retrievals of each of the high-dimensional vector dataset, the low-dimensional vector dataset, and the target dataset; When the number of retrievals exceeds a preset retrieval threshold, the high-dimensional vector dataset / the low-dimensional vector dataset / the target dataset is adjusted to a priority node of a corresponding index structure, wherein the priority node is used to represent a priority retrieval position of data.
7. A data retrieval method, characterized in that: The method comprises: Obtain target retrieval features of the target data set to be retrieved; If the target retrieval feature is a low-dimensional retrieval feature, determining a low-dimensional vector data set and a low-dimensional vector feature corresponding to the low-dimensional vector data set in a first vector layer of a database according to the low-dimensional retrieval feature; Determine a high-dimensional vector data set and high-dimensional vector features corresponding to the high-dimensional vector data set in a second vector layer of the database according to the low-dimensional vector features after high-dimensional conversion; According to the high-dimensional vector features after low-dimensional conversion, a target data set corresponding to the high-dimensional vector features is determined from the original data layer of the database, wherein the retrieved target data set includes original data features, the original data features match the target retrieval features, the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other, the first vector layer, the second vector layer and the original data layer are all provided with corresponding index structures, there is a mapping association relationship between adjacent index structures, and each upper-level index structure contains a pointer to the lower-level index structure.
8. A data storage system, characterized in that: The system comprises: An acquisition module, used to acquire a target data set to be stored and extract a data type of the target data set, wherein the target data set includes original data features; A high-dimensional conversion module, used for performing high-dimensional conversion on the target data set according to the data type to obtain a high-dimensional vector data set, wherein the high-dimensional vector data set includes high-dimensional vector features; A low-dimensional conversion module, used for performing low-dimensional conversion on the high-dimensional vector data set according to the data type to obtain a low-dimensional vector data set, wherein the low-dimensional vector data set includes low-dimensional vector features; A storage module is used to store the low-dimensional vector data set, the high-dimensional vector data set and the target data set in the first vector layer, the second vector layer and the original data layer of the database respectively according to a pre-set index structure, so that when searching according to the low-dimensional vector features or the high-dimensional vector features, the target data set is obtained, wherein the high-dimensional vector features, the low-dimensional vector features and the original data features are associated with each other, the first vector layer, the second vector layer and the original data layer are all provided with corresponding index structures, there is a mapping association relationship between adjacent index structures, and each upper-level index structure contains a pointer to the lower-level index structure.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data storage method according to any one of claims 1 to 6 or the data retrieval method according to claim 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 or the data retrieval method according to claim 7 is implemented.
Citation Information
Patent Citations
Text searching method and device
CN101140583A
System and method for incrementally converting high dimensional data into low dimensional data
CN104318046A
High-dimensional image feature matching method and device
CN108629345A
High-dimensional vector data visualization method and system based on double-layer anchor point graph projection optimization
CN110188098A
Voiceprint retrieval method and device, equipment and storage medium
CN114925230A
