Data storage method and device, equipment, storage medium and program product

Abstract

The invention provides a data storage method and device, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of computers, in particular to the technical field of data processing such as data retrieval, data reading and data transmission. The method comprises the following steps: in response to received target data, obtaining a first preset number of index data similar to the target data in a data index center; determining a plurality of corresponding vector storage engines according to the first preset number of index data, extracting a second preset number of data similar to the target data from each vector storage engine, and calculating the sum of the distances between the target data and the second preset number of data in each vector storage engine; and storing the target data into the vector storage engine with the minimum distance sum. According to the method, the flexibility and efficiency of data storage are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of data processing technology such as data retrieval, data reading, and data transmission, and particularly to a data storage method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the ever-increasing demand for large-scale data processing and storage, traditional centralized storage can no longer meet the requirements of high availability, scalability, and high performance. To address these challenges, data is typically divided into multiple independent parts and stored on different storage instances, which can effectively achieve data load balancing and scalability. Summary of the Invention

[0003] This disclosure provides a data storage method, apparatus, electronic device, computer-readable storage medium, and computer program product, which improves the flexibility and efficiency of data storage.

[0004] In a first aspect, embodiments of this disclosure propose a data storage method, comprising: in response to receiving target data, acquiring a first preset number of index data that are close to the target data in a data index center; determining a plurality of corresponding vector storage engines based on the first preset number of index data, retrieving a second preset number of data that are close to the target data from each vector storage engine, calculating the sum of distances between the target data and the second preset number of data in each vector storage engine; and storing the target data in the vector storage engine with the smallest sum of distances.

[0005] Secondly, embodiments of this disclosure propose a data storage system, including a data sending end, a data storage end, and a data indexing center; the data sending end is used to send target data to the data storage end; the data storage segment is used to store the received target data into a vector storage engine, including: in response to receiving the target data, obtaining a first preset number of index data that are close to the target data in the data indexing center; determining a plurality of corresponding vector storage engines based on the first preset number of index data, retrieving a second preset number of data that are close to the target data from each vector storage engine, calculating the sum of distances between the target data and the second preset number of data in each vector storage engine; storing the target data into the vector storage engine with the smallest sum of distances; the data indexing center is used to store the index data of each vector storage engine.

[0006] Thirdly, embodiments of this disclosure propose a data storage device, comprising: a data acquisition module configured to, in response to receiving target data, acquire a first preset number of index data that are close to the target data in a data index center; a distance calculation module configured to, based on the first preset number of index data, determine a plurality of corresponding vector storage engines, extract a second preset number of data that are close to the target data from each vector storage engine, and calculate the sum of the distances between the target data and the second preset number of data in each vector storage engine; and a data storage module configured to store the target data in the vector storage engine with the smallest sum of distances.

[0007] Fourthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the data storage method as described in the first aspect.

[0008] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the data storage method as described in the first aspect when executed.

[0009] In a sixth aspect, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the steps of the data storage method as described in the first aspect.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart illustrating a data storage method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another data storage method provided in this disclosure embodiment; Figure 4 This is a flowchart example of a data storage method in an application scenario provided by an embodiment of the present disclosure; Figure 5A timing diagram of an embodiment of a data storage system provided in this disclosure; Figure 6 A structural block diagram of a data storage device provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device suitable for performing a data storage method, provided as an embodiment of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0013] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0014] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the data storage methods, apparatuses, electronic devices, and computer-readable storage media of the present disclosure can be applied.

[0015] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0016] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include business applications, search applications, and instant messaging applications.

[0017] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0018] Server 105 can provide various services through its built-in applications. Taking a business application that can store data as an example, when running this business application, server 105 can achieve the following: First, when target data is received, it retrieves a first preset number of index data that are close to the target data from the data index center; then, based on the first preset number of index data, it determines multiple corresponding vector storage engines, retrieves a second preset number of data that are close to the target data from each vector storage engine, and calculates the sum of the distances between the target data and the second preset number of data in each vector storage engine; finally, it stores the target data in the vector storage engine with the smallest sum of distances.

[0019] It should be noted that, in addition to being obtained from terminal devices 101, 102, and 103 via network 104, the target data can also be pre-stored locally on server 105 through various means. Therefore, when server 105 detects that this data is already stored locally (e.g., when starting to process previously stored data storage tasks), it can choose to retrieve this data directly from the local storage. In this case, the exemplary system architecture 100 may also exclude terminal devices 101, 102, and 103 and network 104.

[0020] Since storing data in the corresponding vector storage engine always requires significant computing resources and strong computing power, the data storage methods provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the data storage device is generally also located in the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also perform the aforementioned calculations performed by the server 105 through their installed business applications, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but a business application determines that its terminal device has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the data storage device can also be located in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Please refer to Figure 2 , Figure 2 A flowchart of a data storage method provided in this disclosure embodiment, wherein process 200 includes the following steps: Step 201: In response to receiving the target data, retrieve a first preset number of index data that are similar to the target data from the data index center.

[0023] In this embodiment, when the executing entity (e.g. Figure 1 When the server 105 shown receives the target data, it retrieves a first preset number of index data that are similar to the target data from the data index center. The data index center stores multiple index data, which are used to quickly locate the vector storage engine storing that index data.

[0024] Step 202: Determine multiple corresponding vector storage engines based on the first preset number of index data, extract the second preset number of data that are close to the target data from each vector storage engine, and calculate the sum of the distances between the target data and the second preset number of data in each vector storage engine.

[0025] In this embodiment, the executing entity determines multiple corresponding vector storage engines based on the first preset number of index data obtained in step 201, and retrieves a second preset number of data points similar to the target data from each vector storage engine, and calculates the sum of the distances between the target data and the second preset number of data points in each vector storage engine. A vector storage engine is a database or storage system specifically designed for storing and querying vector data. Its core function is to store high-dimensional data (such as processed vector representations of text, images, and audio) in a specific structure and to efficiently perform operations such as Nearest Neighbor Search. A vector storage engine contains index data and other similar data points similar to the index data.

[0026] In this embodiment, the execution entity can retrieve a second preset number of data that are similar to the target data from each vector storage engine through the following steps: calculate the similarity between all data in each vector storage engine and the target data, and use the K-nearest neighbor (K-NN) query function provided by the vector storage engine to return the K vectors that are most similar to the target vector. Here, K can be regarded as the second preset number.

[0027] In this embodiment, the executing entity can use cosine similarity to calculate the sum of the distances between the target data and a second preset number of data in each vector storage engine.

[0028] Step 203: Store the target data in the vector storage engine with the smallest sum of distances.

[0029] In this embodiment, the executing entity stores the target data in the vector storage engine with the smallest sum of distances. The smaller the sum of distances in the vector storage engine, the closer the stored data is to the target data.

[0030] The data storage method disclosed in this embodiment involves, upon receiving target data, first acquiring a first preset number of index data points similar to the target data from a data index center. Then, based on these first preset number of index data points, multiple corresponding vector storage engines are determined. A second preset number of data points similar to the target data are retrieved from each vector storage engine. The sum of the distances between the target data and the second preset number of data points in each vector storage engine is calculated. Finally, the target data is stored in the vector storage engine with the smallest sum of distances. This application utilizes a hierarchical similarity storage strategy to store large-scale data in designated vector storage engines, improving the flexibility and efficiency of data storage.

[0031] Please refer to Figure 3 , Figure 3 A flowchart of another data storage method provided in this disclosure embodiment, wherein process 300 includes the following steps: Step 301: In response to receiving the target data, obtain a first preset number of index data that are similar to the target data in the data index center.

[0032] In this embodiment, when the executing entity receives the target data, it retrieves a first preset number of index data points from the data index center that are similar to the target data. The data index center is constructed as follows: the data in the database is clustered to obtain multiple data groups; each data group is stored in a different vector storage engine, and the central data of each data group is stored uniformly as index data in the data index center, facilitating efficient similarity queries later. The central data represents all data in the data group and helps describe the overall characteristics of the entire data group. Depending on actual needs, it can be various values ​​such as the mean, median, mode, and quantiles of all data in the data group.

[0033] Step 302: Determine multiple corresponding vector storage engines based on the first preset number of index data, extract the second preset number of data that are close to the target data from each vector storage engine, and calculate the sum of the distances between the target data and the second preset number of data in each vector storage engine.

[0034] Step 302 and as follows Figure 2 The steps shown in step 202 are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0035] Step 303: Store the target data in the vector storage engine with the smallest sum of distances.

[0036] Step 303 and as follows Figure 2 The steps shown in step 203 are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0037] Step 304: Bind the target data to the vector storage engine that stores the target data, and establish a mapping relationship between the target data and the vector storage engine.

[0038] In this embodiment, the executing entity binds the target data to the vector storage engine that stores the target data, establishing a mapping relationship between the target data and the vector storage engine. Specifically, the executing entity can obtain the data identifier of the target data, bind the data identifier to the vector storage engine that stores the target data, and establish a mapping relationship between the data identifier of the target data and the vector storage engine.

[0039] In this embodiment, the executing entity can also bind the target data to the vector storage engine that stores the target data in the following way to establish a mapping relationship between the target data and the vector storage engine: the target data is tagged, and the target data is bound to the vector storage engine that stores the target data according to the tags, thus establishing a mapping relationship between the target data and the vector storage engine. The executing entity can tag the target data manually, semi-automatically, or fully automatically according to actual needs, and data with the same tags can be bound to the same vector storage engine. This embodiment improves the efficiency of data query by establishing a mapping relationship between the target data and the vector storage engine.

[0040] Step 305: In response to receiving a data query instruction, extract the target data from the vector storage engine according to the mapping relationship.

[0041] In this embodiment, when the executing entity receives a data query instruction, it extracts the target data from the vector storage engine according to the mapping relationship. Specifically, when the executing entity receives a data query instruction, it uses the mapping relationship to query the vector storage engine corresponding to the target data and extracts the target data from the vector storage engine.

[0042] Step 306: Send the target data to the corresponding message channel.

[0043] In this embodiment, the executing entity sends the target data to the corresponding message channel. Here, a message channel refers to a channel used for transmitting data.

[0044] In this embodiment, the executing entity can determine the message channel corresponding to the vector storage engine storing the target data based on the preset mapping relationship between the vector storage engine and the message channel, and send the target data to the message channel. When the received data query instruction contains information about a specified message channel, the executing entity can also directly send the target data to the specified message channel.

[0045] In this embodiment, the difference is... Figure 2In the illustrated embodiment, step 301 corresponds to step 201, step 302 corresponds to step 202, and step 303 corresponds to step 203. Step 304 specifically provides an implementation method for establishing a mapping relationship between target data and the vector storage engine; step 305 specifically provides an implementation method for extracting target data from the vector storage engine; and step 306 specifically provides an implementation method for sending target data to the corresponding message channel. There is no causal or dependency relationship between the three implementation methods, and they do not necessarily need to be applied simultaneously in one embodiment. They can be applied separately in different embodiments as needed for establishing the mapping relationship between target data and the vector storage engine, extracting target data from the vector storage engine, or sending target data to the corresponding message channel. This embodiment exists only as a preferred embodiment that simultaneously includes three specific implementation methods.

[0046] In some optional embodiments of this example, before storing the target data in the vector storage engine with the smallest sum of distances, the executing entity may perform the following steps: sorting the target data from the sum of distances to a second preset number of data points in each vector storage engine in ascending order; calculating the difference between the current data volume and a preset maximum data volume in each vector storage engine according to the sorting order; and determining the first vector storage engine with a difference greater than a preset threshold as the vector storage engine with the smallest sum of distances. In this embodiment, to avoid excessive data volume in a particular vector storage engine leading to excessive storage pressure, when writing new data, the current data volume of the vector storage engine is checked. When the data volume of a vector storage engine exceeds the second preset threshold, the new data needs to be routed to other vector storage engines, thus avoiding overload and ensuring load balancing of the vector storage engines.

[0047] This embodiment discloses a data storage method that, upon receiving target data, utilizes a hierarchical similarity storage strategy to store the target data in the vector storage engine with the smallest sum of distances, and establishes a mapping relationship between the target data and the vector storage engines. Subsequently, the target data can be extracted from the vector storage engines based on the mapping relationship and sent to the corresponding message channel. This application utilizes a hierarchical similarity storage strategy to store large-scale data in corresponding vector storage engines and directly extract data from the corresponding vector storage engines based on the mapping relationship, thereby improving the flexibility of data storage and the efficiency of data retrieval.

[0048] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the example below. Figure 4 The process shown is 400. Figure 4In application scenarios, the executing entity can store the received target data into a suitable vector storage engine, and extract the target data from the corresponding vector storage engine when it receives a data query instruction.

[0049] Step 401: When the execution entity receives the target data, it obtains m (first preset number) index data that are similar to the target data in the data index center.

[0050] Step 402: Determine the corresponding v (v≤m) vector storage engines based on m index data, and extract n (second preset number) data that are close to the target data from each vector storage engine. Calculate the sum of the distances between the target data and the n data in each vector storage engine.

[0051] Step 403: Sort the target data and the n data in each vector storage engine in ascending order based on the sum of the distances between them. For example, the sorted vector storage engines are V1, V2, V3...Vv.

[0052] Step 404: Calculate the difference between the current data volume and the preset maximum data volume in each vector storage engine according to the sorting order.

[0053] Step 405: The vector storage engine whose difference from the preset threshold is the first among the sorted vector storage engines is identified as the vector storage engine with the smallest sum of distances. For example, if the difference between the current data volume and the preset maximum data volume in vector storage engine V1 is less than the preset threshold, then the data volume of this vector storage engine is considered too large and unsuitable for further data storage. The process then continues to check if the difference between the current data volume and the preset maximum data volume in vector storage engine V2 is greater than the preset threshold. If the difference is greater, then the current data volume of vector storage engine V2 is considered sufficient to meet the needs of further data storage, and vector storage engine V2 is identified as the vector storage engine with the smallest sum of distances. Otherwise, the process continues according to the sorting order until a vector storage engine with a difference between its current data volume and the preset maximum data volume greater than the preset threshold is found, and that vector storage engine is identified as the vector storage engine with the smallest sum of distances.

[0054] Step 406: Store the target data in the vector storage engine that minimizes the sum of distances.

[0055] Step 407: Bind the target data to the vector storage engine that stores the target data, and establish a mapping relationship between the target data and the vector storage engine.

[0056] Step 408: When the executing entity receives a data query instruction, it extracts the target data from the vector storage engine according to the mapping relationship.

[0057] Step 409: Send the target data to the corresponding message channel.

[0058] This embodiment discloses a data storage method that, upon receiving target data, utilizes a hierarchical similarity storage strategy to store the target data in the vector storage engine with the smallest sum of distances, and establishes a mapping relationship between the target data and the vector storage engines. Subsequently, the target data can be extracted from the vector storage engines based on the mapping relationship and sent to the corresponding message channel. This application utilizes a hierarchical similarity storage strategy to store large-scale data in corresponding vector storage engines and directly extract data from the corresponding vector storage engines based on the mapping relationship, thereby improving the flexibility of data storage and the efficiency of data retrieval.

[0059] Further reference Figure 5 The diagram shows a timing diagram 500 of one embodiment of a data storage system.

[0060] Step 501: The data sending end sends the target data to the data storage end.

[0061] Step 502: When the target data is received, the data storage terminal obtains the first preset number of index data that are similar to the target data from the data index center.

[0062] Step 503: The data storage end determines multiple corresponding vector storage engines based on the first preset number of index data, retrieves the second preset number of data that are close to the target data from each vector storage engine, and calculates the sum of the distances between the target data and the second preset number of data in each vector storage engine.

[0063] Step 504: The data storage end stores the target data into the vector storage engine with the smallest sum of distances.

[0064] For details not specifically described in this embodiment, please refer to [the relevant documentation]. Figure 2 , Figure 3 The relevant descriptions of the steps in the corresponding embodiments will not be repeated here.

[0065] The data storage system provided in this embodiment discloses that, upon receiving target data, a hierarchical similarity storage strategy is used to store the target data in the vector storage engine with the smallest sum of distances, and a mapping relationship is established between the target data and the vector storage engines. Subsequently, the target data can be extracted from the vector storage engines according to the mapping relationship and sent to the corresponding message channel. This application utilizes a hierarchical similarity storage strategy to store large-scale data in corresponding vector storage engines and directly extract data from the corresponding vector storage engines according to the mapping relationship, thereby improving the flexibility of data storage and the efficiency of data retrieval.

[0066] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a data storage device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0067] like Figure 6 As shown, the data storage device 600 in this embodiment may include: a data acquisition module 601, a distance calculation module 602, and a data storage module 603.

[0068] The data acquisition module 601 is configured to, in response to receiving target data, acquire a first preset number of index data that are similar to the target data in the data index center.

[0069] The distance calculation module 602 is configured to determine multiple corresponding vector storage engines based on a first preset number of index data, extract a second preset number of data that are close to the target data from each vector storage engine, and calculate the sum of the distances between the target data and the second preset number of data in each vector storage engine.

[0070] Data storage module 603 is configured to store target data in a vector storage engine that minimizes the sum of distances.

[0071] In this embodiment, the specific processing of the data acquisition module 601, the distance calculation module 602, and the data storage module 603 in the data storage device 600, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0072] In some optional implementations of this embodiment, the data index center in the data acquisition module 601 is constructed in the following way: the data in the database is clustered to obtain multiple data groups; each data group is stored in a different vector storage engine, and the central data of each data group is uniformly stored in the data index center as index data.

[0073] In some optional implementations of this embodiment, the data storage device 600 further includes a mapping establishment module 604, configured to bind the target data with the vector storage engine storing the target data, and establish a mapping relationship between the target data and the vector storage engine.

[0074] In some optional implementations of this embodiment, the mapping establishment module 604 includes: an identifier acquisition unit configured to acquire the data identifier of the target data; and a first mapping establishment unit configured to bind the data identifier to the vector storage engine storing the target data, thereby establishing a mapping relationship between the data identifier of the target data and the vector storage engine.

[0075] In some optional implementations of this embodiment, the mapping establishment module 604 includes: a data tagging unit configured to tag the target data; and a second mapping establishment unit configured to bind the target data to the vector storage engine storing the target data according to the tags of the target data, thereby establishing a mapping relationship between the target data and the vector storage engine.

[0076] In some optional implementations of this embodiment, the mapping establishment module 604 includes: a data query unit, configured to extract target data from the vector storage engine according to the mapping relationship in response to receiving a data query instruction; and a data sending unit, configured to send the target data to the corresponding message channel.

[0077] In some optional implementations of this embodiment, the data sending unit in the mapping establishment module 604 is further configured to: determine the message channel corresponding to the vector storage engine storing the target data according to the preset mapping relationship between the vector storage engine and the message channel; and send the target data to the message channel.

[0078] In some optional implementations of this embodiment, the data sending unit in the mapping establishment module 604 is further configured to: send the target data to the specified message channel in response to the data query instruction containing information about the specified message channel.

[0079] In some optional implementations of this embodiment, the data storage device 600 further includes: a sorting module 605, configured to sort the target data from smallest to largest based on the sum of the distances between the target data and a second preset number of data in each vector storage engine; a gap calculation module 606, configured to calculate the gap between the current data volume and the preset maximum data volume in each vector storage engine according to the sorting order; and a storage engine determination module 607, configured to determine the first vector storage engine in the sorted vector storage engines whose gap is greater than a preset threshold as the vector storage engine with the smallest sum of distances.

[0080] This embodiment is a device embodiment corresponding to the method embodiment described above. The data storage device provided in this embodiment discloses that, upon receiving target data, it utilizes a hierarchical similarity storage strategy to store the target data in the vector storage engine with the smallest sum of distances, and establishes a mapping relationship between the target data and the vector storage engines. Subsequently, the target data can be extracted from the vector storage engines according to the mapping relationship and sent to the corresponding message channel. This application utilizes a hierarchical similarity storage strategy to store large-scale data in corresponding vector storage engines and directly extract data from the corresponding vector storage engines according to the mapping relationship, improving the flexibility of data storage and the efficiency of data retrieval.

[0081] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the data storage method described in any of the above embodiments when executed.

[0082] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the data storage method described in any of the above embodiments when executed.

[0083] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the data storage method described in any of the above embodiments.

[0084] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0085] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0086] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as data storage methods. For example, in some embodiments, the data storage method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the data storage method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform data storage methods by any other suitable means (e.g., by means of firmware).

[0088] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0092] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0093] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0094] According to the technical solution of this disclosure, upon receiving target data, a hierarchical similarity storage strategy is used to store the target data in the vector storage engine with the smallest sum of distances, and a mapping relationship is established between the target data and the vector storage engines. Subsequently, the target data can be extracted from the vector storage engines according to the mapping relationship and sent to the corresponding message channel. This application utilizes a hierarchical similarity storage strategy to store large-scale data in corresponding vector storage engines and extract data directly from the corresponding vector storage engines according to the mapping relationship, thereby improving the flexibility of data storage and the efficiency of data retrieval.

[0095] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data storage method, comprising: In response to receiving target data, a first preset number of index data that are similar to the target data are obtained from the data index center; Based on the first preset number of index data, determine the corresponding multiple vector storage engines, extract the second preset number of data that are close to the target data from each of the vector storage engines, and calculate the sum of the distances between the target data and the second preset number of data in each of the vector storage engines; The target data is stored in the vector storage engine with the smallest sum of distances.

2. The method according to claim 1, wherein, The data index center is constructed in the following way: Cluster the data in the database to obtain multiple data groups; Each data group is stored in a different vector storage engine, and the central data of each data group is stored as index data in the data index center.

3. The method according to claim 1, further comprising: The target data is bound to the vector storage engine that stores the target data, thus establishing a mapping relationship between the target data and the vector storage engine.

4. The method according to claim 3, wherein, The step of binding the target data with the vector storage engine that stores the target data, and establishing a mapping relationship between the target data and the vector storage engine, includes: Obtain the data identifier of the target data; The data identifier is bound to the vector storage engine that stores the target data, thereby establishing a mapping relationship between the data identifier of the target data and the vector storage engine.

5. The method according to claim 3, wherein, The step of binding the target data with the vector storage engine that stores the target data, and establishing a mapping relationship between the target data and the vector storage engine, includes: The target data is then labeled; The target data is bound to the vector storage engine that stores the target data based on the tag of the target data, thus establishing a mapping relationship between the target data and the vector storage engine.

6. The method according to claim 3, further comprising: In response to receiving a data query instruction, the target data is extracted from the vector storage engine according to the mapping relationship; The target data is sent to the corresponding message channel.

7. The method according to claim 6, wherein, The step of sending the target data to the corresponding message channel includes: The message channel corresponding to the vector storage engine storing the target data is determined based on the preset mapping relationship between the vector storage engine and the message channel; The target data is sent to the message channel.

8. The method according to claim 6, wherein, Sending the target data to the corresponding message channel includes: In response to the data query instruction containing information about a specified message channel, the target data is sent to the specified message channel.

9. The method according to claims 1-8, wherein, Before storing the target data in the vector storage engine with the minimum sum of distances, the process also includes: The target data is sorted from smallest to largest based on the sum of the distances between the target data and the second preset number of data in each of the vector storage engines; Calculate the difference between the current data volume and the preset maximum data volume in each of the vector storage engines according to the sorting order; The vector storage engine with the largest difference among the sorted vector storage engines is determined as the vector storage engine with the smallest sum of distances.

10. A data storage system, comprising a data sending end, a data storage end, and a data indexing center; The data sending end is used to send target data to the data storage end; The data storage terminal is used to store the received target data into a vector storage engine, including: in response to receiving target data, obtaining a first preset number of index data that are close to the target data in the data index center; determining a plurality of corresponding vector storage engines based on the first preset number of index data, retrieving a second preset number of data that are close to the target data from each of the vector storage engines, calculating the sum of the distances between the target data and the second preset number of data in each of the vector storage engines; and storing the target data into the vector storage engine with the smallest sum of distances. The data index center is used to store the index data of each of the vector storage engines.

11. A data storage device, comprising: The data acquisition module is configured to, in response to receiving target data, acquire a first preset number of index data that are similar to the target data in the data index center; The distance calculation module is configured to determine multiple corresponding vector storage engines based on the first preset number of index data, extract a second preset number of data that are close to the target data from each of the vector storage engines, and calculate the sum of the distances between the target data and the second preset number of data in each of the vector storage engines. The data storage module is configured to store the target data in a vector storage engine that minimizes the sum of distances.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data storage method according to any one of claims 1-9.

13. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data storage method according to any one of claims 1-9.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the data storage method according to any one of claims 1-9.

Images