Vector data preheating method and system
By loading vector data indexes into NAS devices and constructing preheating requests, and utilizing the S3 protocol and TLS encryption, efficient migration of vector data is achieved, solving the high storage cost problem in the field of intelligent driving and improving data loading speed and storage efficiency.
Patent Information
- Application Number
- CN202510870211.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-09-26
AI Technical Summary
In existing technologies, the method of storing full data using vector databases and local disk storage/network attached storage (NAS) cannot meet the flexible and efficient requirements of model training in the field of intelligent driving due to storage media capacity and cost limitations.
By obtaining the index set of vector data, loading the corresponding data into the NAS device and generating a list of indexes to be preheated, a preheat request is constructed and sent to the server to migrate the vector data to the NAS device. The S3 protocol and TLS encryption are used for data interaction and authentication to achieve efficient data preheating.
It greatly reduces the storage scale and cost of data required for training, improves data loading speed, and solves the problem of large-scale and high-cost storage of vector data.
Smart Images
Figure CN120705118A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a vector data preheating method and system. Background Art
[0002] In the field of intelligent driving, with the growth of intelligent driving data and the increase in the amount of data required for single model training, the related technologies based on vector databases and local disk storage / network attached storage (NAS) to store full data have gradually failed to meet training needs due to the capacity and cost limitations of storage media; the storage, indexing and loading methods of training data are also unable to achieve the requirements of flexible and efficient model training. Summary of the Invention
[0003] The present application provides a vector data preheating method and system, which can solve the technical problems of large vector data storage scale and high cost in the prior art.
[0004] In a first aspect, the present application provides a vector data preheating method, the method comprising: Obtain and traverse the index set of vector data, find the vector data corresponding to the index in the NAS device and load it into the memory, and generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set; A preheating request is constructed using the preheating index list and sent to the server, so that the server migrates the corresponding vector data to the NAS device according to the preheating request.
[0005] In conjunction with the first aspect, in one embodiment, before obtaining and traversing the index of the index set of vector data, the method further includes: Receive the user's authentication parameters and construct the S3 protocol handle and NAS storage handle based on the above authentication parameters; The NAS storage handle is used to interact with the NAS device; the S3 access handle is used to interact with the S3 object storage system.
[0006] In conjunction with the first aspect, in one embodiment, after finding the vector data corresponding to the index in the NAS device and loading it into the memory, the method further includes: The unfound vector data is loaded from the S3 object storage system into memory through the S3 access handle.
[0007] In conjunction with the first aspect, in one embodiment, constructing the S3 protocol handle specifically includes: Initialize the underlying S3 client based on the above authentication parameters and enable TLS encryption; The underlying S3 client dynamically generates a signature for each request based on AWS Signature v4 and appends it to the request header to create a callable S3 protocol handle.
[0008] In conjunction with the first aspect, in one embodiment, the authentication parameter includes a NAS mount path; Building a NAS storage handle specifically includes: If the above NAS mount path does not exist, create it. Get the user-defined preheating configuration parameters. If the preheating configuration parameters are True, start the full preheating service and create a NAS access handle with read and write functions based on the above NAS mount path.
[0009] In conjunction with the first aspect, in one embodiment, the server migrates the corresponding vector data to the NAS device according to the preheating request, specifically including: After receiving the warm-up request, it builds an S3 read handle and a NAS write handle. The vector data corresponding to the warm-up request is read from the S3 object storage system through the S3 read handle, and is written to the NAS device through the NAS write handle.
[0010] In conjunction with the first aspect, in one embodiment, after reading the vector data corresponding to the warm-up request from the S3 object storage system through the S3 read handle, the method further includes: The metadata header of the above index set is appended to generate a data verification file and written into the preset directory structure of the NAS device.
[0011] In conjunction with the first aspect, in one embodiment, when constructing a preheating request using the preheating index list, the method further includes: Get the task tracking ID of the warm-up request above to warm-up tracking; Based on the above task tracking ID, start the scheduled task status poller.
[0012] In conjunction with the first aspect, in one embodiment, after starting the scheduled task status poller, the method further includes: When the scheduled task status poller detects that the task status is SUCCESS, a data integrity check is performed on the vector data migrated to the NAS device.
[0013] In a second aspect, the present application provides a vector data preheating system, the system comprising: The client is used to obtain and traverse the index set of vector data, find the vector data corresponding to the index in the NAS device and load it into the memory, generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set; and construct a preheat request based on the preheat index list and send it to the server; The server is used to migrate the corresponding vector data to the above-mentioned NAS device according to the above-mentioned preheating request.
[0014] The beneficial effects of the technical solution provided by this application include: By obtaining and traversing the index set of vector data, finding the vector data corresponding to the index in the NAS device and loading it into the memory, generating a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set, and then constructing a preheating request with the above preheating index list and sending it to the server, so that the server can migrate the corresponding vector data to the above NAS device according to the above preheating request, greatly reducing the storage scale and cost of the data required for training, improving the speed of data loading before training, and solving the technical problems of large vector data storage scale and high cost existing in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a flow chart of an embodiment of the vector data preheating method of the present application; Figure 2 This is a detailed flow chart of step A1 in the embodiment of the present application; Figure 3 This is a detailed flow chart of step A2 in the embodiment of the present application; Figure 4 This is a detailed flow chart of step A3 in the embodiment of the present application; Figure 5 This is a schematic diagram of the architecture of an embodiment of the vector data preheating system of the present application. DETAILED DESCRIPTION
[0016] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0017] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0018] In a first aspect, an embodiment of the present application provides a vector data preheating method.
[0019] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the vector data preheating method of the present application. The vector data preheating method includes: S1. Obtain and traverse the index set of vector data, find the vector data corresponding to the index in the NAS device and load it into memory. Generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set. S2. Construct a preheating request using the preheating index list and send it to the server, so that the server migrates the corresponding vector data to the NAS device according to the preheating request.
[0020] In this embodiment, by obtaining and traversing the index set of vector data, the vector data corresponding to the index is found in the NAS device and loaded into the memory, and the index for which the corresponding vector data is not found in the index set is used to generate a list of indexes to be preheated. Then, a preheating request is constructed with the above preheating index list and sent to the server, so that the server migrates the corresponding vector data to the above NAS device according to the above preheating request, thereby greatly reducing the storage scale and cost of the data required for training, improving the data loading speed before training, and solving the technical problems of large vector data storage scale and high cost existing in related technologies.
[0021] Furthermore, in one embodiment, before obtaining and traversing the index of the index set of the vector data in the above step S1, the following steps are further included: Receive the user's authentication parameters and construct the S3 protocol handle and NAS storage handle based on the above authentication parameters.
[0022] The NAS storage handle is used to interact with the NAS device; the S3 access handle is used to interact with the S3 object storage system.
[0023] Furthermore, in this embodiment, after finding the vector data corresponding to the index in the NAS device and loading it into the memory, the following steps are further included: The unfound vector data is loaded from the S3 object storage system into memory through the S3 access handle.
[0024] In this embodiment, the vector data corresponding to the index not found in the NAS device is loaded from the above-mentioned S3 object storage system to the memory through the S3 access handle, so that the vector data that is not hit in the cache is loaded into the memory, realizing real-time access and synchronous preheating of data, and further improving the preheating efficiency.
[0025] Optionally, the S3 access handle is used to calculate the offset range of the vector data on the S3 object storage system based on the vector ID or row number of the required vector data, and the corresponding data block is pulled from the S3 object storage system in a streaming manner and returned to the training job to avoid pulling all the vector data at once.
[0026] Furthermore, in one embodiment, the authentication parameters include an access password, a security password, and a designated area.
[0027] In this embodiment, the above-mentioned construction of the S3 protocol handle specifically includes: Initialize the underlying S3 client based on the above authentication parameters and enable TLS encryption; The underlying S3 client dynamically generates a signature for each request based on AWS Signature v4 and appends it to the request header to create a callable S3 protocol handle.
[0028] Specifically, building an S3 protocol handle includes the following steps: 1. Load and verify credentials and configuration information: Obtain authentication parameters such as the Access Key (AK), Secret Key (SK), specified region, or custom service endpoint from configuration items or environment variables; 2. Initialize the underlying S3 client based on the above authentication parameters and enable TLS encryption; 3. Automatic application signing mechanism: The underlying S3 client dynamically calculates a signature for each HTTP request initiated according to the AWS Signature v4 (or other signature versions) specification and appends the signature information to the request header to complete the identity authentication and permission verification of the request; 4. Build a high-level encapsulation handle dedicated to Lance: Based on the initialized underlying S3 client, it is further encapsulated into an S3 protocol handle that Lance can directly use. This handle has the following features: multi-segment download function, initiating segment requests for large objects, and merging segments in parallel or on demand to improve download performance, multipart upload of large objects, and object metadata query: 5. Returns a callable Lance-specific high-level wrapper handle.
[0029] Furthermore, in this embodiment, the above authentication parameters include a NAS mounting path.
[0030] In this embodiment, the above-mentioned construction of the NAS storage handle specifically includes: First, determine whether it exists. If so, proceed directly to the next step. Otherwise, that is, when the above NAS mount path does not exist, create the above NAS mount path and then proceed to the next step.
[0031] Then, the user-defined preheating configuration parameters are obtained. If the preheating configuration parameters are True, the full preheating service is started, and a NAS access handle with read and write support is created based on the above NAS mount path.
[0032] Optionally, the preheat configuration parameter is a full copy parameter full_preheat.
[0033] Based on the above embodiment, in this embodiment, the server migrates the corresponding vector data to the NAS device according to the preheating request, specifically including: First, after receiving the warm-up request, an S3 read handle and a NAS write handle are constructed.
[0034] Then, the vector data corresponding to the preheating request is read from the S3 object storage system through the S3 read handle, and the vector data corresponding to the preheating request is written to the NAS device through the NAS write handle.
[0035] Furthermore, in one embodiment, after reading the vector data corresponding to the warm-up request from the S3 object storage system through the S3 read handle, the method further includes: The metadata header of the above index set is appended to generate a data verification file and written into the preset directory structure of the NAS device.
[0036] Furthermore, in one embodiment, when constructing a preheating request using the preheating index list, the process further includes: First, obtain the task tracking ID of the warm-up request above to warm-up tracking; Then, based on the above task tracking ID, start the scheduled task status poller.
[0037] In this embodiment, after starting the scheduled task status poller, the following steps are also included: When the scheduled task status poller detects that the task status is SUCCESS, a data integrity check is performed on the vector data migrated to the NAS device.
[0038] Optionally, the above authentication parameters also include the S3 access path. After receiving the user's authentication parameters, it also includes: Determine whether the user has read permission for the S3 access path. If so, receive the user's authentication parameters and perform subsequent operations. If not, terminate and report an error.
[0039] Specifically, if Figure 2-Figure 4 As shown, the above-mentioned vector data preheating method specifically includes: A1. Client initialization and handle creation: A11. The client receives authentication parameters entered by the user, including the access key (AK), secret key (SK), endpoint, region, NAS mount path, and S3 access path. A12. Determine whether you have read permission for the S3 access path. If so, proceed to A13. Otherwise, terminate.
[0040] A13. Determine whether the prefix of the S3 access path contains S3. If so, proceed to A14 and A15. Otherwise, proceed to A15.
[0041] A14. Build the S3 protocol handle handle-s3.
[0042] A15. Create the NAS storage handle handle-nas.
[0043] The S3 protocol handle handle-s3 is constructed, including: (1) Using AK / SK for AWS Signature V4 signature authentication; (2) Establish a TLS encrypted connection to the S3 bucket in the specified region; (3) Create an S3 access handle with the following capabilities: - Multipart Download; - Object metadata query; - Data checksum verification.
[0044] The NAS storage handle handle-nas is constructed, which includes: (1) NAS mount path validity check, that is, determine whether the NAS mount path exists. If it exists, go to (2). Otherwise, create the NAS mount path and go to (2). (2) Judge the full copy parameter full_preheat. If full_preheat is True, start the full preheat service and enter (3) after the preheat is successful; if full_preheat is False, enter (3); (3) Create a NAS access handle with read and write support based on the NAS mount path.
[0045] A2. Cache Verification and Task Triggering: A21. The client receives and traverses the index collection rowIds s3 =[i1,i2,…,i n ]; A22. Determine whether vector data corresponding to the index in the index set exists on the NAS device. Consider the vector data corresponding to the index that exists on the NAS device as existing data, and consider the vector data corresponding to the index that does not exist on the NAS device as missing data. If the data exists, proceed to A23; if the data is missing, proceed to A24.
[0046] A23. Directly load existing data into memory; the aforementioned existing data is data from existing verification files.
[0047] A24. The client reads missing data from the S3 object storage system based on the S3 protocol handle and generates a list of indexes to be preheated (rowIds) for the indexes corresponding to the missing data. preheat ; A25. Use the list of indexes to be preheated to construct a preheat request containing a timestamp, task priority, and estimated data size. A26. Asynchronously call the preheat service API to trigger the preheat service and obtain the task trace ID for the preheat request. A27. Start the scheduled task status poller (configurable interval, default 30 seconds).
[0048] A3. Preheating task execution: A31. After receiving the task request, the server writes it to the Kafka message queue and records the receipt timestamp. A32. The task scheduler sorts the queues based on QoS policies (including priority, data volume, and wait time). A33. When a task is scheduled, start the task execution thread and establish an S3 read handle and a NAS write handle. A34. Perform data migration according to the order of the indexes in the preheated index list requested by the task: a. Read the specified vector data block consisting of the vector data corresponding to the to-be-warmed index list from the S3 object storage system through the S3 read handle; b. Add rowIds to the specified vector data block s3 The metadata header; c. Write the specified vector data block with the metadata header to the preset directory structure of the NAS device; generate a verification file based on the vector data size and trigger time and write it to the preset directory of the NAS device for the client to perform data integrity verification; d. Update task progress to the status database; A35. After the task is completed, the completion flag file is written and the task status is updated to SUCCESS.
[0049] A4. Data Verification: When the client detects that the task status is SUCCESS, it performs data integrity verification (including but not limited to: file size verification, checksum comparison, and data format detection).
[0050] The vector data preheating method of this embodiment is based on the S3 protocol. By receiving loading data parameters, handle initialization, local cache hit, requesting remote preheating service, rotation training preheating status, data verification and other processes, it realizes real-time access and synchronization of preheating data, greatly reducing the storage scale and cost of the data required for training, and improving the efficiency of data preparation before algorithm training.
[0051] In a second aspect, an embodiment of the present application also provides a vector data preheating system.
[0052] In one embodiment, referring to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the vector data preheating system of the present application. The above-mentioned vector data preheating system includes a client and a server.
[0053] The above-mentioned client is used to obtain and traverse the index set of vector data, find the vector data corresponding to the index in the NAS device and load it into the memory, and generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set; the client is also used to construct a preheating request with the above-mentioned preheating index list and send it to the server.
[0054] The server is used to migrate the corresponding vector data to the NAS device according to the preheating request.
[0055] Preferably, the above-mentioned vector data preheating system also includes a NAS device and an S3 object storage system.
[0056] The S3 object storage system is a distributed storage system for storing raw vector data. The NAS device is a high-speed storage medium mounted locally and used to cache preheated vector data.
[0057] In this embodiment, the client has dual-path processing capabilities, supports creating data handles through local file paths or S3 protocol paths, and implements memory loading functions for vector data.
[0058] The above-mentioned server is a memory-level preheating server, deployed on a remote server, responsible for receiving and processing preheating requests and coordinating the data migration process.
[0059] Furthermore, in one embodiment, before the client obtains and traverses the index of the index set of vector data, it is further configured to: Receive the user's authentication parameters and construct the S3 protocol handle and NAS storage handle based on the above authentication parameters; The NAS storage handle is used to interact with the NAS device; the S3 access handle is used to interact with the S3 object storage system.
[0060] Furthermore, in one embodiment, after the client finds the vector data corresponding to the index in the NAS device and loads it into the memory, it is further configured to: The unfound vector data is loaded from the S3 object storage system into memory through the S3 access handle.
[0061] Furthermore, in one embodiment, the client is further configured to: Initialize the underlying S3 client based on the above authentication parameters and enable TLS encryption; The underlying S3 client dynamically generates a signature for each request based on AWS Signature v4 and appends it to the request header to create a callable S3 protocol handle.
[0062] The above include access keys (AK), secret keys (SK), specified regions (Regions) or custom service endpoints, etc.
[0063] Furthermore, in one embodiment, the authentication parameters further include a NAS mount path; The above client is also used to: If the above NAS mount path does not exist, create it. Get the user-defined preheating configuration parameters. If the preheating configuration parameters are True, start the full preheating service and create a NAS access handle with read and write functions based on the above NAS mount path.
[0064] Furthermore, in one embodiment, the server is used to: After receiving the warm-up request, it builds an S3 read handle and a NAS write handle. The vector data corresponding to the warm-up request is read from the S3 object storage system through the S3 read handle, and is written to the NAS device through the NAS write handle.
[0065] Furthermore, in one embodiment, the server is further configured to: The metadata header of the above index set is appended to generate a data verification file and written into the preset directory structure of the NAS device.
[0066] Furthermore, in one embodiment, the client is further configured to: When constructing a preheating request with the preheating index list, obtain the task tracking ID of the preheating request for preheating tracking; Based on the above task tracking ID, start the scheduled task status poller.
[0067] Furthermore, in one embodiment, the client is further configured to: When the scheduled task status poller detects that the task status is SUCCESS, a data integrity check is performed on the vector data migrated to the NAS device.
[0068] Among them, the functional implementation of each module in the above-mentioned vector data preheating system corresponds to the various steps in the above-mentioned vector data preheating method embodiment, and its functions and implementation processes will not be repeated here one by one.
[0069] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0070] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.
[0071] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.
[0072] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.
[0073] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.
[0074] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.
[0075] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A vector data preheating method, characterized in that: The method comprises: Obtain and traverse the index set of vector data, find the vector data corresponding to the index in the NAS device and load it into the memory, and generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set; A preheating request is constructed using the preheating index list and sent to the server, so that the server migrates the corresponding vector data to the NAS device according to the preheating request.
2. The vector data preheating method according to claim 1, characterized in that: Before getting and traversing the index of the index collection of vector data, it also includes: Receive user authentication parameters and construct an S3 protocol handle and NAS storage handle based on the authentication parameters; The NAS storage handle is used to interact with the NAS device; the S3 access handle is used to interact with the S3 object storage system.
3. The vector data preheating method according to claim 2, characterized in that: After finding the vector data corresponding to the index in the NAS device and loading it into memory, the following steps are also performed: The unfound vector data is loaded from the S3 object storage system into the memory through the S3 access handle.
4. The vector data preheating method according to claim 2, wherein: The construction of the S3 protocol handle specifically includes: Initialize the underlying S3 client based on the authentication parameters and enable TLS encryption; The underlying S3 client dynamically generates a signature for each request based on AWS Signature v4 and appends it to the request header to create a callable S3 protocol handle.
5. The vector data preheating method according to claim 2, wherein: The authentication parameters include the NAS mount path; Building a NAS storage handle specifically includes: When the NAS mount path does not exist, create the NAS mount path; Get the user-defined preheating configuration parameter. If the preheating configuration parameter is True, start the full preheating service and create a NAS access handle with read and write functions based on the NAS mount path.
6. The vector data preheating method according to claim 1, characterized in that: The server migrates the corresponding vector data to the NAS device according to the preheating request, specifically including: After receiving the warm-up request, construct an S3 read handle and a NAS write handle; The vector data corresponding to the warm-up request is read from the S3 object storage system through the S3 read handle, and is written to the NAS device through the NAS write handle.
7. The vector data preheating method according to claim 6, characterized in that: After reading the vector data corresponding to the warm-up request from the S3 object storage system through the S3 read handle, the method further includes: The metadata header of the index set is attached to generate a data verification file, and the file is written into the preset directory structure of the NAS device.
8. The vector data preheating method according to claim 1, wherein: When constructing a preheating request using the preheating index list, the method further includes: Obtain the task tracking ID of the preheating request for preheating tracking; Based on the task tracking ID, a scheduled task status poller is started.
9. The vector data preheating method according to claim 8, characterized in that: After starting the scheduled task status poller, it also includes: When the scheduled task status poller detects that the task status is SUCCESS, a data integrity check is performed on the vector data migrated to the NAS device.
10. A vector data preheating system, characterized in that: The system comprises: The client is configured to obtain and traverse an index set of vector data, find the vector data corresponding to the index in the NAS device and load it into memory, generate a list of indexes to be preheated for the indexes for which no corresponding vector data is found in the index set; and construct a preheat request using the preheat index list and send it to the server; The server is configured to migrate the corresponding vector data to the NAS device according to the preheating request.