Data processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a medium. The method comprises the steps of obtaining current time, to-be-processed vector data and a persistent data table corresponding to the to-be-processed vector data; determining a vector data processing task according to the to-be-processed vector data under the condition that the current time meets a preset processing time rule; issuing the vector data processing task to a preset asynchronous task processing queue to enable the asynchronous task processing queue to complete the vector data processing task; under the condition that the preset asynchronous task processing queue completes the vector data processing task, updating a persistent data table corresponding to the to-be-processed vector data; the process ensures the dynamic migration and updating of the data among a plurality of instances, and effectively realizes the rebalance of the data. Through efficient processing of the asynchronous task processing queue, the data can flow in time and orderly among different instances, and the phenomenon of data skew is avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] In the application scenarios of vector databases, with the rapid growth of data volume and the continuous changes in business needs, horizontal expansion capability has become a key indicator for measuring database performance. In order to cope with large-scale data processing needs, vector databases usually adopt a multi-instance storage strategy to expand storage and computing capabilities by adding server instances.

[0003] However, in a multi-instance storage environment, a significant technical challenge gradually emerges: data skew. Due to the dynamic changes and uneven distribution of data, the data storage and processing tasks undertaken by different instances often differ significantly. This data skew phenomenon not only causes some instances to be overloaded and have low processing efficiency, but may also cause performance bottlenecks, seriously affecting the stability and availability of the entire vector database.

[0004] The existence of data skew makes it difficult for vector databases to fully utilize the advantages of multiple instances during horizontal expansion. Even if new instances are added, the overall performance may not be effectively improved due to uneven data distribution. Therefore, how to effectively solve the data skew problem and achieve load balancing among multiple instances has become the key to improving the horizontal expansion capabilities of vector databases.

[0005] In summary, load imbalance caused by data skew is the main technical problem that needs to be solved in the current vector database in a multi-instance storage environment. Solving this problem will help optimize data distribution and improve resource utilization, thereby significantly improving the performance and scalability of the vector database. Summary of the invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a data processing method, apparatus, device and medium that overcome the above problems or at least partially solve the above problems.

[0007] In order to solve the above problem, an embodiment of the present invention discloses a data processing method, which includes:

[0008] Obtaining the current time, the vector data to be processed, and the persistent data table corresponding to the vector data to be processed;

[0009] Under the condition that the current time satisfies a preset processing time rule, determining a vector data processing task according to the vector data to be processed;

[0010] Sending the vector data processing task to a preset asynchronous task processing queue so that the asynchronous task processing queue completes the vector data processing task;

[0011] Under the condition that the preset asynchronous task processing queue completes the vector data processing task, the persistent data table corresponding to the vector data to be processed is updated.

[0012] Optionally, under the condition that the current time satisfies a preset processing time rule, determining the vector data processing task according to the vector data to be processed includes:

[0013] determining a candidate vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0014] In the case that the candidate vector data processing task satisfies the preset data task rule, a vector data processing task is determined according to the candidate vector data processing task.

[0015] Optionally, the method further comprises:

[0016] When the candidate vector data processing task does not satisfy the preset data task rule, a reminder event is output to the user so that the user can manually set the vector data processing task.

[0017] Optionally, the vector data to be processed includes multiple data components, each of the multiple data components includes multiple vector data, and the method further includes:

[0018] Obtaining the amount of the vector data in each of the multiple data components;

[0019] The step of determining the vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule includes:

[0020] Under the condition that the current time satisfies a preset processing time rule, a vector data processing task is determined according to the amount of the vector data in each of the multiple data components.

[0021] Optionally, under the condition that the current time satisfies a preset processing time rule, determining the vector data processing task according to the amount of the vector data in each of the multiple data components includes:

[0022] Under the condition that the current time satisfies a preset processing time rule, determining the quantity value of the reference vector data of the data component according to the quantity of the vector data in each of the multiple data components and a preset data processing rule;

[0023] A vector data processing task is determined according to the quantity of the vector data in each data component and the quantity value of the reference vector data.

[0024] Optionally, determining the vector data processing task according to the quantity of the vector data in each data component and the quantity value of the reference vector data includes:

[0025] Determine a reference processed data value according to a difference between the quantity of the vector data in each data component and the quantity value of the reference vector data;

[0026] A vector data processing task is determined according to the reference processing data value.

[0027] Optionally, determining the vector data processing task according to the reference processed data value includes:

[0028] Determining the type of the vector data processing task according to the positive or negative value of the reference processing data value;

[0029] determining the number of vector data processing tasks according to the absolute value of the reference processed data value;

[0030] The vector data processing tasks are determined according to the types of the vector data processing tasks and the number of the vector data processing tasks.

[0031] On the other hand, an embodiment of the present invention discloses a data processing device, the device comprising:

[0032] A data acquisition module, used to acquire the current time, the vector data to be processed and a persistent data table corresponding to the vector data to be processed;

[0033] A processing task acquisition module, used to determine a vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0034] A task processing module, used for sending the vector data processing task to a preset asynchronous task processing queue, so that the asynchronous task processing queue completes the vector data processing task;

[0035] The data updating module is used to update the persistent data table corresponding to the vector data to be processed under the condition that the preset asynchronous task processing queue completes the vector data processing task.

[0036] Optionally, the processing task acquisition module includes:

[0037] A first processing task determination submodule is used to determine a candidate vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0038] The second processing task determination submodule is used to determine a vector data processing task according to the vector data processing candidate task when the vector data processing candidate task meets the preset data task rule.

[0039] Optionally, the device further comprises:

[0040] The manual processing submodule is used to output a reminder event to the user when the vector data processing candidate task does not meet the preset data task rules, so that the user can manually set the vector data processing task.

[0041] Optionally, the vector data to be processed includes multiple data components, each of the multiple data components includes multiple vector data, and the device further includes:

[0042] A data quantity determination submodule, used for obtaining the quantity of the vector data in each of the multiple data components;

[0043] The processing task acquisition module includes:

[0044] The third processing task determination submodule is used to determine the vector data processing task according to the amount of the vector data in each of the multiple data components, under the condition that the current time satisfies the preset processing time rule.

[0045] Optionally, the third processing task determination submodule includes:

[0046] A first data determination unit, configured to determine the quantity value of the reference vector data of the data component according to the quantity of the vector data in each of the multiple data components and a preset data processing rule, under the condition that the current time satisfies a preset processing time rule;

[0047] The fourth processing task determination unit is used to determine the vector data processing task according to the quantity of the vector data in each data component and the quantity value of the reference vector data.

[0048] Optionally, the fourth processing task determining unit includes:

[0049] A reference processed value acquisition subunit, used for determining a reference processed data value according to a difference between the quantity of the vector data in each data component and the quantity value of the reference vector data;

[0050] The fifth processing task subunit is used to determine the vector data processing task according to the reference processing data value.

[0051] Optionally, the fifth processing task subunit includes:

[0052] A task type determination component, used to determine the type of the vector data processing task according to the positive or negative value of the reference processing data value;

[0053] A task value determination component, used to determine the number of vector data processing tasks according to the absolute value of the reference processing data value;

[0054] The sixth processing task component is used to determine the vector data processing task according to the type of the vector data processing task and the number of the vector data processing tasks.

[0055] Accordingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned data processing method embodiment when executed by the processor.

[0056] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned data processing method embodiment is implemented.

[0057] The embodiment of the present invention includes the following advantages: first, the current time, the vector data to be processed and the persistent data table corresponding to the vector data to be processed are obtained. Under the condition that the current time meets the preset processing time rule, the vector data processing task is determined according to the vector data to be processed. These tasks are then sent to the preset asynchronous task processing queue, which is responsible for completing the data processing tasks. Once the asynchronous task processing queue completes these tasks, the persistent data table corresponding to the vector data to be processed will be updated. This process ensures the dynamic migration and update of data between multiple instances, and effectively realizes the rebalancing of data. Through the efficient processing of the asynchronous task processing queue, data can flow between different instances in a timely and orderly manner, avoiding the occurrence of data skew. This dynamic rebalancing of data not only improves the data processing capability of each instance, but also enables the entire vector database to more flexibly expand storage and computing capabilities when facing large-scale data processing needs.

[0058] Therefore, the embodiments of the present invention significantly improve the horizontal expansion capability of the vector database, enabling it to better cope with the growth of data volume and changes in business needs. At the same time, the dynamic rebalancing of data also brings about an improvement in overall performance, improves the processing efficiency and response speed of the database, and provides users with a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of steps of an embodiment of a data processing method of the present invention;

[0060] Figure 2 is a flow chart of an embodiment of a data processing method of the present invention;

[0061] Figure 3 is a timing diagram of an embodiment of a data processing method of the present invention;

[0062] Figure 4 It is a data migration schematic diagram of an embodiment of a data processing method of the present invention;

[0063] Figure 5 It is a structural block diagram of an embodiment of a data processing device of the present invention. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Data skew is a common problem in big data and machine learning. It refers to the phenomenon that in the data processing process, the data distribution is uneven, resulting in some processing units (such as computing nodes, tasks, etc.) needing to process significantly more data than other processing units. This unevenness often leads to a decline in system performance, a waste of resources, and may cause computing bottlenecks.

[0066] Asynchronous processing queues are a programming pattern where tasks are not executed immediately, but are put into a queue to be processed. Other threads or processes will take the tasks from the queue and execute them. By putting the tasks in the queue, the main thread can continue to execute other tasks without having to wait for a time-consuming operation to complete. This improves the performance and responsiveness of the application.

[0067] One of the core concepts of the embodiments of the present invention is to solve the data skew problem by designing data rebalancing rules, combining scheduled tasks and asynchronous processing queues.

[0068] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, which may specifically include the following steps:

[0069] Step 101, obtaining the current time, the vector data to be processed and the persistent data table corresponding to the vector data to be processed;

[0070] The system first obtains the current system time, which is used to determine whether data processing tasks need to be performed. Then, the system obtains the vector data to be processed, which may be unevenly distributed, that is, there is a data skew problem. At the same time, the system also obtains the persistent data table corresponding to the vector data to be processed, so as to update the data status in the table after processing the data.

[0071] In one example, each vector data to be processed is uniquely coded for storage in a persistent database table.

[0072] Step 102, under the condition that the current time satisfies a preset processing time rule, determining a vector data processing task according to the vector data to be processed;

[0073] When the current time meets the preset processing time rules, for example, every morning or at regular intervals, the system will preliminarily determine a batch of data processing candidate tasks based on the vector data to be processed.

[0074] In one example, data processing is performed once at 0:00 every day, and the vector data to be processed is processed according to preset rules; the vector data to be processed is stored in a corresponding instance, that is, a running copy or node of the database.

[0075] In one embodiment, step 102 includes the following sub-steps:

[0076] Sub-step S11, determining a candidate vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0077] Sub-step S12, when the vector data processing candidate task meets the preset data task rule, the vector data processing task is determined according to the vector data processing candidate task. The operation under special circumstances is further explained, making data rebalancing more flexible, supporting expansion, and adapting to different environments.

[0078] In one embodiment, the method further comprises:

[0079] When the candidate vector data processing task does not satisfy the preset data task rule, a reminder event is output to the user so that the user can manually set the vector data processing task.

[0080] If the candidate task does not meet the preset data task rules, the system can output a reminder event to the user, prompting the user to manually set or adjust the data processing task to deal with special situations or data skew problems.

[0081] In one embodiment, the vector data to be processed includes multiple data components, each of the multiple data components includes multiple vector data, and the method further includes

[0082] Obtaining the amount of the vector data in each of the multiple data components;

[0083] Step 102 also includes the following sub-steps:

[0084] Sub-step S21, under the condition that the current time satisfies a preset processing time rule, determines a vector data processing task according to the amount of the vector data in each of the multiple data components.

[0085] For example, the preset rule may be to calculate based on the actual data volume of the vector database and rebalance based on the calculation result;

[0086] For example, rebalancing is performed by calculating the average, that is, the amount of data for each instance = (instance 1 + instance 2 + ... + instance N) / N;

[0087] In one example, the name of each vector data follows a specific rule, for example, the vector data ID consists of 19 digits, which are divided into three sections with different meanings through the business number digit, the data date digit, and the data sequence number digit;

[0088] Business number bit: consists of [1-2] bits, used to isolate vector data according to business. If a single business uses a fixed value of 99;

[0089] Data date bit: consists of bits [3-10], which are 8-bit dates in the "yyyyMMdd" format. They are used to identify the day the vector data is from, so that some data can be rebalanced on a daily basis.

[0090] Data sequence number: It consists of [11-19] bits, and increases from 0 every day according to "business number + data date", and can support up to 1 billion data volumes per day;

[0091] According to this naming method, the preset rules can also be:

[0092] According to the business code number of each vector data, the vector data belonging to the same business are determined, and calculations are performed according to the instances where these vector data are located, thereby achieving business-level data rebalancing;

[0093] In another example, the preset rules may also be formulated based on business hotspots;

[0094] For example: after determining the data of the same business according to the business number of the vector data, obtain the popularity of the business code. If the business is a hot business, the vector data corresponding to the hot business will be evenly distributed to each storage instance to ensure the query efficiency of the hot business; and correspondingly, the rebalancing rules can also be adjusted accordingly. For example, after calculating the amount of data to be allocated to each example, query the amount of hot data in the data that needs to be migrated in this balancing instance, and redistribute it according to a certain threshold to further ensure the query efficiency of the hot data. Through the above examples, a more flexible and efficient way is provided for determining the data processing tasks of the vector database.

[0095] In one embodiment, sub-step S21 includes the following sub-steps:

[0096] Sub-step S31, under the condition that the current time satisfies the preset processing time rule, according to the quantity of the vector data in each of the multiple data components and the preset data processing rule, determining the quantity value of the reference vector data of the data component;

[0097] In one example, the migration quantity is determined based on the calculated data quantity of each instance and the current data quantity in each instance;

[0098] Sub-step S32, determining the vector data processing task according to the quantity of the vector data in each data component and the quantity value of the reference vector data.

[0099] In one embodiment, sub-step S32 includes the following sub-steps:

[0100] Sub-step S321, determining a reference processed data value according to a difference between the quantity of the vector data in each data component and the quantity of the reference vector data;

[0101] Furthermore, the system determines the amount of data that needs to be processed or migrated for each instance by calculating the difference between the amount of vector data in each data component and the reference vector data amount value, which is the average value of all instance data or the target value calculated according to other rules. This difference reflects the gap between the current instance and the ideal state and is the basis for formulating processing tasks. By calculating the difference, the system can accurately quantify the amount of data that needs to be adjusted for each instance, providing an accurate basis for subsequent processing tasks.

[0102] Sub-step S322, determining the vector data processing task according to the reference processing data value.

[0103] By performing processing tasks determined by the difference, the system can effectively balance the load between the instances, avoiding the situation where some instances are overloaded while others are idle.

[0104] In one embodiment, sub-step S322 includes the following sub-steps:

[0105] Sub-step S3221, determining the type of the vector data processing task according to the positive or negative value of the reference processing data value;

[0106] By judging the positive or negative value of the reference processing data, the system can clearly know whether the current data status of each instance is higher or lower than the reference value. This directly affects the type of processing task. If the value is positive, it means that the instance needs to migrate out some data to reduce the load, and if the value is negative, it means that the instance needs to migrate in data to achieve balance. This clear classification provides a clear direction for subsequent processing tasks.

[0107] Sub-step S3222, determining the number of vector data processing tasks according to the absolute value of the reference processing data value;

[0108] The absolute value of the reference processing data value reflects the gap between the current instance and the reference value, that is, the amount of data that needs to be processed. By calculating the absolute value, the system can accurately quantify the amount of data that each instance needs to process, thereby determining the number of processing tasks. This helps the system allocate resources reasonably and ensure that each instance can be processed effectively.

[0109] Sub-step S3223, determining the vector data processing tasks according to the types of the vector data processing tasks and the number of the vector data processing tasks.

[0110] By accurately determining the type and quantity of tasks, the system can efficiently perform processing tasks, reducing unnecessary resource consumption and waiting time. The system can dynamically adjust processing tasks according to the actual data status to ensure that the load between instances remains balanced, improving system stability and performance. This approach does not rely on fixed rules or thresholds, but dynamically adjusts processing tasks according to the actual data status, so it has strong flexibility and scalability. As the amount of data grows and business needs change, the system can easily adapt and make corresponding adjustments.

[0111] Step 103, sending the vector data processing task to a preset asynchronous task processing queue, so that the asynchronous task processing queue completes the vector data processing task;

[0112] The system sends the determined data processing tasks to a preset asynchronous task processing queue so that these tasks can be executed asynchronously to improve processing efficiency.

[0113] In one example, the asynchronous task processing queue can be a Kafka queue. Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is based on the publish / subscribe model and is a distributed, high-throughput message queue system. Kafka is mainly used to process real-time data streams and batch data, and provides an efficient, scalable, reliable, and replicable message processing platform.

[0114] It should be noted that, in addition to using Kafka, the asynchronous task processing queue can also select other task processing queues, such as MQ (Message Queue) series queues, and the present invention does not limit this.

[0115] Step 104 , under the condition that the preset asynchronous task processing queue completes the vector data processing task, updating the persistent data table corresponding to the vector data to be processed.

[0116] After the asynchronous task processing queue completes the data processing task, the system will update the persistent data table corresponding to the vector data to be processed to reflect the processed data status.

[0117] First, the current time, the vector data to be processed, and the persistent data table corresponding to the vector data to be processed are obtained. Under the condition that the current time meets the preset processing time rules, the vector data processing tasks are determined according to the vector data to be processed. These tasks are then sent to the preset asynchronous task processing queue, which is responsible for completing the data processing tasks. Once the asynchronous task processing queue completes these tasks, the persistent data table corresponding to the vector data to be processed will be updated. This process ensures the dynamic migration and update of data between multiple instances, and effectively realizes the rebalancing of data. Through the efficient processing of the asynchronous task processing queue, data can flow between different instances in a timely and orderly manner, avoiding the occurrence of data skew. This dynamic rebalancing of data not only improves the data processing capability of each instance, but also enables the entire vector database to more flexibly expand storage and computing capabilities when facing large-scale data processing needs.

[0118] Therefore, the embodiments of the present invention significantly improve the horizontal expansion capability of the vector database, enabling it to better cope with the growth of data volume and changes in business needs. At the same time, the dynamic rebalancing of data also brings about an improvement in overall performance, improves the processing efficiency and response speed of the database, and provides users with a better user experience.

[0119] Reference Figure 2 , showing a flow chart of an embodiment of a data processing method of the present invention;

[0120] First, the system will maintain a scheduled task, for example, it will be executed at 0:00 every day. Then at 0:00 every day, the system will automatically start the scheduled task, scan the data of each current instance, and calculate the data that needs to be rebalanced according to a certain calculation logic. Taking the figure as an example, it is assumed that there are three instances, namely instance 1, instance 2 and instance 3, and their data are instance 1, 4000 data; instance 2, 3500 data; instance 3, 1500 data; then calculate according to the preset rules. For example, when the calculation rule is the average rule, the result of the next calculation is that the amount of data that each instance should have = the average of the amount of data of all instances, that is, the amount of data applied to each instance is (4000+3500+1500) / 3=3000; at this time, it means that the data of each instance needs to be migrated according to the standard of 3000 data per instance. Correspondingly, instance 1 originally has 4000 data, and the amount of data that needs to be migrated is 4000-3000= 1000, the value is a positive number, which means migration out, and 1000 represents the amount of data, that is, instance 1 needs to migrate out 1000 data; instance 2 originally has 3500 data, and after corresponding calculation, the data migration logic of instance 2 is to migrate out 500 data; and for instance 3, its original data volume is 1500, and the amount of data to be migrated is 1500-3000=-1500, and the value is a negative number, which means that the data migration type is migration in, and the value of 1500 means that 1500 data need to be migrated, that is, instance 3 needs to migrate in 1500 data. It should be noted that this calculation process is only one of the calculation methods in the embodiment of the present invention, and the calculation method can also be other, and the processing method is also improved accordingly. For example, the data migration amount can be the calculated amount of data applied to each instance minus the actual amount of data of each current instance, and the corresponding positive and negative values ​​represent the opposite data migration logic, that is, positive numbers represent migrated-in data, and negative numbers represent migrated-out data;

[0121] According to the calculation results, the data migration task is generated through the rebalancing task module, and two tasks are obtained. Task 1 is to migrate 1,000 data from instance 1 to instance 3, and task 2 is to migrate 500 data from instance 2 to instance 3. At this point, the data migration task generation is completed.

[0122] Reference Figure 3 , showing a timing diagram of an embodiment of a data processing method of the present invention;

[0123] The vector database Balancer is a component designed specifically to manage and optimize the performance of vector databases. It is responsible for balancing the distribution of data between multiple nodes or instances to ensure efficient operation and load balancing of the system. By monitoring the performance and load of the vector database, the Balancer can dynamically adjust the data distribution to avoid overloading some nodes while other node resources are idle. In the data migration process, the Balancer plays a key role. It coordinates the data migration between the source instance and the target instance to ensure the accuracy and integrity of the data. At the same time, the Balancer is also responsible for processing data balancing tasks. By calculating, querying and migrating data, and processing the insertion of new data in the target instance, it ensures the balanced distribution of data between the source instance and the target instance, thereby improving the overall performance and stability of the database.

[0124] First, the process begins with a rebalancing task, which is the starting point of the entire data migration and balancing process.

[0125] The application sends a rebalancing task to the vector database Balancer component. After receiving the task, the Balancer calculates the amount of data that needs to be balanced in the source instance to determine the scale of data migration. Next, the Balancer interacts with the vector database to query and filter out the specific data that needs to be balanced. The queried data is then returned and prepared for migration. At this point, the source instance enters the data migration phase.

[0126] During the data migration phase, data is migrated from the vector database of the source instance to the vector database of the target instance. First, the vector database Balancer of the source instance interacts with the vector database Balancer of the target instance to start data migration. The vector database Balancer of the target instance migrates data to the corresponding vector database. After the migration is completed, the vector database of the target instance sends a successful migration flag to the corresponding vector database Balancer, such as returning a successful new addition prompt. After receiving the prompt of successful migration, the vector database Balancer of the target instance returns the prompt to the vector database Balancer of the source instance. At this time, the vector library Balancer of the source instance interacts with the corresponding vector database, changes the reference relationship of the migrated data, and deletes the migrated data. After the completion, the result of successful deletion is returned to the corresponding vector database Balancer. The vector library Balancer of the source instance then sends the result of this rebalancing to the user. This rebalancing process is completed.

[0127] Reference Figure 4 , which shows a data migration schematic diagram of an embodiment of a data processing method of the present invention:

[0128] This embodiment takes the Kafka queue as an example to illustrate the process of data migration;

[0129] First, after receiving the task execution instruction, the system calculates the rebalance of resources and determines which data needs to be migrated.

[0130] After determining the scope and quantity of data to be migrated, the system starts to read the determined data scope in a loop and prepare for migration. When the data reading is completed, the system will push an end mark, otherwise the system will push the data that needs to be migrated but has not been read, and will return to the re-reading step;

[0131] After pushing the end mark, the system pushes the read data to the Kafka queue and sends a message in the format of topic: XXX (where XXX represents the target instance identifier). The message body includes the end mark and data.

[0132] Then, the target instance consumes the corresponding data in Kafka and stores the corresponding data indicated in it. When the data is stored, a message to delete the data is pushed to the Kafka queue.

[0133] The format is topic: YYY (where YYY represents the source instance ID), and the message body includes the deletion ID and data.

[0134] Then, the source instance consumes the corresponding deletion message in the Kafka queue and deletes the vector data first. After all the vector data is deleted, the state of the end flag is changed from false to true. At this time, the relevant data in the corresponding instance is deleted to complete this migration.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 5 , shows a structural block diagram of a data processing device embodiment of the present invention, which may specifically include the following modules:

[0137] The data acquisition module 201 is used to acquire the current time, the vector data to be processed and the persistent data table corresponding to the vector data to be processed;

[0138] A processing task acquisition module 202 is used to determine a vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0139] The task processing module 203 is used to send the vector data processing task to a preset asynchronous task processing queue so that the asynchronous task processing queue completes the vector data processing task;

[0140] The data updating module 204 is used to update the persistent data table corresponding to the vector data to be processed under the condition that the preset asynchronous task processing queue completes the vector data processing task.

[0141] In one embodiment, the processing task acquisition module includes:

[0142] A first processing task determination submodule is used to determine a candidate vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule;

[0143] The second processing task determination submodule is used to determine a vector data processing task according to the vector data processing candidate task when the vector data processing candidate task meets the preset data task rule.

[0144] In one embodiment, the device further comprises:

[0145] The manual processing submodule is used to output a reminder event to the user when the vector data processing candidate task does not meet the preset data task rules, so that the user can manually set the vector data processing task.

[0146] In one embodiment, the vector data to be processed includes multiple data components, each of the multiple data components includes multiple vector data, and the device further includes:

[0147] A data quantity determination submodule, used for obtaining the quantity of the vector data in each of the multiple data components;

[0148] The processing task acquisition module includes:

[0149] The third processing task determination submodule is used to determine the vector data processing task according to the amount of the vector data in each of the multiple data components, under the condition that the current time satisfies the preset processing time rule.

[0150] In one embodiment, the third processing task determination submodule includes:

[0151] A first data determination unit, configured to determine the quantity value of the reference vector data of the data component according to the quantity of the vector data in each of the multiple data components and a preset data processing rule, under the condition that the current time satisfies a preset processing time rule;

[0152] The fourth processing task determination unit is used to determine the vector data processing task according to the quantity of the vector data in each data component and the quantity value of the reference vector data.

[0153] In one embodiment, the fourth processing task determination unit includes:

[0154] A reference processed value acquisition subunit, used for determining a reference processed data value according to a difference between the quantity of the vector data in each data component and the quantity value of the reference vector data;

[0155] The fifth processing task subunit is used to determine the vector data processing task according to the reference processing data value.

[0156] In one embodiment, the fifth processing task subunit includes:

[0157] A task type determination component, used to determine the type of the vector data processing task according to the positive or negative value of the reference processing data value;

[0158] A task value determination component, used to determine the number of vector data processing tasks according to the absolute value of the reference processing data value;

[0159] The sixth processing task component is used to determine the vector data processing task according to the type of the vector data processing task and the number of the vector data processing tasks.

[0160] First, the current time, the vector data to be processed, and the persistent data table corresponding to the vector data to be processed are obtained. Under the condition that the current time meets the preset processing time rules, the vector data processing tasks are determined according to the vector data to be processed. These tasks are then sent to the preset asynchronous task processing queue, which is responsible for completing the data processing tasks. Once the asynchronous task processing queue completes these tasks, the persistent data table corresponding to the vector data to be processed will be updated. This process ensures the dynamic migration and update of data between multiple instances, and effectively realizes the rebalancing of data. Through the efficient processing of the asynchronous task processing queue, data can flow between different instances in a timely and orderly manner, avoiding the occurrence of data skew. This dynamic rebalancing of data not only improves the data processing capability of each instance, but also enables the entire vector database to more flexibly expand storage and computing capabilities when facing large-scale data processing needs.

[0161] Therefore, the embodiments of the present invention significantly improve the horizontal expansion capability of the vector database, enabling it to better cope with the growth of data volume and changes in business needs. At the same time, the dynamic rebalancing of data also brings about an improvement in overall performance, improves the processing efficiency and response speed of the database, and provides users with a better user experience.

[0162] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0163] An embodiment of the present invention further provides an electronic device, including:

[0164] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0165] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0167] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0168] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0173] The data processing method, device, equipment and medium provided by the present invention are introduced in detail above. The principles and implementation methods of the present invention are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining the current time, the vector data to be processed, and the persistent data table corresponding to the vector data to be processed; Under the condition that the current time satisfies a preset processing time rule, determining a vector data processing task according to the vector data to be processed; Sending the vector data processing task to a preset asynchronous task processing queue so that the asynchronous task processing queue completes the vector data processing task; Under the condition that the preset asynchronous task processing queue completes the vector data processing task, the persistent data table corresponding to the vector data to be processed is updated.

2. The method according to claim 1, characterized in that The step of determining the vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule includes: determining a candidate vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule; In the case that the candidate vector data processing task satisfies the preset data task rule, a vector data processing task is determined according to the candidate vector data processing task.

3. The method according to claim 2, characterized in that The method further comprises: When the candidate vector data processing task does not satisfy the preset data task rule, a reminder event is output to the user so that the user can manually set the vector data processing task.

4. The method according to claim 1, characterized in that: The vector data to be processed includes a plurality of data components, each of the plurality of data components includes a plurality of vector data, and the method further includes: Obtaining the amount of the vector data in each of the multiple data components; The step of determining the vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule includes: Under the condition that the current time satisfies a preset processing time rule, a vector data processing task is determined according to the amount of the vector data in each of the multiple data components.

5. The method according to claim 4, characterized in that The step of determining the vector data processing task according to the amount of the vector data in each of the multiple data components under the condition that the current time satisfies a preset processing time rule includes: Under the condition that the current time satisfies a preset processing time rule, determining the quantity value of the reference vector data of the data component according to the quantity of the vector data in each of the multiple data components and a preset data processing rule; A vector data processing task is determined according to the quantity of the vector data in each data component and the quantity value of the reference vector data.

6. The method according to claim 5, characterized in that The determining of the vector data processing task according to the quantity of the vector data in each data component and the quantity value of the reference vector data comprises: Determine a reference processed data value according to a difference between the quantity of the vector data in each data component and the quantity value of the reference vector data; A vector data processing task is determined according to the reference processing data value.

7. The method according to claim 6, characterized in that The step of determining the vector data processing task according to the reference processing data value comprises: Determining the type of the vector data processing task according to the positive or negative value of the reference processing data value; determining the number of vector data processing tasks according to the absolute value of the reference processed data value; The vector data processing tasks are determined according to the types of the vector data processing tasks and the number of the vector data processing tasks.

8. A data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire the current time, the vector data to be processed and a persistent data table corresponding to the vector data to be processed; A processing task acquisition module, used to determine a vector data processing task according to the vector data to be processed under the condition that the current time satisfies a preset processing time rule; A task processing module, used for sending the vector data processing task to a preset asynchronous task processing queue, so that the asynchronous task processing queue completes the vector data processing task; The data updating module is used to update the persistent data table corresponding to the vector data to be processed under the condition that the preset asynchronous task processing queue completes the vector data processing task.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images