Data processing method and device, equipment, storage medium and product

Abstract

This application discloses a data processing method, apparatus, device, storage medium, and product. The method includes: acquiring a data operation request, which is used to update a retrieval cluster, the retrieval cluster comprising M nodes; adding the data operation request to the message queues corresponding to the M nodes, the data operation request being the k-th queue element added to the message queue; and, in response to a consumption confirmation message sent by any node for the (k-1)-th queue element, sending a data operation request to that node so that the node can consume the data operation request. It is evident that the message queues and consumption confirmation message ensure that the order in which each node in the retrieval cluster consumes the data operation request is consistent, thereby ensuring that the final state of each node in the retrieval cluster matches.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, a data processing apparatus, a computer device, a computer-readable storage medium, and a data processing product. Background Technology

[0002] With the rapid development of computer technology, massive amounts of data are generated every moment, such as image data, text data, and audio data. To manage this data more efficiently, it is typically converted into vectors in a specific vector space. When this data is needed, vector retrieval can be used to quickly obtain the required information. Research has found that retrieval clusters are commonly used to meet the demands of massive retrieval. In addition to distributing retrieval traffic, each node in the retrieval cluster also needs to update based on data operation requests. Ensuring that the updated states of each node in the retrieval cluster match has become a hot research topic. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, computer-readable storage medium, and product that can ensure the matching of updated states of each node in a retrieval cluster.

[0004] On one hand, embodiments of this application provide a data processing method, including:

[0005] Get a data operation request. The data operation request is used to update the retrieval cluster, which consists of M nodes, where M is an integer greater than 1.

[0006] Add the data operation request to the message queues corresponding to M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1.

[0007] In response to the consumption confirmation message of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request; the consumption confirmation message of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

[0008] In this embodiment, a data operation request is obtained. This data operation request is used to update the retrieval cluster, which includes M nodes. The data operation request is added to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request. It can be seen that the message queues and consumption confirmation information ensure that the order in which each node in the retrieval cluster consumes data operation requests is consistent, thereby ensuring that the updated states of each node in the retrieval cluster match.

[0009] On one hand, embodiments of this application provide a data processing method, including:

[0010] Retrieve data operation requests to be consumed from the message queue. The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1.

[0011] Consume the data operation requests to be consumed;

[0012] In response to a successful consumption of a data operation request, a consumption confirmation message is returned to the message queue, enabling the message queue to provide the k-th queue element that has been added to the message queue.

[0013] In this embodiment, a data operation request to be consumed is obtained from the message queue. This request is the (k-1)th queue element added to the message queue. The request is then consumed. Upon successful consumption, a consumption confirmation message is returned to the message queue, enabling the message queue to provide the kth queue element added to it. Therefore, by using the message queue and the consumption confirmation message, the order in which the request is consumed is ensured to be consistent with the order of data operation requests consumed by other nodes, thereby ensuring the matching of updated states across the retrieval cluster.

[0014] On one hand, embodiments of this application provide a data processing apparatus, which includes:

[0015] The acquisition unit is used to acquire data operation requests, which are used to update the retrieval cluster. The retrieval cluster consists of M nodes, where M is an integer greater than 1.

[0016] The processing unit is used to add data operation requests to the message queues corresponding to M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1.

[0017] The sending unit is used to send a data operation request to any node in response to the consumption confirmation information of the (k-1)th queue element sent by any node, so that the node can consume the data operation request; the consumption confirmation information of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

[0018] In one implementation, the data operation request includes data indication information and operation indication information; the processing unit is used to add the data operation request to the message queues corresponding to M nodes, specifically for:

[0019] In the database, the target data is operated on according to the operation instructions. The target data is the data indicated by the data instructions.

[0020] In response to the completion of the operation on the target data, the data operation request is added to the message queues corresponding to M nodes.

[0021] In one embodiment, the processing unit is further configured to:

[0022] Based on the database, the state of M nodes is periodically checked to obtain the state check results of M nodes;

[0023] The status detection result is used to indicate whether the search results for the same data in the database and the M nodes match.

[0024] In one implementation, the processing unit is used to periodically perform state checks on M nodes based on a database to obtain the state check results of the M nodes, specifically for:

[0025] Get the data set to be retrieved, which includes N data items, where N is a positive integer;

[0026] Send N data retrieval requests to each node and obtain S response messages from M nodes, where S is a natural number less than or equal to M×N;

[0027] Based on S response messages, determine the state detection results of M nodes.

[0028] In one implementation, each response message includes a retrieval result of data in a node; the processing unit is configured to determine the state detection results of M nodes based on the S response messages, specifically for:

[0029] Based on the search results of N data in the database, count the number of expected response messages among S response messages; any expected response message contains search results that match the corresponding data in the database.

[0030] The matching rate of the results for M nodes is calculated by the number of responses that meet the expectations and the total number of responses, S.

[0031] If the matching rate is greater than or equal to the first probability threshold, the availability of M nodes is calculated based on the total amount of response information S and the total amount of retrieval requests M×N.

[0032] If the availability rate is greater than or equal to the second probability threshold, then the search results of M nodes for the same data are determined to be a match.

[0033] In one implementation, N data points are the vectors to be retrieved; the i-th response information is obtained by the h-th node after retrieving the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processing unit counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0034] If the ith response information does not match the retrieval result of the jth data in the database, then obtain the candidate vector set of the jth data from the hth node; the candidate vector set includes at least two candidate vectors, and the similarity between any candidate vector and the jth data is greater than the similarity threshold.

[0035] If there is a candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to meet expectations.

[0036] If there is no candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to be unsatisfactory.

[0037] In one implementation, N data points are the vectors to be retrieved; the i-th response information is obtained by the h-th node after retrieving the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processing unit counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0038] If the i-th response does not match the retrieval result of the j-th data in the database, then the database is refreshed.

[0039] If the i-th response information matches the search result of the j-th data in the refreshed database, then the i-th response information is determined to meet expectations.

[0040] If the i-th response information does not match the search result of the j-th data in the refreshed database, then the i-th response information is determined to be unsatisfactory.

[0041] In one implementation, consuming a data operation request by any node means that the node updates the memory according to the data operation request; the processing unit is further configured to:

[0042] M nodes are blocked in turn according to a preset order and preset time, so that the blocked nodes store element indication information on the disk and data in memory;

[0043] The element indication information is used to indicate the target queue element in the message queue. The target queue element is the first queue element consumed by the blocked node after loading the memory data stored on the disk.

[0044] In one embodiment, the processing unit is further configured to:

[0045] A queue element request is retrieved. The queue element request includes element indication information, which is used to indicate the target queue element in the message queue.

[0046] Return the target queue element to the sender of the queue element request so that the sender can restore the state based on the target queue element.

[0047] In one embodiment, the processing unit is further configured to:

[0048] Obtain the baseline database for the nodes to be expanded;

[0049] The baseline database is imported into the memory of M-1 initialization nodes to obtain the retrieval cluster corresponding to the node to be expanded. The retrieval cluster includes the node to be expanded and M-1 replica nodes of the node to be expanded.

[0050] Establish the correspondence between the retrieval cluster and the message queue.

[0051] On one hand, embodiments of this application provide a data processing apparatus, which includes:

[0052] The retrieval unit is used to retrieve data operation requests to be consumed from the message queue. The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1.

[0053] The processing unit is used to process data operation requests to be consumed.

[0054] The sending unit is used to respond to a successful consumption request for data to be consumed by returning consumption confirmation information to the message queue, so that the message queue can provide the kth queue element added to the message queue.

[0055] In one embodiment, the processing unit is further configured to:

[0056] The data backup time is determined based on the number of nodes and the size parameter corresponding to the message queue. The size parameter is used to indicate the number of nodes to be backed up in the same batch.

[0057] When the current time matches the data backup time, copy the backup data and element indication information from the backup node to the disk;

[0058] Among them, the backup data is obtained by the backup node storing the data in memory, and the data in the backup node's memory is obtained by the backup node consuming the queue elements in the message queue; the element indication information is used to indicate the target queue element in the message queue, and the target queue element is: the first queue element consumed after loading the backup data.

[0059] In one implementation, the processing unit is configured to determine the data backup time based on the number of nodes and scale parameters corresponding to the message queue, specifically for:

[0060] Get the start time and copy duration. The start time is the time when the backup node completes the backup. The copy duration is used to indicate the duration of each copy of backup data and element indication information from the backup node.

[0061] The synchronization batch is determined by the number and size parameters of the nodes;

[0062] The data backup time is calculated based on the start time, synchronization batch, and copy duration.

[0063] Accordingly, this application provides a computer device comprising:

[0064] Memory, which stores computer programs;

[0065] A processor is used to load computer programs to implement the above data processing methods.

[0066] Accordingly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method.

[0067] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data processing method.

[0068] In this embodiment, a data operation request is obtained. This data operation request is used to update the retrieval cluster, which includes M nodes. The data operation request is added to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request. It can be seen that the message queues and consumption confirmation information ensure that the order in which each node in the retrieval cluster consumes data operation requests is consistent, thereby ensuring that the final state of each node in the retrieval cluster matches. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A data processing scenario diagram provided in an embodiment of this application;

[0071] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0072] Figure 3 A flowchart illustrating another data processing method provided in this application embodiment;

[0073] Figure 4a A data processing architecture diagram provided for an embodiment of this application;

[0074] Figure 4b A schematic diagram of state detection provided in an embodiment of this application;

[0075] Figure 4c A schematic diagram of a control architecture provided for an embodiment of this application;

[0076] Figure 5 A flowchart illustrating another data processing method provided in this application embodiment;

[0077] Figure 6a A data synchronization architecture diagram provided for an embodiment of this application;

[0078] Figure 6b This is a schematic diagram of a node state recovery process provided in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0080] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0081] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0083] This application provides a data processing scheme that can ensure the matching of the updated states of each node in the retrieval cluster. Specifically, it can be applied to data processing scenarios involving the retrieval cluster, such as vector retrieval, media data retrieval, and file retrieval. Figure 1 A data processing scenario diagram provided for an embodiment of this application, such as Figure 1 As shown, the data processing scenario provided in this application includes a server 101 and nodes 102, and at least two nodes 102 can jointly form a retrieval cluster. The data processing scheme provided in this application can be executed by either server 101 or node 102. Specifically, node 102 can be a terminal device or a server. Terminal devices can include, but are not limited to, smartphones (such as Android phones, iOS phones, etc.), tablets, portable personal computers, mobile internet devices (MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc. This application embodiment does not limit this. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application embodiment does not limit this.

[0084] It should be noted that, Figure 1 The number of servers and nodes is for illustrative purposes only and does not constitute an actual limitation of this application. Server 101 and node 102 can be connected via wired or wireless means, and this application does not impose any restrictions on this.

[0085] The general flow of the data processing solution provided in this application is as follows:

[0086] (1) Server 101 obtains a data operation request. The data operation request is used to update the retrieval cluster, which includes M nodes, where M is an integer greater than 1. The data operation request may include data indication information and operation indication information. The data indication information is used to indicate the data to be operated on, and the operation indication information is used to indicate the specific operation, such as adding, deleting, or modifying.

[0087] (2) Server 101 adds the data operation request to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1. For the first queue element added to the message queue, server 101 can cache the queue element in the message queue and send it to each node in the retrieval cluster. For the kth queue element added to the message queue, server 101 caches the queue element in the message queue and continues to execute step (3). In one implementation, server 101 can first consume the data operation request in its own database, and after consumption, add the data operation request to the message queues corresponding to the M nodes.

[0088] (3) In response to the consumption confirmation message for the (k-1)th queue element sent by any node, server 101 sends a data operation request to that node so that the node can consume the data operation request. The consumption confirmation message for the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the information sender (i.e., a node in the retrieval cluster). That is to say, for each node in the retrieval cluster, it can only obtain the kth queue element from server 101 after successfully consuming the (k-1)th queue element.

[0089] In this embodiment, a data operation request is obtained. This data operation request is used to update the retrieval cluster, which includes M nodes. The data operation request is added to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request. It can be seen that the message queues and consumption confirmation information ensure that the order in which each node in the retrieval cluster consumes data operation requests is consistent, thereby ensuring that the updated states of each node in the retrieval cluster match.

[0090] Based on the above data processing scheme, this application proposes a more detailed data processing method. The data processing method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0091] Please see Figure 2 , Figure 2 A flowchart illustrating a data processing method provided in this application embodiment, which can be executed by a computer device; for example, by... Figure 1 The server 101 shown is executing.

[0092] like Figure 2 As shown, the data processing method may include the following steps S201-S203:

[0093] S201, Data Acquisition Operation Request.

[0094] Data operation requests are used to update the retrieval cluster, which consists of M nodes, where M is an integer greater than 1. In a retrieval task scenario, a node specifically refers to a retrieval task computation node, and nodes within the same retrieval cluster are (retrieval) replicas of each other. Data operation requests can include data indication information and operation indication information. Data indication information indicates the data to be operated on, while operation indication information indicates the specific operation, such as adding, deleting, or modifying.

[0095] S202. Add the data operation request to the message queues corresponding to the M nodes.

[0096] Message queues have single-point write and multi-point consumption capabilities (such as plusar[], etc.).

[0097] In one implementation, the data operation request is the first queue element added to the message queues corresponding to the M nodes. In one embodiment, the computer device can directly cache the data operation request in the message queues corresponding to the M nodes and send the data operation request to the M nodes so that each node can consume the data operation request. In another embodiment, the computer device can first consume the data operation request in its own database. In one implementation, the data operation request includes data indication information and operation indication information. The computer device consuming the data operation request in its own database means: performing an operation on the target data in the database according to the operation indication information; wherein, the target data is the data indicated by the data indication information. After the data operation request is successfully consumed (i.e., the target data operation is completed), the computer device caches the data operation request in the message queues corresponding to the M nodes and sends the data operation request to the M nodes so that each node can consume the data operation request.

[0098] In another implementation, the data operation request is the k-th queue element added to the message queues corresponding to the M nodes, where k is an integer greater than 1. In one embodiment, the computer device can directly cache the data operation request in the message queues corresponding to the M nodes. In another embodiment, the computer device can first consume the data operation request in its own database, and after successful consumption, cache the data operation request in the message queues corresponding to the M nodes. By consuming the data operation request in the database beforehand, the following effect can be achieved: even if the data operation request cached in the message queue is lost, it can be recovered through the database.

[0099] In another implementation, the message queue management strategy includes removing the queue element from the message queue when the cached duration of the queue element reaches a duration threshold. Compared to removing the queue element after sending it to M nodes, caching the queue element for a certain duration before removal is suitable for node recovery scenarios. In one embodiment, the duration threshold is greater than the data backup interval of each node.

[0100] For example, suppose a computer device has provided the k-th queue element to all nodes in the retrieval cluster. Node A has cached queue elements 1 through k in its memory and has stored (backed up) queue elements 1 through k-1 on its disk. After restarting node A, node A can restore queue elements 1 through k-1 to its memory based on the backup data on the disk, and then retrieve the k-th queue element from the message queue for consumption. However, if the computer device removes the k-th queue element after providing it to all nodes in the retrieval cluster, then the k-th queue element cannot be provided after node A restarts.

[0101] S203. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, send a data operation request to that node so that the node can consume the data operation request.

[0102] The consumption confirmation information for the (k-1)th queue element indicates that the (k-1)th queue element has been successfully consumed by the information sender (i.e., a node in the retrieval cluster). In one implementation, a data operation request includes data indication information and operation indication information. A node consuming a data operation request can be understood as performing operations on the target data in its own memory according to the operation indication information; where the target data is the data indicated by the data indication information. A node consuming a data operation request can also be understood as adding or deleting corresponding indexes in the node's retrieval process based on the data operation request.

[0103] In this embodiment, a data operation request is obtained. This data operation request is used to update the retrieval cluster, which includes M nodes. The data operation request is added to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request. It can be seen that the message queues and consumption confirmation information ensure that the order in which each node in the retrieval cluster consumes data operation requests is consistent, thereby ensuring that the updated states of each node in the retrieval cluster match.

[0104] Please see Figure 3 , Figure 3 A flowchart illustrating another data processing method provided in this application embodiment, which can be executed by a computer device; for example, by... Figure 1 The server 101 shown is executing.

[0105] like Figure 3 As shown, the data processing method may include the following steps S301-S306:

[0106] S301. Expand the node to be expanded to obtain the retrieval cluster corresponding to the node to be expanded.

[0107] The nodes to be expanded can be standalone vector integration tools (such as Faiss) or standalone vector retrieval engines. The retrieval cluster consists of M nodes, each a copy of the others, where M is an integer greater than 1. In one implementation, the computer device obtains the baseline database of the nodes to be expanded and imports the baseline database into the memory of M-1 initialization nodes, resulting in M-1 nodes whose states match the nodes to be expanded. State matching can be understood as: the retrieval results for the same data are consistent across all nodes. For example, if a vector A is added to the retrieval cluster, then any node in the retrieval cluster can retrieve vector A. If a node in the retrieval cluster cannot retrieve vector A, it means that the state of that node does not match that of other nodes in the retrieval cluster.

[0108] In approximate retrieval scenarios, state matching can be understood as follows: the retrieval results for the same data to be retrieved in one node are contained in the retrieval set of another node, and this retrieval set contains at least two retrieval results that are approximately similar to the data to be retrieved (with a similarity greater than a similarity threshold). For example, in node 1, an approximate retrieval of the data to be retrieved, A, yields vector C. In node 2, an approximate retrieval of the data to be retrieved, A, yields vector D. At this point, the retrieval results of node 1 and node 2 are different. Increasing the recall count of node 2 (e.g., increasing it to 5) yields the retrieval set {vector D, vector F, vector C, vector J, vector H}. Since vector C is contained in the retrieval set, node 1 and node 2 are determined to be state-matched.

[0109] It should be noted that since the retrieval cluster is extended from the nodes to be extended, any custom method applicable to the nodes to be extended is also applicable to the retrieval cluster, making it more flexible and convenient compared to the distributed retrieval cluster (which does not provide an interface for custom retrieval algorithms).

[0110] S302. Establish the correspondence between the retrieval cluster and the message queue.

[0111] Message queues are used to cache data operation requests, which are then used to update the retrieval cluster. Specifically, each node in the retrieval cluster updates its own memory based on the data operation request; for example, if the data operation request is to add data A, then each node in the retrieval cluster adds data A to its own memory; conversely, if the data operation request is to delete data B, then each node in the retrieval cluster deletes data B from its own memory.

[0112] In one implementation, after the M-1 initialization nodes import the baseline database into their own memory, the correspondence between the retrieval cluster and the message queue is established by subscribing to the message queue.

[0113] S303, Data Acquisition Operation Request.

[0114] S304. Add the data operation request to the message queues corresponding to the M nodes.

[0115] S305. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, send a data operation request to that node so that the node can consume the data operation request.

[0116] For detailed implementation methods of steps S303-S305, please refer to [link / reference]. Figure 2 The implementation methods in steps S201-S203 will not be described in detail here.

[0117] In one implementation, any node in the retrieval cluster can request a corresponding queue element from the computer device via a queue element request after a preset condition is met (such as restart, data recovery, data loss, etc.). The queue element request may include element indication information, which indicates the target queue element (i.e., the element the node requests to retrieve) in the message queue. After receiving the queue element request, the computer device returns the target queue element to the sender of the queue element request (any node in the retrieval cluster), enabling the sender to perform state recovery based on the target queue element.

[0118] Figure 4a This is a data processing architecture diagram provided for an embodiment of this application. For example... Figure 4a As shown, after obtaining a data operation request, the database can be updated based on the data operation request. The database can be any database such as MySQL, depending on the business requirements. Writing to the database has two functions: (1) to archive the data operation request, so that even if all the data in the memory and disk storage of the retrieved copies are lost, there is still reliable database archive data; (2) to meet operational needs; for example, relational databases can be used to perform conditional queries (such as limiting time periods, tags, etc.).

[0119] After the database update is complete, the computer device caches the data operation request to the message queue, and then each node (retrieval replica) in the retrieval cluster subscribes to and consumes the queue elements in the message queue. The message queue can be a message queue component with single-point write and multi-point consumption capabilities, such as plusar[]. All nodes (retrieval replicas) in the retrieval cluster do not need to elect a leader or communicate with each other; instead, they each consume data in their own message queues. If each node successfully consumes a queue element, the states of all nodes after consuming the queue element are matched. Successful consumption can be understood as completing the operation indicating the data in the node's memory. It can be understood that during the consumption of queue elements, the index in the retrieval process is updated (e.g., added or deleted). For any node, after successfully consuming a queue element, it sends a consumption confirmation message to the message queue to obtain the next queue element to be consumed. If a node fails to consume a queue element, it can retry repeatedly. If the maximum number of retries is reached and the queue element still cannot be successfully consumed, a prompt message is sent to the administrator, or the node performs its own state recovery (e.g., restart).

[0120] S306. Based on the database, periodically perform state checks on M nodes to obtain the state check results of M nodes.

[0121] The status detection result is used to indicate whether the retrieval results of M nodes in the database and retrieval cluster match for the same data. In one implementation, the computer device acquires a set of data to be retrieved, which includes N data items, where N is a positive integer. The computer device sends retrieval requests for N data items to each node (i.e., the computer device sends M×N retrieval requests to M nodes in total), and obtains S response messages returned by the M nodes. Each response message includes the retrieval result for one data item in one node, where S is a natural number less than or equal to M×N. Cases where S is less than M×N include situations where some nodes (affected by factors such as network fluctuations) do not return response messages. After obtaining the S response messages, the computer device determines the status detection results of the M nodes based on the S response messages.

[0122] In one embodiment, a computer device, based on the search results of N data points in a database, counts the number of expected response messages among S response messages; any expected response message contains a search result that matches the search result of the corresponding data in the database. Search result matching can be understood as: the search result for the same data in any node is consistent with the search result in the database. In an approximate search scenario, search result matching can be understood as: the search results in the same database to be searched are included in the search set of any node in the search cluster, and this search set contains at least two search results that are approximately similar to the data to be searched (similarity greater than a similarity threshold).

[0123] In one implementation, N data points are vectors to be retrieved; the i-th response is obtained by retrieving the j-th data point from the h-th node, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N. If the i-th response does not match the retrieval result of the j-th data point in the database, the computer device obtains a candidate vector set for the j-th data point from the h-th node; the candidate vector set includes at least two candidate vectors (i.e., expanding the recall of the h-th node), and the similarity between any candidate vector and the j-th data point is higher than a similarity threshold. If there is a candidate vector in the candidate vector set of the j-th data point that matches the retrieval result in the database, the computer device determines that the i-th response is as expected; conversely, if there is no candidate vector in the candidate vector set of the j-th data point that matches the retrieval result in the database, the computer device determines that the i-th response is not as expected. In other words, in approximate retrieval scenarios, if the retrieval results for the target data at a certain node are inconsistent with the retrieval results in the database, the recall of the retrieval results at that node can be increased. After increasing the recall of retrieval results, if there are retrieval results among the recalled multiple retrieval results that are consistent with the retrieval results in the database, then the response information is determined to meet expectations; otherwise, the response information is determined to not meet expectations.

[0124] In another implementation, N data points form a vector to be retrieved; the i-th response is obtained by the h-th node after retrieving the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N. The process by which the computer device determines the status detection results of the M nodes based on the S response points includes: if the i-th response does not match the retrieval result of the j-th data point in the database, the computer device refreshes the database and, after refreshing, continues to check whether the i-th response matches the retrieval result of the j-th data point in the refreshed database. If the i-th response matches the retrieval result of the j-th data point in the refreshed database, the computer device determines that the i-th response meets expectations; conversely, if the i-th response does not match the retrieval result of the j-th data point in the refreshed database, the computer device determines that the i-th response does not meet expectations.

[0125] After counting the number of responses that meet the expectations out of the S responses, the computer calculates the matching rate of the M nodes using the number of responses that meet the expectations and the total number of responses S. This can be expressed as:

[0126]

[0127] Among them, R C N represents the matching rate of the results. C Let S be the number of responses that meet the expectations out of the S responses, where S is the total number of responses.

[0128] Furthermore, if the matching rate is greater than or equal to the first probability threshold (e.g., 100%), the computer device calculates the availability of M nodes based on the total amount of response information S and the total amount of retrieval requests M×N, which can be specifically expressed as:

[0129]

[0130] Among them, R S Let S be the availability rate, S be the total number of response messages, and M×N be the total number of retrieval requests sent by the computer device. If the availability rate is greater than or equal to the second probability threshold, the computer device determines that the retrieval results of M nodes for the same data match.

[0131] It is understandable that the two implementation methods mentioned above can be combined, and the above state detection method can be applied to the data operated on at any time in the node. By periodically detecting the state, problems such as failure to add data or loss of data can be detected in a timely manner.

[0132] Figure 4b This is a schematic diagram of a state detection method provided in an embodiment of this application. Figure 4b As shown, data preparation begins first. Taking a vector retrieval scenario as an example, the computer device can retrieve recently added or deleted vectors (such as vector A) from the database. Assuming a complete check takes 10 minutes, a check can be performed every 10 minutes, and the computer device should retrieve vectors added / deleted more than 10 minutes ago each time, ensuring that every added / deleted vector is detected. Next, the computer device sends retrieval requests to all nodes in the retrieval cluster. It then checks whether vector A retrieved from the database can be retrieved by each node. Finally, the computer device calculates the matching rate based on the response information from each node. The matching rate is calculated based on the expected response information and the total number of response information; "expected" means that if a vector exists in the database, the response information returned by each node should include that vector; if it does not exist in the database, the response information returned by each node should not include that vector. For example, if the Top 1 search result of retrieval process 1 matches the database, then retrieval process 1 is determined to match the database status. Although the Top 1 search result of retrieval process 2 is inconsistent with the database, vector A is included in the Top 10 search results of retrieval process 2, so retrieval process 2 is still determined to match the database status. If the Top 1 search results of retrieval processes 2 and 3 are inconsistent with the database, and vector A is not included in the Top 10 search results of retrieval process 3, then the computer device can continue to detect whether vector A is included in the Top 100 of retrieval process 3 or refresh the database, and compare the search results of vector A in retrieval process 3 and the refreshed database again. Further, if the matching rate passes the detection, the computer device can further calculate the availability rate. The specific implementation method can refer to the above formula (2), which will not be repeated here. If the availability rate also passes the detection, the computer device determines that the status of M nodes matches. Otherwise, if either the matching rate or the availability rate has abnormal fluctuations, an investigation is required. The computer device can also report the status detection results each time and execute the above steps in a loop. Similarly, the existing data in the nodes are also tested using the above method. Since the amount of existing data is large and the inspection takes a long time, the computer equipment can randomly sample a portion of the existing data for inspection each time.

[0133] Optionally, computer equipment can also assist in retrieving data from individual nodes in the cluster for backup or state recovery. Figure 4c This is a schematic diagram of a control architecture provided for an embodiment of this application. Figure 4cAs shown, the computer device can act as a central control node, taking turns masking nodes in the retrieval cluster (masked nodes do not need to share retrieval traffic), so that the masked nodes back up the data in their memory (obtained from consuming queue elements) and element indication information to their own disks. The element indication information is used to determine the first queue element to consume after a node performs state recovery based on the backup data. For example, assuming the message queue includes queue elements 1-5, and node A has successfully consumed queue elements 1-3, node A backs up the element indication information and the data in memory to disk; at this time, the queue element indicated by the element indication information is queue element 4, meaning that after node A performs state recovery based on the backup data, it will use queue element 4 as the first queue element to consume. Correspondingly, during the node's state recovery process, the computer device obtains queue element requests sent by the nodes. These queue element requests include element indication information, which indicates the target queue element in the message queue. The computer device returns the target queue element to the sender of the queue element request, enabling the sender to perform state recovery based on the target queue element.

[0134] exist Figure 2 Building upon the previous implementation, by expanding the nodes to be expanded (such as a single-machine search engine), the limited search volume supported by a single-machine search engine can be resolved while ensuring the state matching of each node in the search cluster. Furthermore, the operation and maintenance methods for each node in the search cluster are universal as those for stateless services, eliminating the need for a separate operation and maintenance platform and commands, making it more convenient and flexible. Using database data as a benchmark, real-time monitoring of whether the state of each node matches the benchmark can promptly detect inconsistencies such as failed additions and data loss.

[0135] Please see Figure 5 , Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device; for example, by... Figure 1 The node 102 shown is executed.

[0136] like Figure 3 As shown, the data processing method may include the following steps S501-S503:

[0137] S501: Retrieve data operation requests to be consumed from the message queue.

[0138] The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1. The data operation request is used to update the retrieval cluster, which consists of M nodes, each corresponding to a message queue and acting as a replica of the others; M is an integer greater than 1. The data operation request can include data indication information and operation indication information. The data indication information indicates the data to be operated on, while the operation indication information indicates the specific operation, such as adding, deleting, or modifying.

[0139] In one implementation, the data operation request is the first queue element added to the message queue (i.e., k=2). After the data operation request is cached in the message queue, the message queue can send the data operation request to each node in the retrieval cluster without obtaining consumption confirmation information.

[0140] In another implementation, the data operation request is not the first queue element added to the message queue (i.e., k is greater than 2). The computer device needs to send the consumption confirmation information (or element indication information) of the (k-2)th queue element to the message queue in order to retrieve the data operation request from the message queue.

[0141] S502, Consume the data operation request to be consumed.

[0142] In one implementation, the data operation request includes data indication information and operation indication information, and the computer device operates on the target data in memory according to the operation indication information; wherein, the target data is the data indicated by the data indication information.

[0143] S503. In response to the successful consumption of the data operation request to be consumed, return consumption confirmation information to the message queue so that the message queue provides the kth queue element added to the message queue.

[0144] The consumption confirmation message is used to indicate that the (k-1)th queue element has been successfully consumed by the computer device (i.e., the current node).

[0145] In one implementation, the computer device can also copy backup data and element indication information from the backup node to a disk. In the event of a status malfunction or the need for status recovery, the computer device can recover using the backup data and element indication information on the disk. The backup data is obtained by the backup node storing data in its memory, which is obtained by the backup node consuming queue elements in the message queue. The element indication information indicates the target queue element in the message queue; the target queue element is the first queue element consumed after loading the backup data. The backup node can be any one of the M nodes; it can also be a dedicated node that consumes queue elements in the message queue and periodically backs them up, without providing retrieval services.

[0146] In one embodiment, nodes in the retrieval cluster can copy backup data and element indication information from backup nodes in batches to avoid congestion. Specifically, the computer device determines the data backup time based on the number of nodes (M) corresponding to the message queue and a scale parameter; where the scale parameter indicates the number of nodes performing backups in the same batch. In one implementation, the computer device obtains the start time and copy duration; where the start time is the time when the backup node completes the backup, and the copy duration indicates the duration for copying backup data and element indication information from the backup node each time. Then, the computer device determines the synchronization batch using the number of nodes and the scale parameter, and calculates the data backup time (i.e., the time when copying backup data and element indication information from the backup node begins) based on the start time, synchronization batch, and copy duration. This can be specifically expressed as:

[0147]

[0148] Among them, T d Indicates the time when the backup node starts backing up, Δ d T represents the time required to back up the backup node. d +Δ d This is the time it takes for the backup node to complete the backup. Rand() is a random number function used to generate... A random integer, M is the number of nodes in the retrieval cluster, and N is the number of nodes in the retrieval cluster. T For the scale parameter, Δ w This is the copy duration.

[0149] Furthermore, when the current time matches the data backup time, the computer device copies the backup data and element indication information from the backup node to the disk. According to the above implementation, the computer device can periodically copy backup data from the backup node. In one embodiment, assuming the time interval for the backup node to perform data backup is G, the time interval for each node to copy backup data and element indication information from the backup node can also be G, in conjunction with the above formula (3), In addition, G is less than the time threshold for caching queue elements in the message queue, which ensures that each node can recover data after the most recent backup time point through the queue elements cached in the message queue.

[0150] Figure 6a This is a data synchronization architecture diagram provided for an embodiment of this application. For example... Figure 6a As shown, each node in the retrieval cluster is assigned to copy backup data and element indication information from the backup node. Compared to Figure 4c In terms of the architecture, this method is decentralized, and each node can still provide retrieval services normally while copying backup data and element indication information.

[0151] In another implementation, in response to the fulfillment of the state recovery conditions, the computer device loads the backup data from the disk into its own memory. The computer device then determines the first queue element to be consumed after loading is complete based on element indication information, and continues to consume queue elements in the message queue to ensure that the final state matches that of other nodes in the retrieval cluster.

[0152] Figure 6b This is a schematic diagram illustrating a node state recovery process provided in an embodiment of this application. Figure 6b As shown, when a node (retrieval replica) is in a normal state, regular data backups are necessary. Assuming the message queue offset is P at the time of the node's last data backup, the node will generate element indication information based on offset P while backing up the data in memory. When a node malfunctions (e.g., due to scaling down, changes, or going offline), data writing to the message queue is unaffected (i.e., data operation requests can still be cached in the message queue). During recovery, the node first loads the backup data from disk into memory, and then determines the first queue element to consume after loading based on the element indication information. For example, when using Pulsar as the message queue implementation component, the seek interface can be used to set the node's offset to P, and the node will consume data sequentially from offset P onwards. This ensures that the recovered node matches the state of other nodes in the retrieval cluster.

[0153] In this embodiment, a data operation request to be consumed is obtained from the message queue. This request is the (k-1)th queue element added to the message queue. The request is then consumed. Upon successful consumption, a consumption confirmation message is returned to the message queue, enabling the message queue to provide the kth queue element added to it. Thus, the message queue and consumption confirmation message ensure that the order in which the request is consumed is consistent with the order of requests from other nodes, thereby ensuring that the updated states of all nodes in the retrieval cluster match. Furthermore, by using backup data and element indication information, node state recovery can be performed, ensuring that the recovered node matches the state of other nodes in the retrieval cluster.

[0154] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0155] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 7 The data processing device shown can be mounted in a computer device, which can specifically be a terminal device or a server. Figure 7 The data processing apparatus shown can be used to perform the above. Figure 2 and Figure 3 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 7 The data processing device includes:

[0156] The acquisition unit 701 is used to acquire data operation requests. The data operation requests are used to update the retrieval cluster, which includes M nodes, where M is an integer greater than 1.

[0157] Processing unit 702 is used to add data operation requests to the message queues corresponding to M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1.

[0158] The sending unit 703 is used to send a data operation request to any node in response to the consumption confirmation information of the (k-1)th queue element sent by any node, so that the node can consume the data operation request; the consumption confirmation information of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

[0159] In one implementation, the data operation request includes data indication information and operation indication information; the processing unit 702 is used to add the data operation request to the message queues corresponding to M nodes, specifically for:

[0160] In the database, the target data is operated on according to the operation instructions. The target data is the data indicated by the data instructions.

[0161] In response to the completion of the operation on the target data, the data operation request is added to the message queues corresponding to M nodes.

[0162] In one embodiment, the processing unit 702 is further configured to:

[0163] Based on the database, the state of M nodes is periodically checked to obtain the state check results of M nodes;

[0164] The status detection result is used to indicate whether the search results for the same data in the database and the M nodes match.

[0165] In one implementation, the processing unit 702 is configured to periodically perform state checks on M nodes based on a database to obtain the state check results of the M nodes, specifically for:

[0166] Get the data set to be retrieved, which includes N data items, where N is a positive integer;

[0167] Send N data retrieval requests to each node and obtain S response messages from M nodes, where S is a natural number less than or equal to M×N;

[0168] Based on S response messages, determine the state detection results of M nodes.

[0169] In one implementation, each response message includes a retrieval result of data in a node; the processing unit 702 is configured to determine the state detection results of M nodes based on the S response messages, specifically for:

[0170] Based on the search results of N data in the database, count the number of expected response messages among S response messages; any expected response message contains search results that match the corresponding data in the database.

[0171] The matching rate of the results for M nodes is calculated by the number of responses that meet the expectations and the total number of responses, S.

[0172] If the matching rate is greater than or equal to the first probability threshold, the availability of M nodes is calculated based on the total amount of response information S and the total amount of retrieval requests M×N.

[0173] If the availability rate is greater than or equal to the second probability threshold, then the search results of M nodes for the same data are determined to be a match.

[0174] In one implementation, N data points are the vectors to be retrieved; the i-th response information is obtained after the h-th node retrieves the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processing unit 702 counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0175] If the ith response information does not match the retrieval result of the jth data in the database, then obtain the candidate vector set of the jth data from the hth node; the candidate vector set includes at least two candidate vectors, and the similarity between any candidate vector and the jth data is greater than the similarity threshold.

[0176] If there is a candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to meet expectations.

[0177] If there is no candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to be unsatisfactory.

[0178] In one implementation, N data points are the vectors to be retrieved; the i-th response information is obtained after the h-th node retrieves the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processing unit 702 counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0179] If the i-th response does not match the retrieval result of the j-th data in the database, then the database is refreshed.

[0180] If the i-th response information matches the search result of the j-th data in the refreshed database, then the i-th response information is determined to meet expectations.

[0181] If the i-th response information does not match the search result of the j-th data in the refreshed database, then the i-th response information is determined to be unsatisfactory.

[0182] In one implementation, consuming a data operation request by any node means that the node updates the memory according to the data operation request; the processing unit 702 is further configured to:

[0183] M nodes are blocked in turn according to a preset order and preset time, so that the blocked nodes store element indication information on the disk and data in memory;

[0184] The element indication information is used to indicate the target queue element in the message queue. The target queue element is the first queue element consumed by the blocked node after loading the memory data stored on the disk.

[0185] In one embodiment, the processing unit 702 is further configured to:

[0186] A queue element request is retrieved. The queue element request includes element indication information, which is used to indicate the target queue element in the message queue.

[0187] Return the target queue element to the sender of the queue element request so that the sender can restore the state based on the target queue element.

[0188] In one embodiment, the processing unit 702 is further configured to:

[0189] Obtain the baseline database for the nodes to be expanded;

[0190] The baseline database is imported into the memory of M-1 initialization nodes to obtain the retrieval cluster corresponding to the node to be expanded. The retrieval cluster includes the node to be expanded and M-1 replica nodes of the node to be expanded.

[0191] Establish the correspondence between the retrieval cluster and the message queue.

[0192] According to one embodiment of this application, Figure 2 and Figure 3 The data processing method shown may involve some steps that can be derived from... Figure 7 The data processing apparatus shown is executed by each unit within it. For example, Figure 2 Steps S201 and S202 shown can be derived from... Figure 7 The acquisition unit 701 shown is executed, and step S202 can be performed by... Figure 7 The processing unit 702 shown executes the step S203, which can be performed by... Figure 7 The transmitting unit 703 shown is executed; Figure 3 Step S303 shown can be performed by Figure 7 The acquisition unit 701 shown is executed, and steps S301, S302, S304, and S306 can be performed by... Figure 7 The processing unit 702 shown executes the step S305, which can be performed by... Figure 7 The sending unit 703 shown is executed. Figure 7The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0193] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 2 and Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0194] Based on the same inventive concept, the principle and beneficial effects of the data processing device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0195] Please see Figure 8 , Figure 8 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Figure 8 The data processing device shown can be mounted in a computer device, which can specifically be a terminal device or a server. Figure 8 The data processing apparatus shown can be used to perform the above. Figure 5 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 8 The data processing device includes:

[0196] The acquisition unit 801 is used to acquire data operation requests to be consumed from the message queue. The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1.

[0197] Processing unit 802 is used to process data operation requests to be processed.

[0198] Sending unit 803 is used to respond to the successful consumption of the data operation request to be consumed by returning consumption confirmation information to the message queue, so that the message queue provides the kth queue element added to the message queue.

[0199] In one embodiment, the processing unit 802 is further configured to:

[0200] The data backup time is determined based on the number of nodes and the size parameter corresponding to the message queue. The size parameter is used to indicate the number of nodes to be backed up in the same batch.

[0201] When the current time matches the data backup time, copy the backup data and element indication information from the backup node to the disk;

[0202] Among them, the backup data is obtained by the backup node storing the data in memory, and the data in the backup node's memory is obtained by the backup node consuming the queue elements in the message queue; the element indication information is used to indicate the target queue element in the message queue, and the target queue element is: the first queue element consumed after loading the backup data.

[0203] In one implementation, the processing unit 802 is configured to determine the data backup time based on the number of nodes and scale parameters corresponding to the message queue, specifically for:

[0204] Get the start time and copy duration. The start time is the time when the backup node completes the backup. The copy duration is used to indicate the duration of each copy of backup data and element indication information from the backup node.

[0205] The synchronization batch is determined by the number and size parameters of the nodes;

[0206] The data backup time is calculated based on the start time, synchronization batch, and copy duration.

[0207] According to one embodiment of this application, Figure 5 The data processing method shown may involve some steps that can be derived from... Figure 8 The data processing apparatus shown is executed by each unit within it. For example, Figure 5 Step S501 shown can be performed by Figure 8 The acquisition unit 801 shown is executed, and step S502 can be performed by... Figure 8 The processing unit 802 shown executes the step S503, which can be performed by... Figure 8 The sending unit 803 shown is executed. Figure 8The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0208] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 5 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 8 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0209] Based on the same inventive concept, the principle and beneficial effects of the data processing device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0210] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 9As shown, the computer device includes at least a processor 901, a communication interface 902, and a memory 903. The processor 901, communication interface 902, and memory 903 can be connected via a bus or other means. The processor 901 (or Central Processing Unit, CPU) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data. For example, the CPU can parse power-on / off commands issued by objects to the computer device and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device, and so on. The communication interface 902 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.), and under the control of the processor 901, it can be used to send and receive data; the communication interface 902 can also be used for data transmission and interaction within the computer device. The memory 903 is the storage device in the computer device, used to store programs and data. It can be understood that the memory 903 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 903 provides storage space for storing the operating system of the computer device, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0211] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by the processor 901. It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0212] In one embodiment, processor 901 performs the following operations by running a computer program stored in memory 903:

[0213] Get a data operation request. The data operation request is used to update the retrieval cluster, which consists of M nodes, where M is an integer greater than 1.

[0214] Add the data operation request to the message queues corresponding to M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1.

[0215] In response to the consumption confirmation message of the (k-1)th queue element sent by any node, a data operation request is sent to that node so that the node can consume the data operation request; the consumption confirmation message of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

[0216] As an optional embodiment, the data operation request includes data indication information and operation indication information; a specific embodiment in which the processor 901 adds the data operation request to the message queues corresponding to the M nodes is as follows:

[0217] In the database, the target data is operated on according to the operation instructions. The target data is the data indicated by the data instructions.

[0218] In response to the completion of the operation on the target data, the data operation request is added to the message queues corresponding to M nodes.

[0219] As an optional embodiment, the processor 901, by running a computer program in the memory 903, also performs the following operations:

[0220] Based on the database, the state of M nodes is periodically checked to obtain the state check results of M nodes;

[0221] The status detection result is used to indicate whether the search results for the same data in the database and the M nodes match.

[0222] As an optional embodiment, the processor 901 periodically performs state checks on M nodes based on a database to obtain the state check results of the M nodes. A specific embodiment of this is as follows:

[0223] Get the data set to be retrieved, which includes N data items, where N is a positive integer;

[0224] Send N data retrieval requests to each node and obtain S response messages from M nodes, where S is a natural number less than or equal to M×N;

[0225] Based on S response messages, determine the state detection results of M nodes.

[0226] As an optional embodiment, each response message includes a retrieval result of data in a node; the processor 901 determines the state detection results of M nodes based on S response messages in the following specific embodiment:

[0227] Based on the search results of N data in the database, count the number of expected response messages among S response messages; any expected response message contains search results that match the corresponding data in the database.

[0228] The matching rate of the results for M nodes is calculated by the number of responses that meet the expectations and the total number of responses, S.

[0229] If the matching rate is greater than or equal to the first probability threshold, the availability of M nodes is calculated based on the total amount of response information S and the total amount of retrieval requests M×N.

[0230] If the availability rate is greater than or equal to the second probability threshold, then the search results of M nodes for the same data are determined to be a match.

[0231] As an optional embodiment, N data points form a retrieval vector; the i-th response information is obtained by the h-th node after retrieving the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processor 901 counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0232] If the ith response information does not match the retrieval result of the jth data in the database, then obtain the candidate vector set of the jth data from the hth node; the candidate vector set includes at least two candidate vectors, and the similarity between any candidate vector and the jth data is greater than the similarity threshold.

[0233] If there is a candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to meet expectations.

[0234] If there is no candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to be unsatisfactory.

[0235] As an optional embodiment, N data points form a retrieval vector; the i-th response information is obtained by the h-th node after retrieving the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process by which the processor 901 counts the number of expected response information points among the S response information points based on the retrieval results of the N data points in the database includes:

[0236] If the i-th response does not match the retrieval result of the j-th data in the database, then the database is refreshed.

[0237] If the i-th response information matches the search result of the j-th data in the refreshed database, then the i-th response information is determined to meet expectations.

[0238] If the i-th response information does not match the search result of the j-th data in the refreshed database, then the i-th response information is determined to be unsatisfactory.

[0239] As an optional embodiment, for any node to consume a data operation request means that the node updates the memory according to the data operation request; the processor 901, by running the computer program in the memory 903, also performs the following operations:

[0240] M nodes are blocked in turn according to a preset order and preset time, so that the blocked nodes store element indication information on the disk and data in memory;

[0241] The element indication information is used to indicate the target queue element in the message queue. The target queue element is the first queue element consumed by the blocked node after loading the memory data stored on the disk.

[0242] As an optional embodiment, the processor 901, by running a computer program in the memory 903, also performs the following operations:

[0243] A queue element request is retrieved. The queue element request includes element indication information, which is used to indicate the target queue element in the message queue.

[0244] Return the target queue element to the sender of the queue element request so that the sender can restore the state based on the target queue element.

[0245] As an optional embodiment, the processor 901, by running a computer program in the memory 903, also performs the following operations:

[0246] Obtain the baseline database for the nodes to be expanded;

[0247] The baseline database is imported into the memory of M-1 initialization nodes to obtain the retrieval cluster corresponding to the node to be expanded. The retrieval cluster includes the node to be expanded and M-1 replica nodes of the node to be expanded.

[0248] Establish the correspondence between the retrieval cluster and the message queue.

[0249] In another embodiment, processor 901 performs the following operations by running a computer program stored in memory 903:

[0250] Retrieve data operation requests to be consumed from the message queue. The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1.

[0251] Consume the data operation requests to be consumed;

[0252] In response to a successful consumption of a data operation request, a consumption confirmation message is returned to the message queue, enabling the message queue to provide the k-th queue element that has been added to the message queue.

[0253] As an optional embodiment, the processor 901, by running a computer program in the memory 903, also performs the following operations:

[0254] The data backup time is determined based on the number of nodes and the size parameter corresponding to the message queue. The size parameter is used to indicate the number of nodes to be backed up in the same batch.

[0255] When the current time matches the data backup time, copy the backup data and element indication information from the backup node to the disk;

[0256] Among them, the backup data is obtained by the backup node storing the data in memory, and the data in the backup node's memory is obtained by the backup node consuming the queue elements in the message queue; the element indication information is used to indicate the target queue element in the message queue, and the target queue element is: the first queue element consumed after loading the backup data.

[0257] As an optional embodiment, the processor 901 determines the data backup time based on the number of nodes and scale parameters corresponding to the message queue in the following specific embodiment:

[0258] Get the start time and copy duration. The start time is the time when the backup node completes the backup. The copy duration is used to indicate the duration of each copy of backup data and element indication information from the backup node.

[0259] The synchronization batch is determined by the number and size parameters of the nodes;

[0260] The data backup time is calculated based on the start time, synchronization batch, and copy duration.

[0261] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.

[0262] This application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute the data processing method described in the above method embodiments.

[0263] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described above.

[0264] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0265] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0266] In the embodiments of this application, the term "module" or "unit" refers to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0267] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0268] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a data operation request, the data operation request being used to update the retrieval cluster, the retrieval cluster comprising M nodes, where M is an integer greater than 1; The data operation request is added to the message queues corresponding to the M nodes. The data operation request is the kth queue element added to the message queue, where k is an integer greater than 1. In response to the consumption confirmation information of the (k-1)th queue element sent by any node, the data operation request is sent to that node so that the node consumes the data operation request; the consumption confirmation information of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

2. The method as described in claim 1, characterized in that, The data operation request includes data indication information and operation indication information; adding the data operation request to the message queues corresponding to the M nodes includes: The target data is operated on in the database according to the operation instruction information, where the target data is the data indicated by the data instruction information; In response to the completion of the operation on the target data, the data operation request is added to the message queues corresponding to the M nodes.

3. The method as described in claim 2, characterized in that, The method further includes: Based on the database, the state of the M nodes is periodically checked to obtain the state check results of the M nodes; The status detection result is used to indicate whether the retrieval results of the database and the M nodes for the same data match.

4. The method as described in claim 3, characterized in that, The step of periodically performing state checks on the M nodes based on the database to obtain the state check results of the M nodes includes: Obtain the data set to be retrieved, which includes N data items, where N is a positive integer; Send retrieval requests for the N data to each node respectively, and obtain S response messages returned by the M nodes, where S is a natural number less than or equal to M×N; Based on the S response information, the state detection results of the M nodes are determined.

5. The method as described in claim 4, characterized in that, Each response message includes a retrieval result of data in a node; the determination of the state detection results of the M nodes based on the S response messages includes: Based on the search results of the N data in the database, the number of expected response information among the S response information is counted; the search results contained in any expected response information match the search results of the corresponding data in the database; The matching rate of the M nodes is calculated by the number of expected response messages and the total number S of the response messages. If the result matching rate is greater than or equal to the first probability threshold, then the availability rate of the M nodes is calculated based on the total amount S of the response information and the total amount M×N of the retrieval requests. If the availability rate is greater than or equal to the second probability threshold, then the search results of the M nodes for the same data are determined to be matched.

6. The method as described in claim 5, characterized in that, The N data points are the vectors to be retrieved; the i-th response information is obtained after the h-th node retrieves the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process of counting the number of expected response information among the S response information points based on the retrieval results of the N data points in the database includes: If the ith response information does not match the retrieval result of the jth data in the database, then a candidate vector set for the jth data is obtained from the hth node; the candidate vector set includes at least two candidate vectors, and the similarity between any candidate vector and the jth data is greater than the similarity threshold. If there is a candidate vector in the candidate vector set of the j-th data that matches the search result in the database, then the i-th response information is determined to meet expectations; If there is no candidate vector in the candidate vector set of the j-th data that matches the search results in the database, then the i-th response information is determined to be unsatisfactory.

7. The method as described in claim 5, characterized in that, The N data points are the vectors to be retrieved; the i-th response information is obtained after the h-th node retrieves the j-th data point, where i is a positive integer less than or equal to S, h is a positive integer less than or equal to M, and j is a positive integer less than or equal to N; the process of counting the number of expected response information among the S response information points based on the retrieval results of the N data points in the database includes: If the i-th response information does not match the retrieval result of the j-th data in the database, then the database is refreshed; If the i-th response information matches the search result of the j-th data in the refreshed database, then the i-th response information is determined to meet expectations; If the i-th response information does not match the retrieval result of the j-th data in the refreshed database, then the i-th response information is determined to be unsatisfactory.

8. The method as described in claim 1, characterized in that, Consuming the data operation request by any node means that the node updates the memory according to the data operation request; the method further includes: The M nodes are blocked in turn according to a preset order and a preset time, so that the blocked nodes store element indication information on the disk and data in memory. The element indication information is used to indicate the target queue element in the message queue. The target queue element is the first queue element consumed by the blocked node after loading the memory data stored on the disk.

9. The method as described in claim 1, characterized in that, The method further includes: A queue element request is retrieved, the queue element request including element indication information, the element indication information being used to indicate a target queue element in the message queue; The sender of the queue element request returns the target queue element so that the sender of the queue element request can perform state recovery based on the target queue element.

10. The method as described in claim 1, characterized in that, The method further includes: Obtain the baseline database for the nodes to be expanded; The baseline database is imported into the memory of M-1 initialization nodes to obtain the retrieval cluster corresponding to the node to be expanded. The retrieval cluster includes the node to be expanded and M-1 replica nodes of the node to be expanded. Establish the correspondence between the retrieval cluster and the message queue.

11. A data processing method, characterized in that, The method includes: Retrieve data operation requests to be consumed from the message queue, wherein the data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1; Consume the data operation requests to be consumed; In response to the successful consumption of the data operation request to be consumed, a consumption confirmation message is returned to the message queue so that the message queue provides the kth queue element added to the message queue.

12. The method as described in claim 11, characterized in that, The method further includes: The data backup time is determined based on the number of nodes and the scale parameter corresponding to the message queue, wherein the scale parameter is used to indicate the number of nodes to be backed up in the same batch; When the current time matches the data backup time, copy the backup data and element indication information from the backup node to the disk; The backup data is obtained by the backup node storing data in memory, and the data in the backup node's memory is obtained by the backup node consuming queue elements in the message queue; the element indication information is used to indicate the target queue element in the message queue, and the target queue element is the first queue element consumed after loading the backup data.

13. The method as described in claim 12, characterized in that, The step of determining the data backup time based on the number and size parameters of the nodes corresponding to the message queue includes: The start time and copy duration are obtained, wherein the start time is the time when the backup node completes the backup, and the copy duration is used to indicate the duration of each copy of backup data and element indication information from the backup node; The synchronization batch is determined by the number and size parameters of the nodes; The data backup time is calculated based on the start time, the synchronization batch, and the copy duration.

14. A data processing apparatus, characterized in that, The data processing device includes: The acquisition unit is used to acquire data operation requests, which are used to update the retrieval cluster, which includes M nodes, where M is an integer greater than 1. The processing unit is used to add the data operation request to the message queues corresponding to the M nodes, wherein the data operation request is the kth queue element added to the message queue, and k is an integer greater than 1; The sending unit is configured to send the data operation request to any node in response to the consumption confirmation information of the (k-1)th queue element sent by any node, so that the node consumes the data operation request; the consumption confirmation information of the (k-1)th queue element is used to indicate that the (k-1)th queue element has been successfully consumed by the node.

15. A data processing apparatus, characterized in that, The data processing device includes: The acquisition unit is used to acquire data operation requests to be consumed from the message queue. The data operation request to be consumed is the (k-1)th queue element added to the message queue, where k is an integer greater than 1. The processing unit is used to consume the data operation requests to be consumed; The sending unit is configured to respond to the successful consumption of the data operation request to be consumed by returning consumption confirmation information to the message queue, so that the message queue provides the kth queue element added to the message queue.

16. A computer device, characterized in that, include: A memory, wherein a computer program is stored; A processor is configured to load the computer program to implement the data processing method as described in any one of claims 1-10; or, to load the computer program to implement the data processing method as described in any one of claims 11-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the data processing method as described in any one of claims 1-10; or, adapted to be loaded by a processor and executed by the data processing method as described in any one of claims 11-13.

18. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the data processing method as described in any one of claims 1-10; or, adapted to be loaded by a processor and execute the data processing method as described in any one of claims 11-13.

Images