Data synchronization methods, devices, equipment, and computer storage media for retrieval

Abstract

This application discloses a data synchronization method, apparatus, device, and computer storage medium, applicable to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving. By deploying write operation-related services in a feature management cluster and retrieval operation-related services in a feature retrieval cluster, the two clusters are physically isolated and each maintains its own associated database, thereby avoiding the reuse of a single retrieval database in the entire retrieval system and improving the reliability of the retrieval system. Simultaneously, after performing a write operation, the feature management cluster synchronizes the updated features to the feature retrieval cluster through write diffusion, enabling the feature retrieval cluster to synchronously update its own database. To ensure the reliability of synchronization, each retrieval node subscribes to write operation transactions in the feature management database, using transaction information as a fallback strategy to guarantee the reliability of synchronization between the feature management database and the feature retrieval database.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of retrieval technology, providing a retrieval data synchronization method, apparatus, device, and computer storage medium. Background Technology

[0002] Artificial intelligence (AI) technology has wide applications in the field of content understanding. For example, AI technology plays a significant role in image content moderation. Image content moderation refers to reviewing uploaded images to determine if they contain illegal or inappropriate content. The most common method for image content moderation is to extract features from images containing illegal or inappropriate content, store these features in a database, and then perform feature retrieval in the database based on the extracted features from the uploaded image.

[0003] A typical retrieval system consists of three parts: image input, software development kit (SDK) processing, and retrieval service. SDK processing mainly includes image size control, encoding, and feature extraction. The retrieval service is the core of the entire retrieval system, mainly used to manage the feature database, such as implementing operations like adding, deleting, or modifying features, and also used for feature retrieval, that is, performing feature matching based on features in the feature database to output retrieval results.

[0004] It is evident that the current retrieval system reuses a single retrieval database, resulting in significant retrieval pressure and increased time consumption during high-concurrency retrieval requests. Furthermore, network or disk failures in the retrieval database will directly render the retrieval system unusable, reducing the efficiency and availability of content review and retrieval. Summary of the Invention

[0005] This application provides a data synchronization method, apparatus, device, and computer storage medium for improving the reliability of the retrieval system, thereby enhancing retrieval efficiency and usability.

[0006] On the one hand, a data synchronization method for retrieval is provided, applied to a feature management cluster included in a retrieval system, wherein the retrieval system further includes a feature retrieval cluster composed of at least one retrieval node; the method includes:

[0007] In response to a write operation request sent by the client, the corresponding write operation is performed on the associated feature management database, and the corresponding operation transaction information is recorded in the operation log of the feature management database.

[0008] Write diffusion requests are initiated to at least one retrieval node, each carrying feature information corresponding to the write operation, enabling each retrieval node to synchronously update the associated feature retrieval database based on the feature information; and...

[0009] When each retrieval node subscribes to the write operation transaction of the feature management database, it sends the operation transaction information to each retrieval node, so that if the synchronization update by each retrieval node fails, it performs a synchronization operation on the feature retrieval database based on the operation transaction information.

[0010] On the one hand, a data synchronization method is provided, applied to any retrieval node in a feature retrieval cluster included in a retrieval system, wherein the retrieval system further includes a feature management cluster; the method includes:

[0011] Receive a write diffusion request sent by the feature management cluster. The write diffusion request carries feature information corresponding to the write operation of the feature management cluster to the associated feature management database. The write operation is triggered by a write operation request sent by the client.

[0012] Based on the aforementioned feature information, the associated feature retrieval database is synchronously updated; and,

[0013] The system receives operation transaction information sent by the feature management cluster, which records the transaction information corresponding to the write operation.

[0014] If it is determined that the synchronization update is unsuccessful, a synchronization operation is performed on the feature retrieval database based on the operation transaction information.

[0015] On the one hand, a data synchronization device is provided, applied in a feature management cluster included in a retrieval system, wherein the retrieval system further includes a feature retrieval cluster composed of at least one retrieval node; the device includes:

[0016] The feature management unit is used to respond to write operation requests sent by the client, perform corresponding write operations on the associated feature management database, and record the corresponding operation transaction information in the operation log of the feature management database.

[0017] A write diffusion unit is configured to initiate write diffusion requests to the at least one retrieval node, wherein the write diffusion request carries feature information corresponding to the write operation, enabling each retrieval node to synchronously update the associated feature retrieval database based on the feature information; and

[0018] The transaction push unit is used to send the operation transaction information to each retrieval node when each retrieval node subscribes to the write operation transaction of the feature management database, so that if the synchronization update by each retrieval node fails, the feature retrieval database is synchronized based on the operation transaction information.

[0019] Optionally, the device further includes a configuration unit for:

[0020] Configure the transaction mailbox corresponding to the management database. The transaction mailbox includes a message relay module and a transaction message queue. The message relay module subscribes to updates of the operation log.

[0021] Receive a subscription request sent by the at least one retrieval node, the subscription request being used to request subscription to the write operation transaction of the feature management database;

[0022] In response to each subscription request, configure each retrieval node to subscribe to the transaction message queue.

[0023] Optionally, the transaction push unit is specifically used for:

[0024] The operation transaction information is obtained from the operation log through the message relay module, and the operation transaction information is written into the transaction message queue.

[0025] Based on each retrieval node's subscription to the transaction message queue, the operation transaction information is obtained from the transaction message queue, and the operation transaction information is sent to each retrieval node respectively.

[0026] Optionally, the device further includes an initial synchronization unit for:

[0027] Receive an initial synchronization request sent by a newly added retrieval node in the feature retrieval cluster, wherein the initial synchronization request carries the target synchronization method determined by the newly added retrieval node;

[0028] If the target synchronization method is a synchronization method based on operation logs, then the operation logs are sent to the newly added retrieval node;

[0029] If the target synchronization method is a database snapshot-based synchronization method, then determine whether the device's resource status meets the resource requirements of that synchronization method;

[0030] If the conditions are met, a snapshot persistence operation is performed to generate a database snapshot of the feature management database, and the database snapshot is sent to the newly added retrieval node.

[0031] If the conditions are not met, the target synchronization method is changed to a synchronization method based on operation logs, and the operation logs are sent to the newly added retrieval node.

[0032] Optionally, the initial synchronization unit is further configured to:

[0033] During the snapshot persistence operation, if a write operation request is received from the client, the received write operation request is cached.

[0034] After sending the database snapshot to the newly added retrieval node, the method further includes:

[0035] The cached write operation request is sent to the newly added retrieval node, so that the newly added retrieval node performs the corresponding write operation based on the received write operation request.

[0036] Optionally, the feature management unit is specifically used for:

[0037] Based on the write operation request, determine the target operation type corresponding to the write operation;

[0038] If the target operation type is a feature deletion operation, then delete the feature indicated by the write operation request from the feature management database;

[0039] If the target operation type is a feature modification operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are used to update the corresponding features in the feature management database;

[0040] If the target operation type is a feature addition operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are added to the feature management database.

[0041] On the one hand, a data synchronization device is provided, applied to any retrieval node in a feature retrieval cluster included in a retrieval system, wherein the retrieval system further includes a feature management cluster; the device includes:

[0042] The receiving unit is configured to receive a write diffusion request sent by the feature management cluster. The write diffusion request carries feature information corresponding to the write operation of the feature management cluster to the associated feature management database. The write operation is triggered by a write operation request sent by the client.

[0043] A write diffusion execution unit is configured to synchronously update the associated feature retrieval database based on the aforementioned feature information; and,

[0044] The receiving unit is further configured to receive operation transaction information sent by the feature management cluster, wherein the operation transaction information records the transaction information corresponding to the write operation;

[0045] A transactional synchronization unit is used to perform a synchronization operation on the feature retrieval database based on the operation transaction information if it is determined that the synchronization update has failed.

[0046] Optionally, the write diffusion execution unit is further configured to:

[0047] Based on the synchronization update results, update the task status of the write operation request in this retrieval node;

[0048] The transactional synchronization unit is specifically used for:

[0049] If the task status indicates that the write operation request has not yet been completed on this retrieval node, then the synchronization update is determined to have failed.

[0050] Optionally, the device further includes an initial synchronization unit for:

[0051] In response to the instruction of this retrieval node to join the feature retrieval cluster, an initial synchronization request is initiated to the feature management cluster. The initial synchronization request is used to instruct the feature retrieval cluster to fully synchronize the features in the feature management database to the feature retrieval database according to the target synchronization method.

[0052] Receive the operation log returned by the feature management cluster, and sequentially perform synchronization operations on the feature retrieval database according to the operation transaction information recorded in the operation log; or,

[0053] Receive the database snapshot returned by the feature management cluster and load the database snapshot into its own feature retrieval database.

[0054] Optionally, the retrieval system further includes a feature extraction cluster; then the initial synchronization unit is specifically used for:

[0055] Determine whether the resource utilization rate of the feature extraction cluster is greater than the preset threshold;

[0056] If it is determined that the resource utilization rate is greater than the preset threshold, the target synchronization method is determined to be the database snapshot-based synchronization method;

[0057] If it is determined that the resource utilization rate is not greater than the preset threshold, the target synchronization method is determined to be the synchronization method based on operation logs;

[0058] Based on the determined target synchronization method, the initial synchronization request is initiated to the feature retrieval cluster.

[0059] Optionally, the initial synchronization unit is specifically used for:

[0060] The operation log is deserialized to obtain the write operation request corresponding to each operation transaction information;

[0061] Each write operation request is validated for validity, and write operation requests that fail the validation are filtered out.

[0062] Perform deduplication on the filtered write operation requests;

[0063] Execute the write operations indicated by each write operation request after deduplication to fully synchronize the features in the feature management database to the feature retrieval database.

[0064] On the one hand, a data retrieval system is provided, including a feature management cluster and a feature retrieval cluster, wherein the feature retrieval cluster includes at least one retrieval node;

[0065] In response to a write operation request sent by the client, the feature management cluster performs a corresponding write operation on the associated feature management database and records the corresponding operation transaction information in the operation log of the feature management database; it also initiates write diffusion requests to the at least one retrieval node, so that each retrieval node synchronously updates the associated feature retrieval database based on the feature information carried by the write operation request.

[0066] When each retrieval node subscribes to the write operation transaction of the feature management database, the feature management cluster also sends the operation transaction information to the at least one retrieval node, so that if the synchronization update by each retrieval node fails, the feature retrieval database is synchronized based on the operation transaction information.

[0067] Optionally, the data retrieval system may further include an access device;

[0068] The access device receives an operation request sent by the client, and when it determines that the operation request is a write operation request, it distributes the write operation request to the feature management cluster; or, when it determines that the operation request is a retrieval operation request, it distributes the retrieval operation request to the feature retrieval cluster.

[0069] On one hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0070] On the one hand, a computer storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0071] On one hand, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the methods described above.

[0072] The retrieval data synchronization method, apparatus, device, and computer storage medium provided in this application divide the retrieval system into a feature management cluster and a feature retrieval cluster. The feature management cluster is responsible for database-related write operations, while the feature retrieval cluster implements the retrieval function. Each cluster maintains its own associated database, thereby avoiding the reuse of a single retrieval database by the entire retrieval system. This prevents increased retrieval pressure and time consumption during high-concurrency retrieval requests, and avoids the problem that network or disk failures in the retrieval database would directly lead to the unavailability of the retrieval system. This improves the reliability of the retrieval system, thereby enhancing retrieval efficiency and availability. Simultaneously, after performing a write operation, the feature management cluster synchronizes the updated features to the feature retrieval cluster through write diffusion, enabling the feature retrieval cluster to synchronously update its own database. To ensure the reliability of synchronization, this application also uses a method where each retrieval node subscribes to the write operation transactions of the feature management database. Once the write operation of the feature management database is completed and written to the operation log, the corresponding operation transaction information is sent to each retrieval node. This allows the operation transaction information to be used as a fallback strategy in case of synchronization failure, enabling a second synchronization operation to be performed. This ensures the reliability of synchronization between the feature management database and the feature retrieval database, further improving the reliability of the retrieval system. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0074] Figure 1 This is a schematic diagram of the architecture of a 1:N retrieval system in related technologies;

[0075] Figure 2 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0076] Figure 3 This is a schematic diagram of the architecture of the retrieval system provided in the embodiments of this application;

[0077] Figure 4 A schematic diagram illustrating the principle of the read / write separation CQRS framework provided in this application embodiment;

[0078] Figure 5 A schematic diagram of the initial synchronization process provided for an embodiment of this application;

[0079] Figure 6 A flowchart illustrating the data retrieval synchronization method provided in this application embodiment;

[0080] Figure 7A schematic flowchart illustrating the write diffusion process provided in an embodiment of this application;

[0081] Figure 8 A schematic diagram illustrating the principle of the write diffusion combined with the transaction sender box mechanism provided in the embodiments of this application;

[0082] Figure 9 This is an application diagram illustrating an example of an image content review scenario provided in an embodiment of this application.

[0083] Figure 10 A schematic diagram of a data retrieval synchronization device provided in this application embodiment;

[0084] Figure 11 A schematic diagram of another data retrieval synchronization device provided in this application embodiment;

[0085] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0087] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0088] Operation log: Used to record all transaction information performed in the database. A database transaction is a sequence of database operations that access and potentially manipulate various data items. These operations are either all executed or none are executed; it is an indivisible unit of work. A transaction consists of all database operations performed between the start and end of the transaction. In a database system, a transaction is a discrete unit of work; it can be an operation that modifies a feature in the database or adds a feature, etc. For example, the Binlog is a binary operation log used to record operations performed by the retrieval service on the retrieval database, including adding, deleting, and modifying features.

[0089] Kafka is a high-throughput distributed publish-subscribe message queue system.

[0090] Command Query Responsibility Segregation (CQRS) architecture is a read-write separation architecture pattern that separates commands that change the state of a model from queries that change the state of the model. The retrieval system provided in this application is based on the concept of CQRS.

[0091] Write operations refer to operations performed on features in the database, including adding, deleting, and modifying features.

[0092] Write diffusion: In this embodiment, write diffusion refers to the process where updating a feature in the feature management database spreads this update to other feature retrieval databases. For example, when a new feature is added to the feature management database, in order to maintain synchronization between the feature management database and the feature retrieval database and ensure the correctness of the retrieval, the added feature should naturally be updated in the feature retrieval database.

[0093] Transactional mailboxes, also known as application events, refer to the process of performing normal updates based on the same transaction and inserting the messages into a specific mailbox table in the database. This allows subscribed recipients to receive these messages and learn about the update actions corresponding to the transaction. When the feature management database performs a write operation, it writes the relevant transaction to the operation log. This relies on the database's highly guaranteed transaction characteristics. Once written to the operation log, it indicates that the feature management cluster has successfully completed all write operations. Thus, the write operations performed by the feature management cluster are notified to the feature retrieval cluster through transactional messages. This is beneficial in ensuring consistency between the feature management database and the feature retrieval database as a fallback strategy if write diffusion fails.

[0094] Database snapshot: A snapshot is a fully usable copy of a specified set of data, including an image of the corresponding data at a specific point in time (the time the copy begins). A snapshot can be a copy of the data it represents or a replica of the data. Essentially, a snapshot is a reference or pointer to data stored on a storage device; in short, it shows the state of the data at a particular moment. Its core working principle is to create a list of pointers indicating the addresses for reading data, providing a snapshot of the data at a given instant. When the data changes, it is copied again. For databases, the data may all reside in memory. If a sudden system crash occurs, all data will be lost. Therefore, a mechanism is needed to ensure that data is not lost due to failure. This mechanism is called persistence, and snapshots are one such persistence mechanism. In short, a snapshot can be understood as a full backup of the database data at a specific point in time.

[0095] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0096] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0097] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0098] The solutions provided in this application involve the management of features extracted using artificial intelligence technology and the retrieval process based on these features. Specifically, feature extraction models for images, audio, video, or text content trained using AI technologies such as computer vision, speech processing, natural language processing, and machine learning / deep learning can be used to extract features from the content to be extracted, and then the features are stored in a database, so that a related retrieval process can be implemented subsequently based on these features.

[0099] Taking image content moderation, a common scenario in content moderation, as an example, the main function is to determine whether an image contains inappropriate content. Therefore, the database typically contains features corresponding to identified inappropriate content. Feature matching is used to determine whether the image contains these features, thus identifying whether the image contains inappropriate content. A 1:N retrieval system is usually employed. See [link / reference]. Figure 1 The diagram shows a typical 1:N retrieval system architecture. This architecture includes three parts: image input, software development kit (SDK) processing, and retrieval service. The image input part takes in the image to be written or retrieved. The SDK processing mainly implements image processing (including image size control), image box detection, encoding, feature extraction, and other functions. The retrieval service is the core of the entire retrieval system, mainly including feature management such as write operations like adding, deleting, and modifying feature information. In addition, it also includes feature retrieval, which mainly implements feature matching to output retrieval scores.

[0100] As can be seen, the entire retrieval system reuses a single retrieval database. Under high concurrency, this results in significant retrieval pressure and increased processing time. Furthermore, network or disk failures in the database will directly render the retrieval system unavailable, impacting the real-time nature of content review. Additionally, the feature management and feature retrieval modules are housed in the same service, leading to high coupling and significant interdependence, which hinders development. For example, developing or optimizing feature management functionality will affect feature retrieval functionality, potentially requiring the entire retrieval system to be paused. From a resource consumption perspective, future expansion due to increased retrieval requests will need to consider the resources required by both feature management and feature retrieval modules, resulting in significant resource waste due to ineffective expansion of the feature management module.

[0101] In practical applications, it has been found that in scenarios such as content moderation, write operations are usually low-frequency operations in the moderation system. Typically, a batch of features are only added to the database when the retrieval system is idle, while retrieval requests are the main function of the moderation system and are high-frequency operations.

[0102] Considering the characteristic of the retrieval system that "adds and deletes less often than queries", this application provides a retrieval data synchronization method. In this method, write operations such as adding, deleting, and modifying are deployed in a separate feature management cluster, while read operations such as retrieval are deployed in a separate feature retrieval cluster. To ensure high availability of retrieval and that they do not affect each other, the feature management cluster and the feature retrieval cluster are physically isolated from each other, and data consistency is ensured through a synchronization mechanism.

[0103] Furthermore, a corresponding synchronization mechanism was proposed, which may include initial synchronization and dynamic synchronization.

[0104] Initial synchronization is a fundamental module in the retrieval database synchronization mechanism. Based on the resource utilization status of the retrieval node, it can dynamically select either a synchronization method based on operation logs or a synchronization method based on database snapshots. Initial synchronization is generally used when the retrieval node first connects to the feature retrieval cluster.

[0105] Dynamic synchronization is the core module of the retrieval database synchronization mechanism, mainly including two strategies: write diffusion and transactional mailbox. Write diffusion is a real-time update strategy introduced after the feature management cluster completes write operations such as adding, deleting, and modifying, and it updates the status of the feature retrieval database associated with each retrieval node in the feature retrieval cluster in real time. That is, after the write operation in the feature management cluster successfully updates the feature management database, multiple write operations are initiated to multiple retrieval nodes. To ensure that the write operations are not blocked and to ensure real-time performance, write diffusion is implemented asynchronously, so there is a possibility of failure. Transactional mailbox is a fallback strategy introduced when the write diffusion request to update the retrieval node fails. By subscribing to the database operation log of the feature management cluster, operation transaction information is generated and written to a message queue. Each retrieval node in the feature retrieval cluster obtains operation transaction information by subscribing to the message queue, thereby ensuring the consistency between the feature management data and the feature retrieval database.

[0106] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0107] This application can be applied to high-availability, low-write, high-retrieval scenarios for image review. After introducing this solution, the public cloud image review system will be able to retrieve information in a more real-time, highly available, and high-concurrency manner, so as to formulate more reasonable and secure operation strategies based on the timely retrieval results.

[0108] The solutions provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. For example, they are suitable for most read-intensive retrieval scenarios with less writing, such as image content review scenarios (e.g., face recognition, image peering), public cloud image content review scenarios, voice feature retrieval scenarios, and fingerprint feature retrieval scenarios, etc., which will not be listed here individually. Figure 2 The diagram shown is an application scenario provided by an embodiment of this application. In this scenario, a terminal device 101, an access device 102, a feature management cluster 103, and a feature retrieval cluster 104 may be included.

[0109] Terminal device 101 can be, for example, a mobile phone, tablet computer (PAD), laptop computer, desktop computer, smart wearable device, smart voice interaction device, smart home appliance, vehicle terminal, or aircraft. Terminal device 101 can be installed with a target application that initiates write operation requests or feature retrieval requests, such as a video application, instant messaging application, image application, music application, or browser. The application involved in this application embodiment can be a software client, or a webpage, mini-program, etc. The server is a backend server corresponding to the software, webpage, mini-program, etc., and the specific type of client is not limited.

[0110] Access device 102 is used to perform functions such as parameter verification and task distribution for received task requests, that is, to distribute write operation requests to the feature management cluster and feature retrieval requests to the feature retrieval cluster.

[0111] The feature management cluster 103 is used to implement write operations such as adding, deleting, and modifying features. The feature retrieval cluster 104 is used to implement feature matching calculation and then return the retrieval score and hit result. The feature retrieval cluster 104 includes multiple retrieval nodes. When the access device distributes feature retrieval requests, the requests are routed to different retrieval nodes through load balancing, thereby ensuring and supporting the horizontal expansion of retrieval capabilities and ensuring the high availability and high concurrency of massive retrieval requests.

[0112] Taking a public cloud image content review scenario as an example, the feature management database and feature retrieval database are mainly used to store the image features of illegal content, such as the image features of illegal faces and objects. When a write operation request such as adding, deleting, or modifying the database is required, the access device 102 can obtain the corresponding write operation request and distribute it to the feature management cluster 103. The feature management cluster 103 can perform the corresponding write operation, update the feature management database, and synchronize the updated content to each retrieval node of the feature retrieval cluster 104 to update the feature retrieval database of each of these retrieval nodes.

[0113] In the public cloud image content review scenario, compared with the retrieval operation of the feature retrieval cluster 104, the feature management operation of the feature management cluster 103 is a low-frequency operation, which mainly involves adding features and requires little resources. However, the retrieval operation has a query rate per second (QPS) of tens of thousands. Therefore, there are high requirements for the high availability, high concurrency, and real-time performance of the feature retrieval cluster 104. Thus, distributed retrieval nodes can be used to meet the requirements through load balancing.

[0114] Similarly, in other similar scenarios, the functions of each device are implemented in a similar manner, so they will not be described in detail here.

[0115] Access device 102, feature management cluster 103, and feature retrieval cluster 104 can all be implemented using server devices with certain computing capabilities. These can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, i.e., content delivery networks (CDN), and big data and artificial intelligence platforms, but are not limited to these.

[0116] It should be noted that, in order to increase the reliability of the retrieval system and avoid the retrieval system becoming unavailable due to equipment failure, the feature management cluster 103 and the feature retrieval cluster 104 are different physical devices and are physically isolated from each other.

[0117] In this embodiment of the application, the retrieval system may further include a feature extraction cluster (in Figure 2 (Not shown in the text) It is used to implement the related functions of feature extraction, and it can provide feature extraction functions triggered by processes such as write operation requests, database synchronization and feature retrieval requests.

[0118] Each retrieval node in the feature management cluster 103 and the feature retrieval cluster 104 may include one or more processors, memory, and interactive I / O interfaces, and may also be configured with its own database for storing extracted features. The memory of each device may store program instructions for the retrieval data synchronization method provided in this application embodiment. When these program instructions are executed by the corresponding processor, they can be used to implement the steps of the retrieval data synchronization method provided in this application embodiment, thereby achieving data synchronization between the feature management database and the feature retrieval database.

[0119] The aforementioned devices can communicate directly or indirectly through one or more networks. This network can be a wired network or a wireless network; for example, a wireless network could be a mobile cellular network or a Wireless-Fidelity (WIFI) network, or any other possible network. This embodiment of the invention does not limit the types of networks used.

[0120] It should be noted that, where possible, the aforementioned feature management cluster 103 and access device 102 can be implemented using the same device. Similarly, the function of the feature extraction cluster can also be implemented using the feature management cluster 103 or the access device 102. This application embodiment does not impose any limitations on this.

[0121] like Figure 3 The diagram shown is an architectural schematic of a retrieval system provided in an embodiment of this application. The retrieval system includes three parts: a retrieval access module, a feature management cluster, and a feature retrieval cluster.

[0122] (1) The retrieval access module is used to implement functions such as parameter verification and task distribution of received task requests, that is, to distribute write operation requests to the feature management cluster and feature retrieval requests to the feature retrieval cluster.

[0123] (2) The feature management cluster includes features management, library management, write diffusion management and initial synchronization. Features management is to implement the relevant write operation process for the feature management database in response to write operation requests. Library management is to implement the relevant management functions for the feature management database. Write diffusion management and initial synchronization are both functions corresponding to maintaining the synchronization between the feature management database and the feature retrieval database.

[0124] (3) The feature retrieval cluster includes features retrieval and synchronization management functions. Feature retrieval refers to the corresponding retrieval process performed in response to a feature retrieval request, that is, to complete the feature matching calculation and then return the retrieval score and hit result. Synchronization management is the synchronization process performed to maintain the synchronization between the feature management database and the feature retrieval database.

[0125] In one possible implementation, the retrieval system can be a distributed retrieval system based on a read-write separation CQRS framework, see [link to relevant documentation]. Figure 4 The diagram shown is a schematic of the read / write separation CQRS framework.

[0126] Read-write separation CQRS is a system architecture that separates the responsibilities of commands and queries, enabling the separation of commands that change the model state from queries that change the model state. Therefore, in this embodiment, feature management and feature retrieval are implemented using different physical devices. The command end, i.e., the feature management cluster, primarily focuses on optimizing the writing of feature data. Upon receiving a write operation request, it implements the write operation process through the feature management service of the feature management cluster, updating the feature management database. The query end, i.e., the feature retrieval cluster, primarily focuses on optimizing the reading of feature data. Upon receiving a feature retrieval request, it implements the retrieval process through the feature retrieval service of the corresponding retrieval node, returning the retrieval results. The command end and query end are deployed independently, physically isolated, and a data synchronization mechanism ensures the real-time performance and eventual consistency of the retrieved data.

[0127] In one possible application scenario, the relevant data (such as feature data) and model parameters involved in the embodiments of this application can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of storage devices (or storage nodes) of various types in the network through application software or application interfaces to work together and jointly provide data storage and business access functions to the outside world.

[0128] In one possible application scenario, to reduce communication latency during retrieval, retrieval nodes can be deployed in various regions. Alternatively, for load balancing, different retrieval nodes can serve terminal devices in different regions. For example, if a terminal device is located at location a, it establishes a communication connection with the retrieval node serving location a; if the terminal device is located at location b, it establishes a communication connection with the retrieval node serving location b. Multiple retrieval nodes and a feature management cluster can form a data sharing system, which uses blockchain to achieve data sharing.

[0129] Each retrieval node or feature management cluster in the data sharing system has a node identifier corresponding to that retrieval node or feature management cluster. Each retrieval node or feature management cluster in the data sharing system can store the node identifiers of other retrieval nodes or feature management clusters in the data sharing system, so that the generated blocks can be broadcast to other retrieval nodes in the data sharing system according to the node identifiers of other retrieval nodes. For example, when a write operation request occurs, the feature management cluster can synchronize features in this way.

[0130] Of course, the methods provided in the embodiments of this application are not limited to... Figure 2 The application scenarios shown or Figure 3The architecture can also be used in other possible application scenarios, and the embodiments of this application do not impose limitations. Figure 2 or Figure 3 The functions that each device or module in the application scenario shown can achieve will be described in the subsequent method embodiments, and will not be elaborated on here.

[0131] Based on the aforementioned read-write separation CQRS architecture, CRUD operations are deployed on the feature management cluster, while retrieval operations are deployed on the feature retrieval cluster. The two clusters are physically isolated and heterogeneous, avoiding the problem of unreliable data storage and system unavailability caused by equipment failure. This ensures high concurrency retrieval requirements even with limited resources. Furthermore, when retrieval requests increase, only horizontal scaling of the feature retrieval cluster is needed, avoiding unnecessary expansion of the feature management cluster. Synchronization between the feature management cluster and the feature retrieval cluster is achieved through a synchronization process. Therefore, this application provides a retrieval data synchronization method for the aforementioned retrieval system to achieve synchronization between the feature management database and the feature retrieval database. The synchronization between the feature management database and the feature retrieval database includes an initial synchronization process when a retrieval node is initially added, and a dynamic synchronization process during the use of the retrieval node, which will be described separately below.

[0132] See Figure 5 The diagram shown is a schematic representation of the initial synchronization process provided in this application embodiment. Initial synchronization refers to the process where, when a retrieval node initially joins the feature retrieval cluster, its associated feature retrieval database does not contain any features. Therefore, it is necessary to synchronize all features from the feature management database to the feature retrieval database. This is a full synchronization process of feature data, which includes the following steps:

[0133] Step 501: Add a new retrieval node to the feature retrieval cluster and start the initial synchronization process.

[0134] When the retrieval demand of the feature retrieval cluster increases, retrieval nodes can be easily added to improve the high concurrency of the entire retrieval system for retrieval requests. When a new retrieval node is successfully added to the feature retrieval cluster, an initial synchronization process is required.

[0135] When a new retrieval node successfully joins the feature retrieval cluster, it can indicate that the new retrieval node has successfully joined the feature retrieval cluster, and then, in response to the instruction that the retrieval node has joined the feature retrieval cluster, it can initiate an initial synchronization request to the feature management cluster.

[0136] In one possible implementation, the feature retrieval cluster may have a management module. When a new retrieval node needs to be added, an application is submitted to the management module. Once the deployment work related to the new retrieval node is completed, the management module can send an indication to the new retrieval node that it has successfully joined the feature retrieval cluster.

[0137] In one possible implementation, the addition of new search nodes can be completed with the assistance of relevant personnel. Once the deployment work related to the new search nodes is completed, the initial synchronization process of the new search nodes can be initiated.

[0138] For example, a newly added retrieval node can provide a display interface through which relevant personnel can operate and initiate the initial synchronization process. When the newly added retrieval node receives the corresponding operation, it means that it has received an instruction to join the feature retrieval cluster.

[0139] Alternatively, relevant personnel can also perform operations on the management page of the retrieval system to initiate the initial synchronization process. When a newly added retrieval node receives the initial synchronization process triggered by this, it is also considered to have received an instruction to join the feature retrieval cluster.

[0140] Step 502: Add a new retrieval node and determine the target synchronization method.

[0141] In this embodiment of the application, the synchronization method can be one of the following:

[0142] (1) Synchronization method based on operation log

[0143] The operation log records all operations performed on the feature management database. By following the operation transaction information in the operation log, the same operation can be performed to obtain the same results as the feature management database.

[0144] For example, when 10 new features are added to the feature management database, the corresponding operation transaction information will be recorded in the operation log. Similarly, after the new retrieval node obtains the operation transaction information from the operation log, it can perform the same write operation again to add 10 features to its associated feature retrieval data.

[0145] However, since the write operation may involve feature extraction, which consumes feature extraction resources, it is necessary to determine whether the feature extraction resources meet the requirements of the first synchronization method mentioned above.

[0146] Specifically, the retrieval system may also include a feature extraction cluster, responsible for feature extraction operations involved in write operations or feature retrieval. Since the synchronization method based on operation logs requires re-extracting features, it places high demands on the resource requirements of the feature extraction cluster. Therefore, adding a new retrieval node can determine whether the current state of the feature extraction cluster meets the requirements of the aforementioned synchronization method.

[0147] (2) Synchronization method based on database snapshot

[0148] When using this synchronization method, the feature management cluster persists a snapshot of the feature management database and sends the snapshot to newly added retrieval nodes. The new retrieval nodes receive and load the snapshot to complete the initial synchronization. This database snapshot-based synchronization method increases the load on the feature management cluster's central processing unit (CPU), thus placing certain requirements on the device status of the feature management equipment.

[0149] Of course, other synchronization methods can also be used, and this application does not limit the implementation of these methods.

[0150] In a specific implementation, the newly added retrieval node can obtain the resource utilization rate of the feature extraction cluster and determine whether the resource utilization rate of the feature extraction cluster is greater than the preset threshold. If it is greater than the preset threshold, it indicates that the current load of the feature extraction cluster is high. If the synchronization method based on operation logs is used, it will further increase the pressure on the feature extraction cluster. Therefore, in order to ensure the computing resources required for online retrieval, other synchronization methods can be used. Otherwise, if the resource utilization rate is not greater than the preset threshold, the synchronization method based on operation logs can be used.

[0151] Taking image content review as an example, the review objects are mainly images. Therefore, when the feature extraction cluster performs feature extraction, the main resource consumed is the graphics processing unit (GPU) resource. Thus, adding a retrieval node can obtain the GPU utilization of the feature extraction cluster. When the GPU load is too high, it indicates that the online computing resources are tight. In order not to affect the online computing, other synchronization methods can be used at this time. When the GPU load is not high, a synchronization method based on operation logs can be used.

[0152] It should be noted that the target synchronization method used for initial synchronization can be pre-specified by the newly added retrieval node. This is used as an example here. However, in actual applications, the feature management cluster can also make the decision after receiving the initial synchronization request.

[0153] Step 503: The newly added retrieval node initiates an initial synchronization request to the feature management cluster.

[0154] In this embodiment of the application, the initial synchronization request is used to instruct the feature retrieval cluster to fully synchronize the features in the feature management database to the feature retrieval database according to the target synchronization method determined above.

[0155] Step 504: The feature management cluster determines the target synchronization method indicated by the initial synchronization request.

[0156] Step 505: If the target synchronization method is based on operation logs, the feature management cluster will send the operation logs to the newly added retrieval node.

[0157] After the feature management cluster receives the initial synchronization request, it obtains the target synchronization method indicated in the initial synchronization request. When the target synchronization method is the operation log-based synchronization method, the feature management cluster sends the current operation log of the associated feature management database to the newly added retrieval node.

[0158] Step 506: The newly added retrieval node performs synchronization operations on the feature retrieval database according to the operation transaction information recorded in the operation log.

[0159] The operation log records the operation transaction information performed on the feature management database in sequence. When a new retrieval node is added, it is executed sequentially, which can achieve synchronization with the feature management database.

[0160] In one possible implementation, the operation log can be a binlog, which records the write operation requests corresponding to the write operations performed sequentially on the feature management database. To save bandwidth resources, the sequence of write operation requests in the binlog can be converted into binary form. When a new retrieval node receives the binlog, it performs deserialization processing to obtain the write operation requests corresponding to each operation transaction, such as the request structure corresponding to each write operation request, which contains the type of write operation to be performed and the write operation object. For image content review scenarios, the write operation object can be an image; for text content review scenarios, the write operation object can be text.

[0161] To avoid resource waste caused by invalid and duplicate requests, the newly added retrieval node can perform legality verification on each write operation request, filter out write operation requests that fail verification, and perform deduplication processing on each of the filtered write operation requests. For each deduplicated write operation request, the corresponding write operation is executed to write each feature to the associated feature retrieval database, so as to fully synchronize the features in the feature management database to the feature retrieval database.

[0162] Specifically, during deduplication, duplicate write requests with the same identifier can be deduplicated. Furthermore, write operations with the same characteristic can be merged to reduce the number of write operations required. For example, if a characteristic is added to the database and then deleted, it's as if the characteristic no longer exists; therefore, the addition and deletion operations are unnecessary, as the final database result is consistent. Similarly, if a characteristic is added to the database and then modified, only the modified characteristic is retained, so only one modification operation is needed. Based on this, operations on characteristics with the same identifier (ID) in the database can be merged to minimize the amount of data processed by write operations.

[0163] After the initial synchronization is completed, the newly added retrieval node can clean up data such as operation logs.

[0164] Step 507: If the target synchronization method is a database snapshot-based synchronization method, the feature management cluster determines whether its own device resource status meets the resource requirements of this synchronization method.

[0165] In this embodiment of the application, since the synchronization method based on database snapshots may increase the CPU load and disk load of the feature management cluster, in order to ensure that online requests are not affected, the feature management cluster needs to determine whether its own device resource status (such as CPU and disk status) meets the resource requirements of the synchronization method, and select whether to modify the target synchronization method based on the determined result.

[0166] Step 508: If satisfied, the feature management cluster performs a snapshot persistence operation to generate a database snapshot of the feature management database.

[0167] Step 509: The feature management cluster sends a database snapshot to the newly added retrieval node.

[0168] Step 510: Add a new retrieval node and load the database snapshot into its own feature retrieval database.

[0169] Step 511: If not satisfied, the feature management cluster modifies the target synchronization method to a synchronization method based on operation logs.

[0170] Step 512: The feature management cluster sends the operation log to the newly added retrieval node.

[0171] If the feature management cluster determines that the CPU load is too high, it will change the target synchronization method to a synchronization method based on operation logs. At the same time, it will reduce the concurrency of the feature extraction cluster to minimize the impact on the online environment. If the CPU load is determined to be normal, it will perform a database snapshot process to complete the initial synchronization.

[0172] In this embodiment of the application, when the feature management cluster determines that its CPU load is too high, it can also negotiate with the newly added retrieval node to see if the initial synchronization time can be postponed. If so, a new initial synchronization time can be determined. For example, a time when the retrieval system is relatively idle can be selected, which can be obtained based on historical data statistics.

[0173] In this embodiment of the application, considering that the initial synchronization process may take a long time when the feature management database is large, new write operation requests may also be received during the initial synchronization process. For the newly received write operation requests, when the synchronization method based on operation logs is adopted, the operation transaction information corresponding to these write operation requests can be sent to the newly added retrieval node.

[0174] When using a database snapshot-based synchronization method, if a write operation request is received from the client during the snapshot persistence operation, an incremental synchronization method can be used. This involves caching the received write operation request and sending the cached write operation request to the newly added retrieval node, so that the newly added retrieval node can perform the corresponding write operation based on the received write operation request.

[0175] In this embodiment, initial synchronization addresses the synchronization of all search data when a new search node joins the feature search cluster for the first time. After joining the feature search cluster and completing initial synchronization, the search node can be put into use. In subsequent use, it is necessary to synchronize the difference in data between the feature management cluster and the feature search cluster in real time to ensure the eventual consistency of the search database. Based on this, this embodiment also introduces dynamic synchronization, which mainly includes write diffusion and transaction mailbox strategies. Write diffusion ensures that most write operation requests from the feature management cluster can be updated to the feature search cluster in a timely manner. Since the transaction mailbox mechanism serves as a fallback strategy, it utilizes the high availability transaction characteristics of the database and the persistence of message queues to ensure that updated features are eventually written to the search node's search database.

[0176] See Figure 6 The diagram shown is a flowchart illustrating the data synchronization method for data retrieval provided in an embodiment of this application.

[0177] Step 601: The feature management cluster receives write operation requests sent by the client.

[0178] In this embodiment, when the access layer of the retrieval system determines that the received operation request is a write operation request, it will distribute the write operation request to the feature management cluster. The access layer will perform a validity check on the write operation request, and the feature management cluster can also perform a secondary check on the distributed write operation request to determine whether the write operation request is invalid. If it is invalid, the write operation request will be ignored and the process will end. Only if it is a valid request will the subsequent process be executed to perform the write operation.

[0179] Step 602: The feature management cluster performs corresponding write operations on the associated feature management database.

[0180] Step 603: The feature management cluster records the corresponding operation transaction information in the operation log of the feature management database.

[0181] In this embodiment of the application, relying on the high-guarantee transaction characteristics of the database, that is, when the feature management cluster successfully completes all the write operations indicated in the above write operation request, the corresponding operation transaction information will be written to the operation log. If it is not successfully completed, or not all of them are completed, it will not be written to the operation log. This also means that once the operation transaction information is recorded in the operation log, it indicates that the feature management cluster must have successfully completed all the write operations indicated in the above write operation request.

[0182] Specifically, the operation transaction information can include all transaction information corresponding to this write operation, including but not limited to the request structure of the write operation request, the process information of executing the write operation, etc.

[0183] Step 604: The feature management cluster sends write diffusion requests to at least one retrieval node.

[0184] In this embodiment of the application, the write diffusion request is initiated when the client sends a write operation request to the feature management cluster, and the feature management cluster automatically triggers the request after completing feature management, retrieval database entry, and service binlog appending. The write diffusion request sent by the feature retrieval cluster carries the feature information corresponding to the write operation. For deletion operations, the feature information can be the feature ID. For modification or addition operations, the feature information can include the modified or added feature data.

[0185] In one possible implementation, to avoid network latency or time consumption caused by retrieval node processing, the write diffusion request in this embodiment adopts an asynchronous write method. That is, after the feature management cluster initiates write diffusion, it returns a response directly without waiting for the processing result of the retrieval node.

[0186] Step 605: Each retrieval node updates the associated feature retrieval database synchronously based on the feature information.

[0187] In this embodiment of the application, synchronous update refers to the operation performed directly on features in the feature retrieval database, such as adding, modifying or deleting.

[0188] Specifically, when a retrieval node receives a write diffusion request carrying modified feature data, the retrieval node replaces the corresponding original feature data with the received feature data; when a retrieval node receives a write diffusion request carrying newly added feature data, the retrieval node adds the received feature data to the feature management database; when a retrieval node receives a write diffusion request carrying a deleted feature ID, the retrieval node searches for and deletes the feature data with that ID.

[0189] Step 606: The feature management cluster sends the operation transaction information to each retrieval node.

[0190] In this embodiment of the application, considering that write diffusion may fail due to various factors, a more reliable synchronization method is needed as a fallback strategy to avoid inconsistencies between the feature management database and the feature retrieval database.

[0191] Therefore, considering the highly guaranteed transaction characteristics of the database, the write operations performed by the feature management database can be obtained by having each retrieval node subscribe to the write operation transactions of the feature management database. In other words, when each retrieval node subscribes to the write operation transactions of the feature management database, the feature management cluster will send the operation transaction information to each retrieval node once the operation log of the feature management database is updated.

[0192] In one possible implementation, the above process can be achieved using a transaction mailbox mechanism, see [link to relevant documentation]. Figure 6 As shown, the operation log of the database can be managed by subscribing to features. When the operation log is updated, the corresponding operation transaction information is written to the transaction message queue. Each retrieval node also subscribes to the topic of the transaction message queue, so that when the transaction message queue is updated, the operation transaction information is sent to each retrieval node.

[0193] Step 607: Determine whether the synchronization update was successful by retrieving the node.

[0194] If the result of step 607 is yes, then the operation transaction information is ignored and not executed.

[0195] Step 608: If the result of step 607 is negative, the retrieval node performs a synchronization operation on the feature retrieval database based on the operation transaction information.

[0196] In this embodiment of the application, since the operation transaction information and the above-mentioned write diffusion request are essentially the same update, the idempotency property of the database can be used to avoid repeated execution of the update.

[0197] In one possible implementation, since write operation requests require updates to the associated database in both the feature management cluster and the feature retrieval cluster, the corresponding task status can be maintained for the same write operation request in both the feature management cluster and each retrieval node. For example, in the feature management cluster, when the write operation process corresponding to the write operation request has been completed, the corresponding task status is updated to "completed"; otherwise, the corresponding task status is updated to "incomplete." Similarly, in each retrieval node, the corresponding task status can also be updated to "completed" when the write operation process corresponding to the write operation request has been completed, and otherwise, the corresponding task status is updated to "incomplete."

[0198] Therefore, after the retrieval node performs a synchronous update based on the write diffusion request, it can update the task status of the write operation request on this retrieval node based on the synchronous update result. That is, if the synchronous update is successful, the task status is updated to "completed"; if the synchronous update fails, the task status is updated to "incomplete". When operation transaction information is received, the retrieval node determines whether the above synchronous update was successful based on the task status. That is, if the task status indicates that the write operation request is still incomplete on this retrieval node, then the synchronous update is determined to have failed. In this case, a synchronous operation can be performed on the feature retrieval database based on the operation transaction information. This means updating the feature retrieval database based on the features in the operation transaction information, or re-extracting features and updating the feature retrieval database based on the request structure of the write operation request in the operation transaction information.

[0199] In one possible implementation, when updating the database based on operation transaction information, it can be confirmed whether the same update has been performed before, that is, whether the operation object, operation content, and write operation request are consistent. If they are consistent, it indicates that the synchronous update has been completed and there is no need to perform a duplicate update; if the same update has not been performed before, the feature retrieval database is updated.

[0200] In the embodiments of this application, see Figure 7 The diagram shown is a flowchart illustrating the diffusion process.

[0201] Step 701: The feature management cluster receives write operation requests.

[0202] Step 702: The feature management cluster determines the target operation type corresponding to the write operation based on the write operation request, and executes the corresponding processing flow according to the operation type.

[0203] Step 703: If the target operation type is a feature deletion operation, the feature management cluster deletes the feature indicated by the write operation request from the feature management database.

[0204] For example, if a write operation request instructs the deletion of a feature with the ID "XXX", the feature management cluster will locate the feature based on the ID and delete it.

[0205] Step 704: If the target operation type is a feature modification operation or a feature addition operation, the feature management cluster performs feature extraction on the content carried in the write operation request. Specifically, the feature management cluster can request the feature extraction cluster to extract features from the carried content.

[0206] Step 705: The feature management cluster determines whether feature extraction was successful.

[0207] Step 706: If successful, the feature management cluster will determine the operation type again. If it fails, the process ends.

[0208] Step 707: Add operations for features, and the feature management cluster constructs a unique ID for the feature data.

[0209] Step 708: The feature management cluster will store the extracted features in the database.

[0210] For example, in the scenario of image content review, if a write operation request instructs that the features corresponding to the image be added to the feature management database, then the feature extraction cluster is invoked to extract features from the image carried in the write operation request and assign an ID to the feature so as to add the corresponding feature to the feature management database.

[0211] Step 709: For feature modification operations, the feature management cluster updates the corresponding features in the feature management database with the obtained features.

[0212] For example, in the scenario of image content review, if a write operation request instructs that the feature with ID "XXX" be modified to the feature corresponding to the image carried, then the feature extraction cluster is invoked to extract the features of the image carried in the write operation request and replace the original feature with ID "XXX" in the feature management database.

[0213] Step 710: The feature management cluster serializes write operation requests and writes them to the operation log.

[0214] Step 711: The feature management cluster determines whether the write operation failed.

[0215] Step 712: If successful, the feature management cluster asynchronously sends a write diffusion request to each retrieval node of the feature retrieval cluster, so that each retrieval node receives the write diffusion request and performs synchronous update.

[0216] Step 713: If the operation fails, the feature management cluster deletes the feature from the feature management database. That is, if writing to the binlog fails, it indicates that the write operation has failed, and the feature history database needs to be rolled back to its state before the operation. The feature management cluster can use lazy deletion to delete features from the feature management database.

[0217] In the embodiments of this application, see Figure 8 The diagram shown is a schematic representation of the write diffusion combined with transaction sender box mechanism provided in an embodiment of this application.

[0218] In the embodiments of this application, see Figure 8 As shown, when a write operation occurs, the feature management service of the feature management cluster performs a corresponding persistent update to the associated feature management database. Upon successful update, it synchronously updates each retrieval node in the feature retrieval cluster via write diffusion. However, the write diffusion process is asynchronous; that is, after sending a write diffusion request, the feature management cluster does not wait for the retrieval nodes to return results but instead returns the result of this write operation to the front end. Therefore, it cannot be guaranteed that all retrieval nodes will write successfully. See [link to documentation]. Figure 8 As shown, it's possible that a feature update at a certain retrieval node might fail. To ensure data consistency, a transactional mailbox mechanism is introduced as a fallback strategy. Based on the high availability of the database's transactional characteristics and the message queue's persistence of messages, it can effectively guarantee that write requests will eventually be successfully written to the retrieval node's database.

[0219] In this embodiment of the application, before the dynamic synchronization process is executed, it is necessary to configure the transaction sending box corresponding to the feature management database. See [link to relevant documentation]. Figure 8 As shown, the transaction message queue mainly consists of two modules: a message relay module and a transaction message queue. The message relay module subscribes to the operation logs of the feature management cluster database, such as the binlog. Specifically, it subscribes to updates to the operation logs. When the feature management cluster successfully completes consecutive operations such as feature extraction and database writing, it writes an operation transaction message to the operation log. The message relay module, which has subscribed to this message, retrieves and caches it, then writes it to a specific topic in the message queue. Furthermore, the feature management cluster also receives subscription requests from various retrieval nodes. These requests are used to subscribe to write operation transactions in the feature management database, and the cluster responds to these requests by configuring each retrieval node to subscribe to the transaction message queue. In other words, each retrieval node also subscribes to a topic in this message queue. Once the message queue is updated, it retrieves the corresponding operation transaction information and updates its own database, thus ensuring data consistency.

[0220] Furthermore, the message relay module can retrieve operation transaction information from the operation log and write it to the transaction message queue. Based on each retrieval node's subscription to the transaction message queue, the module retrieves operation transaction information from the queue and sends it to each node. The message relay module's subscription to the database's operation log relies heavily on the database's high-guarantee transaction characteristics. Once information is written to the operation log, it indicates that the feature management cluster has successfully completed all write operations for the current write request. The message queue primarily decouples the system and ensures information persistence to prevent data loss. After receiving operation transaction information from the message queue, each retrieval node must ensure that its updates are consistent with the order of messages in the feature management cluster, thus avoiding out-of-order updates. Simultaneously, the update idempotency mechanism prevents data duplication.

[0221] See Figure 9 The diagram illustrates an application scenario using image content review as an example. When a new retrieval node is added to the feature retrieval cluster, the features in the feature management database are synchronized to the feature retrieval database associated with that retrieval node using the initial synchronization method provided in this embodiment. The administrator client of the image content review platform initiates a write operation request based on the uploaded image corresponding to the identified illegal content. The feature management cluster then extracts features from the image and writes them to the feature management database. Simultaneously, after successful data entry, the write diffusion and transaction mailbox mechanisms provided in this embodiment synchronize and update each retrieval node in the feature retrieval cluster. Furthermore, when a user client uploads an image and requires content review (e.g., checking for illegal or prohibited items, people, or faces), the feature retrieval cluster receives the corresponding feature retrieval request, calls the feature extraction cluster to extract features from the image to be reviewed, performs a search in the feature retrieval database, outputs the search results to help determine whether the review is successful, and returns the results to the user client.

[0222] In summary, this application proposes a distributed retrieval data synchronization scheme based on a combined write diffusion and transactional mailbox mechanism. This scheme can be applied to public cloud image content review scenarios based on CV technology, enabling more real-time, highly available, and high-concurrency retrieval information. This allows for the formulation of more reasonable and secure operational strategies based on timely retrieval results. It can also be applied to other similar retrieval scenarios, such as feature-based retrieval services based on sound or fingerprint. This distributed retrieval system data synchronization scheme is applicable to other hardware platforms, including PCs and servers, significantly improving the availability of the review retrieval system. Furthermore, by introducing a read-write separation CQRS framework, it greatly ensures stable operation under high concurrency. The dynamic synchronization mechanism employing a combined real-time write diffusion and transactional mailbox mechanism achieves a real-time write diffusion update success rate of over 99% and low synchronization latency. While ensuring retrieval real-time performance, the transactional mailbox acts as a fallback in case of write diffusion failure. Based on the highly reliable transaction guarantee of the database and message queue persistence, it enables the final update of failed data, thereby achieving eventual consistency between the two cluster retrieval databases.

[0223] Please see Figure 10 Based on the same inventive concept, this application also provides a data retrieval synchronization device 100, applied in a feature management cluster included in a retrieval system, wherein the retrieval system further includes a feature retrieval cluster composed of at least one retrieval node; the device includes:

[0224] The feature management unit 1001 is used to respond to the write operation request sent by the client, perform the corresponding write operation on the associated feature management database, and record the corresponding operation transaction information in the operation log of the feature management database.

[0225] The write diffusion unit 1002 is used to initiate write diffusion requests to at least one retrieval node, each write diffusion request carrying feature information corresponding to the write operation, enabling each retrieval node to synchronously update the associated feature retrieval database based on the feature information; and,

[0226] The transaction push unit 1003 is used to send operation transaction information to each retrieval node when each retrieval node subscribes to the write operation transaction of the feature management database, so that if the synchronization update of each retrieval node fails, the feature retrieval database is synchronized based on the operation transaction information.

[0227] Optionally, the device further includes a configuration unit 1004 for:

[0228] Configure the transaction mailbox corresponding to the database. The transaction mailbox includes a message relay module and a transaction message queue. The message relay module subscribes to updates to the operation log.

[0229] Receive at least one subscription request from a retrieval node. The subscription request is used to request a subscription feature to manage write operation transactions in the database.

[0230] In response to each subscription request, configure each retrieval node to subscribe to the transaction message queue.

[0231] Optionally, the transaction push unit 1003 is specifically used for:

[0232] The message relay module retrieves operation transaction information from the operation log and writes the operation transaction information into the transaction message queue.

[0233] Based on each retrieval node's subscription to the transaction message queue, operation transaction information is obtained from the transaction message queue and then sent to each retrieval node.

[0234] Optionally, the device also includes an initial synchronization unit 1005, used for:

[0235] Receive the initial synchronization request sent by the newly added retrieval node in the feature retrieval cluster. The initial synchronization request carries the target synchronization method determined by the newly added retrieval node.

[0236] If the target synchronization method is based on operation logs, then the operation logs will be sent to the newly added retrieval node;

[0237] If the target synchronization method is a database snapshot-based synchronization method, then determine whether the device's resource status meets the resource requirements of this synchronization method;

[0238] If the conditions are met, a snapshot persistence operation is performed to generate a database snapshot of the feature management database, and the database snapshot is sent to the newly added retrieval node.

[0239] If the conditions are not met, the target synchronization method will be changed to the operation log-based synchronization method, and the operation log will be sent to the newly added retrieval node.

[0240] Optionally, the initial synchronization unit 1005 is also used for:

[0241] During the snapshot persistence operation, if a write operation request is received from the client, the received write operation request is cached.

[0242] After sending the database snapshot to the newly added retrieval node, the method also includes:

[0243] The cached write operation request is sent to the newly added retrieval node, so that the newly added retrieval node performs the corresponding write operation based on the received write operation request.

[0244] Optionally, the feature management unit 1001 is specifically used for:

[0245] Based on the write operation request, determine the target operation type corresponding to the write operation;

[0246] If the target operation type is a feature deletion operation, then delete the feature indicated by the write operation request from the feature management database;

[0247] If the target operation type is a feature modification operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are used to update the corresponding features in the feature management database.

[0248] If the target operation type is a feature addition operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are added to the feature management database.

[0249] This device can be used to execute the methods performed by the feature management cluster in the various embodiments of this application. Therefore, the functions that each functional module of this device can achieve can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0250] Please see Figure 11 Based on the same inventive concept, this application also provides a data retrieval synchronization device 110, applied to any retrieval node included in the feature retrieval cluster of the retrieval system, the retrieval system further including a feature management cluster; the device includes:

[0251] The receiving unit 1101 is used to receive a write diffusion request sent by the feature management cluster. The write diffusion request carries the feature information corresponding to the write operation of the feature management cluster to the associated feature management database. The write operation is triggered by a write operation request sent by the client.

[0252] The write diffusion execution unit 1102 is used to synchronously update the associated feature retrieval database based on feature information; and,

[0253] The receiving unit 1101 is also used to receive operation transaction information sent by the feature management cluster. The operation transaction information records the transaction information corresponding to the write operation.

[0254] The transactional synchronization unit 1103 is used to perform synchronization operations on the feature retrieval database based on the operation transaction information if it is determined that the synchronization update has failed.

[0255] Optionally, the write diffusion execution unit 1102 is also used for:

[0256] Based on the synchronous update results, update the task status of the write operation request in this retrieval node;

[0257] The transactional synchronization unit 1103 is specifically used for:

[0258] If the task status indicates that the write operation request has not been completed on this retrieval node, then the synchronization update has failed.

[0259] Optionally, the device also includes an initial synchronization unit 1104, used for:

[0260] In response to the instruction for this retrieval node to join the feature retrieval cluster, an initial synchronization request is initiated to the feature management cluster. The initial synchronization request is used to instruct the feature retrieval cluster to fully synchronize the features in the feature management database to the feature retrieval database according to the target synchronization method.

[0261] Receive the operation logs returned by the feature management cluster, and sequentially perform synchronization operations on the feature retrieval database according to the operation transaction information recorded in the operation logs; or,

[0262] Receive the database snapshot returned by the feature management cluster and load the database snapshot into its own feature retrieval database.

[0263] Optionally, the above retrieval system also includes a feature extraction cluster; then the initial synchronization unit 1104 is specifically used for:

[0264] Determine whether the resource utilization rate of the feature extraction cluster is greater than a preset threshold;

[0265] If it is determined that the resource utilization rate is greater than the preset threshold, the target synchronization method is determined to be the database snapshot-based synchronization method;

[0266] If it is determined that the resource utilization rate is not greater than the preset threshold, the target synchronization method is determined to be the synchronization method based on operation logs;

[0267] Based on the determined target synchronization method, an initial synchronization request is initiated to the feature retrieval cluster.

[0268] Optionally, the initial synchronization unit 1104 is specifically used for:

[0269] The operation log is deserialized to obtain the write operation requests corresponding to each operation transaction information;

[0270] Perform validity checks on each write operation request and filter out write operation requests that fail the checks;

[0271] Perform deduplication on the filtered write operation requests;

[0272] Execute the write operations indicated by each write operation request after deduplication to fully synchronize the features in the feature management database to the feature retrieval database.

[0273] This device can be used to execute the methods performed by the retrieval nodes in the feature retrieval cluster in the various embodiments of this application. Therefore, the functions that each functional module of this device can achieve can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0274] Please see Figure 12 Based on the same technical concept, embodiments of this application also provide a computer device. In one embodiment, the computer device can be... Figure 1 The node in the feature management cluster or the retrieval node in the feature retrieval cluster shown, such as the computer device Figure 12 As shown, it includes a memory 1201, a communication module 1203, and one or more processors 1202.

[0275] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0276] Memory 1201 may be volatile memory, such as random-access memory (RAM); memory 1201 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1201 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1201 may be a combination of the above-described memories.

[0277] The processor 1202 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1202 is used to implement the aforementioned data retrieval synchronization method when calling a computer program stored in the memory 1201.

[0278] The communication module 1203 is used to communicate with terminal devices and other servers.

[0279] This application embodiment does not limit the specific connection medium between the memory 1201, communication module 1203, and processor 1202. This application embodiment... Figure 12 The memory 1201 and the processor 1202 are connected via a bus 1204, and the bus 1204 is in Figure 12 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1204 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 12 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0280] The memory 1201 stores a computer storage medium containing computer-executable instructions for implementing the data retrieval synchronization method of the embodiments of this application. The processor 1202 is used to execute the data retrieval synchronization methods of the above embodiments.

[0281] In some possible implementations, various aspects of the data retrieval synchronization method provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the data retrieval synchronization method according to various exemplary embodiments of this application described above. For example, the computer device can perform the steps of the various embodiments.

[0282] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0283] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0284] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0285] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0286] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0287] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0288] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0289] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0290] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0291] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for synchronizing retrieved data, characterized in that, The method is applied to a feature management cluster within a retrieval system, wherein the retrieval system further includes a feature retrieval cluster composed of at least one retrieval node; the method includes: In response to a write operation request sent by the client, the corresponding write operation is performed on the associated feature management database, and the corresponding operation transaction information is recorded in the operation log of the feature management database. Upon successful execution of the write operation, an asynchronous message-based write diffusion request is triggered, sending a write diffusion request to at least one retrieval node. This request carries feature information corresponding to the write operation, enabling each retrieval node to synchronously update its associated feature retrieval database based on this feature information. Furthermore, the task status of the write operation request is updated based on the result of this synchronous update. The feature information includes: the feature ID to be deleted, or feature data to be modified or added; and... When each retrieval node subscribes to the write operation transaction of the feature management database, it sends the operation transaction information to each retrieval node, so that if the synchronization update is determined to be unsuccessful based on the task status, each retrieval node performs a synchronization operation on the feature retrieval database based on the operation transaction information.

2. The method as described in claim 1, characterized in that, Before each retrieval node sends the operation transaction information to each retrieval node when each retrieval node subscribes to the write operation transaction of the feature management database, the method further includes: Configure the transaction mailbox corresponding to the management database. The transaction mailbox includes a message relay module and a transaction message queue. The message relay module subscribes to updates of the operation log. Receive a subscription request sent by the at least one retrieval node, the subscription request being used to request subscription to the write operation transaction of the feature management database; In response to each subscription request, configure each retrieval node to subscribe to the transaction message queue.

3. The method as described in claim 2, characterized in that, When each retrieval node subscribes to a write operation transaction of the feature management database, the operation transaction information is sent to each retrieval node, including: The operation transaction information is obtained from the operation log through the message relay module, and the operation transaction information is written into the transaction message queue. Based on each retrieval node's subscription to the transaction message queue, the operation transaction information is obtained from the transaction message queue, and the operation transaction information is sent to each retrieval node respectively.

4. The method as described in claim 1, characterized in that, The method further includes: Receive an initial synchronization request sent by a newly added retrieval node in the feature retrieval cluster, wherein the initial synchronization request carries the target synchronization method determined by the newly added retrieval node; If the target synchronization method is a synchronization method based on operation logs, then the operation logs are sent to the newly added retrieval node; If the target synchronization method is a database snapshot-based synchronization method, then determine whether the device's resource status meets the resource requirements of that synchronization method; If the conditions are met, a snapshot persistence operation is performed to generate a database snapshot of the feature management database, and the database snapshot is sent to the newly added retrieval node. If the conditions are not met, the target synchronization method is changed to a synchronization method based on operation logs, and the operation logs are sent to the newly added retrieval node.

5. The method as described in claim 4, characterized in that, The method further includes: During the snapshot persistence operation, if a write operation request is received from the client, the received write operation request is cached. After sending the database snapshot to the newly added retrieval node, the method further includes: The cached write operation request is sent to the newly added retrieval node, so that the newly added retrieval node performs the corresponding write operation based on the received write operation request.

6. The method according to any one of claims 1 to 5, characterized in that, In response to a write operation request sent by the client, perform corresponding write operations on its own feature management database, including: Based on the write operation request, determine the target operation type corresponding to the write operation; If the target operation type is a feature deletion operation, then delete the feature indicated by the write operation request from the feature management database; If the target operation type is a feature modification operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are used to update the corresponding features in the feature management database; If the target operation type is a feature addition operation, then feature extraction is performed on the content carried by the write operation request, and the obtained features are added to the feature management database.

7. A method for synchronizing retrieved data, characterized in that, The method is applied to any retrieval node within a feature retrieval cluster included in a retrieval system, wherein the retrieval system further includes a feature management cluster; the method includes: The feature management cluster receives a write diffusion request sent asynchronously. The write diffusion request carries feature information corresponding to the write operation of the feature management cluster on the associated feature management database. The write operation is triggered by a write operation request sent by the client. The feature information includes: the feature ID to be deleted, or the feature data to be modified or added. Based on the aforementioned feature information, the associated feature retrieval database is synchronously updated, and the task status of the write operation request is updated based on the result of the synchronous update; and, The system receives operation transaction information sent by the feature management cluster, which records the transaction information corresponding to the write operation. If the synchronization update fails based on the task status, a synchronization operation is performed on the feature retrieval database based on the operation transaction information.

8. The method as described in claim 7, characterized in that, After synchronously updating the associated feature retrieval database based on the aforementioned feature information, the method further includes: Based on the synchronization update results, update the task status of the write operation request in this retrieval node; The synchronization update was determined to have failed, including: If the task status indicates that the write operation request has not yet been completed on this retrieval node, then the synchronization update is determined to have failed.

9. The method as described in claim 7, characterized in that, Before receiving the write diffusion request sent by the feature management cluster, the method further includes: In response to the instruction of this retrieval node to join the feature retrieval cluster, an initial synchronization request is initiated to the feature management cluster. The initial synchronization request is used to instruct the feature retrieval cluster to fully synchronize the features in the feature management database to the feature retrieval database according to the target synchronization method. Receive the operation log returned by the feature management cluster, and sequentially perform synchronization operations on the feature retrieval database according to the operation transaction information recorded in the operation log; or, Receive the database snapshot returned by the feature management cluster and load the database snapshot into its own feature retrieval database.

10. The method as described in claim 9, characterized in that, The retrieval system also includes a feature extraction cluster; in response to the instruction of this retrieval node to join the feature extraction cluster, an initial synchronization request is initiated to the feature management cluster, including: Determine whether the resource utilization rate of the feature extraction cluster is greater than a preset threshold; If it is determined that the resource utilization rate is greater than the preset threshold, the target synchronization method is determined to be the database snapshot-based synchronization method; If it is determined that the resource utilization rate is not greater than the preset threshold, the target synchronization method is determined to be the synchronization method based on operation logs; Based on the determined target synchronization method, the initial synchronization request is initiated to the feature retrieval cluster.

11. The method as described in claim 9, characterized in that, The step of receiving the operation log returned by the feature management cluster and sequentially performing synchronization operations on the feature retrieval database according to the operation transaction information recorded in the operation log includes: The operation log is deserialized to obtain the write operation request corresponding to each operation transaction information; Perform validity checks on each write operation request and filter out write operation requests that fail the checks; Perform deduplication on the filtered write operation requests; Execute the write operations indicated by each write operation request after deduplication to fully synchronize the features in the feature management database to the feature retrieval database.

12. A data retrieval synchronization device, characterized in that, The device is applied to a feature management cluster within a retrieval system, wherein the retrieval system further includes a feature retrieval cluster composed of at least one retrieval node; the device includes: The feature management unit is used to respond to write operation requests sent by the client, perform corresponding write operations on the associated feature management database, and record the corresponding operation transaction information in the operation log of the feature management database. A write diffusion unit is configured to, upon successful execution of the write operation, trigger asynchronous message-based write diffusion requests to at least one retrieval node. Each write diffusion request carries feature information corresponding to the write operation, enabling each retrieval node to synchronously update the associated feature retrieval database based on this feature information, and to update the task status of the write operation request based on the result of the synchronous update. The feature information includes: a feature ID to be deleted, or feature data to be modified or added; and... The transaction push unit is used to send the operation transaction information to each retrieval node when each retrieval node subscribes to the write operation transaction of the feature management database, so that if each retrieval node determines that the synchronization update has failed based on the task status, it can perform a synchronization operation on the feature retrieval database based on the operation transaction information.

13. A data retrieval synchronization device, characterized in that, It is applied to any retrieval node in the feature retrieval cluster of the retrieval system, which also includes a feature management cluster; The device includes: The receiving unit is used to receive a write diffusion request sent by the feature management cluster in an asynchronous message manner. The write diffusion request carries feature information corresponding to the write operation of the feature management cluster to the associated feature management database. The write operation is triggered by a write operation request sent by the client. The feature information includes: the feature ID to be deleted, or the feature data to be modified or added. A write diffusion execution unit is configured to synchronously update the associated feature retrieval database based on the feature information, and update the task status of the write operation request based on the result of the synchronous update; and, The receiving unit is further configured to receive operation transaction information sent by the feature management cluster, wherein the operation transaction information records the transaction information corresponding to the write operation; A transactional synchronization unit is used to perform a synchronization operation on the feature retrieval database based on the operation transaction information if it is determined from the task status that the synchronization update has failed.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6 or 7 to 11.

15. A computer storage medium storing computer program instructions thereon, characterized in that, When executed by a processor, the computer program instructions implement the steps of the method according to any one of claims 1 to 6 or 7 to 11.

16. A computer program product comprising computer program instructions, characterized in that, When executed by a processor, the computer program instructions implement the steps of the method according to any one of claims 1 to 6 or 7 to 11.

Images