A method, device and medium for real-time synchronization of source data based on pet management

Real-time data synchronization of pet management system is achieved through Kafka and Hadoop mechanisms, solving conflicts and errors caused by data synchronization lag, and ensuring data reliability and consistency.

CN116860873BActive Publication Date: 2025-08-22INSPUR ZHUOSHU BIG DATA IND DEV CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310817539.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-07-04
Publication Date
2025-08-22
Estimated Expiration
2043-07-04

AI Technical Summary

Technical Problem

In the existing pet management system, data synchronization lag leads to data conflicts or errors, increasing management costs and real-time unification and synchronization cannot be achieved.

Method used

The Kafka distributed publishing and subscription message system and Hadoop parallel loading mechanism are adopted to realize the synchronization of different data sources of pet information, house information and personnel information, and data association processing is carried out through producer and consumer ports to ensure real-time synchronization and consistency of data.

Benefits of technology

It effectively avoids conflicts or errors caused by data synchronization lag, and ensures data reliability and data information unity between different services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116860873B_ABST
    Figure CN116860873B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time source data synchronization method, device, and medium based on pet management, belonging to the field of computer technology. The method is used to solve the technical problem that existing pet management systems are prone to data conflicts or data errors, which increases the cost of the pet management system and is not conducive to the real-time unification and synchronization of pet information data. The method includes: sending management data in a distributed cluster to obtain distributed data; storing the distributed data on a producer port; performing association modification processing on personnel information data and housing information data in the distributed data to obtain first pet management information; performing association deletion processing on the pet information data through a consumer port corresponding to the producer port to obtain second pet management information; and performing data synchronization processing on the pet management system based on the first pet management information and the second pet management information to complete the information unification of the pet management system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and medium for real-time synchronization of source data based on pet management. Background Art

[0002] With the development of information technology, vast amounts of data are generated and consumed on the Internet every day. When data is generated and consumed depends on applications, not the data itself. To balance the processing power of data producers and consumers and to decouple data caching and applications, the producer-consumer model has emerged. It describes a finite buffer and two types of processes: producers and consumers. Producers place products into the buffer, while consumers remove products from it. When the buffer is full, producers must wait until more space is available before continuing to produce. When the buffer is empty, consumers must wait until more products are available before continuing to read. The producer-consumer model has been widely used across various industries and technologies.

[0003] Kafka is a distributed publish-subscribe messaging system with good performance, consisting of data producers (Producers), brokers (Brokers), and consumers (Consumers) running in a cluster.

[0004] In existing pet management systems, it is difficult to achieve real-time data synchronization from the source data source to the target data source for the population data and housing data corresponding to the pet management data. This can easily lead to data conflicts or errors caused by data synchronization lags, and cannot guarantee data reliability and the unification of data information between different services. This greatly increases the management cost of the pet management system and is not conducive to the real-time unification and synchronization of pet information. Summary of the Invention

[0005] The embodiments of the present application provide a method, device and medium for real-time synchronization of source data based on pet management, which are used to solve the following technical problems: the existing pet management system is prone to data conflicts or errors caused by data synchronization lags, which increases the cost of the pet management system and is not conducive to the real-time unification and synchronization of pet information.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a real-time synchronization method for source data based on pet management, including: sending the management data in a distributed cluster according to the management data in the producer port to obtain distributed data; wherein the distributed data includes: pet information data, house information data and personnel information data; storing the distributed data in the producer port; through the consumer port corresponding to the producer port, performing association modification processing on the personnel information data and the house information data in the distributed data to obtain pet management first information; through the consumer port corresponding to the producer port, performing association deletion processing on the pet information data to obtain pet management second information; based on the first pet management information and the second pet management information, performing data synchronization processing on the pet management system to complete the information unification of the pet management system.

[0008] The embodiment of the present application uses Kafka to run a distributed publish-subscribe message system consisting of data producers (Producers), brokers (Brokers), and consumers (Consumers) in a cluster, and then unifies online and offline message processing based on the Hadoop parallel loading mechanism, synchronizes pet information data, house information data, and personnel information data from different data sources, and realizes real-time data synchronization from the source data source to the target data source, effectively avoiding data conflicts or errors caused by data synchronization delays, and ensuring data reliability and the unification of data information between different services.

[0009] In a feasible implementation manner, the management data is sent to a distributed cluster based on the management data in the producer port to obtain distributed data, specifically including: receiving the management data sent by the data producer to the Kafka proxy port; wherein the data producer is the data generator in the pet management system; based on the distributed message subscription mechanism, the management data is classified according to the basic data types in the production port, and the subject data is determined; wherein the basic data types include at least: message topics, message partitions, message data formats, synchronous or asynchronous threads, batch sending threads and single or multiple threads; receiving change information data corresponding to the subject data; wherein the change information data includes: personnel change information, pet change information and house change information; encapsulating the subject data and the change information data; and based on the distributed cluster sending mechanism, pushing the encapsulated information data to the Kafka proxy port to determine the distributed data.

[0010] In a feasible implementation manner, the personnel information data and the house information data in the distributed data are associated and modified through the consumer port corresponding to the producer port to obtain the first pet management information, specifically including: pulling the distributed data stored in the agent port through the consumer port; based on the change information data in the distributed data, the corresponding pet information data is associated and queried through the personnel information data; if the pet quantity information in the pet information data is zero, the data synchronization management of the pet management system is directly ended; if the pet quantity information in the pet information data is not zero, the house information data in the distributed data is judged whether it is associated and changed to obtain the first pet management information; wherein, the first pet management information includes: each house ID and the pet tag information or the pet tag-free information associated with each house ID.

[0011] In a feasible implementation manner, a judgment is performed on whether the house information data in the distributed data is associated with a change, specifically including: classifying the house information data according to the change time node to obtain a house information group before the change and a house information group after the change; judging the existence of a house ID for the house information group; if the house ID does not exist in the house information group before the change, traversing the house ID before the change corresponding to the house information before the change; if the house ID does not exist in the house information group after the change, deleting the pet information data associated with the house ID, and marking the house ID as having no pet tag information; otherwise, marking the house ID as having pet tag information.

[0012] In a feasible implementation, before determining the existence of a house ID for the house information group, the method further includes: traversing the house information group before the change and the house information group after the change in the agent port; obtaining the house ID before the change corresponding to the house information group before the change and the house ID after the change corresponding to the house information group after the change through a preset list; wherein the house ID is composed of the house ID before the change and the house ID after the change.

[0013] In a feasible embodiment, after deleting the pet information data associated with the house ID and determining that the house ID has no pet tag information, the method further includes: obtaining the personnel information data associated with the house ID; and extracting the pet information data of the remaining personnel information data associated with the personnel information data; performing data existence judgment on the pet information data associated and bound with the remaining personnel information data; if the pet information data of the remaining personnel is blank data, marking the house ID as having no pet tag information; otherwise, marking the house ID as having pet tag information.

[0014] In a feasible embodiment, the pet information data is associated and deleted through the consumer port corresponding to the producer port to obtain the second pet management information, which specifically includes: pulling the distributed data stored in the agent port through the consumer port; obtaining the corresponding deleted personnel ID and the corresponding deleted house ID based on the personnel information data and house information data that have been deleted and changed in the distributed data; performing an associated deletion operation on the pet information data to be deleted through the deleted personnel ID and the deleted house ID, and determining the deleted house ID as having no pet tag information; obtaining the remaining personnel information data associated with the deleted house ID; performing data existence judgment on the pet information data associated and bound with the remaining personnel information data; if the pet information data of the remaining personnel is blank data, marking the deleted house ID as having no pet tag information; otherwise, marking the deleted house ID as having pet tag information; determining the second pet management information based on the deleted personnel ID, the deleted house ID and its corresponding pet tag information or no pet tag information.

[0015] In a feasible implementation, data synchronization processing is performed on the pet management system based on the first pet management information and the second pet management information, specifically including: according to the first pet management information and the second pet management information stored in the consumer port, the pet information data, the house information data and the personnel information data are synchronized for multi-port data consistency to complete the information unification of the pet management system; wherein, the multi-port includes: a producer port, an agent port and a consumer port.

[0016] In the second aspect, an embodiment of the present application also provides a real-time synchronization device for source data based on pet management, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute a real-time synchronization method for source data based on pet management as described in any of the above embodiments.

[0017] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium, characterized in that the storage medium is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by the terminal, the terminal executes a real-time synchronization method of source data based on pet management described in any of the above embodiments.

[0018] This application provides a real-time synchronization method, device and medium for source data based on pet management. It uses Kafka to run a distributed publish-subscribe message system consisting of data producers (Producers), brokers (Brokers), and consumers in a cluster, and then unifies online and offline message processing based on the Hadoop parallel loading mechanism. It synchronizes pet information data, house information data, and personnel information data from different data sources, realizing real-time data synchronization from the source data source to the target data source, effectively avoiding data conflicts or errors caused by data synchronization lags, and ensuring data reliability and the unification of data information between different services. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 A flowchart of a method for real-time synchronization of source data based on pet management provided in an embodiment of the present application;

[0021] Figure 2 A structural diagram of a real-time synchronization device for source data based on pet management provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] The embodiment of the present application provides a real-time synchronization method of source data based on pet management, such as Figure 1 As shown, the real-time synchronization method of source data based on pet management specifically includes steps S101-S105:

[0024] It should be noted that Kafka is a distributed publish-subscribe messaging system with good working performance, consisting of data producers (Producers), brokers (Brokers), and consumers (Consumers) running in a cluster. It unifies online and offline message processing through the Hadoop parallel loading mechanism, and has the characteristics of fast persistence, high throughput, and a fully distributed system.

[0025] The Producer-Consumer model ensures that the processing capabilities (rates) of product producers and consumers are relatively independent of each other. Producers simply place their "produced" products into a channel and can continue processing them, without having to wait for the corresponding "products" to be processed by the consumer. Consumers, running in their own worker threads, simply retrieve "products" from the channel and process them, without having to worry about the details of who "produced" these "products" or how they were "produced." As a result, the consumer's processing power relatively does not affect the producer, while maintaining a loosely coupled relationship with the producer. On the other hand, when the consumer's processing power exceeds that of the producer, the channel may become empty. In this case, the consumer's worker thread is suspended until the producer "produces" new "products." This effectively creates a situation where the consumer waits for the producer. Similarly, when the consumer's processing power is less than that of the producer, the channel may become full, causing the producer thread to be suspended until the consumer "consumes" some of the "products" in the channel, freeing up storage space. This effectively creates a situation where the producer waits for the consumer. The independent processing capabilities of the producer and consumer are relative.

[0026] S101: Send the management data in a distributed cluster according to the management data in the producer port to obtain distributed data, wherein the distributed data includes: pet information data, house information data, and personnel information data.

[0027] Specifically, the management data sent by the data producer to the Kafka agent port is first received. The data producer is the data generator in the pet management system.

[0028] Furthermore, based on the distributed message subscription mechanism, the management data is classified into basic data types related to the production port and the subject data is determined. The basic data types include at least: message topic, message partition, message data format, synchronous or asynchronous thread, batch sending thread, and single or multi-thread.

[0029] Furthermore, the system receives change information corresponding to the topic data. This change information includes personnel change information, pet change information, and house change information. The topic data and change information are then packaged and processed. Based on the distributed cluster sending mechanism, the packaged information is pushed to the Kafka broker port to determine the distributed data.

[0030] In one embodiment, a data producer sends a message to a broker port, allowing any broker to discover the location of other brokers. Each message has a unique topic. Basic requirements for data producers include topics, message partitioning, message data format (son.prob1 in characters), synchronous or asynchronous transmission, batch transmission, and single-threaded or multi-threaded transmission.

[0031] S102: Storing the distributed data in the producer port.

[0032] S103. Performing association modification processing on the personnel information data and the housing information data in the distributed data through the consumer port corresponding to the producer port to obtain the first pet management information.

[0033] Specifically, the distributed data stored in the agent port is pulled through the consumer port, and then the corresponding pet information data is associated with the change information data in the distributed data and the personnel information data.

[0034] If the number of pets in the pet information data is zero, the data synchronization management of the pet management system is directly terminated.

[0035] If the number of pets in the pet information data is not zero, the house information in the distributed data is judged to determine whether there is any associated change, thereby obtaining first pet management information. The first pet management information includes: each house ID and pet tag information or no pet tag information associated with each house ID.

[0036] Furthermore, the housing information data in the distributed data is judged to determine whether it has been changed. The specific judgment steps are as follows: first, the housing information data is classified according to the change time node to obtain the housing information group before the change and the housing information group after the change. Then, the housing information group before the change and the housing information group after the change in the agent port are traversed.

[0037] Furthermore, the pre-change house ID corresponding to the pre-change house information group and the post-change house ID corresponding to the post-change house information group are obtained through the preset list, wherein the house ID is composed of the pre-change house ID and the post-change house ID.

[0038] Furthermore, the existence of the house ID is checked for the house information group. If the house ID does not exist in the pre-change house information group, the pre-change house ID corresponding to the pre-change house information is traversed. If the pre-change house ID does not exist in the post-change house information group, the pet information data associated with the house ID is deleted.

[0039] Furthermore, the personnel information data associated with the house ID is first obtained. The pet information data associated with the remaining personnel information data is extracted. Then, the existence of the pet information data associated with the remaining personnel information data is determined.

[0040] If the pet information data of other people is blank, the house ID is marked as having no pet tag information. Otherwise, the house ID is marked as having pet tag information. Then the house ID is marked as having no pet tag information. Otherwise, the house ID is marked as having pet tag information.

[0041] Specifically, when the number of pets in the pet information data is not zero, the house information in the distributed data is determined to determine whether there has been an associated change, thereby obtaining first pet management information. The first pet management information includes each house ID and information related to each house ID, indicating whether a pet tag is present or absent.

[0042] As a feasible implementation, the basic requirements of the consumer end (consumer port) are which topics' data needs to be processed and how to process these topics. Specifically, it needs to determine whether to start multiple data consumers to form a group (cluster), whether to use batch processing, and whether to use single-threaded or multi-threaded processing. It also needs to implement the use of change information for the personnel information data, housing information data, and pet information data within the pet management system.

[0043] In one embodiment, when personnel information is modified, considering that a piece of pet information is associated with a person ID and a house ID, the corresponding pet information count is first queried using the person ID. If the count is zero, the data synchronization process ends. If the pet information count is greater than zero, indicating that the person has associated pet information, the process continues by determining whether the house information has changed. The broker port is traversed to retrieve the pre- and post-modification house information groups for the personnel change information, and the house IDs are obtained using two lists.

[0044] In one embodiment, if the house information group before the change is not empty, the previous house ID is traversed. If the house information group after the change does not contain the previous house ID, the pet information data associated with the house is deleted, and it is determined whether the house is also associated with the pet information of other people. If not, the pet tag of the house is deleted, that is, the house ID is marked as having no pet tag information. If so, it is retained, that is, the house ID is marked as having pet tag information.

[0045] S104. Performing associated deletion processing on the pet information data through the consumer port corresponding to the producer port to obtain the second pet management information.

[0046] Specifically, the distributed data stored in the agent port is pulled through the consumer port, and based on the personnel information data and house information data that have been deleted or changed in the distributed data, the corresponding deleted personnel ID and the corresponding deleted house ID are obtained.

[0047] Furthermore, by deleting the person ID and the house ID, the pet information data to be deleted is subjected to an associated deletion operation, and the deleted house ID is determined as having no pet tag information.

[0048] Furthermore, the remaining personnel information data associated with the deleted house ID is obtained, and then the pet information data associated with the remaining personnel information data is judged to be present.

[0049] If the pet information data of other people is blank, the house ID will be marked as having no pet tag information. Otherwise, the house ID will be marked as having pet tag information.

[0050] Furthermore, based on the deleted person ID, deleted house ID and their corresponding pet tag information or pet tag-free information, the second pet management information is determined.

[0051] In one embodiment, when deleting associated personnel information, the broker first retrieves the person ID and the house ID of the deleted person. These two IDs are used to lock the pet information for deletion. After the deletion is complete, the house ID is used to determine whether there is any pet information associated with the house and other personnel. If there is no other pet information, the deleted house ID is marked as having no pet tag information. Otherwise, the deleted house ID is marked as having pet tag information.

[0052] S105: Based on the first pet management information and the second pet management information, perform data synchronization processing on the pet management system to complete information unification of the pet management system.

[0053] Specifically, based on the first pet management information and the second pet management information stored in the consumer port, the pet information data, the housing information data, and the personnel information data are synchronized for data consistency across multiple ports to achieve information unification of the pet management system. The multiple ports include: a producer port, an agent port, and a consumer port.

[0054] In addition, the embodiment of the present application also provides a source data real-time synchronization device based on pet management, such as Figure 2 As shown, the source data real-time synchronization device 200 based on pet management specifically includes:

[0055] At least one processor 201. And a memory 202 in communication with the at least one processor 201. The memory 202 stores instructions that can be executed by the at least one processor 201, so that the at least one processor 201 can execute:

[0056] According to the management data in the producer port, the management data is sent to a distributed cluster to obtain distributed data; wherein the distributed data includes: pet information data, house information data and personnel information data;

[0057] Store distributed data on the producer port;

[0058] Through the consumer port corresponding to the producer port, the personnel information data and the house information data in the distributed data are associated and modified to obtain the first pet management information;

[0059] Performing associated deletion processing on the pet information data through the consumer port corresponding to the producer port to obtain the second pet management information;

[0060] Based on the first pet management information and the second pet management information, data synchronization processing is performed on the pet management system to complete the information unification of the pet management system.

[0061] This application uses Kafka to run a distributed publish-subscribe message system consisting of data producers (Producers), brokers (Brokers), and consumers (Consumers) in a cluster, and then unifies online and offline message processing based on the Hadoop parallel loading mechanism. It synchronizes pet information data, house information data, and personnel information data from different data sources, and realizes real-time data synchronization from the source data source to the target data source, effectively avoiding data conflicts or errors caused by data synchronization delays, and ensuring data reliability and the unification of data information between different services.

[0062] The various embodiments in this application are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant parts, refer to the descriptions of the method embodiments.

[0063] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0064] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0067] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0068] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0069] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included within the scope of the claims of the present application.

Claims

1. A real-time synchronization method for source data based on pet management, characterized in that: The method comprises: According to the management data in the producer port, the management data is sent in a distributed cluster to obtain distributed data; wherein the distributed data includes: pet information data, house information data and personnel information data; Storing the distributed data in the producer port; By using the consumer port corresponding to the producer port, the personnel information data and the housing information data in the distributed data are subjected to association modification processing to obtain first pet management information, specifically including: Pulling the distributed data stored in the broker port through the consumer port; Based on the change information data in the distributed data, the corresponding pet information data is associated with the personnel information data for query and judgment; If the number of pets in the pet information data is zero, then the data synchronization management of the pet management system is directly terminated; If the pet quantity information in the pet information data is not zero, determining whether the house information data in the distributed data has been associated with a change, and obtaining the first pet management information; wherein the first pet management information includes: each house ID and pet tag information or no pet tag information associated with each house ID; The pet information data is subjected to an associated deletion process through the consumer port corresponding to the producer port to obtain the second pet management information, specifically including: Pulling the distributed data stored in the broker port through the consumer port; Based on the personnel information data and house information data that have been deleted or changed in the distributed data, obtaining the corresponding deleted personnel ID and the corresponding deleted house ID; Performing an associated deletion operation on the pet information data to be deleted by using the deleted person ID and the deleted house ID, and determining the deleted house ID as having no pet tag information; Obtaining the remaining personnel information data associated with the deleted house ID; Performing data existence determination on the pet information data associated with and bound to the remaining personnel information data; If the pet information data of the remaining persons is blank data, the deleted house ID is marked as having no pet tag information; otherwise, the deleted house ID is marked as having pet tag information; Determining the second pet management information based on the deleted person ID, the deleted house ID, and the corresponding pet tag information or pet tag-free information; Based on the first pet management information and the second pet management information, data synchronization processing is performed on the pet management system to complete the information unification of the pet management system.

2. A method for real-time synchronization of source data based on pet management according to claim 1, characterized in that: According to the management data in the producer port, the management data is sent to the distributed cluster to obtain distributed data, specifically including: Receiving the management data sent by the data producer to the Kafka agent port; wherein the data producer is the data generator in the pet management system; Based on a distributed message subscription mechanism, the management data is classified into basic data types in the production port and subject data is determined; wherein the basic data types include at least: message subject, message partition, message data format, synchronous or asynchronous thread, batch sending thread, and single or multi-thread; Receive change information data corresponding to the subject data; wherein the change information data includes: personnel change information, pet change information and house change information; The subject data and the change information data are encapsulated and processed; and based on a distributed cluster sending mechanism, the encapsulated information data is pushed to a Kafka agent port to determine the distributed data.

3. A method for real-time synchronization of source data based on pet management according to claim 1, characterized in that: The determination process of whether the housing information data in the distributed data has been associated with a change specifically includes: Classify the housing information data according to the change time nodes to obtain a housing information group before the change and a housing information group after the change; Determining the existence of a house ID for the house information group; If the house ID does not exist in the house information group before the change, traverse the house IDs before the change corresponding to the house information before the change; If the house ID before the change does not exist in the house information group after the change, the pet information data associated with the house ID is deleted and the house ID is marked as having no pet tag information; otherwise, the house ID is marked as having pet tag information.

4. A method for real-time synchronization of source data based on pet management according to claim 3, characterized in that: Before determining the existence of the house ID on the house information group, the method further includes: Traversing the housing information group before the change and the housing information group after the change in the agent port; Through a preset list, the pre-change house ID corresponding to the pre-change house information group and the post-change house ID corresponding to the post-change house information group are obtained respectively; wherein, the house ID is composed of the pre-change house ID and the post-change house ID.

5. A method for real-time synchronization of source data based on pet management according to claim 3, characterized in that: After deleting the pet information data associated with the house ID and determining that the house ID does not have pet tag information, the method further includes: Acquire the personnel information data associated with the house ID; and extract the pet information data of the remaining personnel information data associated with the personnel information data; Performing data existence determination on the pet information data associated with and bound to the remaining personnel information data; If the pet information data of the remaining persons is blank data, the house ID is marked as having no pet tag information; otherwise, the house ID is marked as having pet tag information.

6. A method for real-time synchronization of source data based on pet management according to claim 1, characterized in that: Based on the first pet management information and the second pet management information, data synchronization processing is performed on the pet management system, specifically including: According to the first pet management information and the second pet management information stored in the consumer port, the pet information data, the house information data and the personnel information data are synchronized for multi-port data consistency to complete the information unification of the pet management system; wherein, the multi-port includes: a producer port, an agent port and a consumer port.

7. A real-time synchronization device for source data based on pet management, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the real-time synchronization method of source data based on pet management according to any one of claims 1-6.

8. A non-volatile computer storage medium, characterized in that The storage medium is a non-volatile computer-readable storage medium, which stores at least one program. Each of the programs includes instructions, and when the instructions are executed by the terminal, the terminal executes a real-time synchronization method of source data based on pet management according to any one of claims 1-6.

Citation Information

Patent Citations

  • Data management method, middleware and data management system

    CN108197233A

  • Configuration data synchronization method, service data system, computer system and medium

    CN112069265A