Real-time dimension table synchronization method and device, electronic equipment and storage medium
By synchronizing the real-time dimension table data of the vehicle to the message queue and inversely query the database, the problems of insufficient real-time, poor sharing and data synchronization delay in real-time dimension table synchronization are solved, and efficient and accurate data sharing and access are achieved.
Patent Information
- Application Number
- CN202411706251.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, real-time dimension table synchronization has problems such as insufficient real-time, poor sharing and data synchronization delay, which cannot meet the needs of real-time computing.
By obtaining the real-time dimension table data of the vehicle, it is stored in the target relational database and synchronized to the target message queue. Determine whether the data in the message queue changes. If it changes, extract the key field information, reverse query the database and store the data in the preset terminal, and provide an API interface for data sharing access.
Real-time dimension table synchronization is realized, data timeliness and accuracy is improved, and problems of poor sharing and synchronization delay are solved.
Smart Images
Figure CN119537475A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle technology, and in particular to a synchronization method, device, electronic device and storage medium for a real-time maintenance table. Background Art
[0002] In the context of big data, the use of real-time computing to process Internet of Vehicles data is increasing day by day. When processing a large amount of data in real time and involving dimension table associations, the following problems arise:
[0003] (1) Lack of real-time performance: In the big data processing of related technologies, dimension table information can only be updated periodically, which cannot meet the needs of real-time computing; (2) Poor sharing: Dimension table data can usually only be loaded into specific applications and cannot be shared across systems and platforms; (3) Data synchronization delay: Data synchronization often has a large delay, which affects the accuracy of real-time computing. Summary of the invention
[0004] The present application provides a real-time dimension table synchronization method, device, electronic device and storage medium to solve the problems of insufficient real-time synchronization, poor sharing and data synchronization delay of real-time dimension table synchronization. The present application synchronizes the dimension table that needs to be queried in real time during program use to a third-party memory storage medium. Each application can use the API opened by the third-party memory storage medium to share and access data, thereby ensuring the timeliness and accuracy of the data.
[0005] The first embodiment of the present application provides a real-time dimension table synchronization method, comprising the following steps:
[0006] Acquire real-time dimension table data of the vehicle, wherein the real-time dimension table data is stored in a target relational database;
[0007] Synchronize the real-time dimension table data to the target message queue, and determine whether the real-time dimension table data in the target message queue has changed;
[0008] If the real-time dimension table data in the target message queue changes, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried according to the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through the at least one API interface.
[0009] Optionally, in some embodiments, after determining whether the real-time dimension table data in the target message queue has changed, the method further includes:
[0010] If the real-time dimension table data in the target message queue has not changed, the real-time dimension table data in the target message queue is monitored.
[0011] Optionally, in some embodiments, the target relational database is one of MySQL, Oracle and PostgreSQL.
[0012] Optionally, in some embodiments, the target relational database opens a log subscription function, and the acquiring of real-time dimension table data of the vehicle further includes:
[0013] Subscribe to the log data of the target relational database.
[0014] A second aspect of the present application provides a real-time dimension table synchronization device, including:
[0015] An acquisition module, used for acquiring real-time dimension table data of a vehicle, wherein the real-time dimension table data is stored in a target relational database;
[0016] A judgment module, used for synchronizing the real-time dimension table data to a target message queue, and judging whether the real-time dimension table data in the target message queue has changed;
[0017] A synchronization module is used to extract key field information of the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue changes, and reversely query the target relational database according to the key field information to obtain the target dimension table data, and store the target dimension table data in a preset terminal, wherein the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through the at least one API interface.
[0018] Optionally, in some embodiments, after determining whether the real-time dimension table data in the target message queue has changed, the determination module further includes:
[0019] The monitoring unit is used for monitoring the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue does not change.
[0020] Optionally, in some embodiments, the target relational database is one of MySQL, Oracle and PostgreSQL.
[0021] Optionally, in some embodiments, the target relational database opens a log subscription function, and the acquisition module further includes:
[0022] The subscription unit is used to subscribe to the log data of the target relational database.
[0023] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the real-time dimension table synchronization method as described in the above embodiment.
[0024] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the real-time dimension table synchronization method as described in the above embodiment.
[0025] Therefore, by acquiring the real-time dimension table data of the vehicle, the real-time dimension table data is stored in the target relational database, and the real-time dimension table data is synchronized to the target message queue, it is determined whether the real-time dimension table data in the target message queue has changed. If the real-time dimension table data in the target message queue has changed, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried based on the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface. Thus, the problems of insufficient real-time synchronization, poor sharing, and data synchronization delay of real-time dimension table synchronization are solved. This application synchronizes the dimension table that needs to be queried in real time during the use of the program to a third-party memory storage medium. Each application can use the API opened by the third-party memory storage medium to share data access, ensuring the timeliness and accuracy of the data.
[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0028] Figure 1 A flowchart of a real-time dimension table synchronization method provided according to an embodiment of the present application;
[0029] Figure 2 A schematic diagram of synchronizing a real-time dimension table to a target message queue according to an embodiment of the present application;
[0030] Figure 3 A schematic diagram of the principle of synchronizing target dimension table data to a target application according to an embodiment of the present application;
[0031] Figure 4 A flowchart of a real-time dimension table synchronization method provided according to an embodiment of the present application;
[0032] Figure 5 A block diagram of a real-time dimension table synchronization device provided according to an embodiment of the present application;
[0033] Figure 6 A block diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0035] The following describes the synchronization method, device, electronic device and storage medium of the real-time dimension table of the embodiment of the present application with reference to the accompanying drawings. In view of the problems of insufficient real-time performance, poor sharing and data synchronization delay of the real-time dimension table synchronization mentioned in the above background technology, the present application provides a synchronization method of the real-time dimension table. In the method, by acquiring the real-time dimension table data of the vehicle, the real-time dimension table data is stored in the target relational database, and the real-time dimension table data is synchronized to the target message queue, and it is determined whether the real-time dimension table data in the target message queue has changed. If the real-time dimension table data in the target message queue has changed, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried according to the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface. Thus, the problems of insufficient real-time performance, poor sharing and data synchronization delay of the real-time dimension table synchronization are solved. The present application synchronizes the dimension table that needs to be queried in real time during the use of the program to a third-party memory storage medium, and each application can use the API opened by the third-party memory storage medium to share and access the data, thereby ensuring the timeliness and accuracy of the data.
[0036] Specifically, Figure 1 A flowchart of a real-time dimension table synchronization method provided in an embodiment of the present application.
[0037] like Figure 1 As shown, the real-time dimension table synchronization method includes the following steps:
[0038] In step S101, real-time dimension table data of the vehicle is obtained, and the real-time dimension table data is stored in a target relational database.
[0039] Among them, the target relational database is one of MySQL, Oracle and PostgreSQL, the target relational database opens a log subscription function, and the embodiment of the present application subscribes to the log data of the target relational database.
[0040] Specifically, the embodiment of the present application can obtain real-time dimension table data of the vehicle, and the dimension table data may include: vehicle model, color, manufacturer, configuration, VIN (vehicle identification number) and other information,
[0041] In step S102, the real-time dimension table data is synchronized to the target message queue, and it is determined whether the real-time dimension table data in the target message queue has changed.
[0042] Specifically, combined Figure 2 As shown in the figure, for some upstream dimension table data (usually stored in relational databases such as MySQL, Oracle, PostgreSQL, etc.), the real-time change data of the dimension table is synchronized to the target message queue by subscribing to binglog or other log files, and the overall delay is basically in milliseconds.
[0043] Optionally, in some embodiments, after determining whether the real-time dimension table data in the target message queue has changed, the method further includes: if the real-time dimension table data in the target message queue has not changed, monitoring the real-time dimension table data in the target message queue.
[0044] Specifically, when the real-time dimension table data in the target message queue does not change, the real-time dimension table data in the target message queue may be continuously monitored.
[0045] In step S103, if the real-time dimension table data in the target message queue changes, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is inversely queried based on the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, which includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface.
[0046] The preset terminal may be a third-party storage medium.
[0047] Specifically, after subscribing to the data in the target message queue and consuming the relevant change information of the dimension table, the target database (usually a relational database such as MySQL, Oracle, PostgreSQL, etc.) of the real-time dimension table can be reversely queried based on the obtained consumption records, and other tables can be associated to obtain the dimension table information required by the business. After obtaining the dimension table information, the data is flushed to a third-party memory storage medium (such as Redis, Memcached). From the subscription in step S101 to the writing in step S103, the current normal framework processing can be guaranteed within 1S. After writing, the real-time and accurate dimension table information is stored in the third-party memory storage medium. There is generally a general open API for third-party storage media. For example, Java can use the Jedis client to access Redis. When various applications need to use the dimension table association, they can first access the memory and then output the program after the processing is completed.
[0048] In order to enable relevant technical personnel in the field to further understand the real-time dimension table synchronization method of the embodiment of the present application, it is described in detail below in conjunction with specific embodiments.
[0049] like Figure 4 As shown, Figure 4 The present invention is a flowchart of a method for synchronizing a real-time dimension table according to an embodiment of the present application.
[0050] The real-time dimension table data is stored in the target relational database, and the target relational database opens the log subscription function; the real-time dimension table data is sent to the message queue, and the log data in the message queue is subscribed, and a real-time processing program is developed to consume the real-time data in the message queue; if the real-time processing program does not consume the change data of the dimension table, it will not do any processing. If the change data of the dimension table is consumed, the key field information of the dimension table is extracted, and if some related fields are involved, the dimension table relational database is checked in reverse to wait for the complete data of the dimension table. (For example, if a real-time dimension table of a vehicle model needs to be established, and the current relevant business fields cannot directly obtain the association between the vehicle and the vehicle model, then the embodiment of the present application can consume the change information of a vehicle in real time, and then use the key fields (such as material number) in the vehicle information table to obtain the vehicle model information of a vehicle in another material number table, and combine them to generate the dimension table model required by the business).
[0051] Write the obtained dimension table data into the third-party memory storage medium, and develop applications according to business needs. If the application needs to use the dimension table information, it can be directly obtained from the third-party memory storage medium. It can be accessed through some common API methods, such as Java can access the Redis database through Jedis; after the application completes the association processing, the specific processing flow of each program can continue without affecting each other.
[0052] In summary, the embodiments of the present application ensure the accuracy and timeliness of data by synchronizing the real-time change data of the dimension table to a third-party memory storage medium; using a message queue (such as Kafka) to capture and transmit change information of the dimension table to ensure that the changes can be processed in real time; developing a real-time processing program to consume the data in the message queue and update the changed dimension information to the memory storage medium to achieve real-time update of data; providing a general API so that multiple applications can share access to real-time updated dimension table information, thereby improving data sharing and utilization efficiency.
[0053] According to the synchronization method of the real-time dimension table proposed in the embodiment of the present application, the real-time dimension table data of the vehicle is obtained, the real-time dimension table data is stored in the target relational database, and the real-time dimension table data is synchronized to the target message queue to determine whether the real-time dimension table data in the target message queue has changed. If the real-time dimension table data in the target message queue has changed, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried based on the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface. Thus, the problems of insufficient real-time synchronization, poor sharing, and data synchronization delay of the real-time dimension table synchronization are solved. The present application synchronizes the dimension table that needs to be queried in real time during the use of the program to a third-party memory storage medium. Each application can use the API opened by the third-party memory storage medium to share and access the data, ensuring the timeliness and accuracy of the data.
[0054] Next, a real-time dimension table synchronization device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.
[0055] Figure 5 It is a block diagram of a real-time dimension table synchronization device according to an embodiment of the present application.
[0056] like Figure 5 As shown, the real-time dimension table synchronization device 10 includes: an acquisition module 100 , a judgment module 200 and a synchronization module 300 .
[0057] The acquisition module 100 is used to acquire the real-time dimension table data of the vehicle, and the real-time dimension table data is stored in the target relational database.
[0058] The determination module 200 is used to synchronize the real-time dimension table data to the target message queue and determine whether the real-time dimension table data in the target message queue has changed.
[0059] The synchronization module 300 is used to extract the key field information of the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue changes, and reversely query the target relational database based on the key field information to obtain the target dimension table data, and store the target dimension table data in a preset terminal. The preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface.
[0060] Optionally, in some embodiments, after determining whether the real-time dimension table data in the target message queue has changed, the determination module 200 further includes: a monitoring unit.
[0061] The monitoring unit is used for monitoring the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue has not changed.
[0062] Optionally, in some embodiments, the target relational database is one of MySQL, Oracle and PostgreSQL.
[0063] Optionally, in some embodiments, the target relational database opens a log subscription function, and the acquisition module further includes: a subscription unit.
[0064] The subscription unit is used to subscribe to the log data of the target relational database.
[0065] It should be noted that the above explanation of the embodiment of the real-time dimension table synchronization method is also applicable to the real-time dimension table synchronization device of this embodiment, and will not be repeated here.
[0066] According to the synchronization device of the real-time dimension table proposed in the embodiment of the present application, by acquiring the real-time dimension table data of the vehicle, the real-time dimension table data is stored in the target relational database, and the real-time dimension table data is synchronized to the target message queue, and it is determined whether the real-time dimension table data in the target message queue has changed. If the real-time dimension table data in the target message queue has changed, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried based on the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through at least one API interface. Thus, the problems of insufficient real-time synchronization, poor sharing, and data synchronization delay of the real-time dimension table synchronization are solved. The present application synchronizes the dimension table that needs to be queried in real time during the use of the program to a third-party memory storage medium, and each application can use the API opened by the third-party memory storage medium to access the data for data sharing, thereby ensuring the timeliness and accuracy of the data.
[0067] Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0068] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .
[0069] When the processor 602 executes the program, the real-time dimension table synchronization method provided in the above embodiment is implemented.
[0070] Furthermore, the electronic device further comprises:
[0071] The communication interface 603 is used for communication between the memory 601 and the processor 602 .
[0072] The memory 601 is used to store computer programs that can be executed on the processor 602 .
[0073] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.
[0074] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0075] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.
[0076] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.
[0077] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned real-time dimension table synchronization method is implemented.
[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0080] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0081] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.
[0082] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0083] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A real-time dimension table synchronization method, characterized in that: The following steps are involved: Acquire real-time dimension table data of the vehicle, wherein the real-time dimension table data is stored in a target relational database; Synchronize the real-time dimension table data to the target message queue, and determine whether the real-time dimension table data in the target message queue has changed; If the real-time dimension table data in the target message queue changes, the key field information of the real-time dimension table data in the target message queue is extracted, and the target relational database is reversely queried according to the key field information to obtain the target dimension table data, and the target dimension table data is stored in a preset terminal, and the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through the at least one API interface.
2. The method according to claim 1, characterized in that: After determining whether the real-time dimension table data in the target message queue has changed, the method further includes: If the real-time dimension table data in the target message queue has not changed, the real-time dimension table data in the target message queue is monitored.
3. The method according to claim 1, characterized in that The target relational database is one of mysql, oracle and postgre.
4. The method according to claim 1, characterized in that: The target relational database opens a log subscription function, and the real-time dimension table data of the vehicle is obtained, and further includes: Subscribe to the log data of the target relational database.
5. A real-time dimension table synchronization device, characterized in that: include: An acquisition module, used for acquiring real-time dimension table data of a vehicle, wherein the real-time dimension table data is stored in a target relational database; A judgment module, used for synchronizing the real-time dimension table data to a target message queue, and judging whether the real-time dimension table data in the target message queue has changed; A synchronization module is used to extract key field information of the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue changes, and reversely query the target relational database according to the key field information to obtain the target dimension table data, and store the target dimension table data in a preset terminal, wherein the preset terminal includes at least one API interface to synchronize the target dimension table data to the target application through the at least one API interface.
6. The device according to claim 5, characterized in that After determining whether the real-time dimension table data in the target message queue has changed, the determination module further includes: The monitoring unit is used for monitoring the real-time dimension table data in the target message queue when the real-time dimension table data in the target message queue does not change.
7. The device according to claim 5, characterized in that The target relational database is one of mysql, oracle and postgre.
8. The device according to claim 5, characterized in that The target relational database opens a log subscription function, and the acquisition module further includes: The subscription unit is used to subscribe to the log data of the target relational database.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the real-time dimension table synchronization method according to any one of claims 1 to 4.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the real-time dimension table synchronization method as described in any one of claims 1 to 4.