Data Synchronization Method, Device, Equipment and Medium

Abstract

The present application discloses a data synchronization method, apparatus, device and medium, relating to the field of computer technology, including: determining a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and reading source data to be synchronized from the target message queue; judging whether it is necessary to reassign a message queue for the source data to be synchronized based on the target configuration information, and if not, judging whether it is necessary to perform a second-party interface call based on the target configuration information; if it is determined that the second-party interface call needs to be performed, determining the corresponding second-party interface type and the field type mapping relationship corresponding to the source data to be synchronized; and storing the source data to be synchronized into a target data source corresponding to the source data to be synchronized based on the second-party interface type and the field type mapping relationship. Through the above solution, the cost can be reduced and the efficiency can be improved when completing the data synchronization task.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a data synchronization method, apparatus, device and medium. Background Art

[0002] Currently, when the backup file system (i.e., dump) completes each data synchronization task (i.e., dump task) in each business line, developers of the corresponding business line need to write corresponding code for each data synchronization task, that is, each data synchronization task requires a corresponding piece of code. Therefore, when there are many data synchronization tasks, developers need to spend a lot of time writing code, resulting in increased costs; when different data synchronization tasks need to call the same interface, corresponding code still needs to be written each time in order to write the source data into the target data source of ES (ElasticSearch), thus causing the writing of duplicate code and reducing efficiency.

[0003] In summary, how to reduce costs and improve efficiency when completing data synchronization tasks is a problem to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data synchronization method, apparatus, device and medium, which can reduce costs and improve efficiency when completing data synchronization tasks. The specific solutions are as follows:

[0005] In a first aspect, the present application discloses a data synchronization method, including:

[0006] Determine a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and read source data to be synchronized from the target message queue;

[0007] Based on the target configuration information, determine whether it is necessary to reassign a message queue for the source data to be synchronized. If not, based on the target configuration information, determine whether it is necessary to make a second-party interface call;

[0008] If it is determined that the second-party interface call is required, determine the corresponding second-party interface type and the field type mapping relationship corresponding to the source data to be synchronized;

[0009] Store the source data to be synchronized into the target data source corresponding to the source data to be synchronized based on the second-party interface type and the field type mapping relationship.

[0010] Optionally, before determining the target message queue by using the target configuration information configured through the preset synchronization task configuration platform, it further includes:

[0011] The preset synchronization task configuration platform encapsulates the general logic of the source data to be synchronized based on the obtained target data synchronization command to obtain the encapsulated general logic;

[0012] The preset synchronization task configuration platform extracts target configurable items corresponding to the target data synchronization command from the encapsulated general logic, and configures the target configurable items based on the target data synchronization command to obtain target configuration information.

[0013] Optionally, the preset synchronization task configuration platform extracts target configurable items corresponding to the target data synchronization command from the encapsulated general logic, and configures the target configurable items based on the target data synchronization command to obtain target configuration information, including:

[0014] If the preset synchronization task configuration platform reads that the target data synchronization command includes a preset service orchestration configuration command, the preset synchronization task configuration platform extracts interface configurable items from the encapsulated general logic;

[0015] The preset synchronization task configuration platform configures the interface configurable items based on the preset service orchestration configuration command to obtain interface configuration information.

[0016] Optionally, the preset synchronization task configuration platform configures the interface configurable items based on the preset service orchestration configuration command to obtain interface configuration information, including:

[0017] The preset synchronization task configuration platform configures any one or several of the interface configurable items such as the second-party interface type, interface address, interface version, interface alias, interface dependency information, interface input parameter information, the orchestration script of the interface input parameter information, and interface call order based on the preset service orchestration configuration command to obtain interface configuration information.

[0018] Optionally, determining whether to perform a second-party interface call based on the target configuration information includes:

[0019] Determining whether to perform a second-party interface call based on the interface configuration information in the target configuration information.

[0020] Optionally, using the target configuration information configured by the preset synchronization task configuration platform to determine a target message queue, and reading the source data to be synchronized from the target message queue, including:

[0021] Read the basic configuration information in the target configuration information configured through the preset synchronization task configuration platform, and determine the target message queue by using the basic configuration information; wherein, the basic configuration information includes any one or several configuration information of target task name information, target task type information, target service level information, target person in charge information, target ES cluster information, and target ES index information;

[0022] If the target message queue determined by using the basic configuration information is a preset public message queue, read the source data to be synchronized from the preset public message queue; if the target message queue determined by using the basic configuration information is a preset independent message queue, read the source data to be synchronized from the preset independent message queue.

[0023] Optionally, before reading the basic configuration information in the target configuration information configured through the preset synchronization task configuration platform, it further includes:

[0024] Read the target data synchronization command through the preset synchronization task configuration platform to obtain the basic information configuration command, and extract the basic information configurable items corresponding to the basic information configuration command from the encapsulated general logic through the preset synchronization task configuration platform.

[0025] In a second aspect, the present application discloses a data synchronization device, including:

[0026] A source data reading module, configured to determine a target message queue by using the target configuration information configured through the preset synchronization task configuration platform, and read the source data to be synchronized from the target message queue;

[0027] A judgment module, configured to judge whether the source data to be synchronized is to be re-delivered based on the target configuration information, and if not, judge whether to perform a two-party interface call based on the target configuration information;

[0028] An information determination module, configured to determine the two-party interface type and the field type mapping relationship corresponding to the source data to be synchronized if it is determined to perform the two-party interface call;

[0029] A storage module, configured to store the source data to be synchronized into the target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relationship.

[0030] In a third aspect, the present application discloses an electronic device, including:

[0031] A memory, configured to store a computer program;

[0032] A processor, configured to execute the computer program to implement the steps of the data synchronization method disclosed above.

[0033] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the data synchronization method disclosed above are implemented.

[0034] It can be seen that the present application determines a target message queue using target configuration information configured through a preset synchronization task configuration platform, and reads source data to be synchronized from the target message queue; based on the target configuration information, it is determined whether it is necessary to reassign a message queue for the source data to be synchronized, and if not, it is determined based on the target configuration information whether it is necessary to perform a two-party interface call; if it is determined that the two-party interface call needs to be performed, the corresponding two-party interface type and the field type mapping relationship corresponding to the source data to be synchronized are determined; based on the two-party interface type and the field type mapping relationship, the source data to be synchronized is stored in a target data source corresponding to the source data to be synchronized. Thus, it can be seen that the present application introduces a preset synchronization task configuration platform, and the target configuration information is obtained through configuration by the preset synchronization task configuration platform. Since the workload of writing code when the preset synchronization task configuration platform performs configuration is greatly reduced compared to the workload of writing code for data synchronization in the prior art, the cost is reduced; subsequently, based on the target configuration information, a target message queue can be determined to read the source data to be synchronized from the target message queue, it can be determined whether the source data to be synchronized needs to be reassigned a message queue, it can be determined whether the source data to be synchronized needs to perform a two-party interface call, and the field type mapping relationship corresponding to the source data to be synchronized can be obtained, and then the source data to be synchronized is stored in the target data source based on the field type mapping relationship, where basically no code writing work is required, the situation of redundant code is reduced, and the data synchronization efficiency is improved. Description of the Drawings

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0036] Figure 1 It is a flowchart of a data synchronization method disclosed in the present application;

[0037] Figure 2 It is a flowchart of a specific data synchronization method disclosed in the present application;

[0038] Figure 3 It is a flowchart of a specific data synchronization method disclosed in the present application;

[0039] Figure 4A specific data synchronization method flowchart disclosed in this application;

[0040] Figure 5 A specific data synchronization method flowchart disclosed in this application;

[0041] Figure 6 A schematic structural diagram of a data synchronization device disclosed in this application;

[0042] Figure 7 A structural diagram of an electronic device disclosed in this application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0044] Currently, when the backup file system (i.e., dump) completes each data synchronization task (i.e., dump task) in each business line, developers of the corresponding business line need to write corresponding code for each data synchronization task, that is, each data synchronization task requires a corresponding piece of code. Therefore, when there are many data synchronization tasks, developers need to spend a lot of time writing code, resulting in increased costs; when different data synchronization tasks need to call the same interface, corresponding code still needs to be written each time to write the source data into the target data source of ES (ElasticSearch), thus causing the writing of duplicate code and reducing efficiency.

[0045] Therefore, this application correspondingly provides a data synchronization solution, which can reduce costs and improve efficiency when completing data synchronization tasks.

[0046] See Figure 1 As shown, the embodiments of this application disclose a data synchronization method, including:

[0047] Step S11: Determine a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and read the source data to be synchronized from the target message queue.

[0048] In this embodiment, the preset synchronization task configuration platform configures each configurable item of the source data to be synchronized based on the obtained target data synchronization command to obtain the target configuration information. For example Figure 2A specific data synchronization method flow chart is shown. Based on the preset basic information configuration command in the target data synchronization command, the basic information configurable items can be configured to obtain the basic configuration information. The basic information configurable items include any one or several configurable items among the target task name, target task type, target service level, target person in charge, target ES cluster information, and target ES index information. And the target task type can be incremental dump or full dump. Based on the preset service orchestration configuration command in the target data synchronization command, the interface configurable items can be configured to obtain the interface configuration information. The interface configurable items include any one or several configurable items among the second-party interface type, interface address, interface version, interface alias, interface dependency information, interface input parameter information, the orchestration script of the interface input parameter information, and the interface call order. And the second-party interface type can be rpc (Remote Procedure Call) or http (Hyper Text Transfer Protocol). Based on the trigger rule configuration command in the target data synchronization command, the trigger configurable items can be configured to obtain the trigger configuration information. The trigger configurable items can include any one or several configurable items among the data source type, topic, group Id, alias, and whether to reallocate the message queue. And the data source type can be canal or MQ (Message Queue). Based on the resource control configuration command in the target data synchronization command, the resource configurable items can be configured to obtain the resource configuration information. The resource configurable items include any one or several configurable items among the target thread number, target exception policy, and number of retries. And the target exception policy can be failure retry or failure discard. Based on the result mapping configuration command in the target data synchronization command, the mapping configurable items can be configured to obtain the mapping configuration information. The mapping configurable items include ES index mapping information, field type, field type mapping relationship, and the processing script corresponding to the field type mapping relationship. When the system starts the data synchronization service, the system reads the target configuration information obtained after being configured through the preset synchronization task configuration platform, and can determine the target message queue, so as to read the source data to be synchronized from the target message queue.

[0049] Step S12: Based on the target configuration information, determine whether it is necessary to reallocate the message queue for the source data to be synchronized. If not, based on the target configuration information, determine whether it is necessary to perform a second-party interface call.

[0050] In this embodiment, it is possible to determine whether it is necessary to reallocate a message queue for the source data to be synchronized based on the trigger configuration information in the target configuration information. When it is determined that it is necessary to reallocate a message queue for the source data to be synchronized, the reallocated message queue is determined based on the trigger configuration information. For example, when it is determined that the reallocated message queue is a preset independent message queue, the source data to be synchronized is reallocated to the preset independent message queue to ensure resource independence when the subsequent source data to be synchronized performs a data synchronization task. When it is determined that it is not necessary to reallocate a message queue for the source data to be synchronized, it is possible to determine whether it is necessary to perform a secondary interface call based on the interface configuration information in the target configuration information, where the interface configuration information can be obtained by configuring the configurable items of the interface through a preset synchronization task configuration platform based on a preset service orchestration configuration command.

[0051] Step S13: If it is determined that the secondary interface call needs to be performed, determine the corresponding secondary interface type and the field type mapping relationship corresponding to the source data to be synchronized.

[0052] In this embodiment, if it is determined that a secondary interface call needs to be performed, the corresponding secondary interface type is determined. It can be understood that the target configuration information including the interface configuration information is obtained by configuring the secondary interface type in the configurable items of the interface through a preset synchronization task configuration platform based on a preset service orchestration configuration command. When the system reads the interface configuration information in the target configuration information, the secondary interface type is determined. For example, the determined secondary interface type is http; when the system reads the mapping configuration information in the target configuration information, the field type mapping relationship corresponding to the source data to be synchronized is obtained.

[0053] Step S14: Store the source data to be synchronized in the target data source corresponding to the source data to be synchronized based on the secondary interface type and the field type mapping relationship.

[0054] In this embodiment, the source data to be synchronized is stored in the corresponding target data source in ES based on the secondary interface type and the field type mapping relationship. For example, if the determined secondary interface type is rpc, the rpc interface can be called using the generic call method of dubbo to perform secondary processing on the source data to be synchronized, and the processed source data to be synchronized is stored in the corresponding target data source in ES based on the field type mapping relationship.

[0055] It can be seen that the present application determines a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and reads source data to be synchronized from the target message queue; determines whether it is necessary to reassign a message queue for the source data to be synchronized based on the target configuration information, and if not, determines whether it is necessary to perform a second-party interface call based on the target configuration information; if it is determined that the second-party interface call needs to be performed, determines the corresponding second-party interface type and the field type mapping relationship corresponding to the source data to be synchronized; and stores the source data to be synchronized in a target data source corresponding to the source data to be synchronized based on the second-party interface type and the field type mapping relationship. Thus, the present application introduces a preset synchronization task configuration platform, which is configured to obtain target configuration information. Since the workload of writing code when the preset synchronization task configuration platform is configured is greatly reduced compared with the workload of writing code for data synchronization in the prior art, the cost is reduced; subsequently, based on the target configuration information, a target message queue can be determined to read the source data to be synchronized from the target message queue, it can be determined whether the source data to be synchronized needs to be reassigned a message queue, it can be determined whether the source data to be synchronized needs to perform a second-party interface call, and the field type mapping relationship corresponding to the source data to be synchronized can be obtained, and then the source data to be synchronized is stored in the target data source based on the field type mapping relationship, where basically no code writing work is required, the situation of redundant code is reduced, and the data synchronization efficiency is improved.

[0056] See Figure 3 As shown, an embodiment of the present application discloses a specific data synchronization method, including:

[0057] Step S21: Read the basic configuration information in the target configuration information configured through a preset synchronization task configuration platform, and use the basic configuration information to determine a target message queue; wherein, the basic configuration information includes any one or several of target task name information, target task type information, target service level information, target person in charge information, target ES cluster information, and target ES index information.

[0058] In this embodiment, before reading the basic configuration information in the target configuration information configured through a preset synchronization task configuration platform, it further includes: reading a target data synchronization command through the preset synchronization task configuration platform to obtain a basic information configuration command, and extracting, through the preset synchronization task configuration platform, basic information configurable items corresponding to the basic information configuration command from the encapsulated general logic. It can be understood that if the basic information configuration command includes the start time of the data synchronization task, the preset synchronization task configuration platform can configure the start time configurable item to obtain time configuration information, so that the subsequent system can start the data synchronization task based on the time configuration information, realizing the controllability of the data synchronization task time.

[0059] Step S22: If the target message queue determined by using the basic configuration information is a preset common message queue, read the source data to be synchronized from the preset common message queue; if the target message queue determined by using the basic configuration information is a preset independent message queue, read the source data to be synchronized from the preset independent message queue.

[0060] In this embodiment, as Figure 4 shown in a specific data synchronization method flowchart, if the basic information configuration command in the target data synchronization command read by the preset synchronization task configuration platform indicates that the source data to be synchronized 1 is of a high level, that is, it needs to be processed preferentially, or the processing link corresponding to the source data to be synchronized 1 is a core link, the preset synchronization task configuration platform can perform corresponding configuration on the target service level configurable item to obtain the target service level information indicating that the source data to be synchronized 1 needs to be read from the preset independent message queue, thereby avoiding the situation where the resources for processing this dump task cannot be guaranteed. When the system reads this target service level information in the target configuration information, it reads the source data to be synchronized 1 from the preset independent message queue; it can be understood that if the basic information configuration command of the source data to be synchronized N is characterized as a normal level, the preset synchronization task configuration platform can configure the target service level configurable item as the normal level so that the system reads the source data to be synchronized N from the preset common message queue.

[0061] Step S23: Based on the target configuration information, determine whether it is necessary to reassign a message queue for the source data to be synchronized. If not, based on the target configuration information, determine whether it is necessary to perform a two-party interface call.

[0062] In this embodiment, when it is determined that it is necessary to reassign a message queue for the source data to be synchronized, the source data to be synchronized can be default reassigned to the preset common message queue, or the message queue for reassignment can be determined based on the basic configuration information in the target configuration information. For example, when it is determined through the basic configuration information that the message queue for reassignment is a preset independent message queue, the preset independent message queue receives the data corresponding to the source data to be synchronized. It should be noted that the corresponding target configuration information can also be obtained by configuring any one or several configurable items such as the required thread pool, MQ consumer client, task context, and execution node through the preset synchronization task configuration platform, so as to reassign the source data to be synchronized to the corresponding message queue by using the target configuration information.

[0063] Step S24: If it is determined that the two-party interface call needs to be performed, determine the corresponding two-party interface type and the field type mapping relationship corresponding to the source data to be synchronized.

[0064] Step S25: Store the source data to be synchronized into the target data source corresponding to the source data to be synchronized based on the mapping relationship between the two-party interface type and the field type.

[0065] It can be seen that in this application, the preset synchronization task configuration platform configures the basic information configurable items to obtain the basic configuration information. When the subsequent system reads the basic configuration information, it can use the basic configuration information to determine the target message queue. Furthermore, data synchronization tasks with different levels can be in different message queues. Therefore, it can ensure that high-level data synchronization tasks have independent resources, improve the quality of task completion, and the system basically does not need to write code during the task execution process, reducing costs and improving efficiency.

[0066] See Figure 5 As shown, an embodiment of this application discloses a specific data synchronization method, including:

[0067] Step S31: The preset synchronization task configuration platform encapsulates the general logic of the source data to be synchronized based on the obtained target data synchronization command to obtain the encapsulated general logic; the preset synchronization task configuration platform extracts the target configurable items corresponding to the target data synchronization command from the encapsulated general logic, and configures the target configurable items based on the target data synchronization command to obtain the target configuration information.

[0068] In this embodiment, it should be noted that the preset synchronization task configuration platform can visually configure the target configuration information required for the subsequent system to execute the data synchronization task during the data synchronization process, such as the address information and data structure of the source data to be synchronized; the configuration of whether independent resources are required determined according to the business link priority; the configuration corresponding to the two-party interface involved in the subsequent data assembly process; the mapping relationship between the assembled data and the field types of the target data source, etc.

[0069] In this embodiment, the step of the preset synchronization task configuration platform extracting the target configurable items corresponding to the target data synchronization command from the encapsulated general logic and configuring the target configurable items based on the target data synchronization command to obtain the target configuration information includes: if the preset synchronization task configuration platform reads that the target data synchronization command includes a preset service orchestration configuration command, the preset synchronization task configuration platform extracts the interface configurable items from the encapsulated general logic; the preset synchronization task configuration platform configures the interface configurable items based on the preset service orchestration configuration command to obtain the interface configuration information.

[0070] In this embodiment, the step of configuring the interface configurable items based on the preset service orchestration configuration command through the preset synchronization task configuration platform to obtain interface configuration information includes: configuring any one or more of the following interface configurable items through the preset synchronization task configuration platform based on the preset service orchestration configuration command: the second-party interface type, interface address, interface version, interface alias, interface dependency information, interface input parameter information, the orchestration script of the interface input parameter information, and the interface call sequence, so as to obtain interface configuration information.

[0071] Step S32: Read the target configuration information, use the target configuration information to determine a target message queue, and read the source data to be synchronized from the target message queue.

[0072] Step S33: Determine whether it is necessary to reassign a message queue for the source data to be synchronized based on the target configuration information. If not, determine whether it is necessary to perform a second-party interface call based on the target configuration information.

[0073] In this embodiment, determining whether to perform a second-party interface call based on the target configuration information specifically includes: determining whether to perform a second-party interface call based on the interface configuration information in the target configuration information. It can be understood that the preset synchronization task configuration platform configures the interface configurable items based on the preset service orchestration configuration command to obtain interface configuration information. When the preset service orchestration configuration command indicates that the source data to be synchronized needs to be processed twice, the preset synchronization task configuration platform configures the interface configurable items to obtain interface configuration information indicating that a second-party interface call is required. It should be noted that when the preset synchronization task configuration platform configures the interface configurable items based on the preset service orchestration configuration command, the request parameters and response result data required for subsequent second-party interface calls can also be orchestrated using the JsonPath technology. When orchestrating the request parameters, the current interface extracts the cached data in the task context based on the interface request sequence and the preset service orchestration configuration command, and then assembles it into the request parameters for the next interface call; when orchestrating the response data, all the call result data in the task context is assembled into a big json object for use when writing the data to be synchronized into the target data source later.

[0074] Step S34: If it is determined that the second-party interface call is required, determine the corresponding second-party interface type and the field type mapping relationship corresponding to the source data to be synchronized.

[0075] In this embodiment, if it is determined that a secondary interface call is required, the corresponding secondary interface type is determined. For example, when the secondary interface type is an HTTP interface, any one or several pieces of information such as the call address, port, and request method of the HTTP interface can be obtained through the target configuration information, and then the request parameter data is constructed and called through the call method of httpClient. After the call is completed, the call result data is saved into the corresponding task context. When the secondary interface type is an RPC interface, a ReferenceConfig (client call configuration) required for generalization call can be constructed through the version, class name, method name, and grouping information. The actual call IP address (Internet Protocol Address) and port of the dubbo service of the invoker object inside the ReferenceConfig are obtained through the Java reflection method. Using the dubbo telnet technology, the parameter types of the dubbo service are obtained based on the IP address and port, so as to construct a GenericService object required for the dubbo generalization call method, construct the request parameter data, and then the RPC interface call can be performed using the dubbo generalization call method. After the call is completed, the call interface data is saved into the task context.

[0076] Step S35: Store the source data to be synchronized into the target data source corresponding to the source data to be synchronized based on the secondary interface type and the field type mapping relationship.

[0077] In this embodiment, for example, if the determined secondary interface type is an RPC interface, the RPC interface is called to obtain the assembled big json. Before storing the source data to be synchronized into the corresponding target data source based on the field type mapping relationship, it further includes performing a one-to-one mapping process on the index fields in the big json and the ES index information through a preset synchronization task configuration platform according to the JsonPath technology and the Groovy technology to obtain the field type mapping relationship. The JsonPath technology is mainly used to extract the values of the big json, and the Groovy technology performs corresponding processing on the fields that need special processing, such as time data formatting and data type conversion operations, and then assembles them into the data format required by the ES index, so as to store the source data to be synchronized into the corresponding target data source.

[0078] It can be seen that in this application, the relevant configuration of whether a secondary interface call is required is completed by the preset synchronization task configuration platform, and the system only needs to complete the corresponding data synchronization task based on the target configuration information, reducing the burden on the system and thus making the system run more stably.

[0079] See Figure 6As shown in the figure, an embodiment of the present application discloses a data synchronization device, including:

[0080] A source data reading module 11, configured to determine a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and read source data to be synchronized from the target message queue;

[0081] A judgment module 12, configured to judge whether the source data to be synchronized needs to be redelivered based on the target configuration information, and if not, judge whether to perform a two-party interface call based on the target configuration information;

[0082] An information determination module 13, configured to determine the two-party interface type and the field type mapping relationship corresponding to the source data to be synchronized if it is determined to perform the two-party interface call;

[0083] A storage module 14, configured to store the source data to be synchronized into a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relationship.

[0084] It can be seen that the present application determines a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and reads source data to be synchronized from the target message queue; judges whether it is necessary to reallocate a message queue for the source data to be synchronized based on the target configuration information, and if not, judges whether it is necessary to perform a two-party interface call based on the target configuration information; determines the corresponding two-party interface type and the field type mapping relationship corresponding to the source data to be synchronized if it is determined to perform the two-party interface call; stores the source data to be synchronized into a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relationship. Thus, it can be seen that the present application introduces a preset synchronization task configuration platform, and the target configuration information is obtained through the configuration of the preset synchronization task configuration platform. Since the workload of writing code during the configuration of the preset synchronization task configuration platform is greatly reduced compared to the workload of writing code for data synchronization in the prior art, the cost is reduced; subsequently, based on the target configuration information, a target message queue can be determined to read the source data to be synchronized from the target message queue, it can be judged whether the source data to be synchronized is reallocated with a message queue, it can be judged whether the source data to be synchronized needs to perform a two-party interface call, and the field type mapping relationship corresponding to the source data to be synchronized can be obtained, and then the source data to be synchronized is stored into the target data source based on the field type mapping relationship, where basically no code writing work is required, the situation of redundant code is reduced, and the data synchronization efficiency is improved.

[0085] Figure 7Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data synchronization method executed by the electronic device disclosed in any of the foregoing embodiments.

[0086] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0087] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computing operations related to machine learning.

[0088] In addition, the memory 22, as a carrier for resource storage, may be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc., and the storage method may be short-term storage or permanent storage.

[0089] Among them, the operating system 221 is used to manage and control each hardware device and computer program 222 on the electronic device, so as to implement the operation and processing of the massive data 223 in the memory 22 by the processor 21. It can be Windows, Unix, Linux, etc. In addition to the computer program that can be used to complete the data synchronization method executed by the electronic device disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks. The data 223 may include not only the data transmitted by external devices received by the electronic device, but also the data collected by its own input / output interface 25, etc.

[0090] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium. A computer program is stored in the storage medium. When the computer program is loaded and executed by a processor, the method steps executed during the data synchronization process disclosed in any of the foregoing embodiments are implemented.

[0091] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0092] The above has introduced in detail a data synchronization method, device, equipment and medium provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A data synchronization method, characterized in that, Including: Determine a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and read source data to be synchronized from the target message queue; the target configuration information includes resource configuration information, basic configuration information, interface configuration information, trigger configuration information, and mapping configuration information; Based on the target configuration information, determine whether it is necessary to reassign a message queue for the source data to be synchronized. If not, determine whether it is necessary to perform a secondary interface call based on the target configuration information; If it is determined that the secondary interface call needs to be performed, determine the corresponding secondary interface type and the field type mapping relationship corresponding to the source data to be synchronized; Based on the secondary interface type and the field type mapping relationship, store the source data to be synchronized in a target data source corresponding to the source data to be synchronized; Among them, before determining the target message queue by using the target configuration information configured through the preset synchronization task configuration platform, it further includes: Encapsulate the general logic of the source data to be synchronized through the preset synchronization task configuration platform based on the obtained target data synchronization command to obtain the encapsulated general logic; extract the target configurable items corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform, and configure the target configurable items based on the target data synchronization command to obtain the target configuration information; The determining the target message queue by using the target configuration information configured through the preset synchronization task configuration platform and reading the source data to be synchronized from the target message queue includes: Read the basic configuration information in the target configuration information configured through the preset synchronization task configuration platform, and use the basic configuration information to determine the target message queue; wherein, the basic configuration information includes any one or several configuration information of target task name information, target task type information, target service level information, target person in charge information, target ES cluster information, and target ES index information; if the target message queue determined by using the basic configuration information is a preset public message queue, read the source data to be synchronized from the preset public message queue; if the target message queue determined by using the basic configuration information is a preset independent message queue, read the source data to be synchronized from the preset independent message queue.

2. The data synchronization method according to claim 1, wherein The extracting the target configurable items corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform and configuring the target configurable items based on the target data synchronization command to obtain the target configuration information includes: If it is read through the preset synchronization task configuration platform that the target data synchronization command includes a preset service orchestration configuration command, extract the interface configurable items in the encapsulated general logic through the preset synchronization task configuration platform; Configure the interface configurable items through the preset synchronization task configuration platform based on the preset service orchestration configuration command to obtain the interface configuration information.

3. The data synchronization method according to claim 2, wherein Configuring the interface configurable items based on the preset service orchestration configuration command through the preset synchronization task configuration platform to obtain interface configuration information includes: Configuring any one or several of the interface configurable items, such as the second-party interface type, interface address, interface version, interface alias, interface dependency information, interface input parameter information, the orchestration script of the interface input parameter information, and interface call sequence, based on the preset service orchestration configuration command through the preset synchronization task configuration platform to obtain interface configuration information.

4. The data synchronization method according to claim 2, wherein Judging whether to perform a second-party interface call based on the target configuration information includes: Judging whether to perform a second-party interface call based on the interface configuration information in the target configuration information.

5. The data synchronization method according to claim 1, wherein Before reading the basic configuration information in the target configuration information configured by the preset synchronization task configuration platform, it further includes: Reading the target data synchronization command through the preset synchronization task configuration platform to obtain the basic information configuration command, and extracting the basic information configurable items corresponding to the basic information configuration command from the encapsulated general logic through the preset synchronization task configuration platform.

6. A data synchronization device, characterized in that, It includes: A source data reading module, configured to determine a target message queue by using the target configuration information configured by the preset synchronization task configuration platform, and read the source data to be synchronized from the target message queue; the target configuration information includes resource configuration information, basic configuration information, interface configuration information, trigger configuration information, and mapping configuration information; A judgment module, configured to judge whether to perform re-delivery processing on the source data to be synchronized based on the target configuration information, and if not, judge whether to perform a second-party interface call based on the target configuration information; An information determination module, configured to determine the second-party interface type and the field type mapping relationship corresponding to the source data to be synchronized if it is determined to perform the second-party interface call; A storage module, configured to store the source data to be synchronized into the target data source corresponding to the source data to be synchronized based on the second-party interface type and the field type mapping relationship; Among them, the data synchronization device is specifically configured to: Encapsulating the general logic of the source data to be synchronized based on the obtained target data synchronization command through the preset synchronization task configuration platform to obtain the encapsulated general logic; extracting the target configurable items corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform, and configuring the target configurable items based on the target data synchronization command to obtain the target configuration information; The source data reading module is specifically configured to: Read the basic configuration information in the target configuration information configured through the preset synchronization task configuration platform, and determine the target message queue by using the basic configuration information; wherein, the basic configuration information includes any one or several configuration information of target task name information, target task type information, target service level information, target person in charge information, target ES cluster information, and target ES index information; if the target message queue determined by using the basic configuration information is a preset public message queue, read the source data to be synchronized from the preset public message queue; if the target message queue determined by using the basic configuration information is a preset independent message queue, read the source data to be synchronized from the preset independent message queue.

7. An electronic device, characterized in that, Including: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the data synchronization method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, For storing a computer program; wherein, when the computer program is executed by the processor, the steps of the data synchronization method according to any one of claims 1 to 5 are implemented.

Images