Data processing method and device, electronic device and computer-readable storage medium

Abstract

The present disclosure provides a data processing method and device, an electronic device, and a computer-readable storage medium, which can be applied to the fields of data processing technology and financial technology. The data processing method includes: in response to a first scheduled task being triggered, obtaining data to be processed according to first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the data to be processed has at least one object type; according to each object type in the at least one object type, determining configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier; processing the data to be processed according to at least one preset rule corresponding to each object type to obtain processed data; and processing the processed data according to at least one preset field identifier corresponding to each object type to obtain target data.

Description

Technical Field

[0001] The present disclosure relates to the fields of data processing technology and financial technology, and more specifically, to a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of data processing technology, the demand for data synchronization between data providers and data recipients is increasing. Data providers can refer to the system that provides data. Data recipients can refer to the system that receives data.

[0003] Data synchronization refers to the process of synchronizing different types of data from a data provider to a data recipient. Data synchronization methods can generally include one of the following: data synchronization methods based on a separate interface and data synchronization methods based on a big data platform.

[0004] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technologies: in the process of realizing data synchronization, the development workload and resource consumption are large. Summary of the Invention

[0005] In view of this, the present disclosure provides a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0007] In response to the first scheduled task being triggered, obtaining the data to be processed according to the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is used to process the data to be processed, and the data to be processed has at least one object type;

[0008] Determining, according to each object type of the at least one object type, configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier;

[0009] Processing the data to be processed according to at least one preset rule corresponding to each of the object types to obtain processed data; and

[0010] The processed data is processed according to at least one preset field identifier corresponding to each object type to obtain target data.

[0011] According to an embodiment of the present disclosure, each object type in the at least one object type is used to represent a target object, and the target object includes at least one of the following: a target interface object, a target file object, and a target database object.

[0012] According to an embodiment of the present disclosure, the determining, based on each object type of the at least one object type, configuration information corresponding to each object type includes:

[0013] In a case where the at least one object type represents that the target object includes the target interface object, determining configuration information corresponding to the target interface object according to the first mapping relationship;

[0014] In a case where the at least one object type represents that the target object includes the target file object, determining configuration information corresponding to the target file object according to the second mapping relationship; and

[0015] In a case where the at least one object type represents that the target object includes the target database object, configuration information corresponding to the target database object is determined according to the third mapping relationship.

[0016] According to an embodiment of the present disclosure, the first mapping relationship includes at least one first key-value pair, and the at least one first key-value pair is determined according to at least one interface object and configuration information corresponding to the at least one interface object.

[0017] According to an embodiment of the present disclosure, the second mapping relationship includes at least one second key-value pair, and the at least one second key-value pair is determined according to at least one file object and configuration information corresponding to each of the at least one file object.

[0018] According to an embodiment of the present disclosure, the third mapping relationship includes at least one third key-value pair, and the at least one third key-value pair is determined according to at least one database object and configuration information corresponding to the at least one database object.

[0019] According to an embodiment of the present disclosure, the data to be processed is processed according to at least one preset rule corresponding to each object type to obtain the processed data, including:

[0020] For each of the at least one object type,

[0021] Determining a data type corresponding to the object type, wherein the data type includes at least one of the following: structured data, semi-structured data, and unstructured data;

[0022] Determining the at least one preset rule according to the object type and the data type, wherein the preset rule includes at least one of the following: a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule; and

[0023] According to at least one of the data combination rule, the data conversion rule, the field addition rule and the data encryption and decryption rule, the data to be processed corresponding to the object type is processed to obtain the processed data.

[0024] According to an embodiment of the present disclosure, the processed data is processed according to at least one preset field identifier corresponding to each object type to obtain target data, including:

[0025] For each of the at least one object type,

[0026] Determining the at least one preset field identifier corresponding to the object type and at least one field identifier in the processed data;

[0027] Determining, based on the at least one preset field identifier and the at least one field identifier, a similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier;

[0028] In response to the similarity being greater than or equal to a preset threshold, determining the processed data corresponding to the target field identifier as the processed data corresponding to the target preset field identifier; and

[0029] The target data is determined based on the processed data corresponding to the target preset field identifier.

[0030] According to an embodiment of the present disclosure, the first task information includes task association information, the at least one address information includes at least one source address information, and the at least one source address information each corresponds to an active system.

[0031] According to an embodiment of the present disclosure, in response to the first scheduled task being triggered, obtaining the data to be processed according to the first task information in the first scheduled task and at least one address information corresponding to the data to be processed includes:

[0032] In response to the first scheduled task being triggered, obtaining the first task information in the first scheduled task; and

[0033] When the task association information indicates that the first scheduled task has no associated task, the data to be processed is obtained from a source system corresponding to each of the at least one source address information according to the at least one source address information corresponding to the data to be processed.

[0034] According to an embodiment of the present disclosure, the above method further includes:

[0035] If the task association information indicates that the first scheduled task has an associated task, determining at least one second scheduled task, wherein the at least one second scheduled task includes a scheduled task having a forward association relationship with the first scheduled task; and

[0036] In response to detecting that the second scheduled task is completed, the data to be processed is obtained from a source system corresponding to each of the at least one source address information according to the at least one source address information corresponding to the data to be processed.

[0037] According to an embodiment of the present disclosure, the above-mentioned first task information also includes data paging information corresponding to the above-mentioned data to be processed. The above-mentioned data paging information is determined based on the total data volume of the above-mentioned data to be processed. The above-mentioned data paging information includes at least one page number information and the data volume corresponding to each of the above-mentioned at least one page number information.

[0038] According to an embodiment of the present disclosure, the step of obtaining the data to be processed from a source system corresponding to each of the at least one source address information according to the at least one source address information corresponding to the data to be processed includes:

[0039] Based on the at least one page number information and the amount of data corresponding to each of the at least one page number information, and according to the at least one source address information, obtaining, from the source system, portions of the to-be-processed data corresponding to each of the at least one page number information; and

[0040] The data to be processed is determined according to the portion of the data to be processed corresponding to each of the at least one page number information.

[0041] According to an embodiment of the present disclosure, the configuration information corresponding to the target interface object includes at least one of the following: a target interface address, a target interface user name, and a target interface password.

[0042] According to an embodiment of the present disclosure, the configuration information corresponding to the target file object includes at least one of the following: a target file transfer protocol address, a target file transfer protocol user name, a target file transfer protocol password, a target port, and a target file path.

[0043] According to an embodiment of the present disclosure, the configuration information corresponding to the target database object includes at least one of the following: a target database address, a target database name, a target instance name, a target database user name, and a target database password.

[0044] According to an embodiment of the present disclosure, the at least one address information further includes at least one target address information, and each of the at least one target address information corresponds to a target system.

[0045] According to an embodiment of the present disclosure, the method further includes, after processing the processed data according to at least one preset field identifier corresponding to each object type to obtain target data:

[0046] The target data is sent to a target system corresponding to each of the at least one target address information, so that the target system can store the target data.

[0047] According to another aspect of the present disclosure, there is provided a data processing apparatus, comprising:

[0048] an acquisition module, configured to, in response to a first scheduled task being triggered, acquire the data to be processed based on first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is configured to process the data to be processed, and the data to be processed has at least one object type;

[0049] a determination module, configured to determine, based on each object type of the at least one object type, configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier;

[0050] A first processing module is configured to process the data to be processed according to at least one preset rule corresponding to each of the object types to obtain processed data; and

[0051] The second processing module is configured to process the processed data according to at least one preset field identifier corresponding to each object type to obtain target data.

[0052] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0053] one or more processors;

[0054] a memory for storing one or more instructions,

[0055] When the one or more instructions are executed by the one or more processors, the one or more processors implement the method described in the present disclosure.

[0056] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the processor implements the method described in the present disclosure.

[0057] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed, they are used to implement the method described in the present disclosure.

[0058] According to an embodiment of the present disclosure, since the data to be processed is obtained based on the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, the data to be processed can be automatically obtained in response to the first scheduled task being triggered, thereby improving the efficiency of data acquisition. In addition, since the configuration information corresponding to each object type is determined based on each object type in at least one object type, the data to be processed is processed according to at least one preset rule included in the configuration information corresponding to each object type to obtain processed data. On this basis, the processed data is processed according to at least one preset field identifier included in the configuration information corresponding to each object type to obtain target data, and the data to be processed can be automatically processed, thereby at least partially overcoming the technical problem of large development workload and resource consumption in the process of realizing data synchronization in the related art, reducing the development workload and resource consumption, and thereby improving the efficiency and accuracy of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0060] Figure 1 The system architecture to which the data processing method according to the embodiment of the present disclosure can be applied is schematically shown;

[0061] Figure 2 The following schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;

[0062] Figure 3A Schematically illustrates an example of a method for determining configuration information corresponding to each object type according to each object type of at least one object type according to an embodiment of the present disclosure;

[0063] Figure 3BSchematically illustrates an example diagram of a method for determining configuration information corresponding to each object type according to each object type of at least one object type according to another embodiment of the present disclosure;

[0064] Figure 3C Schematically illustrates an example diagram of a method for determining configuration information corresponding to each object type according to each object type of at least one object type according to another embodiment of the present disclosure;

[0065] Figure 4 Schematically illustrates an example schematic diagram of a method for processing data to be processed according to at least one preset rule corresponding to each object type to obtain processed data according to an embodiment of the present disclosure;

[0066] Figure 5 Schematically illustrates an example schematic diagram of a method for processing processed data according to at least one preset field identifier corresponding to each object type to obtain target data according to an embodiment of the present disclosure;

[0067] Figure 6 Schematically illustrates an example of a method for obtaining data to be processed according to first task information in the first scheduled task and at least one address information corresponding to the data to be processed in response to a first scheduled task being triggered according to an embodiment of the present disclosure;

[0068] Figure 7 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0069] Figure 8 A block diagram schematically shows an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0071] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0072] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0073] When expressions such as “at least one of A, B, and C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). When expressions such as “at least one of A, B, or C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, or C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.).

[0074] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0075] Data synchronization can be used to describe the process of synchronizing different types of data from a data provider to a data receiver. In related technologies, data synchronization methods generally include one of the following: a data synchronization method based on a separate interface and a data synchronization method based on a big data platform.

[0076] However, the data synchronization method based on a single interface requires the corresponding interface of the other party to be adapted and changed when any interface of the data provider or the data receiver is changed, which results in a large development workload.

[0077] In addition, since the data synchronization method based on the big data platform involves many components and is complex to deploy, it leads to high resource consumption and thus cannot guarantee the cost of data synchronization.

[0078] In order to at least partially solve the technical problems existing in the related art, the present disclosure provides a data processing method and device, an electronic device and a computer-readable storage medium, which can be applied to the field of data processing technology and the field of financial technology. The data processing method includes: in response to a first scheduled task being triggered, according to the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, obtaining the data to be processed, the data to be processed having at least one object type; according to each object type in the at least one object type, determining the configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier; according to the at least one preset rule corresponding to each object type, processing the data to be processed to obtain processed data; and according to the at least one preset field identifier corresponding to each object type, processing the processed data to obtain target data.

[0079] It should be noted that the data processing methods and devices provided in the embodiments of the present disclosure can be used in the fields of data processing technology and financial technology, for example, in data access and data synchronization between different systems. The data processing methods and devices provided in the embodiments of the present disclosure can also be used in any field other than the fields of data processing technology and financial technology, for example, in the field of Internet technology. The application fields of the data processing methods and devices provided in the embodiments of the present disclosure are not limited.

[0080] Figure 1 The system architecture to which the data processing method according to the embodiment of the present disclosure can be applied is schematically shown. Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0081] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0082] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0083] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0084] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received user requests and other data, and feed back processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0085] It should be noted that the data processing method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiments of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0086] Alternatively, the data processing method provided in the embodiments of the present disclosure may be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be executed by another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing apparatus provided in the embodiments of the present disclosure may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0087] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0088] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0089] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0090] like Figure 2 As shown, the data processing method 200 includes operations S210 to S240.

[0091] In operation S210, in response to the first scheduled task being triggered, the data to be processed is obtained according to the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is used to process the data to be processed, and the data to be processed has at least one object type.

[0092] In operation S220, configuration information corresponding to each object type is determined according to each object type of the at least one object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier.

[0093] In operation S230 , the data to be processed is processed according to at least one preset rule corresponding to each object type to obtain processed data.

[0094] In operation S240 , the processed data is processed according to at least one preset field identifier corresponding to each object type to obtain target data.

[0095] According to an embodiment of the present disclosure, a first scheduled task may include triggering execution of a first action at a first predetermined time. The first scheduled task may be associated with data to be processed. The first action may include processing the data to be processed. In this case, the first scheduled task may be used to trigger processing of the data to be processed at a first predetermined time. The first predetermined time may include a first predetermined time point and a first predetermined time period. The first predetermined time may be set according to actual business needs and is not limited here.

[0096] For example, if the first scheduled time is a first scheduled time point, the first scheduled time can be set to 12:00. In this case, the first scheduled task is triggered in response to the current system time reaching 12:00. Alternatively, if the first scheduled time is a first scheduled time period, the first scheduled time can be set to 12 hours. In this case, the first scheduled task is triggered in response to the current system time interval reaching 12 hours.

[0097] According to an embodiment of the present disclosure, the first task information may include task association information and data paging information corresponding to the data to be processed. The at least one address information corresponding to the data to be processed may include at least one source address information and at least one target address information. Each source address information in the at least one source address information may be used to identify a data provider. The data provider may be a system that provides the data to be processed. Each target address information in the at least one target address information may be used to identify a data recipient. The data recipient may be a system that needs to obtain the data to be processed.

[0098] According to an embodiment of the present disclosure, the at least one object type may include an interface object type, a file object type, and a database object type. Each of the at least one object type is used to represent a target object. For example, the interface object type may be used to represent a target interface object. The file object type may be used to represent a target file object. The database object type may be used to represent a target database object.

[0099] According to an embodiment of the present disclosure, in response to a first scheduled task being triggered, first task information in the first scheduled task and at least one address information corresponding to the data to be processed can be determined. The data to be processed can be obtained based on the first task information in the first scheduled task and the at least one address information corresponding to the data to be processed.

[0100] According to an embodiment of the present disclosure, after obtaining the data to be processed, at least one object type corresponding to the data to be processed can be determined based on the data to be processed. Based on each object type in the at least one object type, configuration information corresponding to each object type is determined. The configuration information can be used to characterize the configuration corresponding to each object type. The configuration information may include at least one preset rule and at least one preset field identifier corresponding to the object type. Each preset rule in the at least one preset rule corresponding to the object type can be used to characterize the processing rules for the data to be processed corresponding to the object type. The preset rules may include at least one of the following: data combination rules, data conversion rules, field addition rules, and data encryption and decryption rules. Each preset field identifier in the at least one preset field identifier corresponding to the object type can be used to characterize the target preset field identifier of the data to be processed corresponding to the object type.

[0101] According to an embodiment of the present disclosure, after obtaining configuration information corresponding to each object type, the data to be processed can be processed according to at least one preset rule in the configuration information corresponding to each object type to obtain processed data. The processed data can be used to represent the data processed according to the at least one preset rule corresponding to each object type. The processed data can be processed according to at least one preset field identifier in the configuration information corresponding to each object type to obtain target data. The target data can be used to represent the data processed according to the at least one preset field identifier corresponding to each object type.

[0102] According to an embodiment of the present disclosure, after obtaining target data, the target data can be sent to a target system corresponding to at least one target address information. After receiving the target data, the target system can store the target data in a local database of the target system. The local database of the target system can include a relational database and a non-relational database.

[0103] According to an embodiment of the present disclosure, since the data to be processed is obtained based on the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, the data to be processed can be automatically obtained in response to the first scheduled task being triggered, thereby improving the efficiency of data acquisition. In addition, since the configuration information corresponding to each object type is determined based on each object type in at least one object type, the data to be processed is processed according to at least one preset rule included in the configuration information corresponding to each object type to obtain processed data. On this basis, the processed data is processed according to at least one preset field identifier included in the configuration information corresponding to each object type to obtain target data, and the data to be processed can be automatically processed, thereby at least partially overcoming the technical problem of large development workload and resource consumption in the process of realizing data synchronization in the related art, reducing the development workload and resource consumption, and thereby improving the efficiency and accuracy of data processing.

[0104] Reference below Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 4 、 Figure 5 and Figure 6 , the data processing method 200 according to an embodiment of the present invention is further described.

[0105] According to an embodiment of the present disclosure, operation S220 may include the following operations.

[0106] If at least one object type representing the target object includes a target interface object, configuration information corresponding to the target interface object is determined based on a first mapping relationship. If at least one object type representing the target object includes a target file object, configuration information corresponding to the target file object is determined based on a second mapping relationship. If at least one object type representing the target object includes a target database object, configuration information corresponding to the target database object is determined based on a third mapping relationship.

[0107] According to an embodiment of the present disclosure, each object type in the at least one object type can be used to represent a target object. The target object can include at least one of the following: a target interface object, a target file object, and a target database object.

[0108] According to an embodiment of the present disclosure, a first mapping relationship may be used to characterize a mapping relationship between an interface object and configuration information. For example, the first mapping relationship may include a one-to-one correspondence between the interface object and the configuration information. A second mapping relationship may be used to characterize a mapping relationship between a file object and the configuration information. For example, the second mapping relationship may include a one-to-one correspondence between the file object and the configuration information. A third mapping relationship may be used to characterize a mapping relationship between a database object and the configuration information. For example, the third mapping relationship may include a one-to-one correspondence between the database object and the configuration information.

[0109] According to an embodiment of the present disclosure, when at least one object type representing a target object includes a target interface object, a first similarity can be calculated between an identifier corresponding to the target interface object and an identifier corresponding to an interface object in a first mapping relationship. In response to the first similarity being greater than a first preset threshold, a matching interface object in the first mapping relationship can be determined. Configuration information corresponding to the matching interface object in the first mapping relationship can be determined as the configuration information corresponding to the target interface object.

[0110] According to an embodiment of the present disclosure, when at least one object type characterizing a target object includes a target file object, a second similarity may be calculated between an identifier corresponding to the target file object and an identifier corresponding to a file object in a first mapping relationship. In response to the second similarity being greater than a second preset threshold, a matching file object in the second mapping relationship may be determined. Configuration information corresponding to the matching file object in the second mapping relationship may be determined as the configuration information corresponding to the target file object.

[0111] According to an embodiment of the present disclosure, when at least one object type representing a target object includes a target database object, a third similarity may be calculated between an identifier corresponding to the target database object and an identifier corresponding to a database object in a third mapping relationship. In response to the third similarity being greater than a third preset threshold, a matching database object in the third mapping relationship may be determined. Configuration information corresponding to the matching database object in the third mapping relationship may be determined as the configuration information corresponding to the target database object.

[0112] According to an embodiment of the present disclosure, the first preset threshold, the second preset threshold, and the third preset threshold can be set according to actual business needs and are not limited here. The first preset threshold, the second preset threshold, and the third preset threshold can be the same, for example, they can all be set to 0.95. Alternatively, the first preset threshold, the second preset threshold, and the third preset threshold can also be different, for example, the first preset threshold can be 0.9, the second preset threshold can be 0.85, and the third preset threshold can be 0.8.

[0113] According to an embodiment of the present disclosure, the first mapping relationship includes at least one first key-value pair, and the at least one first key-value pair is determined according to at least one interface object and configuration information corresponding to the at least one interface object.

[0114] According to an embodiment of the present disclosure, the second mapping relationship includes at least one second key-value pair, and the at least one second key-value pair is determined according to at least one file object and configuration information corresponding to the at least one file object.

[0115] According to an embodiment of the present disclosure, the third mapping relationship includes at least one third key-value pair, and the at least one third key-value pair is determined according to at least one database object and configuration information corresponding to the at least one database object.

[0116] According to an embodiment of the present disclosure, an interface object may include a Hypertext Transfer Protocol (HTTP) object. A file object may include a text file object, a binary file object, and a comma-separated value (CSV) file object. A database object may include a relational database object, such as Oracle, SQL Server, Sybase, Informix, Access, DB2, or MySQL. A database object may also include a non-relational database object, such as Hbase, Cassandra, SimpleDB, CouchDB, MongoDB, or Redis.

[0117] According to an embodiment of the present disclosure, at least one first key-value pair may be determined based on at least one interface object and configuration information corresponding to each of the at least one interface object. After determining the at least one first key-value pair, a first mapping relationship may be determined based on the at least one first key-value pair. The first mapping relationship may be used to characterize a mapping relationship between the interface object and the configuration information.

[0118] According to an embodiment of the present disclosure, at least one second key-value pair may be determined based on at least one file object and configuration information corresponding to each of the at least one file object. After determining the at least one second key-value pair, a second mapping relationship may be determined based on the at least one second key-value pair. The second mapping relationship may be used to represent a mapping relationship between the file object and the configuration information.

[0119] According to an embodiment of the present disclosure, at least one third key-value pair may be determined based on at least one database object and configuration information corresponding to each of the at least one database object. After determining the at least one third key-value pair, a third mapping relationship may be determined based on the at least one third key-value pair. The third mapping relationship may be used to represent a mapping relationship between the database object and the configuration information.

[0120] According to an embodiment of the present disclosure, since the first mapping relationship includes at least one first key-value pair, and the at least one first key-value pair is determined based on at least one interface object and the configuration information corresponding to the at least one interface object, the configuration information corresponding to the target interface object can be automatically determined based on the first mapping relationship, thereby improving the efficiency of obtaining the configuration information corresponding to the interface object. In addition, since the second mapping relationship includes at least one second key-value pair, and the at least one second key-value pair is determined based on at least one file object and the configuration information corresponding to the at least one file object, the configuration information corresponding to the target file object can be automatically determined based on the second mapping relationship, thereby improving the efficiency of obtaining the configuration information corresponding to the file object. In addition, since the third mapping relationship includes at least one third key-value pair, and the at least one third key-value pair is determined based on at least one database object and the configuration information corresponding to the at least one database object, the configuration information corresponding to the target database object can be automatically determined based on the third mapping relationship, thereby improving the efficiency of obtaining the configuration information corresponding to the database object.

[0121] According to an embodiment of the present disclosure, the configuration information corresponding to the target interface object includes at least one of the following: a target interface address, a target interface user name, and a target interface password.

[0122] According to an embodiment of the present disclosure, the configuration information corresponding to the target file object includes at least one of the following: a target file transfer protocol address, a target file transfer protocol user name, a target file transfer protocol password, a target port, and a target file path.

[0123] According to an embodiment of the present disclosure, the configuration information corresponding to the target database object includes at least one of the following: a target database address, a target database name, a target instance name, a target database user name, and a target database password.

[0124] According to an embodiment of the present disclosure, the configuration information may include information related to the data source. For example, the information related to the data source may include the data source name, data source type, data source driver, data source Uniform Resource Locator (URL), data source user name, and data source password.

[0125] According to an embodiment of the present disclosure, when at least one object type representing a target object includes a target interface object, the configuration information corresponding to the target interface object includes at least one of the following: a target interface address, a target interface username, and a target interface password. Alternatively, the configuration information corresponding to the target interface object may further include a data paging acquisition method, parameter management, and data return fields.

[0126] According to an embodiment of the present disclosure, the target object may further include an open source stream processing platform (e.g., Kafka). When at least one object type characterizing the target object includes Kafka, the configuration information corresponding to Kafka may include a Kafka cluster address, a Kafka message category (i.e., topic), a Kafka pattern, a data encryption / decryption format, and a Kafka received data field.

[0127] Figure 3A An example diagram of a method for determining configuration information corresponding to each object type according to each object type of at least one object type according to an embodiment of the present disclosure is schematically shown.

[0128] like Figure 3A As shown, in 300A, at least one first key-value pair 303 can be determined based on at least one interface object 301 and configuration information 302 corresponding to the at least one interface object 301. After determining the at least one first key-value pair 303, a first mapping relationship 304 can be determined based on the at least one first key-value pair 303.

[0129] In the case that at least one object type represents that the target object includes a target interface object, configuration information 305 corresponding to the target interface object may be determined according to the first mapping relationship 304 .

[0130] Figure 3B An example diagram of a method for determining configuration information corresponding to each object type in at least one object type according to another embodiment of the present disclosure is schematically shown.

[0131] like Figure 3B As shown in 300B, at least one second key-value pair 308 may be determined based on at least one file object 306 and configuration information 307 corresponding to each of the at least one file object 306. After determining the at least one second key-value pair 308, a second mapping relationship 309 may be determined based on the at least one second key-value pair 308.

[0132] In the case that at least one object type represents that the target object includes a target file object, configuration information 310 corresponding to the target file object may be determined according to the second mapping relationship 309 .

[0133] Figure 3C An example diagram of a method for determining configuration information corresponding to each object type in at least one object type according to another embodiment of the present disclosure is schematically shown.

[0134] like Figure 3C As shown, in 300C, at least one third key-value pair 313 may be determined based on at least one database object 311 and configuration information 312 corresponding to each of the at least one database object 311. After determining the at least one third key-value pair 313, a third mapping relationship 314 may be determined based on the at least one third key-value pair 313.

[0135] In the case that at least one object type represents that the target object includes a target database object, configuration information 315 corresponding to the target database object may be determined according to the third mapping relationship 314 .

[0136] According to an embodiment of the present disclosure, operation S230 may include the following operations.

[0137] For each object type in at least one object type, a data type corresponding to the object type is determined, where the data type includes at least one of the following: structured data, semi-structured data, and unstructured data. Based on the object type and the data type, at least one preset rule is determined, where the preset rule includes at least one of the following: a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule. Based on at least one of the data combination rule, the data conversion rule, the field addition rule, and the data encryption and decryption rule, the data to be processed corresponding to the object type is processed to obtain processed data.

[0138] According to an embodiment of the present disclosure, after obtaining configuration information corresponding to each object type, the data type corresponding to the object type can be determined according to the object type. The data type may include structured data, semi-structured data, and unstructured data.

[0139] According to an embodiment of the present disclosure, structured data may include data obtained from a structured data source. For example, structured data sources may include: DAMO, Oracle, SQL Server, DB2, Sybase, MySQL, MariaDB, and PostgreSQL. Semi-structured data may include data obtained from a semi-structured data source. For example, semi-structured data sources may include MongoDB, Redis, HBase, Json, XML data interfaces, and files. Unstructured data may include data obtained from an unstructured data source. For example, an unstructured data source may include file data such as spreadsheets and logs, and multimedia data such as images, audio, and video.

[0140] According to an embodiment of the present disclosure, after determining the data type, at least one preset rule may be determined based on the object type and the data type. The preset rule may include data combination rules, data conversion rules, field addition rules, and data encryption and decryption rules.

[0141] According to an embodiment of the present disclosure, data combination rules may represent rules for combining data to be processed. Data conversion rules may represent rules for converting data to be processed. For example, data conversion rules may include rules for converting to numerical values, rules for converting to characters, and rules for converting to binary numbers. Field addition rules may represent rules for adding fields to the data to be processed. Data encryption and decryption rules may represent rules for encrypting and decrypting the data to be processed. For example, data encryption and decryption rules may include rules based on the MD5 message digest algorithm (MD5Message-DigestAlgorithm) and rules based on the RSA encryption algorithm (RSA algorithm, RSA).

[0142] According to an embodiment of the present disclosure, for each object type in at least one object type, since the data type is determined according to the object type, at least one preset rule is determined according to the object type and the data type, by processing the data to be processed corresponding to the object type according to the at least one preset rule, processed data is obtained, thereby improving the processing efficiency of the data to be processed.

[0143] Figure 4 An example schematic diagram of a method for processing data to be processed according to at least one preset rule corresponding to each object type to obtain processed data according to an embodiment of the present disclosure is schematically shown.

[0144] like Figure 4 As shown in 400, data to be processed 401 may have at least one object type. Data to be processed 401 may include data to be processed 4011, data to be processed 4012, ..., data to be processed 401n, ..., data to be processed 401N. N may be an integer greater than or equal to 1, n∈{1, 2, ..., (N-1), N}.

[0145] An object type 4021 corresponding to the data to be processed 4011 , an object type 4022 corresponding to the data to be processed 4012 , ..., an object type 402n corresponding to the data to be processed 401n , ..., and an object type 402N corresponding to the data to be processed 401N may be determined.

[0146] A data type 4031 corresponding to the object type 4021, a data type 4032 corresponding to the object type 4022, ..., a data type 403n corresponding to the object type 402n, ..., a data type 403N corresponding to the object type 402N may be determined.

[0147] At least one preset rule 4041 corresponding to the data to be processed 4011 can be determined based on the object type 4021 and the data type 4031. At least one preset rule 4042 corresponding to the data to be processed 4012 can be determined based on the object type 4022 and the data type 4032. At least one preset rule 404n corresponding to the data to be processed 401n can be determined based on the object type 402n and the data type 403n. At least one preset rule 404N corresponding to the data to be processed 401N can be determined based on the object type 402N and the data type 403N.

[0148] Data to be processed 401_1 can be processed according to at least one preset rule 404_1 to obtain processed data 405_1. Data to be processed 401_2 can be processed according to at least one preset rule 404_2 to obtain processed data 405_2. Data to be processed 401_n can be processed according to at least one preset rule 404_n to obtain processed data 405_n. Data to be processed 401_N can be processed according to at least one preset rule 404_N to obtain processed data 405_N.

[0149] According to an embodiment of the present disclosure, operation S240 may include the following operations.

[0150] For each object type in at least one object type, at least one preset field identifier corresponding to the object type and at least one field identifier in the processed data are determined. Based on the at least one preset field identifier and the at least one field identifier, a similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier is determined. In response to the similarity being greater than or equal to a preset threshold, the processed data corresponding to the target field identifier is determined as the processed data corresponding to the target preset field identifier. Target data is determined based on the processed data corresponding to the target preset field identifier.

[0151] According to an embodiment of the present disclosure, after obtaining processed data, at least one field identifier in the processed data may be determined. Alternatively, at least one preset field identifier corresponding to an object type may be determined. After determining the at least one field identifier and the at least one preset field identifier, a similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier may be calculated.

[0152] According to an embodiment of the present disclosure, similarity can be used to characterize the degree of similarity between two objects to be matched. The relationship between the numerical value of the similarity and the degree of similarity can be configured according to actual business needs and is not limited here. For example, the larger the numerical value of the similarity, the greater the degree of similarity between the two objects to be matched. Conversely, the smaller the degree of similarity between the two objects to be matched. Alternatively, the smaller the numerical value of the similarity, the greater the degree of similarity between the two objects to be matched. Conversely, the smaller the degree of similarity between the two objects to be matched.

[0153] According to the embodiments of the present disclosure, the method for determining similarity can be configured according to actual business needs and is not limited here. For example, the method for determining similarity may include at least one of the following: a method based on literal similarity, a method based on text similarity, and a method based on entity similarity. The method based on literal similarity may include at least one of the following: edit distance, Dice coefficient, and Jaccard similarity. The method based on text similarity may include at least one of the following: cosine similarity, relative entropy, KL (Kullback-Leibler, KL) divergence, and probability model similarity.

[0154] According to an embodiment of the present disclosure, the two objects to be matched may include a target preset field identifier in at least one preset field identifier and a target field identifier in at least one field identifier. A fourth similarity between the target preset field identifier in at least one preset field identifier and the target field identifier in at least one field identifier may be determined to obtain at least one fourth similarity. Based on the at least one fourth similarity, a numerical relationship between each fourth similarity in the at least one fourth similarity and a preset threshold is determined. The preset threshold can be set according to actual business needs and is not limited here. For example, the preset threshold may be 0.95.

[0155] According to an embodiment of the present disclosure, in response to the similarity being greater than or equal to a preset threshold, it can be determined that the target preset field identifier and the target field identifier are successfully matched. In this case, the processed data corresponding to the target field identifier can be determined as the processed data corresponding to the target preset field identifier. In response to the similarity being less than the preset threshold, it can be determined that the target preset field identifier and the target field identifier are not successfully matched. In this case, the processed data corresponding to the target field identifier can be deleted.

[0156] According to an embodiment of the present disclosure, for each object type in at least one object type, since at least one preset field identifier and at least one field identifier in the processed data are determined based on the object type, the similarity can characterize the degree of similarity between the target preset field identifier and the target field identifier. When the similarity is greater than or equal to a preset threshold, the target data is determined based on the processed data corresponding to the target preset field identifier, thereby improving the processing efficiency of the processed data.

[0157] Figure 5 An example diagram of a method for processing processed data to obtain target data according to at least one preset field identifier corresponding to each object type according to an embodiment of the present disclosure is schematically shown.

[0158] like Figure 5 As shown, in 500, at least one preset field identifier 502 corresponding to the object type 501 and at least one field identifier 503 in the processed data can be determined. Based on the at least one preset field identifier 502 and the at least one field identifier 503, a similarity 504 between each target preset field identifier in the at least one preset field identifier 502 and each target field identifier in the at least one field identifier 503 is determined.

[0159] A relationship between similarity 504 and a preset threshold 505 is determined. In response to similarity 504 being greater than or equal to preset threshold 505, the processed data corresponding to the target field identifier is determined as processed data 506 corresponding to the target preset field identifier. Target data 507 is determined based on processed data 506 corresponding to the target preset field identifier.

[0160] According to an embodiment of the present disclosure, operation S210 may include the following operations.

[0161] In response to the first scheduled task being triggered, first task information in the first scheduled task is obtained. The task association information indicates that if the first scheduled task has no associated tasks, the data to be processed is obtained from a source system corresponding to each of the at least one source address information corresponding to the data to be processed.

[0162] According to an embodiment of the present disclosure, the first task information may include task association information, the at least one address information may include at least one source address information, and the at least one source address information may each correspond to an active system.

[0163] According to an embodiment of the present disclosure, in response to a first scheduled task being triggered, first task information in the first scheduled task may be obtained. The first task information may include task association information. The task association information may be used to characterize the association relationship between scheduled tasks. The task association information may include one of the following: no associated task exists and an associated task exists.

[0164] According to an embodiment of the present disclosure, when task association information indicates that a first scheduled task has no associated tasks, a source system corresponding to each piece of at least one source address information corresponding to the data to be processed can be determined. The data to be processed can be obtained based on the source system corresponding to each piece of source address information.

[0165] According to an embodiment of the present disclosure, since the first task information is obtained in response to the first scheduled task being triggered, when the task association information in the first task information indicates that there is no associated task for the first scheduled task, the first scheduled task can be executed directly, that is, the data to be processed is obtained from the source system according to the source address information, thereby improving the efficiency of obtaining data that has no associated relationship.

[0166] According to an embodiment of the present disclosure, operation S210 may further include the following operations.

[0167] If the task association information indicates that the first scheduled task has an associated task, at least one second scheduled task is determined, wherein the at least one second scheduled task includes a scheduled task that has a forward association relationship with the first scheduled task. In response to detecting that the second scheduled task has completed execution, the data to be processed is retrieved from a source system corresponding to each of the at least one source address information corresponding to the data to be processed, based on the at least one source address information corresponding to the data to be processed.

[0168] According to an embodiment of the present disclosure, when the task association information indicates the existence of associated tasks, the task association information may also include sequence information of the associated tasks. The dependency relationship between the data can be determined based on the sequence information of the associated tasks. The second scheduled task may be used to process the associated pending data.

[0169] For example, the sequence information of associated tasks may include scheduled task 1 - scheduled task 2 - scheduled task 3. In this case, in response to the completion of scheduled task 1, scheduled task 2 may be triggered, and in response to the completion of scheduled task 2, scheduled task 3 may be triggered. In other words, in response to the completion of processing of pending data associated with scheduled task 1, processing of pending data associated with scheduled task 2 is performed. In response to the completion of processing of pending data associated with scheduled task 2, processing of pending data associated with scheduled task 3 is performed.

[0170] According to an embodiment of the present disclosure, since the first task information is obtained in response to the first scheduled task being triggered, when the task association information in the first task information indicates that there is an associated task with the first scheduled task, a second scheduled task having a forward association relationship with the first scheduled task can be determined. After detecting that the second scheduled task is completed, the first scheduled task is executed, that is, the data to be processed is obtained from the source system according to the source address information, thereby improving the efficiency of obtaining data with an associated relationship.

[0171] According to an embodiment of the present disclosure, according to at least one source address information corresponding to the data to be processed, obtaining the data to be processed from the source system corresponding to each of the at least one source address information may include the following operations.

[0172] Based on at least one page number information and the amount of data corresponding to each of the at least one page number information, and according to at least one source address information, a portion of the to-be-processed data corresponding to each of the at least one page number information is obtained from a source system. Data to be processed is determined based on the portion of the to-be-processed data corresponding to each of the at least one page number information.

[0173] According to an embodiment of the present disclosure, the first task information may further include data paging information corresponding to the data to be processed. The data paging information may be determined based on the total amount of the data to be processed. The data paging information may include at least one page number and the amount of data corresponding to each of the at least one page number.

[0174] According to an embodiment of the present disclosure, data paging information corresponding to the data to be processed can be determined based on the total data volume. At least one page number information may include page number information 1, page number information 2, ..., page number information x, ..., page number information X. X may be an integer greater than or equal to 1, x∈{1, 2, ..., (X-1), X}. The amount of data corresponding to each of the at least one page number information may include data volume 1 corresponding to page number information 1, data volume 2 corresponding to page number information 2, ..., data volume x corresponding to page number information x, ..., data volume X corresponding to page number information X.

[0175] According to an embodiment of the present disclosure, a method of determining data paging information corresponding to the data to be processed based on the total data volume may include one of the following: determining data paging information based on an average value and determining data paging information based on a random value. In the method of determining data paging information based on an average value, at least one page number information may be determined based on the total data volume, and a first data volume may be determined based on the at least one page number information. In this case, the first data volumes corresponding to the at least one page number information may be the same. In the method of determining data paging information based on a random value, at least one second data volume may be determined based on the total data volume, and at least one page number information may be determined based on the number of copies of the at least one second data volume. In this case, the first data volumes corresponding to the at least one page number information may be different.

[0176] For example, if the total data volume is 100 pieces of data, in the method for determining data paging information based on an average value, page number information 1 and page number information 2 can be determined based on the 100 pieces of data, and based on page number information 1 and page number information 2, the first data volume can be determined to be 50 pieces of data. In this case, the first data volume 1 corresponding to page number information 1 can be 50 pieces of data, and the first data volume 1 corresponding to page number information 2 can be 50 pieces of data.

[0177] For example, if the total data volume is 100 pieces of data, in the method for determining data paging information based on random values, second data volume 1 can be determined to be 10 pieces of data, second data volume 2 to be 20 pieces of data, second data volume 3 to be 30 pieces of data, and second data volume 4 to be 40 pieces of data. Based on the number of copies of at least one second data volume, page number information 1, page number information 2, page number information 3, and page number information 4 can be determined. In this case, second data volume 1 corresponding to page number information 1 can be 10 pieces of data, second data volume 2 corresponding to page number information 2 can be 20 pieces of data, second data volume 3 corresponding to page number information 3 can be 30 pieces of data, and second data volume 4 corresponding to page number information 4 can be 40 pieces of data.

[0178] According to an embodiment of the present disclosure, since the data to be processed is determined based on the portion of the data to be processed corresponding to at least one page number information, and the portion of the data to be processed is obtained based on at least one page number information and the amount of data corresponding to at least one page number information, the impact on the data source is reduced, and the amount of data acquisition operations is simplified, thereby improving the speed of data processing.

[0179] Figure 6 The diagram schematically illustrates an example of a method for obtaining data to be processed according to first task information in the first scheduled task and at least one address information corresponding to the data to be processed in response to a first scheduled task being triggered according to an embodiment of the present disclosure.

[0180] like Figure 6 As shown, in step 600, in response to the first scheduled task being triggered, first task information 601 may be obtained. After the first task information 601 is obtained, operation S610 may be performed.

[0181] In operation S610 , it is determined whether the task association information indicates that the first scheduled task has an associated task. If not, the data to be processed 603 is obtained from the source system corresponding to each of the at least one source address information 602 corresponding to the data to be processed.

[0182] If yes, at least one second scheduled task is determined 604. In response to detecting that the second scheduled task 604 is completed, data to be processed is obtained 603 from source systems corresponding to the at least one source address information 602 corresponding to the data to be processed.

[0183] According to an embodiment of the present disclosure, the data processing method 200 may further include the following operations.

[0184] The target data is sent to a target system corresponding to at least one target address information, so that the target system stores the target data.

[0185] According to an embodiment of the present disclosure, the at least one address information may further include at least one target address information. The at least one target address information may each correspond to a target system.

[0186] According to an embodiment of the present disclosure, after obtaining the target data, the target system corresponding to each of the at least one address information can be determined based on the at least one address information. After determining the target system corresponding to each of the at least one address information, the target data can be sent to the target system corresponding to each of the at least one target address information.

[0187] According to an embodiment of the present disclosure, after receiving the target data, the target system may store the target data. For example, the target data may be stored in a local database of the target system. The local database of the target system may include a relational database, such as Oracle, SQL Server, Sybase, Informix, Access, DB2, or MySQL. The local database of the target system may also be a non-relational database, such as HBase, Cassandra, SimpleDB, CouchDB, MongoDB, or Redis.

[0188] The above are merely exemplary embodiments, but are not limited thereto. Other data processing methods known in the art may also be included as long as they can improve data processing efficiency and accuracy.

[0189] Figure 7 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0190] like Figure 7 As shown, the data processing device 700 may include an acquisition module 710 , a determination module 720 , a first processing module 730 , and a second processing module 740 .

[0191] The acquisition module 710 is used to obtain the data to be processed in response to the triggering of the first scheduled task, based on the first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is used to process the data to be processed, and the data to be processed has at least one object type.

[0192] The determination module 720 is configured to determine, based on each object type of at least one object type, configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier.

[0193] The first processing module 730 is configured to process the data to be processed according to at least one preset rule corresponding to each object type to obtain processed data.

[0194] The second processing module 740 is configured to process the processed data according to at least one preset field identifier corresponding to each object type to obtain target data.

[0195] According to an embodiment of the present disclosure, each object type in the at least one object type is used to represent a target object, and the target object includes at least one of the following: a target interface object, a target file object, and a target database object.

[0196] According to an embodiment of the present disclosure, the determination module 720 may include a first determination unit, a second determination unit, and a third determination unit.

[0197] The first determining unit is configured to determine, according to a first mapping relationship, configuration information corresponding to the target interface object when at least one object type characterizing the target object includes a target interface object.

[0198] The second determining unit is configured to determine configuration information corresponding to the target file object according to a second mapping relationship when at least one object type represents that the target object includes a target file object.

[0199] The third determining unit is configured to determine configuration information corresponding to the target database object according to a third mapping relationship when at least one object type characterizing the target object includes a target database object.

[0200] According to an embodiment of the present disclosure, the first mapping relationship includes at least one first key-value pair, and the at least one first key-value pair is determined according to at least one interface object and configuration information corresponding to the at least one interface object.

[0201] According to an embodiment of the present disclosure, the second mapping relationship includes at least one second key-value pair, and the at least one second key-value pair is determined according to at least one file object and configuration information corresponding to the at least one file object.

[0202] According to an embodiment of the present disclosure, the third mapping relationship includes at least one third key-value pair, and the at least one third key-value pair is determined according to at least one database object and configuration information corresponding to the at least one database object.

[0203] According to an embodiment of the present disclosure, for each object type of the at least one object type, the first processing module 730 may include a fourth determining unit, a fifth determining unit, and a first processing unit.

[0204] The fourth determining unit is configured to determine a data type corresponding to the object type, wherein the data type includes at least one of the following: structured data, semi-structured data, and unstructured data.

[0205] The fifth determining unit is configured to determine at least one preset rule according to the object type and the data type, wherein the preset rule includes at least one of the following: a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule.

[0206] The first processing unit is configured to process the data to be processed corresponding to the object type according to at least one of a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule to obtain processed data.

[0207] According to an embodiment of the present disclosure, for each object type of the at least one object type, the second processing module 740 may include a sixth determining unit, a seventh determining unit, an eighth determining unit, and a ninth determining unit.

[0208] The sixth determining unit is configured to determine at least one preset field identifier corresponding to the object type and at least one field identifier in the processed data.

[0209] The seventh determining unit is configured to determine, based on the at least one preset field identifier and the at least one field identifier, a similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier.

[0210] The eighth determining unit is configured to determine, in response to the similarity being greater than or equal to a preset threshold, the processed data corresponding to the target field identifier as the processed data corresponding to the target preset field identifier.

[0211] The ninth determining unit is configured to determine target data according to the processed data corresponding to the target preset field identifier.

[0212] According to an embodiment of the present disclosure, the first task information includes task association information, the at least one address information includes at least one source address information, and the at least one source address information each corresponds to an active system.

[0213] According to an embodiment of the present disclosure, the acquisition module 710 may include a first acquisition unit and a second acquisition unit.

[0214] The first acquiring unit is configured to acquire first task information in the first scheduled task in response to the first scheduled task being triggered.

[0215] The second acquiring unit is configured to acquire the data to be processed from source systems corresponding to the at least one source address information corresponding to the data to be processed, when the task association information indicates that the first scheduled task has no associated task.

[0216] According to an embodiment of the present disclosure, the acquisition module 710 may further include a tenth determining unit and a third acquiring unit.

[0217] The tenth determining unit is configured to determine at least one second scheduled task when the task association information indicates that the first scheduled task has an associated task, wherein the at least one second scheduled task includes a scheduled task having a forward association relationship with the first scheduled task.

[0218] The third acquiring unit is configured to acquire the data to be processed from a source system corresponding to each of the at least one source address information corresponding to the data to be processed in response to detecting that the second scheduled task is completed.

[0219] According to an embodiment of the present disclosure, the first task information also includes data paging information corresponding to the data to be processed. The data paging information is determined based on the total data volume of the data to be processed. The data paging information includes at least one page number information and the data volume corresponding to each of the at least one page number information.

[0220] According to an embodiment of the present disclosure, the second acquiring unit and the third acquiring unit may include a first acquiring subunit and a first determining subunit.

[0221] The first acquisition subunit is configured to acquire, based on at least one page number information and the amount of data corresponding to each of the at least one page number information, a portion of the to-be-processed data corresponding to each of the at least one page number information from a source system according to at least one source address information.

[0222] The first determining subunit is configured to determine the data to be processed according to the portion of the data to be processed corresponding to each of the at least one page number information.

[0223] According to an embodiment of the present disclosure, the configuration information corresponding to the target interface object includes at least one of the following: a target interface address, a target interface user name, and a target interface password.

[0224] According to an embodiment of the present disclosure, the configuration information corresponding to the target file object includes at least one of the following: a target file transfer protocol address, a target file transfer protocol user name, a target file transfer protocol password, a target port, and a target file path.

[0225] According to an embodiment of the present disclosure, the configuration information corresponding to the target database object includes at least one of the following: a target database address, a target database name, a target instance name, a target database user name, and a target database password.

[0226] According to an embodiment of the present disclosure, the at least one address information further includes at least one target address information, and each of the at least one target address information corresponds to a target system.

[0227] According to an embodiment of the present disclosure, the data processing device 700 may further include a sending module.

[0228] The sending module is used to send the target data to the target system corresponding to at least one target address information, so that the target system can store the target data.

[0229] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0230] For example, any multiple of the acquisition module 710, the determination module 720, the first processing module 730 and the second processing module 740 can be combined into one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the acquisition module 710, the determination module 720, the first processing module 730 and the second processing module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 710 , the determination module 720 , the first processing module 730 , and the second processing module 740 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0231] It should be noted that the data processing device part in the embodiment of the present disclosure corresponds to the data processing method part in the embodiment of the present disclosure. The description of the data processing device part specifically refers to the data processing method part and will not be repeated here.

[0232] Figure 8 A block diagram schematically shows an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0233] like Figure 8 As shown, the computer electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 809 into a random access memory (RAM) 803. The processor 801 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0234] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the programs in the ROM 802 and / or RAM 803 to perform various operations of the method flow according to the embodiment of the present disclosure. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also execute the programs stored in the one or more memories to perform various operations of the method flow according to the embodiment of the present disclosure.

[0235] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 808 including a hard disk; and a communication portion 809 including a network interface card such as a LAN card or a modem. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 810 as needed, so that a computer program read therefrom can be installed into the storage portion 808 as needed.

[0236] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0237] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0238] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0239] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or the RAM 803 described above and / or one or more memories other than the ROM 802 and the RAM 803 .

[0240] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiment of the present disclosure.

[0241] When the computer program is executed by the processor 801, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0242] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0243] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0244] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0245] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: In response to a first scheduled task being triggered, obtaining the data to be processed according to first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is used to process the data to be processed, and the data to be processed has at least one object type; Determining, according to each object type of the at least one object type, configuration information corresponding to each object type, wherein the configuration information corresponding to each object type includes at least one preset rule and at least one preset field identifier; For each of the object types, determining a data type corresponding to the object type, wherein the data type includes at least one of the following: structured data, semi-structured data, and unstructured data; determining at least one preset rule based on the object type and the data type, wherein the preset rule includes at least one of the following: a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule; processing the to-be-processed data corresponding to the object type according to at least one of the data combination rule, the data conversion rule, the field addition rule, and the data encryption and decryption rule to obtain processed data; and For each of the object types, determine the at least one preset field identifier corresponding to the object type and the at least one field identifier in the processed data; based on the at least one preset field identifier and the at least one field identifier, determine the similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier; in response to the similarity being greater than or equal to a preset threshold, determine the processed data corresponding to the target field identifier as the processed data corresponding to the target preset field identifier; and determine the target data based on the processed data corresponding to the target preset field identifier.

2. The method according to claim 1, wherein Each object type in the at least one object type is used to represent a target object, wherein the target object includes at least one of the following: a target interface object, a target file object, and a target database object; The determining, according to each object type of the at least one object type, configuration information corresponding to the each object type includes: In a case where the at least one object type represents that the target object includes the target interface object, determining configuration information corresponding to the target interface object according to the first mapping relationship; In a case where the at least one object type represents that the target object includes the target file object, determining configuration information corresponding to the target file object according to a second mapping relationship; and In a case where the at least one object type represents that the target object includes the target database object, configuration information corresponding to the target database object is determined according to the third mapping relationship.

3. The method according to claim 2, wherein: The first mapping relationship includes at least one first key-value pair, where the at least one first key-value pair is determined according to at least one interface object and configuration information corresponding to each of the at least one interface object; The second mapping relationship includes at least one second key-value pair, and the at least one second key-value pair is determined according to at least one file object and configuration information corresponding to the at least one file object; and The third mapping relationship includes at least one third key-value pair, and the at least one third key-value pair is determined according to at least one database object and configuration information corresponding to the at least one database object.

4. The method according to any one of claims 1 to 3, wherein The first task information includes task association information, the at least one address information includes at least one source address information, and the at least one source address information each corresponds to an active system; Wherein, in response to the first scheduled task being triggered, obtaining the data to be processed according to the first task information in the first scheduled task and at least one address information corresponding to the data to be processed includes: In response to the first scheduled task being triggered, obtaining the first task information in the first scheduled task; and When the task association information indicates that the first scheduled task has no associated task, the data to be processed is acquired from source systems corresponding to the at least one source address information corresponding to the data to be processed.

5. The method according to claim 4, further comprising: If the task association information indicates that the first scheduled task has an associated task, determining at least one second scheduled task, wherein the at least one second scheduled task includes a scheduled task having a forward association relationship with the first scheduled task; and In response to detecting that the second scheduled task is completed, the data to be processed is acquired from source systems corresponding to the at least one source address information corresponding to the data to be processed.

6. The method according to claim 5, wherein: The first task information further includes data paging information corresponding to the data to be processed, the data paging information being determined based on the total amount of the data to be processed, and the data paging information including at least one page number information and the amount of data corresponding to each of the at least one page number information; Wherein, acquiring the data to be processed from a source system corresponding to each of the at least one source address information according to the at least one source address information corresponding to the data to be processed comprises: Based on the at least one page number information and the amount of data corresponding to each of the at least one page number information, and according to the at least one source address information, obtaining, from the source system, portions of the to-be-processed data corresponding to each of the at least one page number information; and The data to be processed is determined according to the portion of the data to be processed corresponding to each of the at least one page number information.

7. The method according to claim 2 or 3, wherein: The configuration information corresponding to the target interface object includes at least one of the following: a target interface address, a target interface user name, and a target interface password; The configuration information corresponding to the target file object includes at least one of the following: a target file transfer protocol address, a target file transfer protocol user name, a target file transfer protocol password, a target port, and a target file path; and The configuration information corresponding to the target database object includes at least one of the following: a target database address, a target database name, a target instance name, a target database user name, and a target database password.

8. The method according to claim 1, wherein The at least one address information further includes at least one target address information, and each of the at least one target address information corresponds to a target system; The method further includes, after processing the processed data according to at least one preset field identifier corresponding to each object type to obtain target data: The target data is sent to a target system corresponding to each of the at least one target address information, so that the target system stores the target data.

9. A data processing device comprising: an acquisition module, configured to, in response to a first scheduled task being triggered, acquire the data to be processed based on first task information in the first scheduled task and at least one address information corresponding to the data to be processed, wherein the first scheduled task is configured to process the data to be processed, and the data to be processed has at least one object type; a determination module, configured to determine, based on each object type of the at least one object type, configuration information corresponding to the each object type, wherein the configuration information corresponding to the each object type includes at least one preset rule and at least one preset field identifier; a fourth determining unit, configured to determine, for each of the object types, a data type corresponding to the object type, wherein the data type includes at least one of the following: structured data, semi-structured data, and unstructured data; a fifth determining unit, configured to determine, based on the object type and the data type, the at least one preset rule, wherein the preset rule includes at least one of the following: a data combination rule, a data conversion rule, a field addition rule, and a data encryption and decryption rule; a first processing unit, configured to process the to-be-processed data corresponding to the object type according to at least one of the data combination rule, the data conversion rule, the field addition rule, and the data encryption and decryption rule to obtain processed data; and A sixth determination unit is used to determine, for each of the object types, the at least one preset field identifier corresponding to the object type and the at least one field identifier in the processed data; a seventh determination unit is used to determine, based on the at least one preset field identifier and the at least one field identifier, the similarity between each target preset field identifier in the at least one preset field identifier and each target field identifier in the at least one field identifier; an eighth determination unit is used to determine, in response to the similarity being greater than or equal to a preset threshold, the processed data corresponding to the target field identifier as the processed data corresponding to the target preset field identifier; a ninth determination unit is used to determine the target data based on the processed data corresponding to the target preset field identifier.

10. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer executable instructions, wherein the computer executable instructions are used to implement the method according to any one of claims 1 to 8 when executed.

Images