Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a data processing method and device, electronic equipment and a computer readable storage medium. The method comprises: in the case of receiving at least one original data sent by a data sender, creating a write node to write the at least one original data into a data storage unit by using the write node; after the writing of the original data is completed, allocating a target cross-tenant node to a target object, the target object being a data receiver of the original data; sending notification information to the target object by using the target cross-tenant node, so that the target object can obtain the original data from the data storage unit according to the notification information; wherein the notification information comprises completion information of the write node writing the original data into the data storage unit and / or identification information of the original data. The application can significantly improve the efficiency and accuracy of data pushing, and effectively save human resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and more particularly, to a data processing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] At present, push technology is widely used in various business scenarios to realize real-time transmission of data in a business system to a target system or a terminal user. However, the traditional data pushing process often relies on manual sorting and sending data by developers, and this manual process not only consumes a large amount of development resources, but also is prone to errors caused by human factors, especially when dealing with massive data, resulting in low efficiency and poor accuracy of data pushing. SUMMARY

[0003] The present application provides a data processing method, device, electronic equipment and computer readable storage medium, which can improve the efficiency and accuracy of data pushing and save human resources.

[0004] In a first aspect, a data processing method is provided, which includes: receiving at least one original data sent by a data sender; creating a write node and using the write node to write the at least one original data to a data storage unit; in response to a data write completion operation of the write node, allocating a target cross-tenant node to a target object; wherein the target object is a data receiver that receives the at least one original data; using the target cross-tenant node to send notification information to the target object, so that the target object acquires the original data from the data storage unit according to the notification information; wherein the notification information includes completion information of the write node writing the original data to the data storage unit and / or identification information of the original data.

[0005] Through the above technical solution, in the case of receiving at least one original data sent by a data sender, a write node is created to write the at least one original data to a data storage unit using the write node; after writing the original data, a target cross-tenant node is allocated to a target object, which is a data receiver of the original data; the target cross-tenant node is used to send notification information to the target object, so that the target object can acquire the original data from the data storage unit according to the notification information; wherein the notification information includes completion information of the write node writing the original data to the data storage unit and / or identification information of the original data. Compared with the traditional tedious process of manually processing and sending data, the present application can significantly improve the efficiency and accuracy of data pushing and effectively save human resources.

[0006] With reference to the first aspect, in some possible implementation manners, the creating the write node and writing the at least one raw data into the data storage unit by using the write node comprises: creating one write node for each raw data; and writing the raw data into the data storage unit by using the write node for the raw data. Based on the above technical solution, parallelization of data processing can be achieved by creating a write node for each raw data and writing data into the data storage unit by using the nodes, that is, multiple nodes can process different raw data at the same time, thereby speeding up data processing and helping to reduce data delay and improve data processing efficiency.

[0007] With reference to the first aspect and the above implementation manner, in some possible implementation manners, the creating one write node for the raw data comprises: invoking a table creation statement in a software development kit to create an empty node for the raw data; and adding an insertion statement in the software development kit to the empty node to obtain the write node corresponding to the raw data. Based on the above technical solution, the data write node can be quickly created and filled by using the table creation statement and the insertion statement in the software development kit, which can significantly improve the speed of data processing.

[0008] With reference to the first aspect and the above implementation manner, in some possible implementation manners, the writing the raw data into the data storage unit by using the write node comprises: creating a target data structure in the data storage unit by using a table creation statement in the write node, the target data structure being identifiable by a target object; and adding the raw data to the target data structure by using an insertion statement in the write node, so as to write the raw data into the data storage unit. Based on the above technical solution, the raw data can be converted into a format that can be received by the target object by creating a target data structure, and then the processed data is written into the data storage unit, which can ensure the consistency and standardization of the data and facilitate data query and analysis by the target object.

[0009] With reference to the first aspect and the above implementation manner, in some possible implementation manners, each raw data carries object information of a target object receiving the raw data; and the target object is allocated a target cross-tenant node, which comprises: creating an initial cross-tenant node corresponding to the write node; configuring the object information in the initial cross-tenant node to obtain the target cross-tenant node; and allocating the target cross-tenant node to the target object. Based on the above technical solution, by allocating an independent cross-tenant node to each target object, the data of different target objects can be logically isolated, which improves data security, and meanwhile, multiple nodes can process data in parallel, thereby speeding up data processing.

[0010] Combining the first aspect and the above implementation methods, in some possible implementations, creating the initial cross-tenant node corresponding to the write node includes: calling the function in the software development kit (SDK) used to create cross-tenant nodes to create the initial cross-tenant node corresponding to the write node. Based on the above technical solution, creating the initial cross-tenant node corresponding to the write node by calling the function in the SSD can simplify the cross-tenant node creation process, reduce the complexity of manual configuration, enable rapid creation of cross-tenant nodes, and improve creation efficiency.

[0011] Combining the first aspect and the aforementioned implementation methods, in some possible implementations, a robotic process automation (RPA) is employed to log in to the data push application, thereby enabling the data push application. The data push application then performs the steps of receiving at least one piece of raw data sent by the data sender and sending notification information to the target object via a target cross-tenant node assigned to the target object, allowing the target object to retrieve the raw data from the data storage unit based on the notification information. Based on the above technical solution, by automating the login to the data push application using an RPA and enabling the data push application to perform the tasks of receiving data and sending notifications, the automation level of data processing can be significantly improved, effectively saving human resources; furthermore, it can reduce errors caused by human factors and improve the efficiency and accuracy of data processing.

[0012] Secondly, a data processing apparatus is provided, the apparatus comprising:

[0013] A receiving module is used to receive at least one piece of raw data sent by the data sender;

[0014] A processing module is configured to create a write node, use the write node to write at least one piece of raw data to a data storage unit; and, in response to the data write completion operation of the write node, allocate a target cross-tenant node to a target object; wherein the target object is a data receiver that receives at least one piece of raw data; and use the target cross-tenant node allocated to the target object to send a notification message to the target object, so that the target object can retrieve the raw data from the data storage unit according to the notification message; wherein the notification message includes completion information of the write node writing the raw data to the data storage unit and / or identification information of the raw data.

[0015] In conjunction with the second aspect, in some possible implementations, the above processing module includes:

[0016] The first creation unit is used to create a write node for each piece of raw data; and,

[0017] The write unit is used to write the original data to the data storage unit using the write node created for the original data.

[0018] With reference to the second aspect and the foregoing implementation manners, in some possible implementation manners, the first creating unit comprises:

[0019] a first creating sub-unit, configured to invoke a table creation statement in a software development kit to create an empty node for the original data;

[0020] a first inserting sub-unit, configured to add an insertion statement in the software development kit to the empty node to obtain a write node corresponding to the original data.

[0021] With reference to the second aspect and the foregoing implementation manners, in some possible implementation manners, the write unit comprises:

[0022] a second creating sub-unit, configured to create a target data structure in the data storage unit by using a table creation statement in the write node, the target data structure being identifiable by the target object;

[0023] a second inserting sub-unit, configured to add the original data to the target data structure by using an insertion statement in the write node, to write the original data into the data storage unit.

[0024] With reference to the second aspect and the foregoing implementation manners, in some possible implementation manners, the processing module further comprises:

[0025] a second creating unit, configured to create an initial cross-tenant node corresponding to the write node;

[0026] a configuration unit, configured to configure object information of the target object in the initial cross-tenant node to obtain a target cross-tenant node, and assign the target cross-tenant node to the target object; wherein each original data carries object information of a target object receiving the original data.

[0027] With reference to the second aspect and the foregoing implementation manners, in some possible implementation manners, the second creating unit comprises:

[0028] a third creating sub-unit, configured to invoke a function for creating a cross-tenant node in the software development kit to create the initial cross-tenant node corresponding to the write node.

[0029] With reference to the second aspect and the foregoing implementation manners, in some possible implementation manners, the data processing apparatus further comprises:

[0030] a control module, configured to log in to the data pushing application by using the robot process automation, to enable the data pushing application; and perform, by using the data pushing application, the step of sending, to the target object, notification information by using the target cross-tenant node allocated for the target object, to enable the target object to obtain the original data from the data storage unit according to the notification information.

[0031] In a third aspect, an electronic device is provided, comprising:

[0032] a memory configured to store executable program code;

[0033] a processor configured to invoke and run the executable program code from the memory, so that the electronic device performs the data processing method in the first aspect or any possible implementation manner of the first aspect.

[0034] In a fourth aspect, a computer program product is provided, comprising computer program code, which, when executed on a computer, causes the computer to perform the data processing method in the first aspect or any possible implementation manner of the first aspect.

[0035] In a fifth aspect, a computer-readable storage medium is provided, which stores computer program code, which, when executed on a computer, causes the computer to perform the data processing method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 a schematic flow chart of a data processing method provided by an embodiment of the present application is shown;

[0037] Figure 2 a schematic flow chart of another data processing method provided by an embodiment of the present application is shown;

[0038] Figure 3 a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application is shown;

[0039] Figure 4 a structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] The technical solutions in the present application will be described in detail below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, and in addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0041] Hereinafter, the terms "first", "second", etc. are used only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features.

[0042] Currently, push technology is widely used in various business scenarios to achieve real-time transmission of data in a business system to a target system or end user. However, the traditional data pushing process often relies on manual sorting and sending data by developers, which not only consumes a large amount of development resources, but also is prone to errors caused by human factors, especially when dealing with massive data, resulting in low efficiency and poor accuracy of data pushing.

[0043] Based on the problems in the above related technologies, the embodiments of the present application provide a data processing method, device, electronic equipment and computer readable storage medium. The present application automatically generates a write node and a target cross-tenant node in the data pushing process, realizes the complete automation of data sorting and sending, improves the efficiency and accuracy of data pushing, and effectively saves human resources.

[0044] The following is an embodiment of a data processing method provided by the embodiments of the present application.

[0045] Figure 1 A schematic flowchart of a data processing method provided by the embodiments of the present application is shown as follows, Figure 1 The above data processing method includes the following scheme.

[0046] S110: receiving at least one original data sent by a data sender.

[0047] In the embodiments of the present application, the data sender refers to a party that sends data, that is, an entity responsible for generating or collecting data and sending data to a receiver. The data sender can be various types of devices or systems, such as computer systems, smart phones and database management systems, etc. The role of the data sender is to ensure that the data is correctly packaged and sent to the designated data storage unit, such as data lake, downstream system, for further processing and analysis. The original data refers to the data obtained directly from the source by the data sender without any processing or conversion.

[0048] For example, assume that there is a sales data analysis system that needs to obtain sales data provided by the database of the sales system and store it in the data lake for further analysis. In this embodiment, the data sender is the database of the sales system, which regularly sends sales data to the data lake for subsequent data analysis and report generation. Among them, the sales data provided by the database of the sales system is the original data.

[0049] S120: creating a write node, and writing the at least one raw data into the data storage unit by using the write node.

[0050] The creating of the write node and the writing of the at least one raw data into the data storage unit by using the write node can realize full automation of data arrangement and sending, and improve the efficiency and accuracy of data pushing.

[0051] In a possible implementation, the creating of the write node and the writing of the at least one raw data into the data storage unit by using the write node include: creating a write node for each raw data; and writing the raw data into the data storage unit by using the write node for the raw data.

[0052] It should be understood that the write node is used to receive the raw data, and to perform preliminary preprocessing on the raw data, such as data cleaning, format conversion, etc., and to write the processed raw data into the data storage unit. For each raw data, a write node is allocated for each raw data, which can customize the processing logic according to the characteristics of the data or the requirements of the user, making the data processing more flexible, and helping to adapt to changes in different data types and structures, so that the system can better cope with diversified data sources. On the other hand, creating a write node for each raw data, and writing data into the data storage unit by using these nodes can realize parallelization of data processing, that is, multiple nodes can process different raw data at the same time, thereby speeding up the data processing, and helping to reduce data delay and improve the efficiency of data processing. The following describes the process of creating a write node corresponding to a first raw data in at least one raw data, which is an arbitrary raw data in the at least one raw data.

[0053] In a possible implementation, the creating of the write node for the first raw data includes: calling a table creation statement in a software development kit to create an empty node for the first raw data; and adding an insert statement in the software development kit to the empty node to obtain the write node corresponding to the first raw data.

[0054] It should be understood that the empty node created by calling the table creation statement in the software development kit does not contain the first raw data. The insert statement in the software development kit is added to the empty node to obtain the first write node, which is the write node corresponding to the first raw data. The insert statement is used to write the first raw data into the data storage unit. Exemplarily, the development tool used to create the write node can be a software development kit (SDK), and can also be other types of software development tools. The selection of the type of software development kit depends on the needs of different developers, and the present application does not limit it.

[0055] Based on the above technical solution, the write node is quickly created and filled by using the table creation statement and the insertion statement in the software development kit, so that the speed of data processing can be significantly improved.

[0056] Exemplarily, after the write node corresponding to the first original data is created, the first original data is written into the data storage unit by using the first write node.

[0057] In a possible implementation, writing the first original data into the data storage unit by using the first write node includes: creating a first target data structure in the data storage unit by using the table creation statement in the first write node, the first target data structure being identifiable by a first target object; and adding the first original data to the first target data structure by using the insertion statement in the first write node, so as to write the first original data into the data storage unit.

[0058] The first target object is a data receiver that receives the first original data. The first target data structure refers to a data organization form created in the data storage unit, which can be identified and utilized by the first target object. In the embodiment of the application, the target data structure has identification information for uniquely identifying each original data, and the target data structure can be a database table or other types of data containers, such as a document, a key-value pair, etc. The selection of the type of target data structure depends on the requirements of the target object, which is not limited in the application.

[0059] For example, in the above embodiment of the sales data analysis, the first original data is sales data, the identification information of the first original data is b001, and the first original data includes the following fields: id, product ID (product_id), sales date (sales_date), sales quantity (quantity) and unit price (price), wherein the id is a primary key field for uniquely identifying each record, and the first target data structure is as shown in Table 1.

[0060] Table 1

[0061]

[0062] It should be understood that before the first original data is added to the first target data structure by using the insertion statement in the first write node, the first original data needs to be data cleaned by using the preprocessing statement in the first write node, so as to ensure the accuracy and integrity of the data. For example, the data cleaning includes deleting data in the first original data that does not meet the requirements, and / or sorting the first original data to ensure that the first original data is arranged in a certain order.

[0063] After the data cleaning on the first original data, the cleaned first original data is added to the first target data structure to write the first original data into the data storage unit. Exemplarily, the first target data structure to which the first original data is added is shown in Table 2.

[0064] Table 2

[0065]

[0066] Exemplarily, the first target data structure to which the first original data is added includes two sales data records, i.e., the ids, product IDs, sale dates, sale quantities and unit prices of product P001 and product P002.

[0067] Based on the above technical solution, by using the table creation statement to create the target data structure that can be recognized by the target object, the original data can be converted into a format that can be received by the target object, and by using the insert statement, the processed data is written into the data storage unit, which can ensure the consistency and standardization of the data, and facilitate the data query and analysis of the target object.

[0068] S130: In response to the data write completion operation of the write node, a target cross-tenant node is allocated for the target object.

[0069] In the embodiment of the present application, in response to the data write completion operation of the write node, it indicates that the write node has written data into the data storage unit, and the notification information can be sent to the target object. The target cross-tenant node is used to send the notification information to the target object to remind the target object that the data has been written into the data storage unit and can be used. Therefore, after the data write operation of the write node is completed, the target cross-tenant node is allocated for the target object in order to send the notification information to the target object by using the target cross-tenant node. The process of allocating the target cross-tenant node for the target object will be described below by taking the first target cross-tenant node allocated for the first target object receiving the first original data as an example.

[0070] Exemplarily, the first original data carries the object information of the first target object receiving the first original data, and the target cross-tenant node allocated for the first target object includes: creating an initial cross-tenant node corresponding to the first write node, referred to as the first initial cross-tenant node; configuring the object information of the first target object in the first initial cross-tenant node to obtain the first target cross-tenant node; and allocating the first target cross-tenant node to the first target object.

[0071] It should be understood that the created initial cross-tenant node is configured with parameters such as call time, but does not contain object information of the target object. Configuring the object information of the target object for the initial cross-tenant node can send the notification information to the target object by using the target cross-tenant node according to the object information of the target object after the data write operation of the write node is completed. The object information includes the account, password, and address of the target object. For example, the object information of the first target object is configured in the first initial cross-tenant node to obtain the first target cross-tenant node. The first target cross-tenant node is assigned to the first target object to establish a corresponding relationship between the first target cross-tenant node and the first target object, ensuring that the first target cross-tenant node only serves the first target object, thereby realizing high personalization and security of data processing and resource management.

[0072] In a possible implementation manner, the first initial cross-tenant node corresponding to the first write node is created by calling a function for creating a cross-tenant node in a software development kit.

[0073] For example, the function for creating a cross-tenant node in the software development kit encapsulates all the logic and steps required for creating a cross-tenant node, and can quickly create a cross-tenant node according to preset parameters, thereby greatly reducing the complexity of manual configuration and simplifying the entire creation process to improve the creation efficiency. In addition, the software development kit usually provides highly configurable functions, which can flexibly customize the configuration of the cross-tenant node according to different needs to meet specific application scenarios.

[0074] For example, in some embodiments of the present application, the number of initial cross-tenant nodes corresponding to the target objects can be established, and a list of initial cross-tenant nodes is output. The initial cross-tenant nodes in the list of initial cross-tenant nodes are opened in sequence, and the information of the target object associated with the unique identification information of each original data is configured into the initial cross-tenant node corresponding to the identification information to obtain the target cross-tenant node corresponding to each target object. For example, the list of initial cross-tenant nodes is shown in Table 3, which contains the identification information of at least one original data and the initial cross-tenant node.

[0075] Table 3

[0076] Identification information Initial cross-tenant node b001 Node A b002 Node B ... ...

[0077] For example, the identification information of the original data 1 is b001, and the target object receiving the original data 1 is object 1, that is, the object information of the object 1 is configured to the node A corresponding to the identification information b001, and the target cross-tenant node corresponding to the object 1, that is, the node A configured with the object information is obtained; so as to send the notification information to the object 1 by using the node A configured with the object information.

[0078] Based on the above technical solution, by allocating an independent cross-tenant node for each target object, it can be ensured that the data of different target objects is logically isolated, improving the data security, and at the same time, multiple nodes can be used to process data in parallel, accelerating the data processing speed.

[0079] S140: sending the notification information to the target object by using the target cross-tenant node, so that the target object acquires the original data from the data storage unit according to the notification information.

[0080] The notification information includes the completion information of the write node writing the original data into the data storage unit and / or the identification information of the original data. It should be understood that the completion information of the write node writing the original data into the data storage unit is used to indicate that the data required by the target object has been written into the data storage unit, and the target object can acquire the original data from the data storage unit according to the identification information of the original data and apply it.

[0081] It should be understood that the execution order of the write node and the cross-tenant node is that the write node completes the data writing first, and after the data writing is completed, the write node sends a reminder to the cross-tenant node to remind the cross-tenant node that the data writing is completed, and the cross-tenant node executes the operation of sending the notification information at a preset time point; that is, the execution of the cross-tenant node is subject to two conditions: one is to receive the data writing completion reminder of the write node, and the other is the preset sending time point, that is, before receiving the reminder, even if the preset notification time point is reached, the cross-tenant node will not send any message; or, the data writing completion reminder of the write node has been received, but the preset time point has not been reached, and the cross-tenant node will not send the notification information; that is, in the case of meeting the two conditions of receiving the data writing completion reminder of the write node and reaching the preset time point, the cross-tenant node will send the notification information to the target object.

[0082] Exemplarily, assuming that there is a data pushing task, the preset time point is 12 o'clock, the writing node is responsible for receiving data and writing it into the data storage unit, and the cross-tenant node is used to send notification information to the target object to inform it that the data has been written; if the writing node completes the data writing operation at 11 o'clock and sends a data writing completion reminder to the cross-tenant node; after receiving the data writing completion reminder, the cross-tenant node judges whether the preset time point is reached, since the preset sending time point of the cross-tenant node is 12 o'clock, the preset time point is not reached; therefore, the cross-tenant node does not perform the operation of sending notification information, and continues to wait until the preset time point is reached, that is, 12 o'clock, and the operation of sending notification information to the target object is performed; or, if the preset time point is reached, that is, 12 o'clock at night, but the cross-tenant node does not receive the data writing completion reminder sent by the writing node, the cross-tenant node does not perform the operation of sending notification information, and continues to wait until the data writing completion reminder is received, and in response to the data writing completion signal, the notification information is sent to the target object.

[0083] It should be understood that since the cross-tenant node sends the notification information after the writing node is completed and the preset time point is reached, it can ensure that the notification is sent only after the data writing is completed, avoiding the problem of data inconsistency caused by early sending; and complying with the preset time point for sending notification information can ensure the timeliness and regularity of the message, improving the stability and efficiency of the system.

[0084] Through the above technical solution, in the data pushing process, the writing node and the target cross-tenant node are automatically generated, realizing complete automation of data arrangement and sending; compared with the cumbersome process of traditional manual processing and sending data, the present application can significantly improve the efficiency and accuracy of data pushing, effectively saving human resources.

[0085] The above is the scheme flow of the data processing method provided by the embodiment of the present application, and the application process of the data processing method provided by the embodiment of the present application will be described below in combination with a specific application scenario.

[0086] In some possible implementation manners, the robot process automation is used to log in the data pushing application to enable the data pushing application; the data pushing application is used to execute the step of receiving at least one original data sent by the data sender to the target cross-tenant node allocated for the target object to send notification information to the target object, so that the target object acquires the original data from the data storage unit according to the notification information.

[0087] Exemplarily, Figure 2A schematic flowchart of another data processing method provided by an embodiment of the present application is shown. It should be understood that the data processing method provided by an embodiment of the present application is applied to a data pushing application, the present application logs into the data pushing application by using a robotic process automation, to enable the data pushing application; and the process of executing the data processing method by using the data pushing application includes the following steps:

[0088] S201: The robotic process automation logs into the data pushing application.

[0089] In a conventional data pushing operation, the integration, arrangement and sending process of data often completely rely on manual operation, which is not only cumbersome and inefficient, but also prone to errors caused by human factors when processing massive data. In order to significantly improve the efficiency and accuracy of data pushing, the present application uses robotic process automation technology, so that the data integration, arrangement and sending links that originally require human intervention are now completely completed by automated robots. By learning human operations through robotic process automation (RPA) technology, logging into and using the data pushing application, the complete automation of data pushing operation is realized, which greatly improves the work efficiency and effectively reduces the risk of human error, ensuring the accuracy and timeliness of data pushing.

[0090] S202: Obtain a data pushing list.

[0091] Exemplarily, after logging into the data pushing application, a data pushing list that needs to be processed is displayed in the interactive page of the data pushing application. The data pushing list is a list containing all to-be-processed data pushing tasks, which contains at least one original data sent by at least one data sender. For example, the data pushing list can be as shown in Table 4; wherein the data pushing list shown in Table 4 includes 4 pushing tasks, each task contains information of a data sender and original data to be pushed. Among them, the task number is equivalent to the identification information of the original data. For example, the data sender in the data pushing list shown in Table 4 includes the sales department, the procurement department, customer A and customer B; the original data provided by the sales department is sales report 1, the original data provided by the procurement department is purchase list 1, the original data provided by customer A is customer feedback 1, and the original data provided by customer B is customer feedback 2.

[0092] Table 4

[0093] Task number Data sender Original data 001 Sales department Sales report 1 002 Purchasing department Purchase list 1 003 Customer A Customer feedback 1 004 Customer B Customer feedback 2

[0094] S203: Call the SDK to read the data pushing list.

[0095] The SDK can provide functions or interfaces for accessing and processing data sources to extract necessary metadata and other relevant information from the data push manifest, such as column names, data types, data volumes, data formats, etc. The purpose of reading the data push manifest is to better understand and process the data. Facilitating subsequent data cleaning, conversion, and loading. For example, the process of calling the SDK to read the data push manifest shown in Table 4 can include the following steps: 1. Connect to the data source: connect to Table 4 using the functions or interfaces provided by the SDK; 2. Read metadata: read the metadata of Table 4, such as column names, data types, data volumes, etc., to understand the structure and content of the data; 3. Parse data: parse the content of Table 4 and convert it into a processable form, including converting the data into arrays, dictionaries, or other data structures; 4. Save detailed information: save the detailed information of Table 4 obtained for subsequent use, including saving the metadata to a database or file.

[0096] S204: Call the table creation statement and the insert statement to create the write node.

[0097] For example, after reading the content of the data push manifest shown in Table 4, the table creation statement in the software development kit is called to create an empty node for each of the 4 original data in the data push manifest shown in Table 4. The empty node corresponding to each of the 4 original data does not contain the corresponding original data, and the insert statement in the software development kit is added to each of the 4 empty nodes to obtain the write node corresponding to each of the 4 original data.

[0098] S205: Call the SDK to fill in the scheduling information and submit the write node.

[0099] After creating the write node corresponding to each of the 4 original data, the SDK is called to set the scheduling information and submit the write node to the system of the data push application, which means defining the scheduling information of the write node through the interfaces or functions provided by the SDK. The scheduling information includes parameters such as time, frequency, priority, etc., which determine when the write node task starts to execute and the periodicity of execution; after the scheduling information is set, the write node is added to the system, i.e. the write node is added as part of the data processing flow.

[0100] Exemplarily, after the scheduling information setting of the four original data respectively corresponding write nodes is completed, the four original data respectively corresponding write nodes are used to write the four original data respectively into the data storage unit according to the scheduling information of the four original data respectively corresponding write nodes. Specifically, the table creation statements in the four original data respectively corresponding write nodes are used to respectively create four target data structures corresponding to the four original data in the data storage unit, wherein the four target data structures are data structures that can be received by the target objects corresponding to the four original data; the insert statements in the four original data respectively corresponding write nodes are used to add the four original data respectively corresponding to the cleaned data to the corresponding target data structures, so as to write the four original data respectively into the data storage unit.

[0101] S206: Call the SDK to create an initial cross-tenant node and fill in the scheduling information.

[0102] After the four original data are respectively written into the data storage unit, a function for creating a cross-tenant node in the SDK is called to create four original data respectively corresponding initial cross-tenant nodes; after the creation of the four original data respectively corresponding initial cross-tenant nodes is completed, the scheduling information of the cross-tenant node is defined through the interface or function provided by the SDK, and the scheduling information includes parameters such as time, frequency, priority, etc. of task execution, which determines when the cross-tenant node starts to execute and the periodicity of execution.

[0103] S207: Output the initial cross-tenant node list.

[0104] Exemplarily, the initial cross-tenant node list is shown in Table 5, and the initial cross-tenant node list includes the identification information of the four original data and the four original data respectively corresponding initial cross-tenant nodes.

[0105] Table 5

[0106] Identification information Initial cross-tenant node 001 Node 1 002 Node 2 003 Node 3 004 Node 4

[0107] S208: Read the initial cross-tenant node.

[0108] The initial cross-tenant nodes in the initial cross-tenant node list are read in sequence.

[0109] S209: Determine whether the initial cross-tenant node is ready.

[0110] It is judged whether the initial cross-tenant node meets the ready condition, wherein the ready condition includes that the initial cross-tenant node can be searched, and the initial cross-tenant node can normally receive and send data. If both of the above two conditions are met, it is determined that the initial cross-tenant node is ready, and S210 is executed; if any one of the above two conditions is not met, it is determined that the initial cross-tenant node is not ready, and S211 is executed. For example, it is judged whether node 1 shown in Table 5 is ready, which includes searching node 1, if node 1 can be searched, it is detected whether node 1 can normally receive and send data, if node 1 can normally receive and send data, it is determined that node 1 is ready, and S210 is executed; if node 1 cannot be searched or node 1 fails and cannot normally receive and send data, it is determined that node 1 is not ready, and S211 is executed.

[0111] S210: opening the initial cross-tenant node and filling in the account information.

[0112] It should be understood that the created initial cross-tenant node has parameters such as calling time configured, but does not contain object information of the target object. In the case where it is determined that the initial cross-tenant node is ready, the initial cross-tenant node is opened, and the target object of the original data corresponding to the identification information of the initial cross-tenant node is determined according to the identification information of the initial cross-tenant node, and the information of the target object is configured into the initial cross-tenant node to obtain the target cross-tenant node corresponding to the target object.

[0113] S211: recording failure logs.

[0114] If it is determined that the node is not ready, the factors causing the node to be not ready are recorded to identify and solve potential problems. For example, if the factor causing node 1 to be not ready is that node 1 cannot be searched, possible reasons include network problems, configuration errors or nonexistence of node 1, and network connection, configuration file or existence of node 1 can be checked to solve the problem that node 1 is not ready; or if the factor causing node 1 to be not ready is that node 1 cannot normally receive and send data, it is possible that node 1 itself has a problem, and the health status, resource limit or data format of node 1 can be checked to solve the problem that node 1 is not ready; and after the recording is completed, S208 is returned to be executed.

[0115] S212: submitting online.

[0116] Submitting online means submitting the cross-tenant node configured with object information to the system, that is, writing the node as part of the data processing flow, so that the system can use the target cross-tenant node to send notification information to the target object.

[0117] Exemplarily, after the object information configured cross-tenant node is submitted into the system, the cross-tenant node is used to send notification information to the target object corresponding to the original data, so that the target object acquires the original data from the data storage unit according to the notification information. The notification information includes completion information of the write node writing the original data into the data storage unit and / or identification information of the original data; the completion information of the write node writing the original data into the data storage unit is used to indicate that the data required by the target object has been written into the data storage unit, and the target object can acquire the original data from the data storage unit according to the identification information of the original data and perform application.

[0118] It should be understood that the execution of the cross-tenant node is subject to two conditions: one is to receive the data write completion reminder of the write node, and the other is the configured execution time point. For example, the execution time point of the target cross-tenant node corresponding to the original data sales report 1 in the data push list shown in Table 4 is 9 o'clock. After the write node corresponding to the sales report 1 completes the write operation of the sales report 1, the write node sends a data write completion reminder to the cross-tenant node to indicate that the write of the sales report 1 has been completed. The cross-tenant node corresponding to the sales report 1 executes the operation of sending notification information to the target object corresponding to the sales report 1 at 9 o'clock after receiving the data write completion reminder.

[0119] S213: Determine whether the initial cross-tenant node list is read.

[0120] After configuring the object information for each cross-tenant node, it is determined whether all initial cross-tenant nodes in the initial cross-tenant node list have been configured. If yes, S214 is executed; if not, S208 is returned.

[0121] S214: Close the data push application.

[0122] After all initial cross-tenant nodes in the initial cross-tenant node list are configured, that is, all push tasks in the data push list shown in Table 4 are processed, the data push application is closed.

[0123] Based on the above technical solution, the robot process automation is used to log in the data push application, and the data push application is enabled to execute the task of generating the write node and the target cross-tenant node, so that the complete automation of data arrangement and sending is realized, the human resources are effectively saved, the errors caused by human factors are reduced, and the efficiency and accuracy of data processing are improved.

[0124] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0125] Figure 3A structural schematic diagram of a data processing apparatus provided by an embodiment of the present application is shown. As shown in the structural schematic diagram, the data processing apparatus 300 comprises: Figure 3

[0126] A receiving module 310, configured to receive at least one original data sent by a data sender;

[0127] A processing module 320, configured to create a write node, write the at least one original data into a data storage unit by using the write node, and allocate a target cross-tenant node for a target object in response to a data write completion operation of the write node; wherein the target object is a data receiver receiving the at least one original data; and send notification information to the target object by using the target cross-tenant node allocated for the target object, so that the target object acquires the original data from the data storage unit according to the notification information; wherein the notification information comprises completion information of the write node writing the original data into the data storage unit and / or identification information of the original data.

[0128] In a possible implementation manner, the processing module 320 comprises:

[0129] A first creating unit, configured to create a write node for each original data; and

[0130] A writing unit, configured to write the original data into the data storage unit by using the write node created for the original data.

[0131] In a possible implementation manner, the first creating unit comprises:

[0132] A first creating sub-unit, configured to invoke a table creation statement in a software development kit to create an empty node for the original data;

[0133] A first inserting sub-unit, configured to add an insertion statement in the software development kit to the empty node to obtain the write node corresponding to the original data.

[0134] In a possible implementation manner, the writing unit comprises:

[0135] A second creating sub-unit, configured to create a target data structure in the data storage unit by using a table creation statement in the write node, and the target data structure can be recognized by the target object;

[0136] A second inserting sub-unit, configured to add the original data to the target data structure by using an insertion statement in the write node, so as to write the original data into the data storage unit.

[0137] In a possible implementation manner, the processing module 320 further comprises:

[0138] ​The second creating unit is configured to create an initial cross-tenant node corresponding to the write node.

[0139] The configuration unit is configured to configure the object information in the initial cross-tenant node to obtain a target cross-tenant node, and assign the target cross-tenant node to the target object.

[0140] In a possible implementation, the second creating unit includes:

[0141] The third creating sub-unit is configured to invoke a function for creating a cross-tenant node in a software development kit to create the initial cross-tenant node corresponding to the write node.

[0142] In a possible implementation, the data processing apparatus 300 further includes:

[0143] The control module is configured to log in to the data push application by using the robot process automation to enable the data push application, and perform the step of sending the notification information to the target object by using the data push application to send at least one original data sent by the data sender to the target cross-tenant node allocated for the target object, so that the target object acquires the original data from the data storage unit according to the notification information.

[0144] Figure 4 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0145] Exemplarily, as shown in FIG. 4, the electronic device 400 includes a memory 401 and a processor 402, wherein the memory 401 stores executable program code 4011, and the processor 402 is configured to invoke and execute the executable program code 4011 to perform a data processing method. Figure 4

[0146] The embodiment can divide the functional modules of the electronic device according to the above method examples, for example, can correspond to each functional module, or can integrate two or more functions in one processing module, and the integrated module can be implemented in the form of hardware. It should be noted that the division of the modules in the embodiment is illustrative, and is only a logical function division, and another division mode can be used in actual implementation.

[0147] In the case of dividing each functional module according to each function, the electronic device can include an acquisition module, a processing module, a control module, and the like. It should be noted that all related contents of each step involved in the above method embodiments can be referred to the function description of the corresponding functional module, which will not be repeated here.

[0148] ​The electronic device provided by the embodiment is used to execute the data processing method. Therefore, the same effects as the implementation method can be achieved.

[0149] In the case of using the integrated unit, the electronic device can include a processing module and a storage module. The processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute related program codes and data.

[0150] The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, digital signal processing (DSP) and microprocessor combinations, etc. The storage module can be a memory.

[0151] The embodiment also provides a computer readable storage medium, which stores computer program codes. When the computer program codes are executed on a computer, the computer executes the related method steps to realize the data processing method in the above embodiment.

[0152] The embodiment also provides a computer program product, which makes the computer execute the related steps to realize the data processing method in the above embodiment when the computer program product is executed on the computer.

[0153] In addition, the electronic device provided by the embodiment of the present application can be a chip, an assembly or a module. The electronic device can include a connected processor and a memory. The memory is used to store instructions. When the electronic device is running, the processor can call and execute the instructions to make the chip execute the data processing method in the above embodiment.

[0154] The electronic device, the computer readable storage medium, the computer program product or the chip provided by the embodiment are used to execute the corresponding data processing method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding data processing method provided above, which will not be described here.

[0155] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0156] In the embodiments of the present disclosure, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, and the division of the modules or units is merely a logical function division. In actual implementation, another division manner can be adopted, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or in other forms.

[0157] The above merely describes specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present disclosure, which should be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The data processing method includes: Receive at least one piece of raw data sent by the data sender; For each piece of raw data, call the table creation statement in the software development kit to create an empty node for the raw data; Add the insert statement from the software development kit to the empty node to obtain the write node created for the original data; For the write node created for the original data, the table creation statement in the write node is used to create a target data structure in the data storage unit. The target data structure can be recognized by the target object, which is the data receiver that receives the at least one piece of original data. The original data is added to the target data structure using the insert statement in the write node, so as to write the original data into the data storage unit; In response to the data write completion operation of the write node, a target cross-tenant node is allocated to the target object; The target cross-tenant node sends a notification message to the target object so that the target object can retrieve the original data from the data storage unit according to the notification message; wherein, the notification message includes completion information of the writing node writing the original data into the data storage unit and / or the identification information of the original data.

2. The data processing method according to claim 1, characterized in that, Each piece of raw data carries object information of the target object that receives the raw data; The process of allocating a target cross-tenant node to the target object includes: Create the initial cross-tenant node corresponding to the write node; Configure the object information in the initial cross-tenant node to obtain the target cross-tenant node; Assign the target cross-tenant node to the target object.

3. The data processing method according to claim 2, characterized in that, Creating the initial cross-tenant node corresponding to the write node includes: The function in the software development kit used to create cross-tenant nodes is called to create the initial cross-tenant node corresponding to the write node.

4. The data processing method according to any one of claims 1 to 3, characterized in that, The data processing method further includes: The data push application is automated by using a robotic process to log in, thereby enabling the data push application; The step involves using the data push application to receive at least one piece of raw data sent by the data sender to send notification information to the target object via a target cross-tenant node allocated to the target object, so that the target object can retrieve the raw data from the data storage unit based on the notification information.

5. A data processing apparatus, characterized in that, The data processing device includes: A receiving module is used to receive at least one piece of raw data sent by the data sender; The processing module is configured to: for each piece of raw data, call the table creation statement in the software development kit (SDK) to create an empty node for the raw data; add the insert statement from the SSD to the empty node to obtain a write node created for the raw data; for the write node created for the raw data, use the table creation statement in the write node to create a target data structure in the data storage unit, the target data structure being recognizable by a target object, the target object being a data receiver receiving at least one piece of raw data; use the insert statement in the write node to add the raw data to the target data structure to write the raw data into the data storage unit; in response to the data write completion operation of the write node, allocate a target cross-tenant node to the target object; use the target cross-tenant node to send a notification message to the target object, so that the target object can retrieve the raw data from the data storage unit according to the notification message; wherein, the notification message includes completion information of the write node writing the raw data into the data storage unit and / or identification information of the raw data.

6. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the data processing method as described in any one of claims 1 to 4.

Images