Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a filled change description template; wherein the filled change description template comprises change content; the change content comprises table structure change information and data content change information; determining a data link relationship according to the filled modification description template; wherein the data link relationship represents a data influence range of the change content on the data in the data; determining a target modification path based on the data link relationship; and processing the corresponding data in the data table based on the target modification path and the filled modification description template. According to the embodiment of the invention, the change content filled by using the uniform change description template is combined with the target change path, so that the corresponding data in the data-in-the-channel can be automatically processed, and meanwhile, the change of the data in the data-in-the-channel can be reduced, and the maintenance cost of the change of the data in the data-in-the-channel can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] At present, in order to solve the problems of data dispersion, difficulty in asset accumulation, and inconsistent standards, many companies choose to establish a data middle platform to build a data system and realize data empowerment. However, this "funnel-type" architecture also causes the problem of high maintenance costs. The table structure and data content of many data source systems often change, which may cause larger-scale changes in the data-intensive middle platform, thereby causing more downstream consumer system changes, and the maintenance cost of changes is high. Summary of the invention

[0003] The present application provides a data processing method, apparatus, device, storage medium and program product, which can reduce changes to the data middle platform and reduce the maintenance cost of changes to the data middle platform.

[0004] In the first aspect, an embodiment of the present application provides a data processing method, including: obtaining a filled-in change description template; wherein the filled-in change description template includes change content; the change content includes table structure change information and data content change information; determining a data link relationship based on the filled-in change description template; wherein the data link relationship represents the scope of influence of the change content on the data of the data center; determining a target modification path based on the data link relationship; and processing the corresponding data in the data center based on the target modification path and the filled-in change description template.

[0005] In the second aspect, an embodiment of the present application also provides a data processing device, including: an acquisition module, used to acquire a filled-in change description template; wherein the filled-in change description template includes change content; the change content includes table structure change information and data content change information; a data link relationship determination module, used to determine the data link relationship based on the filled-in change description template; wherein the data link relationship represents the scope of influence of the change content on the data of the data center; a target modification path determination module, used to determine the target modification path based on the data link relationship; a data processing module, used to process the corresponding data in the data center based on the target modification path and the filled-in change description template.

[0006] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in the embodiment of the present application.

[0007] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the data processing method as described in the embodiment of the present application.

[0008] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the embodiment of the present application.

[0009] The technical solution of the embodiment of the present application is to obtain a filled-in change description template; wherein the filled-in change description template includes change content; the change content includes table structure change information and data content change information; determine a data link relationship based on the filled-in change description template; wherein the data link relationship represents the scope of influence of the change content on the data of the data center; determine a target modification path based on the data link relationship; and process the corresponding data in the data center based on the target modification path and the filled-in change description template. In the embodiment of the present application, by using the change content filled in by a unified change description template combined with the target modification path, it is possible to realize automatic processing of corresponding data in the data center, and at the same time reduce changes to the data in the data center, thereby reducing the maintenance cost of changes to the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0012] Figure 2 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0013] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be understood that the various steps recorded in the method implementation of the present disclosure can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. The term "including" and its variations used herein are open inclusions, that is, "including but not limited to". It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more". It is understandable that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0016] Figure 1 The following is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by a data processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC, or a server. Figure 1 As shown, the method includes:

[0017] S110. Obtain a filled-in change description template.

[0018] The filled-in change description template includes change content; the change content includes table structure change information and data content change information.

[0019] Among them, the table structure change information may include the table structure changes of the data table, and the data content change information may include the mapping relationship between the new and old fields or the dictionary value change information. The changed content is used for internal analysis of the data center to determine the modification plan with the smallest change, thereby reducing the changes to the data center.

[0020] It should be noted that the format of the change description template is uniformly provided by the data center, but the content of the filled-in change description template is filled in by the data source system to standardize the change content of the data source system. At the same time, it is convenient for automated processing based on the filled-in change description template. There can be multiple data source systems. The data center can be understood as a large-scale database, that is, a database cluster. The data source system can be understood as a small-scale database. When the data in the data source system changes, the data in the data center needs to change synchronously.

[0021] Optionally, after obtaining the filled-in change description template, the step further includes: obtaining supplementary change content.

[0022] In this embodiment, there is no restriction on the supplementary content of the change, and the supplementary content of the change may include the change content in addition to the filled-in change description template. It may also include a description of the change content. For example, the supplementary content of the change may be "debit cards, current accounts, and fixed deposits in Table A are split into three tables A, B, and C after the change."

[0023] S120: Determine the data link relationship according to the filled-in change description template.

[0024] Among them, the data link relationship represents the scope of impact of the changed content on the data of the data center.

[0025] In this embodiment, the filled-in change description template can be analyzed to analyze the scope of data impact on the data center, that is, based on the changed content in the filled-in change description template, the affected data in the data center is determined, and the field-level data link relationship is sorted out from the source, such as Table A-Table B-Table C.

[0026] Optionally, determining the data link relationship based on the filled-in change description template includes: using a first data lineage tool to parse the filled-in change description template to obtain the data link relationship.

[0027] The first data lineage tool can be understood as a data model analysis and transformation tool based on field-level lineage. In this embodiment, the first data lineage tool is not limited, and can be any data lineage model tool that can obtain data link relationships.

[0028] In this embodiment, the data link relationship is obtained by parsing the filled-in change description template using the first data lineage tool. Compared with manually determining the data link relationship, the obtained data link relationship is more accurate.

[0029] S130: Determine a target modification path based on the data link relationship.

[0030] In this embodiment, based on the data link relationship and the filled-in modification description template, a modification path with the smallest data impact range can be determined as the target modification path. The target modification path can be understood as the data modification range.

[0031] Optionally, determining the target modification path based on the data link relationship includes: determining the target node based on a graph traversal algorithm and the data link relationship; and determining the target modification path based on the data link relationship and the target node.

[0032] The target modification path may be composed of a source node and a target node.

[0033] In this embodiment, there is no restriction on the number of data link relationships, and there is no restriction on the number of target modification paths. If the number of data link relationships is one, the number of target modification paths is one. If the number of data link relationships is multiple, the number of target modification paths is multiple. That is, the number of data link relationships is the same as the number of target modification paths.

[0034] The target node can be understood as the key node with the largest data impact range. For example, if there are multiple nodes in the data link relationship, the target node can be an intermediate node, and the target node affects many downstream nodes, that is, the data impact range is large. If there are 2 nodes in the data link relationship, the target node can be a leaf node.

[0035] In this embodiment, there is no restriction on the graph traversal algorithm, such as the depth-first search algorithm and the breadth-first search algorithm.

[0036] In this embodiment, if the data in the data center is modified according to the target modification path, that is, unified modifications can be made at the target node, the scope of data impact can be minimized, the modification of the data in the data center can be reduced, and maintenance costs can be saved.

[0037] Exemplarily, the data center includes C, D, and E. The data source system splits the content of a field in table A into two tables, A and B. In the original processing, table C is processed using table A, and both D and E are processed using table C. If both D and E use data from tables A and B, then only C can be modified, and the modification path can be truncated at C. In this case, the data link relationship can be A+B->C, C->D, CE, that is, the data impact range is A, B, C, D, E. The target modification path is A+B->C.

[0038] For example, if only E uses the data in tables A and B, and D only uses the split data in table B, then tables C and D need to be modified. In this case, the data link relationship can be A+B->C, CE; B->D, that is, the data affected range is A, B, C, D, E. The target modification path is A+B->C; B->D.

[0039] S140: Process the corresponding data in the data based on the target modification path and the filled-in modification description template.

[0040] In this embodiment, the corresponding data in the data center can be modified accordingly according to the target modification path and the modification content in the filled modification description template.

[0041] Optionally, the data corresponding to the data center are processed based on the target modification path and the filled-in change description template, including: generating a modification plan based on the target modification path, the filled-in change description template and the change supplementary content; determining a modification script based on the modification plan; and processing the data corresponding to the data center based on the modification script.

[0042] In this embodiment, the content, format, etc. of the modified supplementary content are reviewed to ensure accuracy. After the review is passed. A modification plan is generated based on the target modification path, the filled-in modification description template, and the modified supplementary content. A modification script is generated according to the modification plan, the modification script is executed, the corresponding database query statement is output, and the database query statement is executed to automatically modify the corresponding data in the data center.

[0043] Optionally, before processing the corresponding data in the data center based on the modification script, it also includes: using a second data lineage tool to determine the corresponding downstream consumption system based on the leaf node of the data link relationship, and pushing the modification plan to the corresponding downstream consumption system.

[0044] The second data lineage tool can be understood as a downstream consumer system data impact range analysis tool based on cross-system table-level lineage. In this embodiment, the second data lineage tool is not limited, and can be any data lineage model tool that can obtain the corresponding downstream consumer system based on the leaf node of the data link relationship.

[0045] In this embodiment, the second data lineage tool is used to obtain the downstream consumption system corresponding to the leaf node of the data link relationship, and the modification plan is pushed to the corresponding downstream consumption system, which can achieve accurate push and minimize the downstream impact range.

[0046] The technical solution of the embodiment of the present application is to obtain a filled-in change description template; wherein the filled-in change description template includes change content; the change content includes table structure change information and data content change information; determine a data link relationship based on the filled-in change description template; wherein the data link relationship represents the scope of influence of the change content on the data of the data center; determine a target modification path based on the data link relationship; and process the corresponding data in the data center based on the target modification path and the filled-in change description template. In the embodiment of the present application, by using the change content filled in by a unified change description template combined with the target modification path, it is possible to realize automatic processing of corresponding data in the data center, and at the same time reduce changes to the data in the data center, thereby reducing the maintenance cost of changes to the data center.

[0047] Figure 2 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the device includes: an acquisition module 210, a data link relationship determination module 220, a target modification path determination module 230 and a data processing module 240;

[0048] The acquisition module 210 is used to acquire a filled-in modification description template; wherein the filled-in modification description template includes modification content; the modification content includes table structure modification information and data content modification information;

[0049] The data link relationship determination module 220 is used to determine the data link relationship according to the filled-in change description template; wherein the data link relationship indicates the scope of influence of the change content on the data of the data middle station;

[0050] A target modification path determination module 230, configured to determine a target modification path based on the data link relationship;

[0051] The data processing module 240 is used to process the corresponding data in the data based on the target modification path and the filled-in change description template.

[0052] Optionally, the above-mentioned device also includes a supplement module, which is used to obtain modified supplementary content; wherein the modified supplementary content is a description of the modified content.

[0053] Optionally, the data link relationship determination module is specifically used to: use a first data lineage tool to parse the filled-in change description template to obtain the data link relationship.

[0054] Optionally, the target modification path determination module is specifically used to: determine the target node based on the graph traversal algorithm and the data link relationship; determine the target modification path based on the data link relationship and the target node.

[0055] Optionally, the data processing module is specifically used to: generate a modification plan based on the target modification path, the filled-in change description template, and the change supplementary content; determine a modification script based on the modification plan; and process corresponding data in the data center based on the modification script.

[0056] Optionally, the above-mentioned device also includes a push module, which is used to use a second data lineage tool to determine the corresponding downstream consumption system based on the leaf node of the data link relationship, and push the modification plan to the corresponding downstream consumption system.

[0057] The technical solution of the embodiment of the present application is to obtain a filled-in change description template through an acquisition module; wherein the filled-in change description template includes change content; the change content includes table structure change information and data content change information; determine the data link relationship according to the filled-in change description template through a data link relationship determination module; wherein the data link relationship represents the scope of influence of the change content on the data of the data center; determine the target modification path based on the data link relationship through a target modification path determination module; and process the corresponding data in the data center based on the target modification path and the filled-in change description template through a data processing module. In the embodiment of the present application, by using the change content filled in by a unified change description template combined with the target modification path, it is possible to realize automatic processing of corresponding data in the data center, and at the same time, it is possible to reduce changes to the data in the data center and reduce the maintenance cost of changes in the data center.

[0058] The data processing device provided in the embodiment of the present application can execute the data processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0059] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0060] like Figure 3As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0061] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0062] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as method data processing.

[0063] In some embodiments, the method data processing may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method data processing described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method data processing in any other suitable manner (e.g., by means of firmware).

[0064] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0065] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0066] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0067] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0068] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0069] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0070] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a data processing method as provided in any embodiment of the present application.

[0071] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0072] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: Obtaining a filled-in modification description template; wherein the filled-in modification description template includes modification content; the modification content includes table structure modification information and data content modification information; Determine the data link relationship according to the filled-in change description template; wherein the data link relationship indicates the scope of influence of the change content on the data of the data middle station; Determine a target modification path based on the data link relationship; The corresponding data in the data center is processed based on the target modification path and the filled-in change description template.

2. The method according to claim 1, characterized in that After obtaining the completed change description template, it also includes: Obtain modified supplementary content; wherein the modified supplementary content is an explanation of the modified content.

3. The method according to claim 1, characterized in that Determine the data link relationship based on the filled-in change description template, including: The filled-in change description template is parsed using a first data lineage tool to obtain the data link relationship.

4. The method according to claim 1, characterized in that: Determining a target modification path based on the data link relationship includes: Determine the target node based on the graph traversal algorithm and the data link relationship; A target modification path is determined based on the data link relationship and the target node.

5. The method according to claim 2, characterized in that: Processing the corresponding data in the data based on the target modification path and the filled-in modification description template includes: Generate a modification plan based on the target modification path, the filled-in modification description template, and the modification supplementary content; Determine a modification script based on the modification scheme; The corresponding data in the data is processed based on the modification script.

6. The method according to claim 5, characterized in that Before processing the corresponding data in the data center based on the modification script, the method further includes: The second data lineage tool is used to determine the corresponding downstream consumption system based on the leaf node of the data link relationship, and the modification plan is pushed to the corresponding downstream consumption system.

7. A data processing device, characterized in that: include: An acquisition module, used to acquire a filled-in modification description template; wherein the filled-in modification description template includes modification content; the modification content includes table structure modification information and data content modification information; A data link relationship determination module, used to determine the data link relationship according to the filled-in change description template; wherein the data link relationship indicates the scope of influence of the change content on the data of the data middle station; A target modification path determination module, used to determine a target modification path based on the data link relationship; The data processing module is used to process the corresponding data in the data based on the target modification path and the filled-in change description template.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 6.

9. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the data processing method according to any one of claims 1 to 6 when executed by a computer processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6.

Images