Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps of obtaining to-be-offline data from a data warehouse, wherein the to-be-offline data comprises to-be-offline tasks and / or data tables; under the condition that downstream dependent data does not exist in the to-be-offline data, the to-be-offline data is offline, the downstream dependent data are downstream data related to the to-be-offline data, and the downstream dependent data comprise at least one of external downstream data, internal downstream data and authorized user data; and under the condition that the downstream dependent data exists in the to-be-offline data, performing offline on the to-be-offline data according to the offline rule corresponding to the downstream dependent data. According to the embodiment of the invention, the data offline efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] Data warehouses have the characteristics of large capacity, high complexity, and wide compatibility. As the data foundation for daily business operations, as the scale of the enterprise grows, the stock and incremental data in the data warehouse are growing. For example, in the optimization process of information technology systems (such as indicator system transformation, link design optimization, password security transformation, etc.), a new set of design links are often launched, resulting in an increase in the amount of data in the data warehouse. In addition, changes in business direction and company strategic direction will also lead to the replacement of old and new data reports, which will also cause an increase in the amount of data in the data warehouse. As the amount of data in the data warehouse increases, the cost of hardware resources also increases dramatically.

[0003] To solve the above problems, the existing technology uses a variety of methods to offline redundant data during data processing. However, the current data processing method is usually for a single or a few tasks, and is processed one by one on the scheduling platform. This processing method is time-consuming. For complex and complicated data-dependent networks, how to efficiently offline a large amount of data to be offline has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a data processing method, device, electronic device and storage medium, aiming to improve the efficiency of data offline.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a data processing method, the method comprising:

[0006] Acquire data to be taken offline from a data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables;

[0007] In the case that the data to be taken offline does not have downstream dependent data, the data to be taken offline is taken offline, the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data, the external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse;

[0008] In the case that the data to be taken offline has the downstream dependent data, the data to be taken offline is taken offline according to the offline rule corresponding to the downstream dependent data.

[0009] In some embodiments, when the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes:

[0010] In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the internal downstream data, acquiring the running status of the internal downstream data;

[0011] When the running state of the internal downstream data is an abnormal working state, the data to be taken offline and the internal downstream data are taken offline, and the abnormal working state is a pause state or a failure state.

[0012] In some embodiments, when the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes:

[0013] In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the external downstream data and / or the authorized user data, switching the link of the external downstream data and / or the authorized user data;

[0014] When the link switching of the external downstream data and / or authorized user data is completed, the data to be taken offline is taken offline, and the link switching is used to switch the data to be taken offline referenced by the external downstream data and / or authorized user data to preset data.

[0015] In some embodiments, after obtaining the to-be-offline data from the data warehouse, the method further includes:

[0016] Acquire metadata related to the data to be taken offline;

[0017] If the data to be taken offline includes tasks to be taken offline, data parsing is performed on the metadata to obtain downstream dependent data of the tasks to be taken offline, where the downstream dependent data includes external downstream data and / or internal downstream data;

[0018] If the data to be taken offline includes a data table to be taken offline, the metadata is parsed to obtain downstream dependent data of the data table to be taken offline, wherein the downstream dependent data includes at least one of external downstream data, internal downstream data and authorized user data.

[0019] In some embodiments, taking the data to be taken offline offline includes:

[0020] If the data to be taken offline includes tasks to be taken offline, deleting the tasks to be taken offline;

[0021] or,

[0022] If the data to be taken offline includes a data table to be taken offline, renaming the data table to be taken offline to obtain a renaming result;

[0023] In the case where the renaming result indicates that the data table to be taken offline is renamed successfully, if no data table rollback instruction is received within a preset time period, deleting the data table to be taken offline;

[0024] or,

[0025] If the data to be taken offline includes tasks to be taken offline and data tables to be taken offline, complete the following steps in sequence:

[0026] Stop the task to be offline, and store the stop time of the task to be offline;

[0027] Rename the data table to be taken offline;

[0028] Deleting the data table to be taken offline;

[0029] Delete the task to be taken offline.

[0030] In some embodiments, the data to be taken offline includes a data table to be taken offline, and downstream dependent data of the data table to be taken offline includes authorized user data;

[0031] Before deleting the data table to be taken offline, the method further includes:

[0032] According to the authorized user data, the authority granted to the data table to be taken offline is reclaimed.

[0033] In some embodiments, before taking the data to be taken offline offline, the method further includes:

[0034] Obtaining metadata related to the data to be taken offline, and performing data analysis on the metadata to obtain analysis results;

[0035] According to the analysis result and the data to be taken offline, a mapping relationship model related to the data to be taken offline is obtained, wherein the mapping relationship model includes a plurality of nodes, wherein the plurality of nodes include a root node and child nodes of the root node, and the child nodes include leaf nodes;

[0036] The data to be taken offline is matched with the data stored in the leaf node in the mapping relationship model, and when the match is successful, the data to be taken offline is taken offline.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data processing device, the device comprising:

[0038] A data acquisition module, used for acquiring data to be taken offline from a data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables;

[0039] A first data processing module is used to take the data to be taken offline offline if there is no downstream dependent data for the data to be taken offline, the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data, the external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse;

[0040] The second data processing module is used for, when the data to be taken offline has the downstream dependent data, taking the data offline according to the offline rule corresponding to the downstream dependent data.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the data processing method described in the first aspect when executing the computer program.

[0042] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data processing method described in the first aspect.

[0043] The data processing method, device, electronic device and storage medium proposed in the present application obtain the data to be taken offline from the data warehouse, and take the data offline when there is no downstream dependent data for the data to be taken offline, or, when there is downstream dependent data for the data to be taken offline, take the data offline according to the offline rules corresponding to the downstream dependent data. According to different situations of the data to be taken offline, that is, whether there is downstream dependent data, the present application adopts the corresponding offline method to process the data to be taken offline. In the case of a large amount of data to be taken offline, the data to be taken offline can be taken offline in an orderly manner, which reduces the complexity of the data offline process, thereby improving the efficiency of data offline. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application;

[0045] Figure 2 yes Figure 1 A flow chart of step S103 in FIG.

[0046] Figure 3 yes Figure 1 Another flow chart of step S103 in FIG. 1 ;

[0047] Figure 4 is another flow chart of the data processing method provided in an embodiment of the present application;

[0048] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] First, some nouns involved in this application are analyzed:

[0054] Data Warehouse (DW): is a data system used for storage, analysis, and reporting. Its purpose is to build an integrated data environment to provide decision support for enterprises. A data warehouse is usually a collection of data that is oriented to a specific business area and changes over time.

[0055] Data offline: It is the process of removing a task or data table in the data warehouse from the production environment so that it can no longer provide services to users. It usually occurs when the data warehouse is managed, maintained, upgraded or repaired.

[0056] Data table: It is the most basic operation object in the data warehouse and the basic unit of data storage. A data table is defined as a collection of fields (or columns), and data is stored in the table in the format of rows and columns. Each row represents a unique record, which is used to store specific data, and each column represents a field or attribute of the record.

[0057] Tasks: including data extraction, transformation, loading (Extract-Transform-Load, ETL), data cleaning, data integration, etc., all centered around data tables. The goal of the task is to extract data from one or more initial data tables, and after a series of processing and transformation, finally load the results into the target data table.

[0058] Dependency: refers to the relationship between data tables, or between data tables and tasks. In a data warehouse, a data table may be generated by one or more other data tables through task processing, so the data table, one or more other data tables and tasks form a dependency relationship.

[0059] To solve the problems of the prior art, the embodiments of the present application provide a data processing method, device, electronic device and storage medium, aiming to improve the efficiency of data offline.

[0060] The data processing method, device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0061] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] The data processing method provided in the embodiment of the present application relates to the field of data processing technology. The data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S103.

[0067] Step S101, obtaining data to be taken offline from a data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables.

[0068] Step S102, when the data to be taken offline does not have any downstream dependent data, the data to be taken offline is taken offline, the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data. The external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse.

[0069] Step S103 , when the data to be taken offline has the downstream dependent data, the data to be taken offline is taken offline according to the offline rule corresponding to the downstream dependent data.

[0070] The data warehouse may be a data warehouse applied to different business scenarios such as financial business, insurance business or medical business. In different business scenarios, the data stored in the data warehouse is different. For example, for the financial business scenario, the data to be taken offline may include financial data cleaning tasks to be taken offline and / or financial data tables to be taken offline. Alternatively, for the insurance business scenario, the data to be taken offline may include insurance data conversion tasks to be taken offline and / or insurance data tables to be taken offline.

[0071] When performing management, maintenance and other operations on the data warehouse, the electronic device can obtain data to be taken offline, that is, data that needs to be cleared from the data warehouse, such as unreferenced tasks, unreferenced data tables, etc. There is no limit on the number of data to be taken offline, and it can be one data or multiple data. Subsequently, the electronic device can analyze the data to be taken offline to determine whether there is downstream dependent data for the data to be taken offline. Among them, the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data. External downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and authorized user data is the authorization information of the data to be taken offline for downstream users outside the data warehouse.

[0072] In the case where the data to be taken offline does not have downstream dependent data, clearing the data to be taken offline at this time will not cause abnormalities in other data. Therefore, the data to be taken offline can be taken offline. In the case where the data to be taken offline has downstream dependent data, according to the different types of downstream dependent data, the electronic device can determine the offline rule corresponding to the downstream dependent data, and process the downstream dependent data according to the corresponding offline rule. After completing the processing of the downstream dependent data, the data to be taken offline can be taken offline. Exemplarily, taking the data to be taken offline as a task to be taken offline as an example, if the downstream dependent data is external downstream data, at this time, the link of the downstream dependent data can be switched, and the reference of the downstream dependent data to the data to be taken offline can be disconnected, and then the data to be taken offline can be processed offline.

[0073] In addition, when it is necessary to take all the data in the data warehouse offline, the authorized user data of the data warehouse can be obtained, and then the link of the authorized user data is switched, and all the data in the data warehouse is taken offline when the switching is completed.

[0074] Steps S101 to S103 shown in the embodiment of the present application are to obtain the data to be taken offline from the data warehouse, and take the data offline if there is no downstream dependent data for the data to be taken offline, or to take the data offline according to the offline rules corresponding to the downstream dependent data if there is downstream dependent data for the data to be taken offline. In view of different situations of the data to be taken offline, that is, whether there is downstream dependent data, the present application adopts the corresponding offline method to process the data to be taken offline. In the case of a large amount of data to be taken offline, the data to be taken offline can be taken offline in an orderly manner, which reduces the complexity of the data offline process, thereby improving the efficiency of data offline.

[0075] In some embodiments, after obtaining the to-be-offline data from the data warehouse, the method further includes:

[0076] Acquire metadata related to the data to be taken offline;

[0077] If the data to be taken offline includes tasks to be taken offline, data parsing is performed on the metadata to obtain downstream dependent data of the tasks to be taken offline, where the downstream dependent data includes external downstream data and / or internal downstream data;

[0078] If the data to be taken offline includes a data table to be taken offline, the metadata is parsed to obtain downstream dependent data of the data table to be taken offline, wherein the downstream dependent data includes at least one of external downstream data, internal downstream data and authorized user data.

[0079] Metadata provides descriptive information about data, such as the source, format, content, structure, creation date, creator, access information, and permissions of the data. Therefore, the electronic device can obtain metadata related to the data to be taken offline and parse the metadata. For the tasks to be taken offline, since the tasks in the data warehouse are used to process and convert data tables, they are not oriented to users outside the data warehouse. Therefore, for the tasks to be taken offline, after parsing the metadata, downstream dependent data that has a dependency relationship with the tasks to be taken offline, that is, external downstream data and / or internal downstream data, can be obtained. Unlike the tasks in the data warehouse, users outside the data warehouse can be granted permissions such as reading and using data tables. Therefore, for the data tables to be taken offline, the electronic device also needs to parse the metadata to obtain downstream dependent data that has a dependency relationship with the data tables to be taken offline, that is, at least one of external downstream data, internal downstream data, and authorized user data. Through the above-mentioned method, it is determined whether the data to be taken offline has downstream dependent data, and when there is downstream dependent data, the specific type of the downstream dependent data is determined, so that it is convenient to further determine the offline method of the data to be taken offline.

[0080] In some embodiments, before taking the data to be taken offline offline, the method further includes:

[0081] Obtaining metadata related to the data to be taken offline, and performing data analysis on the metadata to obtain analysis results;

[0082] According to the analysis result and the data to be taken offline, a mapping relationship model related to the data to be taken offline is obtained, wherein the mapping relationship model includes a plurality of nodes, wherein the plurality of nodes include a root node and child nodes of the root node, and the child nodes include leaf nodes;

[0083] The data to be taken offline is matched with the data stored in the leaf node in the mapping relationship model, and when the match is successful, the data to be taken offline is taken offline.

[0084] Before the data to be taken offline is taken offline, the electronic device can verify the linearity of the data to be taken offline, so as to reduce the possibility of causing abnormal downstream dependent data by mistakenly taking the data offline. Specifically, the electronic device can obtain metadata related to the data to be taken offline, and perform data analysis on the metadata to obtain analysis results. The analysis results include but are not limited to the source, creator, access information and permissions of the data to be taken offline. Then, according to the analysis results, it can be determined whether the data to be taken offline has upstream dependent data and / or downstream dependent data, and then a mapping relationship model related to the data to be taken offline can be constructed, that is, a model reflecting the reference relationship between the data to be taken offline and all upstream dependent data and / or downstream dependent data related to the data to be taken offline. The mapping relationship model includes multiple nodes, including a root node and child nodes of the root node, and the child nodes include leaf nodes. Among them, the leaf node refers to a node whose stored data is in an abnormal working state, or a node that has not generated a next-level child node. Subsequently, the electronic device can match the data to be taken offline with the data stored in the leaf node in the mapping relationship model. When the data to be taken offline successfully matches the data stored in the leaf node in the mapping relationship model, it is determined that the data to be taken offline has no downstream dependent data in normal working state or no downstream dependent data, and the data to be taken offline can be taken offline.

[0085] Figure 2 yes Figure 1 For a flowchart of step S103 in Figure 2 In some embodiments, when the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes:

[0086] Step S201, when the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the internal downstream data, then the running state of the internal downstream data is acquired;

[0087] Step S202, when the running state of the internal downstream data is an abnormal working state, the data to be taken offline and the internal downstream data are taken offline, and the abnormal working state is a pause state or a failure state.

[0088] In the case where the data to be taken offline has downstream dependent data, according to different types of downstream dependent data, the electronic device can determine the offline rule corresponding to the downstream dependent data, and process the downstream dependent data according to the corresponding offline rule. After completing the processing of the downstream dependent data, the data to be taken offline can be taken offline. Specifically, for the internal downstream data included in the downstream dependent data, the electronic device can obtain information such as the operating frequency and operating time of the internal downstream data, so as to determine the operating state of the internal downstream data. The operating state includes a normal operating state and an abnormal operating state, wherein the abnormal operating state can be divided into a pause state or a failure state. In the case where the operating state of the internal downstream data is an abnormal operating state, taking the data to be taken offline offline at this time will not affect the internal downstream data, and since the data to be taken offline is offline, the internal downstream data that has a dependency relationship with the data to be taken offline cannot work normally. Therefore, in order to further reduce the redundant data in the data warehouse, the data to be taken offline and the internal downstream data can be taken offline. In the case where the operating state of the internal downstream data is a normal operating state, the operation of the internal downstream data can be suspended, so that the operating state of the internal downstream data is converted to a pause state, and then the data to be taken offline and the internal downstream data are taken offline. In this way, when the downstream dependent data includes internal downstream data, the data to be taken offline can be taken offline.

[0089] Figure 3 yes Figure 1 For another flowchart of step S103 in Figure 3 In some embodiments, when the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes:

[0090] Step S301, in the case where the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the external downstream data and / or authorized user data, switching the link of the external downstream data and / or authorized user data;

[0091] Step S302, when the link switching of the external downstream data and / or authorized user data is completed, the data to be taken offline is taken offline, and the link switching is used to switch the data to be taken offline referenced by the external downstream data and / or authorized user data to preset data.

[0092] In the case where the data to be taken offline has downstream dependent data, the electronic device can determine the offline rule corresponding to the downstream dependent data according to the different types of the downstream dependent data, and process the downstream dependent data according to the corresponding offline rule. After completing the processing of the downstream dependent data, the data to be taken offline can be taken offline. Specifically, for the external downstream data and / or authorized user data included in the downstream dependent data, the link of the external downstream data and / or the authorized user data can be switched, that is, the data to be taken offline referenced by the external downstream data and / or the authorized user data can be switched to preset data. The preset data can be selected according to the actual situation. When the link switching of the external downstream data and / or the authorized user data is completed, the data to be taken offline is taken offline. In this way, the impact of the offline data to be taken offline on the data outside the data warehouse can be reduced.

[0093] It should be noted that if the downstream dependent data includes internal downstream data, external downstream data, or includes internal downstream data, authorized user data, or includes all three data at the same time, they can be processed separately according to the offline rules in the previous text, and after the processing is completed, the data to be offline can be offline. For example, when the downstream dependent data includes internal downstream data, external downstream data, and authorized user data, the internal downstream data, external downstream data, and authorized user data are processed separately according to the offline rules in the previous text, and when the internal downstream data is in an abnormal working state and the links of the external downstream data and the authorized user data are switched, the data to be offline will be offline.

[0094] In some embodiments, taking the data to be taken offline offline includes:

[0095] If the data to be taken offline includes tasks to be taken offline, deleting the tasks to be taken offline;

[0096] or,

[0097] If the data to be taken offline includes a data table to be taken offline, renaming the data table to be taken offline to obtain a renaming result;

[0098] In the case where the renaming result indicates that the data table to be taken offline is renamed successfully, if no data table rollback instruction is received within a preset time period, deleting the data table to be taken offline;

[0099] or,

[0100] If the data to be taken offline includes tasks to be taken offline and data tables to be taken offline, complete the following steps in sequence:

[0101] Stop the task to be offline, and store the stop time of the task to be offline;

[0102] Rename the data table to be taken offline;

[0103] Deleting the data table to be taken offline;

[0104] Delete the task to be taken offline.

[0105] Specifically, if the data to be taken offline includes tasks to be taken offline, the electronic device can directly delete the tasks to be taken offline. Alternatively, the electronic device can first suspend the tasks to be taken offline and store the stop time of the tasks to be taken offline, so that data query and data rollback of the tasks to be taken offline can be performed later.

[0106] If the data to be taken offline includes a data table to be taken offline, the data table to be taken offline can be renamed to disconnect the reference of the data table to be taken offline by the downstream dependent data, and obtain the renaming result. The renaming result includes a renaming failure result and a renaming success result. When the renaming result indicates that the data table to be taken offline is successfully renamed (that is, the renaming result is a renaming success result), and no data table rollback instruction is received within a preset time period, the data table to be taken offline is deleted. The preset time period can be determined according to actual conditions. The data table rollback instruction can be used to instruct the rollback of the data table to be taken offline.

[0107] There is a possibility that there is a dependency relationship between the task to be taken offline and the data table to be taken offline. Therefore, in order to ensure the orderly offline of the data to be taken offline, when the data to be taken offline is processed offline, if the data to be taken offline includes both the task to be taken offline and the data table to be taken offline, then the task to be taken offline can be stopped first, and the stop time of the task to be taken offline can be stored to facilitate data query and data rollback. Subsequently, the electronic device can rename the data table to be taken offline to disconnect the reference of the downstream dependent data of the data table to be taken offline to the data table to be taken offline. Then, the electronic device can delete the data table to be taken offline and delete the task to be taken offline. In this way, the offline process of the data to be taken offline can be completed.

[0108] In some embodiments, the data to be taken offline includes a data table to be taken offline, and downstream dependent data of the data table to be taken offline includes authorized user data;

[0109] Before deleting the data table to be taken offline, the method further includes:

[0110] According to the authorized user data, the authority granted to the data table to be taken offline is reclaimed.

[0111] Unlike tasks in the data warehouse, users outside the data warehouse can be granted permissions such as reading and using data tables. Therefore, when taking a data table offline, if the downstream dependent data of the data table to be taken offline includes authorized user data, before deleting the data table to be taken offline, it is necessary to determine the permissions that the data table to be taken offline has granted to users outside the data warehouse based on the authorized user data, and then revoke the permissions granted to the data table to be taken offline. In this way, the possibility of abnormalities in the downstream dependent data of the data table to be taken offline can be further reduced after the data table to be taken offline is taken offline.

[0112] Figure 4 is another flow chart of the data processing method provided in the embodiment of the present application, please refer to Figure 4 The following is an example of the data processing method provided in the embodiment of the present application. The data processing method may include:

[0113] Step S401, obtaining data to be offline;

[0114] Step S402, parse the metadata to obtain parsing results;

[0115] Step S403, constructing a mapping relationship model according to the offline data and the analysis results;

[0116] Step S404, determine whether there is external downstream data; if so, proceed to step S406;

[0117] Step S405, determine whether there is authorized user data; if so, proceed to step S406;

[0118] Step S406, switching the link;

[0119] Step S407, determine whether the link switching is completed; if so, proceed to step S409;

[0120] Step S408, determine whether there is internal downstream data; if not, proceed to step S409;

[0121] Step S409, determining whether the data to be offline matches the data stored in the leaf node of the mapping relationship model; if so, proceeding to step S410;

[0122] Step S410, offline the offline data.

[0123] The electronic device can obtain data to be taken offline, such as tasks to be taken offline and / or data tables to be taken offline. Subsequently, the electronic device can parse metadata related to the data to be taken offline, thereby obtaining a parsing result. The parsing result includes but is not limited to information such as the source, creator, access information, and permissions of the data to be taken offline. In this way, the electronic device can construct a mapping relationship model for reflecting the dependency relationship between the data to be taken offline and its upstream dependent data and / or downstream dependent data based on the data to be taken offline and the parsing result. Then, according to the mapping relationship model, the electronic device can determine whether the data to be taken offline has external downstream data, whether it has authorized user data, and whether it has internal downstream data. It should be noted that the order of this judgment process is not limited and can be adjusted according to actual needs. If the data to be taken offline has external downstream data, the link of the external downstream data is switched. If the data to be taken offline has authorized user data, the link of the authorized user data is switched. The electronic device can determine whether the link switching is completed. If the switching is completed, and it is determined that the data to be taken offline has no internal downstream data at this time, it enters the verification stage, that is, it determines whether the data to be taken offline matches the data stored in the leaf node of the mapping relationship model. When the data to be taken offline matches the data stored in the leaf node of the mapping relationship model, the data to be taken offline is taken offline.

[0124] Figure 5 This is a schematic diagram of the structure of the data processing device provided in the embodiment of the present application. Figure 5 The embodiment of the present application further provides a data processing device 500, which can implement the above data processing method. The device 500 includes:

[0125] The data acquisition module 501 is used to acquire data to be taken offline from the data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables;

[0126] A first data processing module 502 is used to take the data to be taken offline offline if there is no downstream dependent data for the data to be taken offline, wherein the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data, wherein the external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse;

[0127] The second data processing module 503 is used to, if the data to be taken offline has the downstream dependent data, take the data offline according to the offline rule corresponding to the downstream dependent data.

[0128] In some embodiments, the second data processing module 503 is further configured to:

[0129] In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the internal downstream data, acquiring the running status of the internal downstream data;

[0130] When the running state of the internal downstream data is an abnormal working state, the data to be taken offline and the internal downstream data are taken offline, and the abnormal working state is a pause state or a failure state.

[0131] In some embodiments, the second data processing module 503 is further configured to:

[0132] In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the external downstream data and / or the authorized user data, switching the link of the external downstream data and / or the authorized user data;

[0133] When the link switching of the external downstream data and / or authorized user data is completed, the data to be taken offline is taken offline, and the link switching is used to switch the data to be taken offline referenced by the external downstream data and / or authorized user data to preset data.

[0134] In some embodiments, the data processing device 500 may further include a data parsing module, which is used to:

[0135] Acquire metadata related to the data to be taken offline;

[0136] If the data to be taken offline includes tasks to be taken offline, data parsing is performed on the metadata to obtain downstream dependent data of the tasks to be taken offline, where the downstream dependent data includes external downstream data and / or internal downstream data;

[0137] If the data to be taken offline includes a data table to be taken offline, the metadata is parsed to obtain downstream dependent data of the data table to be taken offline, wherein the downstream dependent data includes at least one of external downstream data, internal downstream data and authorized user data.

[0138] In some embodiments, the first data processing module 502 or the second data processing module 503 is further configured to:

[0139] If the data to be taken offline includes tasks to be taken offline, deleting the tasks to be taken offline;

[0140] or,

[0141] If the data to be taken offline includes a data table to be taken offline, renaming the data table to be taken offline to obtain a renaming result;

[0142] In the case where the renaming result indicates that the data table to be taken offline is renamed successfully, if no data table rollback instruction is received within a preset time period, deleting the data table to be taken offline;

[0143] or,

[0144] If the data to be taken offline includes tasks to be taken offline and data tables to be taken offline, complete the following steps in sequence:

[0145] Stop the task to be offline, and store the stop time of the task to be offline;

[0146] Rename the data table to be taken offline;

[0147] Deleting the data table to be taken offline;

[0148] Delete the task to be taken offline.

[0149] In some embodiments, the first data processing module 502 or the second data processing module 503 is further configured to:

[0150] Before deleting the data table to be taken offline, the method further includes:

[0151] According to the authorized user data, the authority granted to the data table to be taken offline is reclaimed.

[0152] In some embodiments, the first data processing module 502 or the second data processing module 503 is further configured to:

[0153] Obtaining metadata related to the data to be taken offline, and performing data analysis on the metadata to obtain analysis results;

[0154] According to the analysis result and the data to be taken offline, a mapping relationship model related to the data to be taken offline is obtained, wherein the mapping relationship model includes a plurality of nodes, wherein the plurality of nodes include a root node and child nodes of the root node, and the child nodes include leaf nodes;

[0155] The data to be taken offline is matched with the data stored in the leaf node in the mapping relationship model, and when the match is successful, the data to be taken offline is taken offline.

[0156] The specific implementation of the data processing device is basically the same as the specific implementation of the above-mentioned data processing method, and will not be repeated here.

[0157] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0158] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. Figure 6 . Electronic equipment includes:

[0159] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0160] The memory 602 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the data processing method of the embodiments of this application;

[0161] Input / output interface 603, used to implement information input and output;

[0162] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0163] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0164] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0165] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data processing method when executed by a processor.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The data processing method, device, electronic device and storage medium provided in the embodiments of the present application obtain the data to be taken offline from the data warehouse, and take the data offline when there is no downstream dependent data for the data to be taken offline, or, when there is downstream dependent data for the data to be taken offline, take the data offline according to the offline rules corresponding to the downstream dependent data. According to different situations of the data to be taken offline, that is, whether there is downstream dependent data, the present application adopts the corresponding offline method to process the data to be taken offline. In the case of a large amount of data to be taken offline, the data to be taken offline can be taken offline in an orderly manner, which reduces the complexity of the data offline process, thereby improving the efficiency of data offline.

[0168] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0169] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire data to be taken offline from a data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables; In the case that the data to be taken offline does not have downstream dependent data, the data to be taken offline is taken offline, the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data, the external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse; In the case that the data to be taken offline has the downstream dependent data, the data to be taken offline is taken offline according to the offline rule corresponding to the downstream dependent data.

2. The method according to claim 1, characterized in that When the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes: In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the internal downstream data, acquiring the running status of the internal downstream data; When the running state of the internal downstream data is an abnormal working state, the data to be taken offline and the internal downstream data are taken offline, and the abnormal working state is a pause state or a failure state.

3. The method according to claim 1, characterized in that When the data to be taken offline has the downstream dependent data, taking the data to be taken offline offline according to the offline rule corresponding to the downstream dependent data includes: In the case that the data to be taken offline has the downstream dependent data, if the downstream dependent data includes the external downstream data and / or the authorized user data, switching the link of the external downstream data and / or the authorized user data; When the link switching of the external downstream data and / or authorized user data is completed, the data to be taken offline is taken offline, and the link switching is used to switch the data to be taken offline referenced by the external downstream data and / or authorized user data to preset data.

4. The method according to claim 1, characterized in that After obtaining the data to be taken offline from the data warehouse, the method further includes: Acquire metadata related to the data to be taken offline; If the data to be taken offline includes tasks to be taken offline, data parsing is performed on the metadata to obtain downstream dependent data of the tasks to be taken offline, where the downstream dependent data includes external downstream data and / or internal downstream data; If the data to be taken offline includes a data table to be taken offline, the metadata is parsed to obtain downstream dependent data of the data table to be taken offline, wherein the downstream dependent data includes at least one of external downstream data, internal downstream data and authorized user data.

5. The method according to claim 1, characterized in that Taking the data to be taken offline offline includes: If the data to be taken offline includes tasks to be taken offline, deleting the tasks to be taken offline; or, If the data to be taken offline includes a data table to be taken offline, renaming the data table to be taken offline to obtain a renaming result; In the case where the renaming result indicates that the data table to be taken offline is renamed successfully, if no data table rollback instruction is received within a preset time period, deleting the data table to be taken offline; or, If the data to be taken offline includes tasks to be taken offline and data tables to be taken offline, complete the following steps in sequence: Stop the task to be offline, and store the stop time of the task to be offline; Rename the data table to be taken offline; Deleting the data table to be taken offline; Delete the task to be taken offline.

6. The method according to claim 5, characterized in that The data to be taken offline includes a data table to be taken offline, and downstream dependent data of the data table to be taken offline includes authorized user data; Before deleting the data table to be taken offline, the method further includes: According to the authorized user data, the authority granted to the data table to be taken offline is reclaimed.

7. The method according to claim 1, characterized in that Before taking the data to be taken offline offline, the method further includes: Obtaining metadata related to the data to be taken offline, and performing data analysis on the metadata to obtain analysis results; According to the analysis result and the data to be taken offline, a mapping relationship model related to the data to be taken offline is obtained, wherein the mapping relationship model includes a plurality of nodes, and the plurality of nodes include a root node and child nodes of the root node, and the child nodes include leaf nodes; The data to be taken offline is matched with the data stored in the leaf node in the mapping relationship model, and when the match is successful, the data to be taken offline is taken offline.

8. A data processing device, characterized in that: The device comprises: A data acquisition module, used for acquiring data to be taken offline from a data warehouse, wherein the data to be taken offline includes tasks to be taken offline and / or data tables; A first data processing module is used to take the data to be taken offline offline if there is no downstream dependent data for the data to be taken offline, the downstream dependent data is downstream data related to the data to be taken offline, and the downstream dependent data includes at least one of external downstream data, internal downstream data, and authorized user data, the external downstream data is downstream data outside the data warehouse that has a dependent relationship with the data to be taken offline, the internal downstream data is downstream data inside the data warehouse that has a dependent relationship with the data to be taken offline, and the authorized user data is authorization information of the data to be taken offline for downstream users outside the data warehouse; The second data processing module is used for, when the data to be taken offline has the downstream dependent data, taking the data offline according to the offline rule corresponding to the downstream dependent data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images