Data processing method and device, storage medium and terminal
By performing quality rule checks and label generation on target data during data update events, the problem of difficulty in controlling the quality of large-scale data is solved, and the intuitive display of data quality and the accuracy of analysis are improved.
Patent Information
- Application Number
- CN202410876592.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2026-01-06
AI Technical Summary
The quality of large volumes of data is difficult to control, affecting the accuracy of analysis results and the efficiency of enterprise operations.
In response to data update events in the data warehouse, the system acquires target data, determines data quality rules, performs quality checks, generates labels, and sends the tagged data to the data management platform for use.
By validating quality rules during data updates, we can ensure a clear and intuitive presentation of data quality information, improve the accuracy of data analysis and the ability to identify problems during operations, and ultimately enhance data quality.
Smart Images
Figure CN121277918A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data processing technology, and in particular to a data processing method, apparatus, storage medium and terminal. Background Technology
[0002] With the development and widespread application of the internet, data is being generated continuously. Collecting, analyzing, and interpreting this data helps businesses more accurately understand key information such as market trends and customer needs, thus making data a crucial basis for business decision-making. Furthermore, data helps businesses optimize operations and improve efficiency. By monitoring and analyzing data from various stages such as production, sales, and inventory in real time, businesses can promptly identify problems and make relevant adjustments.
[0003] However, the value of data depends not only on its quantity but also on its quality, which directly affects the accuracy of the analytical results based on it. Therefore, a data processing method is needed to perform quality checks on data in order to optimize its quality. Summary of the Invention
[0004] This application provides a data processing method, apparatus, storage medium, and terminal, which can solve the technical problem of difficulty in controlling the data quality of large-volume data in related technologies.
[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising:
[0006] In response to a data update event in the data warehouse, obtain at least one target data corresponding to the aforementioned data update event;
[0007] Determine the data quality rules corresponding to each target data, perform quality inspection on the data quality of each target data according to the above data quality rules, and generate the corresponding labels for each target data based on the quality inspection results of each target data.
[0008] Each tagged target data is sent to the target data management platform so that users can use the target data based on its tags within the platform.
[0009] In one possible implementation, obtaining at least one target data corresponding to the data update event includes: triggering a data tracking task corresponding to the data warehouse based on the data update event, and collecting at least one target data corresponding to the data update event based on the data tracking task.
[0010] In one possible implementation, the above-mentioned determination of the data quality rules corresponding to each target data includes: determining the data quality rules corresponding to each target data according to the data attributes of each target data.
[0011] In one possible implementation, the above-mentioned determination of the data quality rules corresponding to each target data includes: calling the rule engine based on the above-mentioned data tracking task, so that the rule engine determines the data quality rules matching the data attributes of each target data from the rule management platform; wherein the above-mentioned rule management platform includes at least one preset data quality rule, which is set for the characteristics of at least one data attribute.
[0012] In one possible implementation, the above-mentioned sending of each tagged target data to a target data management platform, so that users can use each target data based on the tags of each target data in the data management platform, includes: sending each tagged target data to a target data management platform, so that the data management platform can perform statistics on each target data based on the tags of each target data, and convert each target data into business indicators.
[0013] In one possible implementation, the target data are statistically analyzed and transformed according to preset indicator generation rules, which are formulated based on the business logic of the business scenario.
[0014] In one possible implementation, the method further includes: responding to a user's search request for a target tag in the target data management platform; retrieving target search data corresponding to the target tag in the target data management platform; and displaying the target search data on the human-computer interaction interface between the target data management platform and the user.
[0015] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising:
[0016] The data update module is used to respond to data update events in the data warehouse and obtain at least one target data corresponding to the aforementioned data update event;
[0017] The data verification module is used to determine the data quality rules corresponding to each target data, perform quality verification on the data quality of each target data according to the above data quality rules, and generate the corresponding label for each target data based on the quality verification results.
[0018] The data usage module is used to send the tagged target data to the target data management platform, so that users can use the target data based on the tags of each target data in the aforementioned data management platform.
[0019] In one possible implementation, the data update module is further configured to trigger a data tracking task corresponding to the data warehouse based on the data update event, and collect at least one target data corresponding to the data update event based on the data tracking task.
[0020] In one possible implementation, the aforementioned data verification module is also used to determine the data quality rules corresponding to each target data according to the data attributes of each target data.
[0021] In one possible implementation, the data verification module is further configured to invoke the rule engine based on the data tracking task, so that the rule engine can determine the data quality rules matching the data attributes of each target data from the rule management platform; wherein the rule management platform includes at least one preset data quality rule, which is set for the characteristics of at least one data attribute.
[0022] In one possible implementation, the aforementioned data usage module is also used to send each tagged target data to the target data management platform, so that the data management platform can perform statistics on each target data based on the tags of each target data and transform each target data into business indicators.
[0023] In one possible implementation, the target data are statistically analyzed and transformed according to preset indicator generation rules, which are formulated based on the business logic of the business scenario.
[0024] In one possible implementation, the device further includes a data retrieval module, configured to respond to a user's retrieval request for a target tag in the target data management platform; retrieve target retrieval data corresponding to the target tag in the target data management platform; and display the target retrieval data on the human-computer interaction interface between the target data management platform and the user.
[0025] Thirdly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method described above.
[0026] Fourthly, embodiments of this application provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the above-described method.
[0027] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following:
[0028] This application provides a data processing method that, in response to a data update event in a data warehouse, acquires at least one target data corresponding to the data update event; determines data quality rules for each target data; performs quality checks on the data quality of each target data according to the data quality rules; and generates labels for each target data based on the quality check results. The tagged target data is then sent to a target data management platform, enabling users to use the target data based on its labels within the data management platform. Because when a data update event occurs in the data warehouse, before using the updated target data, the target data is first validated against data quality rules to check its quality. The check results are then used as labels for the target data, which are stored in the data management platform. This allows users to intuitively understand the data quality information when using the target data based on its labels, facilitating convenient data analysis and maintenance, identifying data problems during operation, and improving data quality. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 An exemplary system architecture diagram of a data processing method provided in this application embodiment;
[0031] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;
[0032] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application;
[0033] Figure 4 A logic flowchart of a data processing method provided in an embodiment of this application;
[0034] Figure 5 A structural block diagram of a data processing apparatus provided in an embodiment of this application;
[0035] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation
[0036] To make the features and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0037] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0038] Currently, data has become the fifth major factor of production after land, labor, capital, and technology, and big data can empower all sectors of society. By collecting, analyzing, and interpreting data, enterprises can more accurately understand key information such as market trends and customer needs, thereby making more informed decisions. Data is not only an important basis for enterprise decision-making, but also helps optimize internal operations and improve efficiency. By monitoring and analyzing data from various stages such as production, sales, and inventory in real time, enterprises can promptly identify problems and make relevant adjustments. In the long run, in the digital age, data is also an important source of technological innovation across all industries. Through the mining and analysis of big data, enterprises can discover new business models, optimize product design and service processes, upgrade technology, and thus improve user experience.
[0039] However, the value of data depends not only on its quantity but, more importantly, on its quality. If data contains errors, omissions, or inconsistencies, decisions made based on it are likely to be biased or misleading. Therefore, data quality directly impacts the accuracy of decisions. Furthermore, low-quality data can lead to system failures, process delays, and other problems, increasing operational costs and risks for businesses. Thus, data quality is also closely related to operational efficiency.
[0040] Therefore, this application provides a data processing method to solve the technical problem of difficulty in controlling the data quality of large amounts of data.
[0041] Please see Figure 1 , Figure 1 An exemplary system architecture diagram of a data processing method provided in an embodiment of this application.
[0042] like Figure 1As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.
[0043] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.
[0044] In this embodiment, firstly, terminal 101 responds to a data update event in the data warehouse and obtains at least one target data corresponding to the data update event; further, terminal 101 determines the data quality rules corresponding to each target data, performs quality checks on the data quality of each target data according to the data quality rules, and generates a tag corresponding to each target data based on the quality check results; at this time, terminal 101 can send each target data with tags to the target data management platform, so that users can use each target data in the data management platform based on the tags of each target data.
[0045] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.
[0046] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this application do not limit this.
[0047] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.
[0048] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal performing data processing, a processor within the terminal performing the data processing method, or a data processing service within the terminal performing the data processing method. For ease of description, the following example uses a processor within a terminal as the execution entity to illustrate the specific execution process of the data processing method.
[0049] like Figure 2 As shown, the data processing method may include at least:
[0050] S202. In response to a data update event in the data warehouse, obtain at least one target data corresponding to the data update event.
[0051] Optionally, since data quality affects the accuracy of analytical results based on data, rigorous data quality checks are necessary to ensure data accuracy and reliability. Typically, data quality checks include steps such as data cleaning, validation, and standardization to ensure data integrity, accuracy, and consistency. However, with the accumulation of data volumes, the continuous generation and updating of large amounts of data, and the increasing complexity and diversity of data types, different data types often require different quality check rules. This makes data checks more difficult and consumes more computational resources during the process.
[0052] Alternatively, typically, enterprise data is stored in data warehouses, and target data is retrieved directly from the data warehouse when needed. However, in this data management approach, databases scattered throughout the organization become isolated data silos, lacking a unified management and sharing mechanism. Furthermore, the same logic can have different implementations and interpretations, making it difficult to standardize understanding and reach consensus. Due to cost and technical considerations, it's impossible to use a single table to store all the data, easily leading to data bias. In this situation, data quality management typically involves statistical analysis at fixed time intervals, resulting in poor timeliness, delayed problem detection, and high manpower costs. This approach is only suitable for short-term, infrequently changing data scenarios. When used in scenarios where data undergoes long-term iteration, inconsistencies in data table and content formats, as well as logical mismatches, can occur, ultimately leading to data loss or errors.
[0053] Optionally, to ensure the accuracy of all data during use, data specifications need to be checked in real time during the daily operation of the data tables. That is, every time new data appears in the data warehouse, a quality check is performed on the new data before it is put into use. This allows for timely quality checks even with long-term data iteration, frequent additions, or changes, thus enabling timely maintenance and operation of data quality. Based on this, in this embodiment, a data warehouse is used to store all data in all data tables. When a data update event occurs, it indicates a data change. Therefore, the system first responds to the data update event in the data warehouse and obtains at least one target data corresponding to the data update event. The target data is the new data whose quality has not been checked.
[0054] S204. Determine the data quality rules corresponding to each target data, perform quality inspection on the data quality of each target data according to the data quality rules, and generate the corresponding labels for each target data based on the quality inspection results of each target data.
[0055] Optionally, different types of data have different data specification standards. Therefore, to perform standardized quality checks on various data types and transform the results into an intuitive and easy-to-use format, the quality specifications corresponding to different types of data can be abstracted into data rules. Based on these pre-defined data rules, corresponding rule constraints are applied to each type of data to achieve data quality checks for all types of data. Specifically, data quality rules corresponding to different attributes can be pre-defined. When target data exists, the data quality rules corresponding to each target data are determined, and the data quality of each target data is checked according to these rules.
[0056] Specifically, considering that the quality standards for various types of data are related to and differ from their attributes—for example, data with attribute A needs to be checked for missing data and correct data format, while data with attribute B needs to be checked for normality—data quality rules are determined for each target data based on its data attributes, and the corresponding data quality rules are then used to perform appropriate checks on the data.
[0057] Furthermore, after data inspection, the quality inspection results of each target data are obtained. The target data should carry its own quality information so that users can intuitively understand the quality of the data when it is used later, which is conducive to improving the accuracy of data analysis. Therefore, it is necessary to generate labels corresponding to each target data based on the quality inspection results of each target data.
[0058] S206. Send each tagged target data to the target data management platform so that users can use each target data based on its tags in the data management platform.
[0059] Optionally, as described above, by converting the quality inspection results of target data into tags, data quality can be quantified. Furthermore, the tags can be used to automatically statistically analyze the target data into indicators, facilitating users' access to them at any time. Specifically, each tagged target data is sent to a target data management platform. As a human-computer interaction platform, the target data management platform allows users to manage target data. After the data and its quality information are quantified into indicators and output to the platform, data producers, data developers, data consumers, and other business stakeholders can operate the data in a targeted manner according to their respective concerns. This allows users to use each target data based on its tags within the data management platform, thereby identifying data problems and improving data quality during operation.
[0060] In this embodiment, a data processing method is provided. In response to a data update event in a data warehouse, at least one target data corresponding to the data update event is obtained; data quality rules corresponding to each target data are determined; the data quality of each target data is checked according to the data quality rules; and a tag is generated for each target data based on the quality check results. The tagged target data is then sent to a target data management platform, enabling users to use the target data based on its tags within the data management platform. Because when a data update event occurs in the data warehouse, before using the updated target data, the target data is first validated according to data quality rules to check its quality. The check results are then used as tags for the target data, and the target data is stored in the data management platform. This allows users to intuitively understand the data quality information when using the target data based on the tags, facilitating convenient data analysis and maintenance, thereby identifying data problems during operation and improving data quality.
[0061] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application.
[0062] like Figure 3 As shown, the data processing method may include at least:
[0063] S302. In response to a data update event in the data warehouse, trigger a data tracking task corresponding to the data warehouse based on the data update event, and collect at least one target data corresponding to the data update event based on the data tracking task.
[0064] Optionally, to perform timely quality checks on target data, a data tracking function can be used on the data warehouse. When a data update event occurs, the quality check process for the target data is automatically triggered. The principle of the data tracking function is mainly based on event tracking technology. Its core purpose is to capture, process, and send relevant data for specific behaviors or events. Specifically, data tracking is implemented by deploying collection SDK code on terminals such as Apps, H5, and PCs. When an event occurs and meets certain conditions, such as entering a page or generating a certain type of data, this code is automatically triggered, recording and storing the relevant operation data. In this embodiment, in response to a data update event in the data warehouse, a corresponding data tracking task is triggered based on the data update event. This data tracking task collects at least one target data corresponding to the data update event, reducing the computational resource consumption for monitoring large volumes of data through data warehouse data tracking technology.
[0065] Optionally, please refer to Figure 4 , Figure 4 This is a logic flowchart of a data processing method provided in an embodiment of this application. Figure 4 As shown, all data from all data tables are stored in the data warehouse. A data specification module is built for the data warehouse to manage data quality inspection tasks. An event tracking function is added to the data specification module. The event tracking function can trigger event tracking tasks based on data update events to collect the target data corresponding to the data update events.
[0066] S304. The rule engine is invoked based on the data tracking task, so that the rule engine can determine the data quality rules that match the data attributes of each target data from the rule management platform, perform quality inspection on the data quality of each target data according to the data quality rules, and generate the corresponding tags for each target data based on the quality inspection results of each target data.
[0067] Optionally, when quality checks are required on the target data, data quality rules can be determined for each target data based on its data attributes. Please refer to the following for details. Figure 4 After the target data is collected by the tracking task, the rule engine is invoked. The rule engine then determines the data quality rules matching the data attributes of each target data point from the rule management platform. The rule management platform includes at least one preset data quality rule, which is set based on the characteristics of at least one data attribute. This allows relevant personnel to manage the rules within the rule management platform, including defining the specific content, triggering conditions, and judgment targets of the rules. Data synchronization between the rule engine and the rule management platform facilitates the invocation of any rule.
[0068] Furthermore, after data inspection, the quality inspection results of each target data are obtained. The target data should carry its own quality information so that users can intuitively understand the quality of the data when it is used later, which is conducive to improving the accuracy of data analysis. Therefore, it is necessary to generate labels corresponding to each target data based on the quality inspection results of each target data.
[0069] S306. Send the tagged target data to the target data management platform so that the data management platform can perform statistics on each target data based on the tags of each target data and transform each target data into business indicators.
[0070] Optionally, by converting the quality inspection results of the target data into labels, data quality can be quantified. This allows for automated statistical analysis of the target data using these labels, making it convenient for users to view the data at any time. Please continue reading. Figure 4 Specifically, tagged target data is sent to a target data management platform. The data collectors on the platform continuously receive data, enabling the platform to statistically analyze the data based on its tags and transform it into business metrics. During this metric transformation, each target data point is statistically analyzed and transformed according to preset metric generation rules, which are derived from business logic within the specific business scenario. This allows users to view data at any time and monitor notifications in real time; continuous operation of the product, data, rules, and metrics continuously improves data quality.
[0071] S308. Respond to the search request initiated by the user for the target tag in the target data management platform.
[0072] Optionally, users can also view and use data for specific tags based on the tags corresponding to the data. When a user needs to find data for a target tag, a search request will be initiated for the target tag. That is, the terminal will respond to the search request initiated by the user for the target tag in the target data management platform.
[0073] S310. Retrieve the target search data corresponding to the target tag in the target data management platform, and display the target search data on the human-computer interaction interface between the target data management platform and the user.
[0074] Optionally, after receiving a user's search request, the system retrieves the target search data corresponding to the target tag in the target data management platform and displays the target search data in the human-computer interaction interface between the target data management platform and the user, so that the user can manage specific types of data according to their own needs.
[0075] In this embodiment, a data processing method is provided. In response to a data update event in a data warehouse, a corresponding data tracking task is triggered based on the data update event. The tracking task collects at least one target data corresponding to the data update event. When the data update event occurs, a quality check process for the target data is automatically triggered, collecting the target data corresponding to the data update event. A rule engine is invoked based on the tracking task, enabling the rule engine to determine data quality rules matching the data attributes of each target data from a rule management platform. This introduces rule management and a rule engine, allowing rule management within the rule management platform and facilitating the invocation of any rule. Each target data item with tags is sent to a target data management platform, enabling the platform to perform statistics on each target data item based on its tags, transforming each target data item into business indicators, quantifying data quality, and facilitating user viewing at any time. Based on the tags of the data after rule verification, data with specific tags can also be retrieved and used within the platform.
[0076] Please see Figure 5 , Figure 5 This is a structural block diagram of a data processing device provided in an embodiment of this application.
[0077] like Figure 5 As shown, the data processing device 500 includes:
[0078] Data update module 510 is used to respond to data update events in the data warehouse and obtain at least one target data corresponding to the data update event;
[0079] The data inspection module 520 is used to determine the data quality rules corresponding to each target data, perform quality inspection on the data quality of each target data according to the data quality rules, and generate the corresponding label for each target data based on the quality inspection results.
[0080] The data usage module 530 is used to send each tagged target data to the target data management platform, so that users can use each target data based on the tags of each target data in the data management platform.
[0081] Optionally, the data update module 510 is also used to trigger the corresponding data tracking task of the data warehouse based on the data update event, and to collect at least one target data corresponding to the data update event based on the data tracking task.
[0082] Optionally, the data verification module 520 is also used to determine the data quality rules corresponding to each target data according to the data attributes of each target data.
[0083] Optionally, the data verification module 520 is also used to call the rule engine based on the data tracking task, so that the rule engine can determine the data quality rules that match the data attributes of each target data from the rule management platform; wherein, the rule management platform includes at least one preset data quality rule, which is set for the characteristics of at least one data attribute.
[0084] Optionally, the data usage module 530 is also used to send each tagged target data to the target data management platform, so that the data management platform can perform statistics on each target data based on the tags of each target data and transform each target data into business indicators.
[0085] Optionally, each target data is statistically analyzed and transformed according to preset indicator generation rules, which are formulated based on business logic in the business scenario.
[0086] Optionally, the data processing device 500 further includes: a data retrieval module, used to respond to a retrieval request initiated by a user for a target tag in the target data management platform; retrieve target retrieval data corresponding to the target tag in the target data management platform; and display the target retrieval data on the human-computer interaction interface between the target data management platform and the user.
[0087] In this embodiment, a data processing apparatus is provided, comprising: a data update module for responding to a data update event in a data warehouse and acquiring at least one target data corresponding to the data update event; a data verification module for determining data quality rules corresponding to each target data, performing quality verification on the data quality of each target data according to the data quality rules, and generating a tag corresponding to each target data based on the quality verification results; and a data usage module for sending the tagged target data to a target data management platform, enabling users to use each target data based on its tags within the data management platform. Because when a data update event occurs in the data warehouse, before putting the updated target data into use, the target data is first verified using data quality rules to check its quality. The verification results are then used as tags for the target data, and the target data is stored in the data management platform. This allows users to intuitively understand the data quality information when using target data based on tags, facilitating convenient data analysis and maintenance, thereby identifying data problems during operation and improving data quality.
[0088] This application also provides a computer storage medium that can store multiple instructions adapted for loading by a processor and executing the steps of any of the methods described in the above embodiments.
[0089] Please see Figure 6 , Figure 6This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 6 As shown, terminal 600 may include: at least one terminal processor 601, at least one network interface 604, user interface 603, memory 605, and at least one communication bus 602.
[0090] The communication bus 602 is used to enable communication between these components.
[0091] The user interface 603 may include a display screen and a camera. Optionally, the user interface 603 may also include a standard wired interface and a wireless interface.
[0092] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0093] The terminal processor 601 may include one or more processing cores. The terminal processor 601 connects to various parts within the terminal 600 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the terminal processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The terminal processor 601 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the terminal processor 601 and may be implemented as a separate chip.
[0094] The memory 605 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 605 may include a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned terminal processor 601. Figure 6 As shown, the memory 605, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data processing program.
[0095] exist Figure 6 In the terminal 600 shown, the user interface 603 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 601 can be used to call the data processing program stored in the memory 605 and specifically perform the following operations:
[0096] In response to a data update event in the data warehouse, retrieve at least one target data corresponding to the data update event;
[0097] Determine the data quality rules corresponding to each target data, perform quality checks on the data quality of each target data according to the data quality rules, and generate labels corresponding to each target data based on the quality check results.
[0098] Each tagged target data is sent to the target data management platform so that users can use the target data based on its tags within the platform.
[0099] In some embodiments, when the terminal processor 601 executes the process of acquiring at least one target data corresponding to a data update event, it specifically performs the following steps: triggering a data warehouse tracking task based on the data update event, and collecting at least one target data corresponding to the data update event based on the tracking task.
[0100] In some embodiments, when the terminal processor 601 executes the steps of determining the data quality rules corresponding to each target data, it specifically performs the following steps: determining the data quality rules corresponding to each target data according to the data attributes of each target data.
[0101] In some embodiments, when the terminal processor 601 executes the data quality rules corresponding to each target data, it specifically performs the following steps: calling the rule engine based on the data tracking task so that the rule engine determines the data quality rules matching the data attributes of each target data from the rule management platform; wherein, the rule management platform includes at least one preset data quality rule, which is set for the characteristics of at least one data attribute.
[0102] In some embodiments, when the terminal processor 601 sends each tagged target data to the target data management platform so that the user can use each target data based on the tags of each target data in the data management platform, it specifically performs the following steps: sending each tagged target data to the target data management platform so that the data management platform can perform statistics on each target data based on the tags of each target data and convert each target data into business indicators.
[0103] In some embodiments, the target data are statistically analyzed and transformed according to preset indicator generation rules, which are formulated based on business logic in the business scenario.
[0104] In some embodiments, the terminal processor 601 further performs the following steps: responding to a search request initiated by a user for a target tag in the target data management platform; retrieving target search data corresponding to the target tag in the target data management platform; and displaying the target search data on the human-computer interaction interface between the target data management platform and the user.
[0105] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0106] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).
[0108] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0110] The above is a description of a data processing method, apparatus, storage medium, and terminal provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data processing method, characterized by, The method includes: In response to a data update event in the data warehouse, at least one target data corresponding to the data update event is obtained; Determine the data quality rules corresponding to each target data, perform quality inspection on the data quality of each target data according to the data quality rules, and generate the corresponding label for each target data based on the quality inspection results of each target data. Each tagged target data is sent to a target data management platform so that users can use the target data based on its tags within the platform.
2. The method of claim 1, wherein, The step of obtaining at least one target data corresponding to the data update event includes: The data update event triggers the corresponding data tracking task in the data warehouse, and the data tracking task collects at least one target data corresponding to the data update event.
3. The method of claim 1, wherein, The determination of the data quality rules corresponding to each target data includes: The data quality rules for each target data are determined based on its data attributes.
4. The method of claim 2, wherein, The determination of the data quality rules corresponding to each target data includes: The rule engine is invoked based on the data tracking task, so that the rule engine can determine the data quality rules that match the data attributes of each target data from the rule management platform; The rule management platform includes at least one preset data quality rule, which is set for the characteristics of at least one data attribute.
5. The method of claim 1, wherein, The step of sending each tagged target data to a target data management platform, so that users can use each target data based on its tags in the data management platform, includes: Each tagged target data is sent to the target data management platform, so that the data management platform can perform statistics on each target data based on the tags of each target data and transform each target data into business indicators.
6. The method of claim 5, wherein, Each target data is statistically analyzed and transformed according to preset indicator generation rules, which are formulated based on business logic in the business scenario.
7. The method of claim 1, wherein, The method further includes: Respond to the search request initiated by the user for the target tag in the target data management platform; The target data corresponding to the target tag is retrieved in the target data management platform, and the target data is displayed on the human-computer interaction interface between the target data management platform and the user.
8. A data processing apparatus, characterized by, The device includes: The data update module is used to respond to a data update event in the data warehouse and obtain at least one target data corresponding to the data update event; The data verification module is used to determine the data quality rules corresponding to each target data, perform quality verification on the data quality of each target data according to the data quality rules, and generate the corresponding label for each target data based on the quality verification results. The data usage module is used to send each tagged target data to the target data management platform, so that users can use each target data based on the tags of each target data in the data management platform.
9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 7.
10. A terminal, characterized by comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 7 when running the program.