Data processing method and device and medium
Through the relationship between configuration tables and monitoring policies, and using tables as media, the problem of frequent reconfiguration during job updates in the existing technology is solved, and efficient data quality monitoring is achieved.
Patent Information
- Application Number
- CN202311155238.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-03-07
AI Technical Summary
When the prior art monitors the data output by the job, it is necessary to frequently reconfigure the relationship between the job and the monitoring strategy, resulting in large amounts of configuration work and low efficiency.
By configuring the relationship between the table and the monitoring policy, using the table as a medium, the monitoring policy corresponding to the table of the target job is determined, thereby performing data quality monitoring.
It reduces the configuration workload, improves the quality monitoring efficiency of big data, reduces the need for reconfiguration during job updates, and reduces the coupling between data quality monitoring and data development.
Smart Images
Figure CN119576908A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of big data, and in particular to a data processing method, device and medium. Background Art
[0002] Modern society is a rapidly developing society with advanced technology and high-speed information flow. Big data is a product of this high-tech era. Big data is a collection of data that is so large that it greatly exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management and analysis. It has the characteristics of massive data scale, fast data flow, diverse data types and low value density. Big data technology has also been applied to various industries, including finance, automobiles, catering, telecommunications, energy, logistics and entertainment.
[0003] In the field of big data development, development operations can be carried out according to specific needs to produce the required data. For example, big data can be used to profile enterprise users and train models. The data produced by development operations needs to be quality-controlled to ensure the accuracy of the data. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, device and medium.
[0005] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:
[0006] Identify whether the target job has data output;
[0007] In response to identifying that the target job has data output, determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job;
[0008] According to the determined monitoring strategy, the quality of the data output by the target operation is monitored.
[0009] Optionally, the identifying whether the target job has data output includes:
[0010] Determine whether the current node of the target job has been executed;
[0011] Determine the table of the target job according to the mapping relationship between the job and the table;
[0012] Determine whether the table partition corresponding to the current node of the target job is established;
[0013] When the current node of the target job has been executed and the table partition corresponding to the current node of the target job has been established, it is determined that the target job has data output.
[0014] Optionally, the determining whether the table partition corresponding to the current node of the target job is established includes:
[0015] According to the audit log in the HDFS, it is determined whether the table partition corresponding to the current node of the target job is established, and the table of the target job is stored in the HDFS.
[0016] Optionally, the method further comprises:
[0017] The mapping relationship between the job and the table is determined according to the audit log in the HDFS, and the table of the target job is stored in the HDFS.
[0018] Optionally, determining whether the table partition corresponding to the current node of the target job is established according to the audit log in the HDFS includes:
[0019] Obtaining first progress information, where the first progress information is used to indicate whether a table partition corresponding to a current node of the target job has been established, and the first progress information is information generated according to an audit log in the HDFS;
[0020] According to the first progress information, it is determined whether the table partition corresponding to the current node of the target job has been established.
[0021] Optionally, the method further comprises:
[0022] Get the mapping relationship between jobs and tables.
[0023] Optionally, determining whether the current node of the target job has been executed completely includes:
[0024] Acquire second progress information, where the second progress information is used to indicate whether the current node of the target job has been executed;
[0025] Determine whether the current node of the target job has been executed completely according to the second progress information.
[0026] According to a second aspect of an embodiment of the present disclosure, a data processing device is provided, the device comprising:
[0027] An identification module is configured to identify whether a target job has data output;
[0028] A first determination module is configured to, in response to identifying that the target job has data output, determine the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job;
[0029] The monitoring module is configured to perform quality monitoring on the data output by the target job according to the determined monitoring strategy.
[0030] According to a third aspect of an embodiment of the present disclosure, there is provided a data processing device, including:
[0031] A second processor;
[0032] a second memory for storing second processor executable instructions;
[0033] Wherein, the second processor is configured as follows:
[0034] Identify whether the target job has data output;
[0035] In response to identifying that the target job has data output, determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job;
[0036] According to the determined monitoring strategy, the quality of the data output by the target operation is monitored.
[0037] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a first processor, the steps of the data processing method provided in the first aspect of the present disclosure are implemented.
[0038] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0039] In the related art, when the quality of the data output by the job is monitored, the association between the job and the monitoring strategy is configured. In the workflow of the job, when a node is completed, the monitoring strategy adopted for the job is determined according to the above association. If the developed job is updated, the association between the job and the monitoring strategy needs to be reconfigured. In this solution, the association between the table and the monitoring strategy is configured, that is, a table is added between the association between the job and the monitoring strategy, with the table as the medium. Since the structural characteristics of the table corresponding to the job are usually unchanged, when performing data quality monitoring, even if the job is updated (such as version iteration), it is usually not necessary to reconfigure the association between the table and the monitoring strategy. Therefore, when performing data quality monitoring, this solution reduces the workload of configuration and improves the efficiency of quality monitoring of big data. In addition, since the object associated with the monitoring strategy is converted from the job to the table, the operator who performs data quality monitoring does not need to configure the association between the job and the monitoring strategy multiple times, and the coupling degree between data quality monitoring and data development is low.
[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0042] Figure 1 The invention is a flowchart of a data processing method according to an exemplary embodiment.
[0043] Figure 2 The diagram is a schematic diagram of a scenario showing a data processing method according to an exemplary embodiment.
[0044] Figure 3 It is a block diagram of a data processing device according to an exemplary embodiment.
[0045] Figure 4 It is a block diagram of a device for data processing according to an exemplary embodiment. DETAILED DESCRIPTION
[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0047] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.
[0048] Figure 1 FIG. 1 is a flow chart of a data processing method according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps.
[0049] In step S101 , it is identified whether the target job has data output.
[0050] In step S102, in response to identifying that the target job has data output, the monitoring strategy corresponding to the table of the target job is determined according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job.
[0051] In step S103, quality monitoring is performed on the data output by the target operation according to the determined monitoring strategy.
[0052] In the field of big data, data is stored in the form of tables. For example, Hive and Iceberg tables are used to store offline data (data that is not updated in real time), where the data production of the table depends on the data development workflow maintained by each developer. A job refers to a series of operations on data. According to pre-defined requirements (for example, user portraits), developers develop the process of the job. During the execution of the job, data is obtained and analyzed according to the developed process, and the output data is updated in the table. That is, when the job is executed, the received upstream data is processed and the output data is written into the table to update the data in the table. For example, the job can be to obtain user consumption data from a predetermined shopping platform, and finally determine the tags of users in the shopping platform according to the pre-developed workflow. Consumption data can include shopping categories, shopping frequency, shopping amounts, etc. User tags can include point of interest classification.
[0053] The execution plan of the target job can be pre-set by the designer. For example, the execution plan can include starting execution at 6 o'clock every day. In the process of executing the target job according to the plan, the data produced by the target job is stored in the table corresponding to the target job. When the job is updated (for example, version iteration), the data of the updated job is also stored in the table corresponding to the job before the update.
[0054] In the related art, when the quality of data produced by a job is monitored, the association between the job and the monitoring strategy is configured, for example, job A executes strategy A, and job B executes strategy B. During the execution of the job, after data is produced, the monitoring strategy adopted for the job is determined based on the pre-configured association. If the job is updated, the original association may no longer apply to the updated job, and the relationship between the job and the monitoring strategy needs to be reconfigured.
[0055] In this solution, the association between the table and the monitoring strategy is configured (for example, the A1 table executes the A1 strategy, and the B1 table executes the B1 strategy). That is, a table is added between the association between the job and the monitoring strategy, with the table as the medium. When it is identified that the target job has data output, the monitoring strategy corresponding to the table of the target job is determined by using the configuration relationship between the table and the monitoring strategy. Even if the job is updated (for example, version iteration), as long as the structural characteristics of the table corresponding to the job remain unchanged, the monitoring strategy can still be determined based on the original configuration relationship between the table and the monitoring strategy, so as to monitor the data quality of the data output by the target job, and there is no need to reconfigure the association between the updated job and the monitoring strategy. For example, the data output by the F job is stored in the F1 table, and the monitoring strategy associated with the F1 table is the F2 strategy. If the F job has data output, the F2 strategy is determined and executed with the F1 table as the medium. When the F job is updated, the data output by the updated F job is also stored in the F1 table. When the updated F job output data is identified, the quality monitoring is still performed according to the F2 strategy associated with the F1 table.
[0056] Through the above technical solution, in the related technology, when the quality of the data output by the job is monitored, the association between the job and the monitoring strategy is configured. In the workflow of the job, when a node is completed, the monitoring strategy adopted for the job is determined according to the above association. If the developed job is updated, the association between the job and the monitoring strategy needs to be reconfigured. In this solution, the association between the table and the monitoring strategy is configured, that is, a table is added between the association between the job and the monitoring strategy, with the table as the medium. Since the structural characteristics of the table corresponding to the job are usually unchanged, when performing data quality monitoring, even if the job is updated (such as version iteration), there is no need to reconfigure the association between the table and the monitoring strategy. Therefore, when performing data quality monitoring, this solution reduces the workload of configuration and improves the efficiency of quality monitoring of big data. In addition, since the object associated with the monitoring strategy is converted from the job to the table, the operator who performs data quality monitoring does not need to configure the association between the job and the monitoring strategy multiple times, and the coupling degree between data quality monitoring and data development is low.
[0057] In yet another embodiment, the step of identifying whether the target operation has data output includes:
[0058] Determine whether the current node of the target job has completed execution;
[0059] Determine the table of the target job based on the mapping relationship between the job and the table;
[0060] Determine whether the table partition corresponding to the current node of the target job has been established;
[0061] When the current node of the target job has been executed and the table partition corresponding to the current node of the target job has been established, it is determined that the target job has data output.
[0062] When the target job is executed according to the execution plan, each execution instance can be regarded as a node of the target job. For example, the instance of the target job executed within the predetermined time period is regarded as a node of the target job. For example, when creating user profiles based on the user consumption data obtained by the shopping platform, the job executed between 18:00 every day and 18:00 the next day can be regarded as a node.
[0063] The mapping relationship between jobs and tables can be a one-to-one relationship, for example, the data produced by job P is written into table P1, and the data produced by job Q is written into table Q1. During the execution of each node of the target job, the table of the target job is determined according to the mapping relationship between the job and the table. When the node execution of the target job is completed, it can be considered that the data produced by the current node has been written into the corresponding table. After the data produced by the current node is written into the corresponding table, a table partition corresponding to the current node is established in the table. Table partitions are storage locations for data division in a table. Data produced by different nodes of a job can be stored in different table partitions, that is, during the execution process of each job node, a table partition corresponding to the current node can be established in the table. For example, if a job is executed once a day, the data can be partitioned and stored in the table corresponding to the job according to the date of the day. The data format written into the table by each node is the same, but the data content is different. When the current node of the target job has been executed and the table partition corresponding to the current node of the target job has been established, it can be determined that the target job has data output, and the quality of the data produced by the target job is monitored.
[0064] In this embodiment, based on the completion of the execution of the current node, a judgment is added on whether the table partition is established to determine whether the target job has data output, thereby triggering the execution of data quality monitoring. That is, the data output time is identified based on the two conditions of the completion of the execution of the current node and the completion of the establishment of the table partition. In this way, when the monitoring strategy corresponding to the table of the target job is determined by using the configuration relationship between the table and the monitoring strategy, and the quality of the data output by the target job is monitored, the reliability of the data in the table is ensured, and the problem of missed detection, wrong detection or untimely monitoring of data due to untimely table partitioning is avoided.
[0065] In yet another embodiment, the above-mentioned determining whether the table partition corresponding to the current node of the target job is established includes:
[0066] According to the audit log in the Hadoop Distributed File System (HDFS), it is determined whether the table partition corresponding to the current node of the target job is established. The table of the target job is stored in the HDFS.
[0067] HDFS is a distributed file system. The data produced by the target job can be stored in HDFS. The data is written to HDFS and the corresponding table partition is established. HDFS can generate the corresponding HDFS audit log. HDFS audit log can record events related to data in HDFS. Events can include file access, permission changes, and file deletion.
[0068] HDFS has high fault tolerance and can provide high-throughput data access, which is suitable for applications on large-scale data sets. Storing the target job table in HDFS is low-cost and highly reliable. Based on the HDFS audit log, it can be reliably and accurately determined whether the table partition corresponding to the current node has been established.
[0069] In yet another embodiment, the above method further includes:
[0070] According to the audit log in HDFS, the mapping relationship between the job and the table is determined, and the table of the target job is stored in HDFS.
[0071] Multiple tables can be stored in HDFS, and each table can correspond to a job. The table into which the job output data is written can be recorded in the HDFS audit log. The mapping relationship between the job and the table determined by the HDFS audit log provides a data basis for determining the table corresponding to the job. The mapping relationship between the job and the table can be offline data, that is, when the quality of the output data of the current node of the job is monitored, the mapping relationship between the job and the table called is the mapping relationship used for quality monitoring of the output data of the historical nodes of the job. The mapping relationship between the job and the table can also be real-time data updated after the execution of each node is completed. When the quality of the output data of each node of the job is monitored, the real-time determined mapping relationship between the job and the table is used.
[0072] In this embodiment, the mapping relationship between jobs and tables can be reliably and accurately determined based on the HDFS audit log.
[0073] In yet another embodiment, the above-mentioned determining whether the table partition corresponding to the current node of the target job is established according to the audit log in the HDFS includes:
[0074] Obtaining first progress information, where the first progress information is used to indicate whether the table partition corresponding to the current node of the target job has been established, and the first progress information is information generated according to the audit log in the HDFS;
[0075] According to the first progress information, it is determined whether the table partition corresponding to the current node of the target job has been established.
[0076] The first progress information may be, for example, an identifier generated by a device (a server, a terminal device, etc.) outside the HDFS after acquiring the audit log in the HDFS.
[0077] Figure 2 This is a scenario diagram of a data processing method according to an exemplary embodiment. The device storing the program for executing the target job is called the "data development center", and the device storing the program for quality monitoring of the job output data is called the "data quality center". This solution can be a method executed by the data quality center. The device storing the data warehouse of metadata is called the "metadata warehouse". Figure 2 As shown, during the execution of the current node of the target job, the data development center writes the data produced by the target job into HDFS and stores it in the table corresponding to the target job. The metadata warehouse obtains the HDFS audit log, generates the first progress information indicating the progress of the table partition according to the HDFS audit log, and generates the mapping relationship between the job and the table according to the HDFS audit log. The data quality center determines whether the table partition corresponding to the current node of the job has been established by querying the first progress information. The data quality center can query the first progress information by querying the message queue Talos Topic, for example.
[0078] In this embodiment, the first progress information is directly obtained, so as to quickly and accurately determine whether the table partition corresponding to the current node of the target job is established.
[0079] In yet another embodiment, the above method further includes: acquiring a mapping relationship between the job and the table.
[0080] Figure 2 In the process, after the metadata warehouse determines the mapping relationship between the job and the table based on the HDFS audit log, the data quality center can obtain the mapping relationship between the job and the table from the metadata warehouse through an online analytical processing (OLAP) query, for example, as the data basis for the quality monitoring of the next node of the job. When monitoring the quality of the next job node, the table corresponding to the job can be determined based on the mapping relationship between the table and the job obtained at the historical node.
[0081] In this embodiment, the mapping relationship between jobs and tables is directly obtained from the outside to perform data quality monitoring, which saves the processing of data analysis and saves storage space and computing power.
[0082] In yet another embodiment, the above-mentioned determining whether the current node of the target job has been executed completely includes:
[0083] Obtaining second progress information, where the second progress information is used to indicate whether the current node of the target job has been executed;
[0084] According to the second progress information, it is determined whether the current node of the target job is executed completely.
[0085] like Figure 2 As shown, during the execution of the current node of the target job, the data development center can generate second progress information for indicating whether the current node of the target job has been executed. The data quality center can determine whether the current node of the target job has been executed by querying the second progress information. The data quality center can query the second progress information, for example, by querying the Rocket message queue (Message Queue, MQ), and the second progress information is real-time information, and the second progress information can be an identifier.
[0086] In related technologies, the judgment of data output is highly coupled to the data development center, so that independent functions such as data exploration and data quality are also strongly coupled to the data development center. If the data development center has a generation that is not forward compatible, this highly coupled judgment method will result in a large workload for data migration.
[0087] In this solution, the data quality center combines the second progress information with the first progress information to determine whether the data has been produced. The first progress information indicating whether the table partition has been established and the mapping relationship between the job and the table are all queried by the data quality center from the metadata warehouse. On this basis, the data quality center only needs to combine the second progress information queried from the data development center to indicate whether the job node has been executed to complete the judgment of whether the data has been produced. The judgment of data output is less coupled with the data development center.
[0088] The data used for quality monitoring can be obtained from HDFS by the data quality center with the help of search engine tools. In one embodiment, a target job is developed in the data development center, and the target job is set to be executed at 9 o'clock every day. After the target job is executed, the data of the corresponding table in HDFS is updated. Then after the job execution is completed at 9 o'clock every day, the data development center writes the output data into the corresponding table in HDFS and generates a second progress information. A table partition corresponding to the current node is established inside the corresponding table in HDFS. At 9:05, the data quality center determines that the node execution is completed by querying the second progress information, and determines the table corresponding to the job through the mapping relationship between the job and the table. Then, by paying attention to whether the table partition is established, it can be determined whether the target job has data output. At 9:06, the data quality center determines that the table partition is established by querying the first progress information, and can determine that the target job has data output.
[0089] After determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, the quality of the data output by the target job can be monitored. The data quality monitoring can be performed immediately after determining the monitoring strategy corresponding to the table of the target job, or after determining the monitoring strategy corresponding to the table of the target job for a predetermined period of time.
[0090] Based on the same inventive concept, the present disclosure also provides a data processing device, Figure 3 FIG. 1 is a block diagram of a data processing device according to an exemplary embodiment. Figure 3 As shown, the data processing device 300 includes an identification module 301 , a first determination module 302 and a monitoring module 303 .
[0091] The identification module 301 is configured to identify whether the target job has data output.
[0092] The first determination module 302 is configured to determine the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy in response to identifying that the target job has data output, wherein the data output by the target job is stored in the table of the target job.
[0093] The monitoring module 303 is configured to perform quality monitoring on the data output by the target job according to the determined monitoring strategy.
[0094] Optionally, the identification module 301 includes a first determination submodule, a second determination submodule, a third determination submodule and a fourth determination submodule.
[0095] The first determination submodule is configured to determine whether the current node of the target job has been executed completely.
[0096] The second determining submodule is configured to determine the table of the target job according to the mapping relationship between the job and the table.
[0097] The third determination submodule is configured to determine whether the table partition corresponding to the current node of the target job is established.
[0098] The fourth determination submodule is configured to determine that the target job has data output when the current node of the target job has been executed and the table partition corresponding to the current node of the target job has been established.
[0099] Optionally, the third determining submodule is further configured to:
[0100] According to the audit log in the HDFS, it is determined whether the table partition corresponding to the current node of the target job is established, and the table of the target job is stored in the HDFS.
[0101] Optionally, the data processing device 300 further includes a second determining module.
[0102] The second determination module is configured to determine a mapping relationship between a job and a table according to an audit log in the HDFS, where the table of the target job is stored.
[0103] Optionally, the third determining submodule is further configured to:
[0104] Obtaining first progress information, where the first progress information is used to indicate whether a table partition corresponding to a current node of the target job has been established, and the first progress information is information generated according to an audit log in the HDFS;
[0105] According to the first progress information, it is determined whether the table partition corresponding to the current node of the target job has been established.
[0106] Optionally, the data processing device 300 further includes an acquisition module.
[0107] The acquisition module is configured to acquire a mapping relationship between a job and a table.
[0108] Optionally, the first determining submodule is further configured to:
[0109] Obtaining second progress information, where the second progress information is used to indicate whether the current node of the target job has been executed;
[0110] According to the second progress information, it is determined whether the current node of the target job is executed completely.
[0111] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0112] Through the above technical solution, in the related technology, when the quality of the data output by the job is monitored, the association between the job and the monitoring strategy is configured. In the workflow of the job, when a node is completed, the monitoring strategy adopted for the job is determined according to the above association. If the developed job is updated, the association between the job and the monitoring strategy needs to be reconfigured. In this solution, the association between the table and the monitoring strategy is configured, that is, a table is added between the association between the job and the monitoring strategy, with the table as the medium. Since the structural characteristics of the table corresponding to the job are usually unchanged, when performing data quality monitoring, even if the job is updated (such as version iteration), there is no need to reconfigure the association between the table and the monitoring strategy. Therefore, when performing data quality monitoring, this solution reduces the workload of configuration and improves the efficiency of quality monitoring of big data. In addition, since the object associated with the monitoring strategy is converted from the job to the table, the operator who performs data quality monitoring does not need to configure the association between the job and the monitoring strategy multiple times, and the coupling degree between data quality monitoring and data development is low.
[0113] The present disclosure also provides a data processing device, comprising:
[0114] A second processor;
[0115] a second memory for storing second processor executable instructions;
[0116] Wherein, the second processor is configured as:
[0117] Identify whether the target job has data output;
[0118] In response to identifying that the target job has data output, determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job;
[0119] According to the determined monitoring strategy, the quality of the data produced by the target operation is monitored.
[0120] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of the data processing method provided by the present disclosure when the program instructions are executed by a first processor.
[0121] Figure 4 800 is a block diagram of a device for data processing according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0122] Reference Figure 4, the device 800 may include one or more of the following components: a processing component 802 , a second memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .
[0123] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more second processors 820 to execute instructions to complete all or part of the steps of the above-mentioned data processing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0124] The second memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The second memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0125] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0126] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0127] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the second memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0128] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0129] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0130] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0131] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned data processing method.
[0132] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a second memory 804 including instructions, and the instructions can be executed by the second processor 820 of the device 800 to complete the above data processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0133] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned data processing method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip including a processor, a memory, and an interface for communicating with other devices. The executable instruction can be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned data processing method is implemented; alternatively, the integrated circuit or chip can receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned data processing method.
[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data processing method, characterized in that: The method comprises: Identify whether the target job has data output; In response to identifying that the target job has data output, determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job; According to the determined monitoring strategy, the quality of the data output by the target operation is monitored.
2. The method according to claim 1, characterized in that The step of identifying whether the target job has data output includes: Determine whether the current node of the target job has been executed; Determine the table of the target job according to the mapping relationship between the job and the table; Determine whether the table partition corresponding to the current node of the target job is established; When the current node of the target job has been executed and the table partition corresponding to the current node of the target job has been established, it is determined that the target job has data output.
3. The method according to claim 2, characterized in that The determining whether the table partition corresponding to the current node of the target job is established includes: According to the audit log in the HDFS, it is determined whether the table partition corresponding to the current node of the target job is established, and the table of the target job is stored in the HDFS.
4. The method according to claim 2, characterized in that: The method further comprises: The mapping relationship between the job and the table is determined according to the audit log in the HDFS, and the table of the target job is stored in the HDFS.
5. The method according to claim 3, characterized in that: The step of determining, according to the audit log in the HDFS, whether the table partition corresponding to the current node of the target job is established completely includes: Obtaining first progress information, where the first progress information is used to indicate whether a table partition corresponding to a current node of the target job has been established, and the first progress information is information generated according to an audit log in the HDFS; According to the first progress information, it is determined whether the table partition corresponding to the current node of the target job has been established.
6. The method according to claim 2, characterized in that The method further comprises: Get the mapping relationship between jobs and tables.
7. The method according to claim 2, characterized in that The determining whether the current node of the target job has been executed completely includes: Acquire second progress information, where the second progress information is used to indicate whether the current node of the target job has been executed; Determine whether the current node of the target job has been executed completely according to the second progress information.
8. A data processing device, characterized in that: The device comprises: An identification module is configured to identify whether a target job has data output; A first determination module is configured to, in response to identifying that the target job has data output, determine the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job; The monitoring module is configured to perform quality monitoring on the data output by the target job according to the determined monitoring strategy.
9. A data processing device, characterized in that: include: A second processor; a second memory for storing second processor executable instructions; Wherein, the second processor is configured as: Identify whether the target job has data output; In response to identifying that the target job has data output, determining the monitoring strategy corresponding to the table of the target job according to the configuration relationship between the table and the monitoring strategy, wherein the data output by the target job is stored in the table of the target job; According to the determined monitoring strategy, the quality of the data output by the target operation is monitored.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by the first processor, the steps of the method described in any one of claims 1 to 7 are implemented.