Data processing methods, apparatus, systems, storage media, devices and software products

Abstract

This application provides a data processing method, apparatus, system, storage medium, device, and program product. The data processing method includes: reading and determining a first reading duration for reading data from a cached data area; if a target reading duration exceeding a preset reading duration threshold exists within the first reading duration, performing at least one regionalization on the target cached data area corresponding to the target reading duration to obtain multiple sub-cached data areas; reading and uploading data from the sub-cached data areas, as well as data from other cached data areas besides the target cached data area, to a quality analysis device. In this application's data processing method, the selective regionalization based on whether the first reading duration exceeds a preset reading duration threshold, and the simultaneous reading of data from multiple sub-cached data areas obtained from the regionalization, effectively improves the efficiency of data acquisition, thereby enhancing the timeliness of data quality analysis and the referenceability of the quality analysis results.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, system, storage medium, device, and program product. Background Technology

[0002] Data quality monitoring is an important part of the data ecosystem, and data processing is crucial. Currently, the market mainly achieves data processing by traversing and scanning to complete data collection.

[0003] However, when the amount of data being monitored reaches a certain level, the data collection process becomes too time-consuming, which greatly reduces the timeliness of data quality monitoring and the reliability of the monitoring results.

[0004] Therefore, how to improve the efficiency of data processing in data quality monitoring has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a data processing method, apparatus, system, storage medium, device, and program product that selectively divides a cached data area into multiple sub-cache data areas, and then collects data from the cached data area or simultaneously collects data from the sub-cache data areas for subsequent quality analysis, thereby increasing the efficiency of data collection and effectively improving the timeliness of data quality monitoring.

[0006] In a first aspect, embodiments of this application provide a data processing method, which includes: reading data from at least one cached data area and determining each first reading duration for reading data from each cached data area; if, within the at least one first reading duration, there exists a target reading duration exceeding a preset reading duration threshold, performing at least one regionalization on the target cached data area corresponding to the target reading duration to obtain at least two sub-cached data areas; reading data from the at least two sub-cached data areas and uploading the data from the at least two sub-cached data areas, as well as the data from other cached data areas in at least one cached data area besides the target cached data area, to a quality analysis device, such that the quality analysis device performs data quality analysis on the uploaded data to obtain a quality analysis result.

[0007] Secondly, embodiments of this application also provide a data processing apparatus, which includes an acquisition unit, a partitioning unit, and an upload unit. The acquisition unit is configured to acquire data from at least one cached data area and determine each first read duration for reading data from each cached data area. The partitioning unit is configured to, if a target read duration exceeding a preset read duration threshold exists within at least one first read duration, perform at least one regionalization partitioning on the target cached data area corresponding to the target read duration to obtain at least two sub-cached data areas. The upload unit is configured to read data from at least two sub-cached data areas and upload the data from at least two sub-cached data areas, as well as data from other cached data areas besides the target cached data area, to a quality analysis device, so that the quality analysis device performs data quality analysis on the uploaded data to obtain quality analysis results.

[0008] Thirdly, embodiments of this application also provide a data processing system, which includes a quality analysis device and the aforementioned data processing device. The quality analysis device is used to receive data uploaded by the data processing device and perform data quality analysis on the uploaded data to obtain quality analysis results.

[0009] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned data processing method.

[0010] Fifthly, embodiments of this application also provide an apparatus including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned data processing method.

[0011] Sixthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the aforementioned data processing method.

[0012] In summary, in this embodiment, data from at least one cached data area is read, and a first reading duration for reading data from the cached data area is determined. If a target reading duration exceeding a preset reading duration threshold exists within the first reading duration, the target cached data area corresponding to the target reading duration is divided into at least two sub-cached data areas. Data in the sub-cached data areas, as well as data in other cached data areas besides the target cached data area, are read and uploaded to the quality analysis device. Based on whether the time required to read data from the cached data area meets the timeliness requirements (i.e., whether the first read time exceeds the preset read time threshold), for cached data areas that meet the timeliness requirements (i.e., the first read time does not exceed the preset read time threshold), their data is directly read and uploaded. For cached data areas that do not meet the timeliness requirements (i.e., the first read time exceeds the preset read time threshold), they are divided into multiple sub-cache data areas, and data from multiple sub-cache data areas is read and uploaded simultaneously. This reduces the excessive time consumption of data collection caused by the large volume of cached data areas, thereby solving the problem of reduced timeliness of data quality analysis, effectively improving the efficiency of data collection, and thus improving the timeliness of data quality analysis and the referenceability of the quality analysis results. Attached Figure Description

[0013] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0014] Figure 2 A flowchart of step S20 of a data processing method provided in an embodiment of this application;

[0015] Figure 3 Another flowchart of a data processing method provided in an embodiment of this application;

[0016] Figure 4 Another flowchart of a data processing method provided in an embodiment of this application;

[0017] Figure 5 Another flowchart of step S20 of a data processing method provided in an embodiment of this application;

[0018] Figure 6 A structural diagram of a data processing apparatus provided in an embodiment of this application;

[0019] Figure 7 A structural diagram of a data processing system provided in an embodiment of this application;

[0020] Figure 8 Another structural diagram of the data processing system provided in the embodiments of this application;

[0021] Figure 9 This is a flowchart illustrating the data processing method according to an embodiment of this application. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0023] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0024] Given that data quality monitoring is a crucial part of the data ecosystem, it serves both as a service quality early warning system and a reliable data support for subsequent optimization and iteration in actual project operation and maintenance. Data processing is particularly critical, and current market practices primarily involve data collection through traversal and scanning. However, when the volume of data for data quality monitoring reaches a certain level, the data collection process becomes excessively time-consuming, significantly reducing the timeliness and reliability of data quality monitoring results. Therefore, improving the efficiency of data processing in data quality monitoring has become an urgent problem to be solved.

[0025] To address the aforementioned issues, this application proposes a data processing method for data quality analysis and monitoring. This method selectively divides the cached data area into multiple sub-cache data areas based on the data scale of the data quality analysis, and then directly collects data from the cached data areas; or it simultaneously collects data from multiple sub-cache data areas for data quality analysis, effectively improving data processing efficiency and thus enhancing the timeliness of data quality monitoring and the reliability of data quality analysis results.

[0026] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the data processing method includes at least steps S10 to S30.

[0027] Step S10: Read data from at least one cached data area and determine the first read duration for each cached data area.

[0028] Here, at least one cached data area contains data from one or more different data sources. The data from these different data sources represents data of different storage types, i.e., heterogeneous multi-source data. The cached data area refers to a temporary storage area for the data.

[0029] In some embodiments, upon receiving a quality analysis task, data is read from the corresponding cached data area as indicated by the quality analysis task. This can be done by reading data from multiple cached data areas sequentially or simultaneously; that is, at least one cached data area is read.

[0030] For example, data is collected from a data source and stored in 10 cached data areas, denoted as the first to the tenth cached data areas. Upon receiving a quality analysis task, assuming the cached data areas indicated by the task are the fourth, fifth, and seventh cached data areas, data can be read sequentially from the fourth, fifth, and seventh cached data areas in any order, with the first read duration determined for each area; alternatively, data from two cached data areas can be read simultaneously, followed by the third; or, data from all three cached data areas can be read simultaneously.

[0031] In some embodiments, determining each first read duration for reading data from each cached data area in step S10 may include: reading data from the cached data area and obtaining the amount of data read over a predetermined duration. The first read duration is determined based on the amount of data read over the predetermined duration and the amount of data in the cached data area. It is understood that the amount of data in each cached data area is predetermined. The predetermined duration may be a unit duration or other durations, determined according to specific circumstances, and this application embodiment does not impose any limitations on this.

[0032] For example, if a quality analysis task indicates that a certain cached data area stores 2560 bytes of data, and the predetermined duration is 1 second, the quality analysis task is received, the data in the cached data area is read, and when the data reading time reaches 1 second, the amount of data read in 1 second is obtained. If the amount of data read in 1 second is 256 bytes, then the first reading time for reading the data in the cached data area is 2560 divided by 256, which equals 10 seconds.

[0033] In some embodiments, the data source may refer to a data source related to the quality of high-precision positioning services, such as service call volume or differential data, or other data sources. This application embodiment does not impose specific limitations.

[0034] Step S20: If there is a target read duration exceeding a preset read duration threshold in at least one first read duration, perform at least one regionalization on the target cache data area corresponding to the target read duration to obtain at least two sub-cache data areas.

[0035] Here, the preset read duration threshold is set based on the timeliness requirements of the quality analysis task. A first read duration exceeding the preset read duration threshold indicates that the time required to directly read the data in the cached data area corresponding to that first read duration cannot meet the timeliness requirements of data quality analysis.

[0036] If the first read duration does not exceed the first read duration threshold, the data in the cached data area corresponding to the first read duration is read directly.

[0037] The target cached data area refers to the cached data area where the time required to read the data cannot meet the timeliness requirements of data quality analysis. In other words, the first read time for data in the target cached data area exceeds a preset read time threshold.

[0038] The target cache data area can be divided into equal or unequal parts. For example, if the target cache data area contains 1024 bytes of data, it can be divided into four equal parts, each containing 256 bytes; or it can be divided into four unequal parts, with each part containing 128 bytes, 256 bytes, 64 bytes, and 576 bytes respectively. It is understood that the number of parts the target cache data area is divided into, and whether it is divided equally, can be determined based on actual timeliness requirements and other factors.

[0039] Step S30: Read the data from at least two sub-cached data areas, and upload the data from at least two sub-cached data areas, as well as the data from at least one cached data area other than the target cached data area, to the quality analysis device, so that the quality analysis device performs data quality analysis on the uploaded data to obtain quality analysis results.

[0040] Here, "other cached data area" refers to the cached data area where the first read time for reading data from other cached data area is less than or equal to a preset read time threshold.

[0041] In some embodiments, the quality analysis device receives uploaded data and performs quality analysis on the data. The quality analysis device compares each received data point with a reasonable range for that data. If a data point is outside the reasonable range, it is considered abnormal data; if a data point is within the reasonable range, it is considered normal data. The reasonable range for the data can be determined by the data processing system based on previously received data. Alternatively, the quality analysis device calculates the difference between every two received data points, obtaining multiple difference values. It then compares these differences with a reasonable range of difference values. If a difference value is outside the reasonable range, the corresponding data is considered abnormal data; if all differences for a data point are within the reasonable range, the data is considered normal data. The reasonable range of difference values ​​is determined based on the obtained multiple difference values. For example, if the multiple difference values ​​include 5, 6, 8, and 20, then a difference value of 20 is considered to be outside the reasonable range.

[0042] The data quality analysis results include whether the data is normal or abnormal, as well as the location and severity of the abnormal data. The severity of abnormality refers to the degree to which the abnormal data deviates from the reasonable range of the data. It can be represented by the smaller absolute value of the difference between the abnormal data and the two extreme values ​​of the reasonable range, or by the average of the absolute values ​​of multiple differences corresponding to the abnormal data. For example, if the abnormal data value is 25, and the reasonable range of the data is 18 to 20, then the severity of abnormality is 5. If the differences corresponding to the abnormal data include 4, 5, 8, and 7, then the severity of abnormality is the average of 4, 5, 8, and 7, which is 6.

[0043] In this embodiment of the application, in steps S10 to S30, the time taken to read data from the cached data area is determined based on the relationship between the first reading time and the preset reading time to determine whether the time taken to read the data in the cached data area can meet the timeliness requirements of the quality analysis. Then, the cached data area whose time cannot meet the timeliness requirements is divided into multiple sub-cached data areas. Data from multiple sub-cached data areas is read simultaneously, which reduces the excessive time consumption of data collection caused by the large amount of data. This solves the problem of reducing the timeliness of data quality analysis, effectively improves the efficiency of data collection, and thus improves the timeliness of data quality analysis and the referenceability of the quality analysis results.

[0044] Figure 2 This is a flowchart of step S20 of a data processing method provided in an embodiment of this application. Figure 2 As shown, in some embodiments, step S20 further includes the following steps:

[0045] Step S21: Divide the target cache data area into regions to obtain at least two sub-target cache areas.

[0046] The number of sub-target cache areas obtained by dividing the target cache data area can be 2, 3, 5, 6, 7, or other values, and this application embodiment does not impose specific limitations on this. Furthermore, dividing the target cache data area into sub-target cache areas can be done in equal or unequal portions, and this application embodiment does not impose specific limitations on this. The specific details of the regional division of the target cache data area are described in the aforementioned step S20 and will not be repeated here.

[0047] Step S22: Determine the second read duration for reading data from each sub-target buffer.

[0048] The method for determining the second read duration is similar to that for determining the first read duration; please refer to the aforementioned embodiments for details, which will not be repeated here.

[0049] Step S23: If the second read duration exceeds the preset read duration threshold, perform at least one region division on the sub-target cache area corresponding to the second read duration exceeding the preset read duration threshold to obtain at least two sub-cache data areas; wherein, the time for reading data from each sub-cache data area is less than or equal to the preset read duration threshold, and / or the number of sub-cache data areas meets the preset conditions.

[0050] In some embodiments, step S23 is a loop step, which involves: dividing the sub-target buffer into regions, reading the data from the sub-target buffer obtained by the region division, and determining the reading duration.

[0051] If the number of sub-target cache regions obtained does not meet the preset conditions and the read time exceeds the preset read time threshold, the loop steps continue to be repeated to divide the obtained sub-target cache regions into regions.

[0052] If the number of sub-target cache areas obtained meets the preset conditions and / or the read time is less than or equal to the preset read time threshold, the repeated loop steps are stopped, and the obtained sub-target cache areas are used as sub-cache data areas.

[0053] In some embodiments, a timing period is set, and when the timing period is reached, it is determined whether the data of the sub-target cache area can be read within a preset reading time threshold. If "can", it indicates that the second reading time has not exceeded the preset reading time threshold. In this case, the sub-target cache area is used as a sub-cache data area, the data of the sub-cache data area is read, and the data of the sub-cache data area is uploaded to the quality analysis device along with the data of other cache data areas. If "cannot", it indicates that the second reading time has exceeded the preset reading time threshold.

[0054] Here, the preset condition refers to the number of sub-cached data areas being sufficient to allow the data processing system to read data simultaneously. This can be determined based on the actual situation, and this application embodiment does not impose specific restrictions on it.

[0055] In this embodiment of the application, in steps S21 to S23, when the time taken to read data from each sub-cache data area is less than or equal to a preset reading time threshold or the number of sub-cache data areas meets a preset condition, that is, when the reading capacity of the data processing system is maximized, the division of the cache data area is stopped. This further refines the number of times the cache data area is divided, making the effect of improving the efficiency of data reading even better. It reduces the excessive time consumption of data collection caused by the large amount of data, thereby solving the problem of reducing the timeliness of data quality analysis, effectively improving the efficiency of data collection, and thus improving the timeliness of data quality analysis and the referenceability of the quality analysis results.

[0056] Figure 3 This is another flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 As shown, in some embodiments, the data processing method further includes the following steps:

[0057] Step S24: Determine the sum of the number of sub-cache data areas and the number of other cache data areas; the number of sub-target cache areas meets the preset condition that the sum is greater than or equal to the maximum number of threads for parallel data reading in the data processing system.

[0058] Here, data from multiple sub-cache data areas and other cache data areas can be read simultaneously. The maximum number of threads that can read data in parallel by the data processing system is the maximum value of the sum.

[0059] It should be noted that if the total number of threads exceeds the maximum number of threads, the data will be read in at least two steps and then uploaded to the quality analysis device.

[0060] For example, with a maximum of 20 threads and 5 other cache areas, if the number of sub-cache data areas is 14, the data processing system will simultaneously read data from 5 other cache areas and 14 sub-cache data areas. If the number of sub-cache data areas is 18, the data processing system can first simultaneously read data from 5 other cache areas and 15 sub-cache data areas; after reading all of them, it will then read data from the remaining 3 sub-cache data areas.

[0061] It is understandable that a cache area is set up to temporarily store the read data. After all the data in the cache data area corresponding to the quality analysis task has been read, it will be uploaded to the quality analysis device.

[0062] For example, the maximum number of threads in the data processing system is 20, the number of other cache areas is 5, and the number of sub-cache data areas is 18. The data processing system can first read the data from 5 other cache areas and 15 sub-cache data areas simultaneously, and store the data from the 5 other cache areas and 15 sub-cache data areas in the cache area. Then, it can continue to read the data from the remaining 3 sub-cache data areas. After the data from the remaining 3 sub-cache data areas has been read, the data from the 3 sub-cache data areas, along with the data from the 5 other cache areas and 15 sub-cache data areas stored in the cache area, are uploaded to the quality analysis device.

[0063] In this embodiment of the application, in step S24, by determining the sum of the number of sub-cache data areas and the number of other cache data areas, and ensuring that the sum is less than or slightly greater than the maximum number of threads in the data processing system, the problem of data needing to be read in batches due to the excessive number of sub-cache data areas is reduced, thereby affecting the reading efficiency. This effectively improves the efficiency of data acquisition, thereby improving the timeliness of data quality analysis and the referenceability of the quality analysis results.

[0064] Figure 4 This is another flowchart illustrating a data processing method provided in an embodiment of this application. Figure 4 As shown, in some embodiments, before step S10, the data processing method further includes the following steps:

[0065] Step S05: Receive the quality analysis task, which carries the data identifier of the cached data area to be analyzed.

[0066] Here, quality analysis tasks are received periodically.

[0067] The cached data area to be analyzed refers to the cached data area corresponding to the quality analysis task. Each cached data area has a data identifier, which can be used to distinguish each cached data area.

[0068] Step S10 includes step S10a, which is: reading the data in the cached data area corresponding to the data identifier.

[0069] The data identifier here is used to identify the cached data area corresponding to the quality analysis task; that is, different cached data areas have different data identifiers.

[0070] In some examples, the target cache data area is divided into sub-target cache areas X1, X2...X n Read subtarget buffer X j The data is from subtarget cache X. j If the second read duration exceeds the preset read duration threshold, then the sub-target cache X will be... jPerform at least one region partitioning. Here, j is a positive integer not greater than n.

[0071] Figure 5 Another flowchart of step S20 of a data processing method provided in an embodiment of this application. For example... Figure 5 As shown, in some embodiments, step S20 includes the following steps:

[0072] Step S20a: Determine the amount of data in the cached data area based on the data identifier.

[0073] Here, the number of cached data areas is known, and the amount of data in the corresponding cached data area can be identified based on the data identifier.

[0074] For example, if a data processing system has five cached data areas, the amount of data in each cached data area is known, and the cached data areas are identified as 1-1, 1-2, 1-3, 1-4, and 1-5 respectively, when a quality analysis task is received, and the data identification of the quality analysis task is 1-4, then the corresponding cached data area is identified based on the data identification 1-4, and the amount of data in that cached data area is determined.

[0075] Step S20b: Based on the amount of data in the cached data area, the target cached data area is divided into at least two sub-cache data areas by regionalization at least once.

[0076] Here, the partitioning scheme is determined based on the amount of data, that is, the number of sub-cached data areas to be divided into and whether to divide them equally.

[0077] For example, if the target cache data area has 1280 bytes of data, it can be determined that the first time it is divided into 4 regions, and then divided equally, resulting in four sub-target cache areas, each with 320 bytes of data. If the target cache data area has 2560 bytes of data, it can be determined that the first time it is divided into 8 regions, and then divided equally, resulting in eight sub-target cache areas, each with 320 bytes of data.

[0078] In some embodiments, the quality analysis results include data quality indicators, which are used by the quality analysis device to generate a verification file, and the verification file is used to accept the data quality indicators.

[0079] Here, the quality analysis result refers to querying the uploaded data and comparing it with the normal range to determine whether the data is normal or abnormal. The normal range can be determined based on previous data from the data processing system or by comparing relationships between data points. Furthermore, the data quality content included in the normal or abnormal data assessment is the data quality indicator. For example, a data quality indicator might be the difference between the data and the normal value, or the difference between two data points.

[0080] The quality analysis device generates a verification file based on data quality indicators. The verification file is used to locate abnormal data and transmit the abnormal data to other downstream modules or data quality monitoring terminals.

[0081] In some embodiments, the verification file includes a data quality name, data for quality analysis, and quality analysis results. Based on the verification file, abnormal data can be located and the authenticity of the data quality indicators for the abnormal data can be verified.

[0082] Figure 6 A structural diagram of a data processing apparatus 20 provided in an embodiment of this application is shown below. Figure 6 As shown, the data processing device includes an acquisition unit 21, a partitioning unit 22, and an upload unit 23. The acquisition unit 21 is used to acquire data from at least one cached data area and determine each first read duration for reading data from each cached data area. The partitioning unit 22 is used to perform at least one regionalization of the target cached data area corresponding to the target read duration, obtaining at least two sub-cached data areas, if a target read duration exceeding a preset read duration threshold exists within at least one first read duration. The upload unit 23 is used to read data from at least two sub-cached data areas and upload the data from at least two sub-cached data areas, as well as data from other cached data areas besides the target cached data area, to a quality analysis device, enabling the quality analysis device to perform data quality analysis on the uploaded data and obtain quality analysis results.

[0083] Figure 7 This is a structural diagram of a data processing system provided in an embodiment of this application. Figure 7 As shown, the data processing system 100 includes the aforementioned data processing device 20 and a quality analysis device 30, wherein,

[0084] The quality analysis device 30 receives data uploaded by the data processing device, performs data quality analysis on the uploaded data, and obtains the quality analysis results.

[0085] In some embodiments, the quality analysis device 30 performs data quality analysis according to a preset set of rules, calculates data quality-related indicators, and generates corresponding verification files. The verification files are used to verify the reliability of the data quality.

[0086] In some embodiments, the partitioning unit 22 is further configured to regionalize the target cache data area to obtain at least two sub-target cache areas, determine a second read duration for reading data from each sub-target cache area, and if the second read duration exceeds a preset read duration threshold, perform at least one regional partitioning on the sub-target cache area corresponding to the second read duration exceeding the preset read duration threshold to obtain at least two sub-cache data areas, wherein the read duration for reading data from each sub-cache data area is less than or equal to the preset read duration threshold, and / or the number of sub-cache data areas meets a preset condition.

[0087] In some embodiments, the partitioning unit 22 is further configured to determine the sum of the number of sub-cache data areas and the number of other cache data areas, wherein the number of sub-target cache areas satisfies a preset condition by the sum being greater than or equal to the maximum number of threads in the data processing system that can read data in parallel.

[0088] Figure 8 This is another structural diagram of the data processing system provided in an embodiment of this application. For example... Figure 8 As shown, in some embodiments, the data processing system 300 further includes a data acquisition device 40, wherein the data acquisition device 40 is used to acquire data from a data source and store it in at least one cached data area.

[0089] The cached data area can be located in the data acquisition device 40, or in other areas outside the data acquisition device 40 within the data processing system 300.

[0090] In some embodiments, the data processing system 300 further includes a scheduling device 50. The data processing device 20 receives a quality analysis task sent by the scheduling device 50, the quality analysis task carrying a data identifier of the cached data area to be analyzed. Then, the acquisition unit 21 reads the data of the cached data area corresponding to the data identifier.

[0091] In some embodiments, the partitioning unit 22 is further configured to determine the amount of data in the cached data area based on the data identifier, and to perform at least one regionalization of the target cached data area based on the amount of data in the cached data area to obtain at least two sub-cache data areas.

[0092] In some embodiments, the data processing system 300 further includes an archiving device 60, wherein the quality analysis device 30 generates a verification file based on data quality indicators and transmits the verification file to the archiving device 60, wherein the data quality indicators are included in the quality analysis results; the archiving device 60 archives the verification file after acceptance.

[0093] The description of the system embodiments above is similar to that of the method embodiments above, and has similar beneficial effects. For technical details not disclosed in the system embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0094] Figure 9 This is a flowchart illustrating the data processing method according to an embodiment of this application. Figure 9 As shown, the method includes the following steps S100 to S600.

[0095] Step S100: The first-level acquisition module (which can be understood as the data acquisition device mentioned above) reads heterogeneous multi-source data (which can be understood as the data mentioned above), partitions and stores it in the cache, and sets a unique identifier (which can be understood as the identifier mentioned above) for each partition (which can be understood as the cache data area mentioned above).

[0096] Step S200: The scheduling module (which can be understood as the scheduling device mentioned above) initiates a data quality monitoring task.

[0097] Here, the scheduling module periodically triggers the secondary acquisition module (which can be understood as the data processing device mentioned earlier) to read the data from the partition. The data quality monitoring task carries the data identifier of the partition to be analyzed.

[0098] Step S300: The secondary acquisition module sets the data fragmentation strategy, divides the data into segments according to the data fragmentation strategy, and reads the data and uploads it to the quality analysis module (which can be understood as the quality analysis device mentioned above).

[0099] Here, "partition" refers to the partition to be analyzed in the data quality monitoring task. There can be one partition or multiple partitions to be analyzed.

[0100] In this embodiment of the application, step S300 includes at least steps S301 to S307.

[0101] Step S301: The secondary acquisition module determines the amount of data in the partition based on the partition's unique identifier.

[0102] Step S302: The secondary acquisition module sets the reading range to a partition and determines the first reading duration for reading partition data.

[0103] Step S303: Determine whether the first reading time exceeds the preset reading time threshold.

[0104] In this embodiment of the application, if the first reading time does not exceed the preset reading time threshold, the process waits to execute step S400; if the first reading time exceeds the preset first reading time threshold, the process executes step S305.

[0105] Step S305: Divide the partition into data shards, with the number of shards being n. The secondary acquisition module updates the reading range to the shards (which can be understood as the sub-cache data area mentioned above), reads the data from the shards, and determines the second reading duration for reading the shard data.

[0106] In the embodiments of this application, n is a positive integer greater than or equal to 2.

[0107] Step S306: Determine whether the second reading time exceeds the preset reading time threshold.

[0108] In this embodiment of the application, if the second reading time exceeds the preset reading time threshold, step S307 is executed; if the second reading time does not exceed the preset reading time threshold, step S400 is waited to be executed.

[0109] Step S307: Determine whether the number of shards has reached the upper limit (which can be understood as the preset condition mentioned above).

[0110] In this embodiment of the application, if the number of fragments has not reached the upper limit, the fragment is treated as a partition and step S305 is executed; if the number of fragments has reached the upper limit, the process waits to execute step S400.

[0111] In step S400, the secondary acquisition module reads the fragmented data and uploads the fragmented data and the data from the unfragmented partitions to the quality analysis module.

[0112] In this embodiment of the application, if all partitions to be analyzed corresponding to the data quality detection task meet the conditions, step S400 is executed, wherein the conditions are: the time taken to read data from the partition or shard is less than a preset reading time threshold; or the number of shards reaches the upper limit.

[0113] Step S500: The quality analysis module performs data analysis on the uploaded data to obtain data quality monitoring results and generates verification credentials (which can be understood as the verification file mentioned above).

[0114] Step S600: The archiving module (which can be understood as the archiving device mentioned above) receives the data quality monitoring results and verification certificates, and archives the verification certificates.

[0115] Based on the same concept, embodiments of this application also propose a storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of a data processing method.

[0116] Based on the same concept, embodiments of this application also propose an apparatus including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a data processing method.

[0117] Based on the same concept, embodiments of this application also propose a computer program product that, when executed by a processor, implements the steps of a data processing method.

[0118] In the embodiments of this application, the specific working principles of the storage medium, device and computer program product can be found in other embodiments, and will not be repeated here.

[0119] It should be noted that the data processing system provided in this application embodiment includes all the logic units, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0120] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0121] It should be noted that the descriptions of the above embodiments of storage media, devices, and computer program products are similar to the descriptions of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the storage media embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0122] In summary, in this embodiment, data stored in at least one cached data area is read, and a first read duration for reading data from the cached data area is determined; if a target read duration exceeding a preset read duration threshold exists within the first read duration, the target cached data area corresponding to the target read duration is regionalized at least once to obtain at least two sub-cached data areas; data in the sub-cached data areas and data in other cached data areas besides the target cached data area are read and uploaded to the quality analysis device. Based on whether the time required to read data from the cached data area meets the timeliness requirements (i.e., whether the first read time exceeds the preset read time threshold), for cached data areas that meet the timeliness requirements (i.e., the first read time does not exceed the preset read time threshold), their data is read directly. For cached data areas that do not meet the timeliness requirements (i.e., the first read time exceeds the preset read time threshold), they are divided into multiple sub-cache data areas, and data from multiple sub-cache data areas is read simultaneously. This reduces the excessive time consumption of data collection caused by the large volume of cached data areas, thereby solving the problem of reduced timeliness of data quality analysis, effectively improving the efficiency of data collection, and thus improving the timeliness of data quality analysis and the referenceability of the quality analysis results.

[0123] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0124] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0126] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0127] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0128] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0129] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0130] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The data processing method includes: Read data from at least one cached data area and determine each first read duration from each of the cached data areas; If at least one of the first reading durations has a target reading duration exceeding a preset reading duration threshold, the target cache data area corresponding to the target reading duration is divided into at least two sub-cache data areas by regionalization at least once. The data in the at least two sub-cache data areas is read, and the data in the at least two sub-cache data areas, as well as the data in other cache data areas in the at least one cache data area except the target cache data area, are uploaded to the quality analysis device, so that the quality analysis device performs data quality analysis on the uploaded data to obtain quality analysis results.

2. The data processing method according to claim 1, characterized in that, The target cache data area corresponding to the target read duration is divided into at least two sub-cache data areas by performing at least one regionalization, including: The target cache data area is divided into regions to obtain at least two sub-target cache areas; Determine a second read duration for reading data from each of the sub-target buffers; If the second read duration exceeds the preset read duration threshold, the sub-target cache area corresponding to the second read duration exceeding the preset read duration threshold is divided into regions at least once to obtain the at least two sub-cache data areas; Wherein, the time taken to read data from each of the sub-cache data areas is less than or equal to the preset read time threshold, and / or the number of the sub-cache data areas meets the preset condition.

3. The data processing method according to claim 2, characterized in that, The method further includes: Determine the sum of the number of the sub-cache data areas and the number of the other cache data areas; The number of sub-target caches satisfying the preset condition means that the total number is greater than or equal to the maximum number of threads for parallel data reading.

4. The data processing method according to claim 1, characterized in that, The method further includes: Receive a quality analysis task, wherein the quality analysis task carries a data identifier of the cached data area to be analyzed; Reading data from at least one cached data area includes: Read the data from the cached data area corresponding to the data identifier.

5. The data processing method according to claim 4, characterized in that, The target cache data area corresponding to the target read duration is divided into at least two sub-cache data areas by performing at least one regionalization, including: The amount of data in the cache data area is determined based on the data identifier; Based on the amount of data in the cached data area, the target cached data area is divided into at least two sub-cache data areas by regionalization.

6. The data processing method according to any one of claims 1 to 5, characterized in that, The quality analysis results include data quality indicators, which are used by the quality analysis device to generate a verification file, and the verification file is used to verify the data quality indicators.

7. A data processing apparatus, characterized in that, The data processing device includes an acquisition unit, a partitioning unit, and an uploading unit, wherein, The acquisition unit is configured to acquire data from at least one cached data area and determine each first read duration for reading data from each of the cached data areas; The partitioning unit is used to perform at least one regionalization of the target cache data area corresponding to the target read time when there is a target read time exceeding a preset read time threshold in at least one of the first read times, to obtain at least two sub-cache data areas. The uploading unit is used to read data from the at least two sub-cached data areas and upload the data from the at least two sub-cached data areas, as well as the data from other cached data areas in the at least one cached data area other than the target cached data area, to the quality analysis device, so that the quality analysis device performs data quality analysis on the uploaded data to obtain quality analysis results.

8. A data processing system, characterized in that, The data processing system includes a quality analysis device and the data processing device as described in claim 7, wherein... The quality analysis device is used to receive data uploaded by the data processing device, perform data quality analysis on the uploaded data, and obtain quality analysis results.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. An apparatus comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images