Data quality inspection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data quality inspection method and device, electronic equipment and a storage medium, and relates to the technical field of data quality intelligent inspection. The method comprises the steps that a headless browser technology, a User-Agent camouflage technology and window simulation are adopted to collect a first screenshot and a second screenshot corresponding to key indexes in a target billboard area of a first terminal and a target billboard area of a second terminal respectively; performing standardization processing and key area enhancement on the first screenshot and the second screenshot; the standardization processing comprises size normalization, color unification and background purification; calling a deep learning model to intelligently identify the processed first screenshot and second screenshot to obtain first terminal data and second terminal data, and comparing to obtain a difference value; and setting a difference threshold value, and automatically triggering an alarm mechanism when the difference value exceeds the difference threshold value. According to the invention, the problems of data inconsistency, difficulty in troubleshooting and lack of real-time monitoring in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent data quality inspection technology, and in particular to a data quality inspection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advent of the big data era, enterprise data volume has exploded, making data visualization a crucial basis for enterprise decision-making. However, data inconsistency is becoming increasingly prominent in the display of data dashboards on primary terminals and mobile devices (APPs), severely impacting data credibility and decision-making efficiency. This problem mainly stems from differences in user groups, usage scenarios, and development teams, leading to discrepancies in data display dimensions, data retrieval logic, and refresh rates between primary and secondary terminals, resulting in inconsistencies in the displayed values ​​of the same data metric across the two platforms.

[0003] Traditional data quality inspection solutions mainly rely on manual rules or API interfaces for data comparison, which has drawbacks such as high implementation costs, limited detection capabilities, high false alarm rates, and lack of real-time monitoring. Specifically, traditional solutions require the development of API interfaces, which is time-consuming and costly; they only support numerical comparisons and cannot perform multi-dimensional detection such as trend and status analysis; they rely on manual rules, resulting in a false alarm rate as high as approximately 15%; and they cannot achieve real-time anomaly detection, leading to a lag in problem discovery.

[0004] Therefore, there is an urgent need for a data quality inspection method that can monitor data consistency in real time, intelligently identify abnormal differences, and trigger alarms in real time. Summary of the Invention

[0005] The present invention provides a data quality inspection method to address the problems of data inconsistency, difficulty in troubleshooting, and lack of real-time monitoring in existing technologies. The technical solution is as follows:

[0006] According to one aspect of the present invention, a data quality inspection method includes: using headless browser technology, User-Agent spoofing technology, and window simulation to collect first and second screenshots corresponding to key indicators in target dashboard areas of a first terminal and a second terminal, respectively; performing standardization processing and key area enhancement on the first and second screenshots; the standardization processing includes size normalization, color unification, and background purification; calling a deep learning model to intelligently identify the processed first and second screenshots to obtain first terminal data and second terminal data, and comparing them to obtain a difference value; setting a difference threshold, and automatically triggering an alarm mechanism when the difference value exceeds the difference threshold; the alarm mechanism includes information push and result feedback.

[0007] In one embodiment, the automatic collection of first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals at a set period using headless browser technology, User-Agent spoofing technology, and window simulation is achieved through the following steps: automatically logging into the backend system of the first terminal and locating the target dashboard area using headless browser technology, and capturing the indicator value area using CSS selectors; the first terminal includes a PC; the headless browser technology includes Selenium and Puppeteer; taking screenshots of the indicator value area at a set time period, saving the screenshots as PNG format and attaching a timestamp to obtain the first screenshot.

[0008] In one embodiment, the automatic collection of first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals according to a set period using headless browser technology, User-Agent spoofing technology, and window simulation technology is achieved through the following steps: automatically logging into the second terminal and locating the target dashboard area using User-Agent spoofing technology and window simulation technology, and capturing the indicator value area through CSS selectors; the second terminal includes an APP; taking screenshots of the indicator value area according to the set time period, saving the screenshots as PNG format and attaching a timestamp to obtain the second screenshot.

[0009] In one embodiment, the standardization and key region enhancement of the first and second screenshots are achieved through the following steps: adjusting the size and resolution of the first and second screenshots to set values; converting the first and second screenshots to RGB three-channel format; and using Gaussian blur to eliminate noise in the first and second screenshots; and using image enhancement techniques to enhance the contrast of key regions in the first and second screenshots; the image enhancement techniques include CLAHE algorithm, AHE algorithm, wavelet transform, and deep learning super-resolution technology.

[0010] In one embodiment, before performing intelligent recognition on the processed first and second screenshots, the method further includes the following steps: unifying the timestamps and indicator standards of the first and second screenshots, and establishing a mapping table; the mapping table is used to unify the data indicators of the first and second terminals.

[0011] In one embodiment, a deep learning model is invoked to intelligently identify the processed first and second screenshots to obtain first terminal data and second terminal data respectively, and the difference value is obtained by comparison. This is achieved through the following steps: the deep learning model is invoked to identify the numerical content in the first and second screenshots based on the mapping table. During the identification process, the positive and negative signs, decimal points, and original units are retained, and the data is output in JSON format to obtain first terminal data and second terminal data respectively; the deep learning model includes the DeepSeekV3 model; the data values ​​of the first terminal data and the second terminal data are compared to obtain the difference value, and the data values ​​with differences and the corresponding data indicators are recorded.

[0012] In one embodiment, the automatic alarm mechanism when the difference value exceeds the difference threshold is implemented through the following steps: when the difference value exceeds the difference threshold, the data value with difference and the corresponding data indicator are pushed to the relevant responsible person, and the first terminal and the second terminal are investigated and corrected according to the data value with difference and the corresponding data indicator.

[0013] According to one aspect of the present invention, a data quality inspection device includes: a data acquisition module, configured to acquire first and second screenshots corresponding to key indicators in target dashboard areas of a first terminal and a second terminal respectively using headless browser technology, User-Agent spoofing technology, and window simulation; a data preprocessing module, configured to perform standardization processing and key area enhancement on the first and second screenshots; the standardization processing includes size normalization, color unification, and background purification; an intelligent comparison module, configured to call a deep learning model to intelligently identify the processed first and second screenshots to obtain first terminal data and second terminal data respectively, and compare them to obtain a difference value; and an anomaly handling module, configured to set a difference threshold, and automatically trigger an alarm mechanism when the difference value exceeds the difference threshold; the alarm mechanism includes information push and result feedback.

[0014] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the data quality inspection method as described above.

[0015] According to one aspect of the present invention, a storage medium has computer-readable instructions stored thereon, which are executed by one or more processors to implement the data quality inspection method as described above.

[0016] The beneficial effects of the technical solution provided by this invention are:

[0017] In the above technical solution, this invention utilizes headless browser technology, User-Agent spoofing technology, and window simulation technology to capture screenshots of the target dashboard area on a first terminal (e.g., a PC) and a second terminal (e.g., an app), ensuring automated and accurate data collection. The captured screenshots undergo standardization and key area enhancement to improve image recognition accuracy. Subsequently, a deep learning model is used to intelligently identify the processed screenshots, obtaining data from the first and second terminals. Comparative analysis is then performed to calculate the difference value. When the difference value exceeds a set threshold, the system automatically triggers an alarm mechanism, pushing the discrepancy data to relevant personnel for investigation and correction. This not only achieves real-time monitoring and intelligent inspection of data quality but also significantly improves data consistency and accuracy across different terminals, reducing the error rate. Therefore, it effectively solves the problems of data inconsistency, difficulty in troubleshooting, and lack of real-time monitoring present in existing technologies. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a data quality inspection method according to an exemplary embodiment;

[0020] Figure 2 This is a flowchart illustrating a data quality inspection method in an exemplary embodiment;

[0021] Figure 3 This is a block diagram of a data quality inspection device according to an exemplary embodiment;

[0022] Figure 4 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0023] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0026] This invention provides a data quality inspection method that achieves real-time comparison and intelligent alarm of multi-terminal data through terminal data screenshots, artificial intelligence image analysis, and data comparison. This effectively solves problems such as data inconsistency, difficulty in troubleshooting, and lack of real-time monitoring in existing technologies. This data quality inspection method is applicable to data quality inspection devices, which can be electronic devices. The data quality inspection method in this invention can be applied to various scenarios, such as data unification on shopping platforms.

[0027] Please see Figure 1 This invention provides a data quality inspection method applicable to electronic devices.

[0028] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0029] like Figure 1 As shown, the method may include the following steps:

[0030] Step 110: Using headless browser technology, User-Agent spoofing technology, and window simulation, first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals are collected respectively.

[0031] In one possible implementation, a headless browser is used to automatically log into the backend system of the first terminal and locate the target dashboard area. The indicator value area is captured using CSS selectors, and screenshots of the indicator value area are taken according to the set time period. The screenshots are saved as PNG format and a timestamp is added to obtain the first screenshot.

[0032] In one possible implementation, User-Agent spoofing and window simulation techniques are used to automatically log in to the second terminal and locate the target dashboard area. The indicator value area is captured through CSS selectors, and screenshots of the indicator value area are taken according to the set time period. The screenshots are saved as PNG format and a timestamp is attached to obtain the second screenshot.

[0033] The first terminal includes PCs and other terminals, the second terminal includes mobile devices such as mobile apps, tablet apps, smartwatch apps, etc., and headless browser technologies include Selenium and Puppeteer, etc., without any specific limitations.

[0034] In the above process, the embodiments of the present invention do not require API interface docking, enabling immediate use and reducing implementation costs. Timed screenshots ensure the real-time nature and consistency of data.

[0035] Step 120: Standardize and enhance key areas of the first and second screenshots. Standardization includes size normalization, color unification, and background purification.

[0036] In one possible implementation, the size and resolution of the first and second screenshots are adjusted to set values, the first and second screenshots are converted to RGB three-channel format, Gaussian blur is used to remove noise from the first and second screenshots, and image enhancement technology is used to enhance the contrast of key areas in the first and second screenshots.

[0037] Image enhancement techniques include CLAHE algorithm, AHE algorithm, wavelet transform and deep learning super-resolution techniques, etc., without being limited here.

[0038] In the above process, the embodiments of the present invention ensure the consistency of image data through standardization processing, improve the accuracy of subsequent recognition, and the key area enhancement technology makes the indicator values ​​clearer and more distinguishable, further improving the recognition effect.

[0039] Step 130: Call the deep learning model to intelligently identify the processed first screenshot and second screenshot to obtain the first terminal data and the second terminal data, and compare them to obtain the difference value.

[0040] In one possible implementation, before intelligently recognizing the first and second screenshots, the timestamps and indicator definitions of the first and second screenshots are unified, and a mapping table is established to unify the data indicators of the first and second terminals.

[0041] In one possible implementation, a deep learning model is invoked to identify the numerical content in the first and second screenshots based on a mapping table. During the identification process, the positive and negative signs, decimal points, and original units are preserved, and the data from the first and second terminals are output in JSON format.

[0042] Furthermore, the data values ​​of the first terminal data and the second terminal data are compared to obtain the difference values, and the data values ​​with differences and the corresponding data indicators are recorded.

[0043] Deep learning models include DeepSeek V3, etc., without being specified here.

[0044] In the above process, the embodiments of the present invention ensure the accuracy and consistency of data comparison through data standardization and mapping. The intelligent recognition capability of the deep learning model greatly improves the accuracy and efficiency of data extraction. Data comparison can promptly detect data differences, providing a basis for subsequent alarm mechanisms.

[0045] Step 140: Set a difference threshold. When the difference value exceeds the difference threshold, an alarm mechanism is automatically triggered.

[0046] The alarm mechanism includes information push and result feedback.

[0047] In one possible implementation, when the difference value exceeds the difference threshold, the data value with the difference and the corresponding data indicator are pushed to the relevant responsible person, and the first terminal and the second terminal are investigated and corrected based on the data value with the difference and the corresponding data indicator.

[0048] In the above process, the embodiment of the present invention sets a difference threshold to ensure the accuracy and timeliness of the alarm mechanism. By automatically triggering the alarm mechanism, the efficiency of problem discovery is improved, the cost of manual investigation is reduced, and the investigation and correction steps ensure the consistency and accuracy of data, thereby improving the enterprise's data management level.

[0049] Through the above process, this invention achieves real-time monitoring and intelligent inspection of data quality across multiple terminals by means of data acquisition, data preprocessing and enhancement, intelligent identification and data comparison, as well as alarm and correction. It has advantages such as no need to connect to API interfaces, low implementation cost, low false alarm rate, support for multi-terminal acquisition and real-time alarm triggering.

[0050] In an exemplary embodiment, ensuring the consistency of sales data displayed on different terminals (PC and APP) is crucial in digital enterprise operations. Taking enterprise sales data consistency monitoring as an application scenario, the detailed process of data quality inspection is demonstrated.

[0051] like Figure 2 As shown, the specific steps may include:

[0052] Step 1: Log in to the PC version of the dashboard page.

[0053] Specifically, headless browser technology (such as Selenium or Puppeteer) is used to automatically log in to the PC backend system, simulating manual operation. The system will enter and submit the login information in the browser based on the preset account and password information, and then enter the sales data dashboard page on the PC.

[0054] In the above process, this embodiment of the invention obtains access permissions to the PC-side sales data dashboard to subsequently capture screenshots of key indicators on the page, ensuring accurate sales data display from the PC. This achieves automatic login without manual intervention, improving data collection efficiency and avoiding errors that may occur with manual login, ensuring login accuracy and stability, and laying the foundation for subsequent data collection work.

[0055] Step 2: Log in to the dashboard page on the APP.

[0056] Specifically, by using User-Agent spoofing and window simulation techniques, a headless browser is used to simulate the access environment of the mobile app. The system modifies the browser's User-Agent identifier to disguise it as an access request from a mobile device, while adjusting the window size to fit the app's page layout. Then, users enter their username and password to log in to the sales data dashboard page on the app.

[0057] In the above process, this embodiment of the invention obtains access permissions to the sales data dashboard on the APP and collects data synchronously with the PC, so as to comprehensively compare the sales data displayed on both ends and discover any possible discrepancies. This overcomes the limitations of data collection on the APP, achieving synchronization and comprehensiveness of data collection between the PC and APP, providing a complete data source for subsequent comparative analysis of the data from both ends, and eliminating the need to develop complex collection interfaces separately for the APP, thus reducing development costs.

[0058] Step 3: Take screenshots of the "Sales Revenue" metric on the PC and the "Sales Revenue" metric on the APP.

[0059] Specifically, on the PC and mobile dashboard pages after successful login, the numerical area containing the "Sales Revenue" metric is precisely located using CSS selectors. Then, at set time intervals (e.g., every 5 minutes), the headless browser's screenshot function is used to capture screenshots of this area, which are saved as PNG format with a timestamp appended.

[0060] In the above process, the embodiments of the present invention obtain intuitive display images of the "sales revenue" indicator on the PC and APP terminals, providing raw materials for subsequent content recognition and data comparison. The addition of timestamps facilitates the tracing and analysis of data over time.

[0061] The timed screenshot feature ensures real-time and consistent data monitoring, enabling timely detection of changes in data display at both ends. PNG format screenshots guarantee image quality, and timestamp recording ensures data traceability, facilitating accurate analysis of the timing and causes of data discrepancies.

[0062] Step 4: Use the DeepSeek V3 model to perform content recognition on the sales screenshots from the PC and the APP.

[0063] Specifically, the collected sales screenshots from the PC and APP are input into a pre-configured DeepSeekV3 model. The model will recognize the numerical content in the screenshots, retaining the plus or minus sign, decimal point, and original units during the recognition process, and output the recognition results in JSON format, including the recognized sales figures and other information.

[0064] In the above process, this embodiment of the invention accurately extracts the numerical information of the "sales revenue" indicator from the screenshot image, transforming the image data into structured data that can be compared and analyzed. Utilizing the powerful recognition capabilities of deep learning models improves the accuracy and efficiency of data extraction, accurately identifying sales revenue values ​​in various formats and styles, reducing the workload and error rate of manual data extraction, and providing an accurate data foundation for subsequent data comparison.

[0065] Step 5: Compare the data values ​​returned by DeepSeek. Compare the data values ​​from the PC and the app, and issue alerts based on the comparison results.

[0066] Specifically, the system automatically compares the sales data values ​​returned by the DeepSeek V3 model from the PC and the APP, calculating the difference between the two. The calculated difference is then compared to a pre-set difference threshold (e.g., a sales difference exceeding 1%). When the difference exceeds the threshold, an alarm mechanism is automatically triggered, pushing information such as the differing data value, the corresponding data metric, and the timestamp of the difference to the relevant responsible personnel (e.g., operations staff, data analysts).

[0067] In the above process, this embodiment of the invention promptly detects discrepancies between sales data on the PC and APP platforms. When these discrepancies exceed the normal range, relevant personnel are promptly notified for processing, ensuring data consistency and accuracy. This ability to quickly and accurately identify data differences and promptly notify relevant personnel for troubleshooting and correction reduces the risks associated with data inconsistencies, improves the enterprise's data management level and decision-making efficiency, and guarantees the stability and accuracy of business operations.

[0068] Through the above process, this embodiment of the invention effectively ensures the consistency of sales data between the PC and APP terminals through a rigorous five-step operation. First, it automatically logs into both dashboard pages using headless browser technology, paving the way for data collection. Next, it accurately locates the "sales revenue" indicator area and takes screenshots periodically to ensure the acquisition of real-time and traceable raw data. Then, it uses the DeepSeek V3 model to intelligently recognize the screenshot content, converting the image into structured data, improving the accuracy and efficiency of data extraction. Finally, by comparing the data values ​​from both ends, it automatically issues warnings based on preset thresholds, enabling timely detection and handling of data discrepancies. The entire solution does not require API interface integration, reducing implementation costs, and utilizes AI technology to reduce false alarms, achieving multi-dimensional detection of values, trends, and status. This provides strong support for enterprise data management and ensures stable business operations.

[0069] In one application scenario, during major e-commerce promotions, sales data dashboards on both PC and mobile apps are crucial for operations staff and decision-makers. However, due to differences in development teams, user groups, focuses, and usage scenarios, the two dashboards exhibit discrepancies, leading to data inconsistency and impacting decision-making efficiency and accuracy. This embodiment addresses this by employing a data collection and AI-powered intelligent data quality inspection method to ensure consistency between PC and mobile app sales data.

[0070] Specifically, first deploy a data acquisition server and install a headless browser (such as Selenium or Puppeteer) and related dependency libraries. Configure the service interface of the DeepSeek V3 deep learning model to ensure that the model can be called normally. Set the screenshot period (e.g., every 5 minutes), the difference threshold (e.g., sales difference exceeding 1%), and the alarm notification method (e.g., email, SMS).

[0071] The installation of the headless browser and its dependencies ensured the automation and efficiency of data collection. Configuring the DeepSeek V3 model enabled intelligent recognition, improving the accuracy and efficiency of data extraction. Setting screenshot periods, difference thresholds, and alarm notification methods ensured the real-time nature and effectiveness of monitoring.

[0072] Furthermore, using headless browser technology to automatically log in to the PC backend system, the system precisely locates the "Sales Revenue" indicator value area in the sales data dashboard using CSS selectors, takes a screenshot, and saves it as a PNG file with a timestamp. Then, using User-Agent spoofing and window simulation technologies, the system simulates an app visit using a headless browser, similarly locating the "Sales Revenue" indicator value area, taking a screenshot, and saving it as a PNG file with a timestamp.

[0073] The application of headless browser technology and User-Agent spoofing technology enables automated data collection on both PC and mobile app platforms, eliminating the need for API integration and reducing implementation costs. Screenshots are saved as PNG files with timestamps for easy data processing and traceability.

[0074] Furthermore, the collected screenshots from the PC and APP are standardized, including size normalization (e.g., adjusting to 800×600 resolution), color unification (converting to RGB three-channel format), and background purification (using Gaussian blur to eliminate noise). Key areas in the screenshots (such as the "sales" numerical area) are enhanced to improve contrast.

[0075] Standardization ensures the consistency of image data and improves the accuracy of subsequent recognition, while key area enhancement makes the indicator values ​​clearer and more distinguishable, further improving the recognition effect.

[0076] Furthermore, the DeepSeek V3 model is invoked to intelligently recognize the standardized screenshots from the PC and APP, extract the "sales" indicator value, retain the plus or minus sign, decimal point and original unit during the recognition process, and output the recognition results in JSON format.

[0077] The powerful recognition capabilities of the DeepSeek V3 model make data extraction more accurate and efficient. The JSON format output facilitates subsequent data processing and comparative analysis.

[0078] Furthermore, the extracted "sales" metric values ​​from the PC and APP sides are compared, the difference value is calculated, and the data values ​​with differences and their corresponding data metrics, as well as the timestamps of the differences, are recorded.

[0079] Among these features, data comparison can promptly identify data discrepancies, providing accurate information for subsequent early warning and problem investigation. The timestamp records of the discrepancies facilitate tracing the time when the problem occurred.

[0080] Finally, when the difference exceeds the set threshold, an alarm mechanism is automatically triggered, pushing the discrepancy data value and corresponding data indicators to the relevant responsible persons (such as operations personnel and data analysts). The relevant responsible persons then check and correct the data on the PC and APP based on the information pushed to ensure data consistency.

[0081] Among them, the early warning notification mechanism can promptly notify relevant personnel to investigate and correct problems, reducing the risks caused by data inconsistency. Problem investigation and correction ensure data consistency and accuracy, thereby improving the company's data management level and decision-making efficiency.

[0082] Through the above process, this embodiment achieves real-time monitoring of the consistency of sales data on PC and APP during e-commerce promotional periods by using data collection and AI-powered intelligent data quality inspection. From system preparation and configuration, data collection, data preprocessing and standardization, intelligent identification and data extraction, data comparison and discrepancy analysis to early warning notifications and problem investigation, the entire process is automated and efficient, reducing implementation costs and improving the accuracy and efficiency of data extraction. Simultaneously, the early warning notification mechanism can promptly detect and notify of data discrepancies, facilitating timely correction and ensuring data consistency and accuracy, providing strong support for stable enterprise operations and decision-making.

[0083] The following are embodiments of the apparatus of the present invention, which can be used to execute the data quality inspection method involved in the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the method embodiments of the data quality inspection method involved in the present invention.

[0084] Please see Figure 3 This invention provides a data quality inspection device 800.

[0085] The data quality inspection device 800 includes, but is not limited to: a data acquisition module 810, a data preprocessing module 830, an intelligent comparison module 850, and an anomaly handling module 870.

[0086] The data acquisition module 810 is used to collect the first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals respectively, using headless browser technology, User-Agent spoofing technology and window simulation.

[0087] The data preprocessing module 830 is used to perform standardization processing and key area enhancement on the first and second screenshots; the standardization processing includes size normalization, color unification and background purification.

[0088] The intelligent comparison module 850 is used to call a deep learning model to intelligently identify the processed first screenshot and second screenshot to obtain the first terminal data and the second terminal data, and compare them to obtain the difference value.

[0089] The exception handling module 870 is used to set the difference threshold. When the difference value exceeds the difference threshold, an alarm mechanism is automatically triggered. The alarm mechanism includes information push and result feedback.

[0090] It should be noted that the data quality inspection provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the data quality inspection device will be divided into different functional modules to complete all or part of the functions described above.

[0091] Furthermore, the data quality inspection device and data quality inspection method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0092] Figure 4 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0093] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on having... Figure 4 One or more components of the exemplary electronic device 2000 shown.

[0094] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 4 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0095] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0096] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this does not constitute a specific limitation.

[0097] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0098] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0099] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 4 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, the data quality inspection device can be considered as application program 253 deployed on the electronic device 2000.

[0100] Data 255 may be signal information, etc., and is stored in memory 250.

[0101] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, a data quality inspection method may be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0102] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0103] Please see Figure 5 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0104] exist Figure 5 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0105] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0107] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0108] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0109] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0110] The computer-readable instructions are executed by one or more processors 4001 to implement the data quality inspection methods in the above embodiments.

[0111] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the data quality inspection method described above.

[0112] This invention provides a computer program product, which includes computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the data quality inspection method described above.

[0113] Compared with related technologies, the beneficial effects of the present invention are:

[0114] 1. This invention enables real-time comparison of sales data between PC and APP terminals; through headless browser technology and screenshot collection methods, it realizes automated data collection and comparison of sales data dashboards on both ends, solving the problem of decision-making errors caused by data inconsistency and improving the credibility of data and the accuracy of decision-making.

[0115] 2. This invention can reduce the false alarm rate of data quality inspection; by adopting the DeepSeek V3 deep learning model AI image recognition and data comparison, it realizes intelligent identification and alarm of abnormal data, solves the problem of high false alarm rate caused by the reliance on manual rules in traditional solutions, and improves the efficiency and accuracy of data quality inspection.

[0116] 3. This invention can reduce implementation costs and time; through a ready-to-use solution that does not require API interface integration, and a combination of screenshot acquisition and AI intelligent recognition technologies, it solves the problems of traditional solutions requiring interface development and long development cycles, thereby reducing implementation costs and improving deployment efficiency.

[0117] 4. This invention can improve the efficiency of problem investigation and handling; through real-time monitoring, intelligent alarm and push mechanism, it can promptly detect and notify data discrepancies, solve the problems of difficult problem investigation and high communication costs, enable managers and operators to quickly align goals and reduce decision-making risks.

[0118] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0119] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data quality inspection method, characterized in that, The method includes: Headless browser technology, User-Agent spoofing technology, and window simulation were used to collect the first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals, respectively. The first and second screenshots are subjected to standardization processing and key area enhancement; the standardization processing includes size normalization, color unification, and background purification. The deep learning model is invoked to intelligently identify the processed first and second screenshots to obtain first terminal data and second terminal data, and the difference values ​​are obtained by comparison. A difference threshold is set, and an alarm mechanism is automatically triggered when the difference value exceeds the difference threshold; the alarm mechanism includes information push and result feedback.

2. The data quality inspection method as described in claim 1, characterized in that, The method employs headless browser technology, User-Agent spoofing technology, and window simulation to automatically collect first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals according to a set cycle, including: The system automatically logs into the backend system of the first terminal and locates the target dashboard area using headless browser technology, and captures the indicator value area using CSS selectors; the first terminal includes a PC; the headless browser technology includes Servlet and Puppeteer. The indicator value area is captured according to the set time period, and the screenshot is saved as a PNG format with a timestamp attached to obtain the first screenshot.

3. The data quality inspection method as described in claim 1, characterized in that, The method employs headless browser technology, User-Agent spoofing technology, and window simulation to automatically collect first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals according to a set cycle, including: The system automatically logs into a second terminal and locates the target dashboard area using User-Agent spoofing and window simulation technologies, and captures the indicator value area through CSS selectors; the second terminal includes an APP. Take screenshots of the indicator value area according to the set time period, save the screenshots as PNG format and add a timestamp to obtain a second screenshot.

4. The data quality inspection method as described in claim 1, characterized in that, The standardization and key area enhancement of the first and second screenshots include: Adjust the size and resolution of the first and second screenshots to the set values, convert the first and second screenshots to RGB three-channel format, and use Gaussian blur to eliminate noise in the first and second screenshots; Image enhancement techniques are used to enhance the contrast of key regions in the first and second screenshots; the image enhancement techniques include CLAHE algorithm, AHE algorithm, wavelet transform and deep learning super-resolution technology.

5. The data quality inspection method as described in claim 1, characterized in that, Before performing intelligent recognition on the processed first and second screenshots respectively, the method further includes: The timestamps and indicator definitions of the first and second screenshots are unified, and a mapping table is established; the mapping table is used to unify the data indicators of the first and second terminals.

6. The data quality inspection method as described in claim 5, characterized in that, The process involves using a deep learning model to intelligently identify the processed first and second screenshots to obtain first terminal data and second terminal data respectively, and then comparing them to obtain difference values, including: The deep learning model is invoked to identify the numerical content in the first and second screenshots based on the mapping table. During the identification process, the positive and negative signs, decimal points, and original units are retained, and the data from the first terminal and the second terminal are output in JSON format, respectively. The deep learning model includes the DeepSeek V3 model. The data values ​​of the first terminal data and the second terminal data are compared to obtain the difference value, and the data values ​​with differences and the corresponding data indicators are recorded.

7. The data quality inspection method as described in claim 6, characterized in that, The automatic alarm mechanism triggered when the difference value exceeds the difference threshold includes: When the difference value exceeds the difference threshold, the data value with the difference and the corresponding data indicator are pushed to the relevant responsible person, and the first terminal and the second terminal are investigated and corrected according to the data value with the difference and the corresponding data indicator.

8. A data quality inspection device, characterized in that, The device includes: The data acquisition module is used to collect the first and second screenshots corresponding to key indicators in the target dashboard areas of the first and second terminals respectively, using headless browser technology, User-Agent spoofing technology and window simulation. The data preprocessing module is used to perform standardization processing and key area enhancement on the first and second screenshots; the standardization processing includes size normalization, color unification, and background purification. The intelligent comparison module is used to call a deep learning model to intelligently identify the processed first screenshot and second screenshot to obtain first terminal data and second terminal data respectively, and compare them to obtain the difference value. An anomaly handling module is used to set a difference threshold. When the difference value exceeds the difference threshold, an alarm mechanism is automatically triggered. The alarm mechanism includes information push and result feedback.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the data quality inspection method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the data quality inspection method as described in any one of claims 1 to 7.

Images