A data checking and cleaning method, device and medium

Abstract

The application discloses a data verification and cleaning method, device and medium, the method comprising: receiving an encrypted data packet sent by a channel merchant, and analyzing an encrypted transmission instruction in the encrypted data packet to extract a channel identifier, an encryption key, an encryption algorithm identifier and a timestamp; decrypting the encrypted transmission instruction based on the encryption key and the encryption algorithm identifier to restore original data, and calculating an MD5 hash value corresponding to the decrypted restored original data; comparing the MD5 hash value with a hash value of the original data to verify whether the encrypted data packet is complete, and if so, storing the original data in a temporary table; triggering an asynchronous multithreaded task through a timer to cyclically perform data verification by the channel merchant and verify whether the inventory data satisfies a balance relationship, and if not, clearing a formal table corresponding to the channel merchant and a corresponding date; and in response to a manual operation instruction, deleting an abnormal temporary table or an abnormal formal table to complete data cleaning.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data verification and cleaning method, device and medium. Background Technology

[0002] Currently, in the supply chain management of fast-moving consumer goods (FMCG) industries such as alcoholic beverages, the sales, purchase, and inventory data of distributors are the core basis for reflecting market dynamics. Traditional data management methods rely on manual offline collection of paper or electronic spreadsheets, which are then transmitted via email or mobile storage devices. This model has significant security risks; sensitive data is easily intercepted or tampered with during transmission, and the lack of integrity verification mechanisms makes it impossible for the receiving end to effectively identify data anomalies. In addition, the manual verification process requires comparing massive amounts of data item by item, and processing efficiency decreases exponentially with the number of distributors, often resulting in erroneous verification results due to human error.

[0003] Existing technologies attempt to improve transmission security through basic encryption, but they only encrypt the raw data and lack a composite verification mechanism that includes identity verification, timestamps, and hash values. This fails to simultaneously guarantee the credibility of the data source and the integrity of the content. In the data verification stage, traditional solutions employ a single-threaded sequential processing mode without asynchronous task scheduling and state locking mechanisms. Concurrent requests from multiple channel providers can easily lead to data overwriting or duplicate verification. Especially when data from a particular channel provider is abnormal, blocking processing can slow down the entire process, causing delays in the analysis of critical business data. Furthermore, for erroneous data cleanup, current solutions generally rely on periodic batch deletion or manual screening. Periodic batch deletion is prone to accidentally deleting valid data, while manual screening requires operators with high professional skills and struggles to accurately locate abnormal entries scattered across temporary and formal tables. Summary of the Invention

[0004] This application provides a data verification and cleaning method, device, and medium to solve the above-mentioned technical problems.

[0005] On the one hand, embodiments of this application provide a data verification and cleaning method, including: The system receives encrypted data packets sent by channel partners, parses the encrypted transmission instructions in the encrypted data packets, and extracts the channel identifier, encryption key, encryption algorithm identifier, and timestamp; the encrypted data packets include: encrypted transmission instructions and original data; The encrypted transmission command is decrypted based on the encryption key and the encryption algorithm identifier to restore the original data, and the MD5 hash value corresponding to the restored original data is calculated. The MD5 hash value is compared with the hash value of the original data to verify whether the encrypted data packet is complete. If it is, the original data is stored in a temporary table. An asynchronous multi-threaded task is triggered by a timer to perform data verification on channel partners in a loop and verify whether the inventory data meets the balance relationship. If not, the corresponding channel partner and the corresponding date's official table are cleared. In response to manual operation instructions, abnormal temporary tables or abnormal formal tables are deleted to complete data cleanup.

[0006] In one implementation of this application, before receiving the encrypted data packet sent by the channel provider, the method further includes: In response to a data transmission request from a channel provider, the channel provider's channel identifier, encryption key, encryption algorithm identifier, data hash value, and timestamp are concatenated, and the concatenated string is symmetrically encrypted using a public key to generate an encrypted transmission command. The original data and encrypted transmission instructions, encrypted with an encryption algorithm, are sent to the target system through a secure transmission channel.

[0007] In one implementation of this application, the MD5 hash value is compared with the hash value of the original data to verify whether the encrypted data packet is complete. If so, the original data is stored in a temporary database, specifically including: The calculated MD5 hash value of the restored original data is compared with the hash value of the original data to determine whether the MD5 hash value and the hash value of the original data are consistent. If there is a discrepancy, it indicates that the encrypted data packet has been corrupted, and a verification failure message is returned to the channel provider to prompt the channel provider to retransmit; If they match, the original data is stored in a temporary table, a data transfer flag is set in Redis, and a preset countdown timer is started to clear the original data.

[0008] In one implementation of this application, an asynchronous multithreaded task is triggered by a timer to perform data verification in a loop by the channel partners, specifically including: Check if the data transmission flag of the current channel provider exists in Redis. If the data transmission flag exists, return directly to execute the next channel provider. If the data transmission flag does not exist, continue to check if the data verification flag of the current channel provider exists in Redis. If the data verification flag exists, proceed directly to the next channel provider; otherwise, set the Redis verification flag. Query the start date based on the verification log, and loop up to the day before the current date to traverse the date range; For the day before the current date, back up the data in the temporary table to the backup table, and after the integrity verification is successful, delete the data corresponding to the day before the current date in the official table. The backup table is migrated to the deleted official table to verify whether the inventory data meets the balance relationship. After the loop is completed within the date range, the next channel partner is executed.

[0009] In one implementation of this application, traversing the date range specifically includes: Check if there is a last verification log record with the type identifier set to the preset threshold; If the verification log record exists, the verification date of the verification log record shall be used as the starting date, and the starting date shall be incremented by one day when the verification is successful; If the verified log record does not exist, the start date is read from the initialization configuration table, and the end date is fixed to the day before the system date.

[0010] In one implementation of this application, it is verified whether the inventory data satisfies the balance relationship; if not, the corresponding channel supplier and the corresponding date in the formal table are cleared, specifically including: Check if the basic data in the official table is complete. If it is incomplete, clear the data in the official table and record the failure log. The basic data includes store basic data and product basic data. Assuming the basic data is complete, the following formula is used to further verify whether the inventory data in the official table for that day is equal:

[0011] in, Indicates beginning inventory. Indicates the quantity received into the warehouse. Indicates sales quantity. Indicates the quantity transferred into the warehouse. Indicates the quantity transferred into the warehouse. This indicates the ending inventory; If they are equal, update the verification log and perform data verification for the next date. If they are not equal, clear the data in the official table and record the verification failure.

[0012] In one implementation of this application, in response to a manual operation instruction, an abnormal temporary table or an abnormal formal table is deleted to complete data cleanup, specifically including: Receive manual operation instructions input from the system interface; the manual operation instructions include a channel identifier, a target data table identifier, and a date range; Based on the channel identifier, the target data table identifier, and the date range, determine whether there is a temporary table with unverified data exceeding the preset storage period. If so, locate the abnormal data row in the abnormal temporary table and determine whether there is a formal table with a history of a specified number of consecutive verification failures. If so, locate the abnormal data row in the abnormal formal table. In response to the manual operation instruction, a deletion operation is performed on the target data row based on the location information of the abnormal data row, so as to complete the cleanup operation of the specified data.

[0013] In one implementation of this application, after triggering the asynchronous multithreaded task via a timer, the method further includes: Each channel partner is assigned an independent thread to perform the verification operation, and the verification flag status of each thread is synchronized through Redis shared memory, so that different threads skip duplicate verification when they detect the same channel partner's verification flag. After completing the data verification for all date ranges of a single channel merchant, the verification flag for the corresponding channel merchant in Redis is deleted. The verification results of each thread are not written to the database log table in real time. The database log table contains the channel merchant ID, verification date, balance relationship verification status, and basic data integrity status.

[0014] On the other hand, embodiments of this application also provide a data verification and cleaning device, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a data verification and cleanup method as described above.

[0015] On the other hand, embodiments of this application also provide a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement a data verification and cleanup method as described above.

[0016] This application provides a data verification and cleaning method, device, and medium, which has at least the following beneficial effects: The system parses encrypted transmission commands and extracts elements such as channel identifiers and timestamps to ensure data traceability and prevent the risk of data injection from illegal channels. It compares the decrypted MD5 hash with the original data hash to accurately identify data tampering during transmission, reducing the error rate of integrity verification. Verified raw data is temporarily stored in a temporary table to decouple the transmission and verification processes, preventing receiver blockage. A timer-triggered asynchronous multi-threaded tasks replace manual polling, executing channel data verification in parallel and reducing processing time. When verification of the inventory balance fails, the system immediately clears the data in the official table, preventing erroneous data from entering the analysis process and avoiding decision-making errors caused by abnormal correlations between beginning and ending inventory. Manual commands drive the deletion of abnormal tables, achieving strong linkage with verification results, accurately locating discrete erroneous data that needs to be cleaned, and reducing the rate of false deletion of valid data. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a data verification and cleaning method provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating a data transmission method provided in an embodiment of this application; Figure 3 A flowchart illustrating a data verification method provided in an embodiment of this application; Figure 4 A flowchart illustrating a data cleaning method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the internal structure of a data verification and cleaning device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a data verification and cleanup method provided in an embodiment of this application.

[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0022] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0023] like Figure 1 As shown in the embodiment of this application, a data verification and cleaning method includes: Step 101: Receive the encrypted data packet sent by the channel provider, parse the encrypted transmission instructions in the encrypted data packet, and extract the channel identifier, encryption key, encryption algorithm identifier and timestamp; the encrypted data packet includes: encrypted transmission instructions and original data.

[0024] In this embodiment, a string is concatenated according to certain rules, namely: channel provider ID: symmetric encryption key: symmetric encryption algorithm: hash value of transmitted data: current timestamp. During the data transmission stage, the concatenated string is encrypted with AES or DES public key to generate an encrypted transmission command, which is then transmitted together with the original data.

[0025] Figure 2 This is a flowchart illustrating a data transmission method provided in an embodiment of this application. Figure 2 As shown, the distributor first encrypts the sales and purchase data using an AES or DES public key, then calls a REST service to transmit the encrypted data packet to the target system. The receiving enterprise verifies the encrypted data packet based on the MD5 hash to determine its integrity. If complete, the decrypted original data is stored in a temporary table; otherwise, the received data is discarded, and a retransmission prompt is returned for the distributor to retransmit.

[0026] In one embodiment of this application, the system receives encrypted data packets sent by channel partners through a preset secure transmission channel, such as HTTPS or a leased network. The encrypted data packets adopt a layered encapsulation structure, with the inner layer containing the original data and the outer layer encapsulating encrypted transmission instructions. Specifically, the channel identifier (such as a unique channel partner code), the encryption key (a key pre-negotiated between the channel partner and the enterprise), the encryption algorithm identifier (such as AES-256 or DES), and the timestamp are concatenated into a string in a preset order. It should be noted that the concatenation format is: Channel Partner ID: Symmetric Encryption Key: Symmetric Encryption Algorithm: Hash Value of Transmitted Data: Current Timestamp.

[0027] The concatenated string is encrypted using the public key of an asymmetric encryption algorithm (such as RSA) to generate an encrypted transmission command. Then, the encrypted transmission command is combined with the original data to form a complete data packet.

[0028] Upon receiving a data packet, the receiving end first parses the packet header information using a proprietary protocol to locate the start of the encrypted transmission command. Then, it decrypts the encrypted command using a pre-stored private key to recover key parameters such as the channel identifier and encryption key. This ensures the confidentiality and integrity of data transmission and prevents man-in-the-middle attacks or data tampering.

[0029] Step 102: Decrypt the encrypted transmission command based on the encryption key and encryption algorithm identifier to restore the original data, and calculate the MD5 hash value corresponding to the restored original data after decryption.

[0030] In this embodiment, the receiving end decrypts the encrypted data packet using the private key, calculates the MD5 hash value corresponding to the encrypted transmission command according to the generated rules, and then compares the MD5 hash value with the hash value of the original data. If there is an error, it indicates that the data integrity has been compromised, and a verification failure prompt is returned to prompt retransmission. The successfully verified data will be stored in a temporary table, waiting for the timer to trigger the data verification logic.

[0031] In one embodiment of this application, the encrypted transmission command is decrypted by calling the corresponding symmetric encryption algorithm, such as AES-256-CBC, based on the extracted encryption key and encryption algorithm identifier, and the original data is restored.

[0032] The decrypted raw data is hashed byte by byte using MD5 hashing to generate a fixed-length hash value. For example, for structured data containing fields such as product code, quantity, and price, it must be ensured that all fields participate in the hash calculation. The calculated MD5 hash value is compared with the original hash value carried in the encrypted transmission command. If they match, the data packet is determined to be untampered and enters the temporary table storage process; if they do not match, an exception alarm is triggered and the data packet is discarded. A hash verification mechanism ensures that the data has not been illegally modified during transmission, providing a reliable data source for subsequent verification processes.

[0033] Step 103: Compare the MD5 hash value with the hash value of the original data to verify whether the encrypted data packet is complete. If so, store the original data in a temporary table.

[0034] In one embodiment of this application, if the MD5 hash value matches the hash value of the original data, the data packet is confirmed to be complete; if they do not match, a log is recorded and subsequent processes are blocked. The verified original data is written to a temporary database table, which adopts a partitioned storage design, isolating data by channel vendor ID and date range.

[0035] Setting data transfer status flags in the Redis cache, such as using the SETNX command to generate unique key-value pairs, can prevent the same data packet from being processed repeatedly. Simultaneously, enabling a data validity period mechanism with a preset duration, such as a 24-hour countdown, automatically cleans up temporary table data after the timeout. Through temporary storage and status flagging mechanisms, data availability is ensured while avoiding resource waste caused by duplicate validation.

[0036] Step 104: Trigger an asynchronous multi-threaded task through a timer to perform data verification for channel partners in a loop, and verify whether the inventory data meets the balance relationship. If not, clear the official table for the corresponding channel partner and the corresponding date.

[0037] Figure 3 This is a flowchart illustrating a data verification method provided in an embodiment of this application. Figure 3 The data verification phase employs a timer-triggered asynchronous multi-threaded task processing. First, it loops through the channel partners, creating asynchronous thread tasks based on each partner. It checks if the current channel partner's data transmission flag exists in Redis; if so, it immediately returns and executes the next channel partner's task. To avoid errors in data verification, it checks if the current channel partner's data verification flag exists in Redis; if so, it immediately returns and executes the next channel partner's task. Otherwise, it continues execution and sets the verification flag.

[0038] Loop through a date range, for example, with yesterday as the end date and yesterday as the start date: Query the last verification log record with TYPE1=50. If the data verification date exists, add one day on success, otherwise leave it unchanged; if not, retrieve the date from the initialization table. Execute the following logic sequentially: Execute the data transfer logic, back up the original data to the backup table, verify data integrity, delete data from the official table, and migrate the original data to the official table; Execute the data verification logic, record the data verification log, and determine if the basic data (store and product data) is complete. If the basic data is missing, return failure and clear the official table data; if successful, continue verifying whether the inventory data in the official table for that day is equal (i.e., whether beginning inventory + receipts - sales + transfer receipts - transfers out equals ending inventory). If they are not equal, clear the official table data and return verification failure; if they are equal, update the verification log and proceed to the next date. After looping through all date ranges for the current channel partner, proceed to the next channel partner, and so on.

[0039] In this embodiment, the system employs a distributed task scheduling framework (such as Quartz) to automate the verification process. A timer triggers an asynchronous task at a preset interval (e.g., every morning at midnight) to retrieve a list of channel partners to be verified from Redis. An independent thread is assigned to each channel partner to execute the verification task. A distributed lock is implemented using the Redis SETNX command to prevent multiple threads from concurrently verifying the same channel partner's data.

[0040] Back up the temporary table data to the historical table and then overwrite it in the official data table. Verify the balance of inventory management data based on business rules; the core formula is:

[0041] in, Indicates beginning inventory. Indicates the quantity received into the warehouse. Indicates sales quantity. Indicates the quantity transferred into the warehouse. Indicates the quantity transferred into the warehouse. This represents the ending inventory. It should be noted that the beginning inventory is taken from the ending value of the previous verification period in the official report, and all types of inbound and outbound data are derived from detailed records uploaded by channel partners.

[0042] If the validation fails, such as due to data imbalance or missing master data, the data for that day is marked as invalid, an alarm is triggered, and the data for that day is cleared from the official table through the database transaction rollback mechanism. This automated validation and transaction rollback mechanism ensures that valid data is always stored in the official table, reducing the cost of manual intervention.

[0043] Step 105: In response to manual operation instructions, delete the abnormal temporary table or abnormal formal table to complete the data cleanup.

[0044] Figure 4 This is a flowchart illustrating a data cleaning method provided in an embodiment of this application, as shown below. Figure 4 As shown, since channel partners may generate junk data, affecting normal data collection and verification, the data cleanup phase mainly involves enterprise staff operating the system to physically delete erroneous data in temporary or formal tables reported by channel partners, thus cleaning up junk data. Specifically, when the manual operating system function is invoked, the system executes the data cleanup logic, determines whether there are abnormal temporary or formal tables, and if so, locates the abnormal data rows in the abnormal temporary table, thereby cleaning the abnormal temporary table. Based on the same processing flow, the abnormal formal table is also cleaned up.

[0045] In this embodiment, the system provides two data cleanup modes to meet different scenario requirements. The first mode triggers cleanup tasks based on preset strategies, such as deleting abnormal data in batches via SQL statements if temporary table data has been stored for more than 90 days or if a formal table fails three consecutive validations. The second mode allows administrators to submit cleanup commands by inputting a channel identifier, target table identifier (temporary table / formal table), and date range through the management interface.

[0046] Based on the command parameters, the system queries the temporary table for overdue, unverified data or the official table for records marked as failed, generating a list of data to be cleaned. Before performing physical deletion, operation permissions and the scope of cleanup must be confirmed again to avoid accidental deletion. After cleanup is complete, the data lifecycle status flag is updated, such as marking "pending cleanup" as "expired," and the operator ID, cleanup time, and affected data range are recorded in the audit log. Through this human-machine collaboration mechanism, timely cleanup of abnormal data is ensured while avoiding the risk of data loss due to accidental operations.

[0047] In this embodiment, when the timer triggers the asynchronous multithreaded task, the system allocates an independent thread to each channel partner to perform the verification operation. Here, "independent thread" refers to a lightweight execution unit at the operating system level, dynamically created by the thread pool manager. Specifically, the thread pool allocates thread resources as needed based on the number of currently active channel partners (e.g., 50 distributors), ensuring that the data verification for each channel partner runs in isolation within its dedicated thread. For example, the verification task for channel partner "DC2023-SD001" is executed by thread-15, while "DC2023-SD002" is executed by thread-16, avoiding interference between data from multiple channel partners in memory.

[0048] Understandably, to avoid state chaos caused by multiple threads concurrently processing the same channel provider data, this solution uses Redis shared memory to synchronize and verify the flag status. As a distributed key-value database, Redis's in-memory storage capabilities enable microsecond-level state read / write. Specifically, before executing channel provider verification, each thread checks if a flag with the key `check_flag:{channel provider ID}` (e.g., `check_flag:DC2023-SD001`) exists in Redis. If the flag exists, it indicates that a thread is already processing the channel provider's data, and the current thread immediately skips this channel provider and moves on to the next one. If the flag does not exist, the current thread sets the flag (valid for 2 hours) and locks the verification privileges for that channel provider. For example, when Thread-15 and Thread-16 simultaneously process the channel provider "DC2023-SD003", the thread that acquires the Redis lock first successfully sets the flag, and the other thread automatically abandons the verification upon detecting the flag, fundamentally preventing duplicate verification.

[0049] It's important to note that after verifying all date ranges for a single channel provider, the system proactively deletes the verification flag for that channel provider from Redis. This operation is achieved by executing the Redis command `DEL check_flag:DC2023-SD001`, freeing up resources for subsequent tasks. Failure to delete it promptly can lead to three problems: first, expired flags may occupy memory resources; second, subsequent scheduled tasks may mistakenly identify the channel provider as currently verifying and skip processing; and third, flag accumulation in high-concurrency scenarios may cause Redis memory overflow. For example, when the 30-day data verification for channel provider "DC2023-SD001" is completed, regardless of success or failure, the thread immediately clears its flag to ensure that verification can start normally in the next cycle.

[0050] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a data verification and cleaning device, the structure of which is as follows: Figure 5 As shown.

[0051] Figure 5 This is a schematic diagram of the internal structure of a data verification and cleaning device provided in an embodiment of this application. Figure 5 As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: Receive encrypted data packets sent by channel partners, parse the encrypted transmission instructions in the encrypted data packets, and extract the channel identifier, encryption key, encryption algorithm identifier, and timestamp; the encrypted data packets include: encrypted transmission instructions and original data; The encrypted transmission command is decrypted based on the encryption key and encryption algorithm identifier to restore the original data, and the MD5 hash value corresponding to the restored original data is calculated. Compare the MD5 hash value with the hash value of the original data to verify whether the encrypted data packet is complete. If it is, store the original data in a temporary table. An asynchronous multi-threaded task is triggered by a timer to perform data verification on channel partners in a loop and verify whether the inventory data meets the balance relationship. If not, the corresponding channel partner and the corresponding date's official table are cleared. In response to manual operation instructions, abnormal temporary tables or abnormal formal tables are deleted to complete data cleanup.

[0052] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: Receive encrypted data packets sent by channel partners, parse the encrypted transmission instructions in the encrypted data packets, and extract the channel identifier, encryption key, encryption algorithm identifier, and timestamp; the encrypted data packets include: encrypted transmission instructions and original data; The encrypted transmission command is decrypted based on the encryption key and encryption algorithm identifier to restore the original data, and the MD5 hash value corresponding to the restored original data is calculated. Compare the MD5 hash value with the hash value of the original data to verify whether the encrypted data packet is complete. If it is, store the original data in a temporary table. An asynchronous multi-threaded task is triggered by a timer to perform data verification on channel partners in a loop and verify whether the inventory data meets the balance relationship. If not, the corresponding channel partner and the corresponding date's official table are cleared. In response to manual operation instructions, abnormal temporary tables or abnormal formal tables are deleted to complete data cleanup.

[0053] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0054] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0060] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0061] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0062] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0063] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data verification and cleaning method, characterized in that, The method includes: The system receives encrypted data packets sent by channel partners, parses the encrypted transmission instructions in the encrypted data packets, and extracts the channel identifier, encryption key, encryption algorithm identifier, and timestamp; the encrypted data packets include: encrypted transmission instructions and original data; The encrypted transmission command is decrypted based on the encryption key and the encryption algorithm identifier to restore the original data, and the MD5 hash value corresponding to the restored original data is calculated. The MD5 hash value is compared with the hash value of the original data to verify whether the encrypted data packet is complete. If it is, the original data is stored in a temporary table. An asynchronous multi-threaded task is triggered by a timer to perform data verification on channel partners in a loop and verify whether the inventory data meets the balance relationship. If not, the corresponding channel partner and the corresponding date's official table are cleared. In response to manual operation instructions, abnormal temporary tables or abnormal formal tables are deleted to complete data cleanup; Asynchronous multithreaded tasks are triggered by a timer to perform data verification in a loop for channel partners, specifically including: Check if the data transmission flag of the current channel provider exists in Redis. If the data transmission flag exists, return directly to execute the next channel provider. If the data transmission flag does not exist, continue to check if the data verification flag of the current channel provider exists in Redis. If the data verification flag exists, proceed directly to the next channel provider; otherwise, set the Redis verification flag. Query the start date based on the verification log, and loop up to the day before the current date to traverse the date range; For the day before the current date, back up the data in the temporary table to the backup table, and after the integrity verification is successful, delete the data corresponding to the day before the current date in the official table. The backup table is migrated to the deleted official table to verify whether the inventory data meets the balance relationship. After the loop is completed within the date range, the next channel partner is executed.

2. The data verification and cleaning method according to claim 1, characterized in that, Before receiving encrypted data packets sent by the channel provider, the method further includes: In response to a data transmission request from a channel provider, the channel provider's channel identifier, encryption key, encryption algorithm identifier, data hash value, and timestamp are concatenated, and the concatenated string is symmetrically encrypted using a public key to generate an encrypted transmission command. The original data and encrypted transmission instructions, encrypted with an encryption algorithm, are sent to the target system through a secure transmission channel.

3. The data verification and cleaning method according to claim 1, characterized in that, The MD5 hash value is compared with the hash value of the original data to verify whether the encrypted data packet is complete. If so, the original data is stored in a temporary table, specifically including: The calculated MD5 hash value of the restored original data is compared with the hash value of the original data to determine whether the MD5 hash value and the hash value of the original data are consistent. If there is a discrepancy, it indicates that the encrypted data packet has been corrupted, and a verification failure message is returned to the channel provider to prompt the channel provider to retransmit; If they match, the original data is stored in a temporary table, a data transfer flag is set in Redis, and a preset countdown timer is started to clear the original data.

4. The data verification and cleaning method according to claim 1, characterized in that, Traversing the date range, specifically including: Check if there is a last verification log record with the type identifier set to the preset threshold; If the verification log record exists, the verification date of the verification log record shall be used as the starting date, and the starting date shall be incremented by one day when the verification is successful; If the verification log record does not exist, the start date is read from the initialization configuration table, and the end date is fixed to the day before the system date.

5. The data verification and cleaning method according to claim 1, characterized in that, Verify whether the inventory data satisfies the balance relationship. If not, clear the official tables for the corresponding channel partners and dates. Specifically, this includes: Check if the basic data in the official table is complete. If it is incomplete, clear the data in the official table and record the failure log. The basic data includes store basic data and product basic data. Assuming the basic data is complete, the following formula is used to further verify whether the inventory data in the official table for that day is equal: in, Indicates beginning inventory. Indicates the quantity received into the warehouse. Indicates sales quantity. Indicates the quantity transferred into the warehouse. Indicates the quantity transferred out of the warehouse. This indicates the ending inventory; If they are equal, update the verification log and perform data verification for the next date. If they are not equal, clear the data in the official table and record the verification failure.

6. The data verification and cleaning method according to claim 1, characterized in that, In response to manual operation instructions, abnormal temporary tables or abnormal formal tables are deleted to complete data cleanup, specifically including: Receive manual operation instructions input from the system interface; the manual operation instructions include a channel identifier, a target data table identifier, and a date range; Based on the channel identifier, the target data table identifier, and the date range, determine whether there is a temporary table with unverified data exceeding the preset storage period. If so, locate the abnormal data row in the abnormal temporary table and determine whether there is a formal table with a history of a specified number of consecutive verification failures. If so, locate the abnormal data row in the abnormal formal table. In response to the manual operation instruction, a deletion operation is performed on the target data row based on the location information of the abnormal data row, so as to complete the cleanup operation of the specified data.

7. The data verification and cleaning method according to claim 1, characterized in that, After triggering the asynchronous multithreaded task via a timer, the method further includes: Each channel partner is assigned an independent thread to perform the verification operation, and the verification flag status of each thread is synchronized through Redis shared memory, so that different threads skip duplicate verification when they detect the same channel partner's verification flag. After completing the data verification for all date ranges of a single channel merchant, the verification flag for the corresponding channel merchant in Redis is deleted. The verification results of each thread are not written to the database log table in real time. The database log table contains the channel merchant ID, verification date, balance relationship verification status, and basic data integrity status.

8. A data verification and cleanup device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a data verification and cleanup method as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement a data verification and cleanup method as described in any one of claims 1-7.

Images