E-commerce platform data quality inspection method and system

Abstract

The present application relates to a method and system for data quality inspection of an e-commerce platform, the method comprising receiving and executing a first general scheduled task, sending a data acquisition request to a data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform; receiving and executing a second general scheduled task, obtaining the data acquisition results stored in the data acquisition system, and storing them in a corresponding table; receiving and executing a third general scheduled task, performing quality inspection on the data acquisition results with reference to pre-configured quality inspection rules, and obtaining quality inspection results; wherein the first general scheduled task, the second general scheduled task, and the third general scheduled task are obtained based on the XXL-JOB configuration, and each has an independent thread pool. The original scheduled task polling initiated by each type of quality inspection item was changed to three fixed general scheduled tasks, saving thread expenses. In addition, an independent thread pool means that scheduled tasks will not be blocked due to thread resources, effectively improving the processing power and response speed of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality detection, and particularly to a method and system for data quality inspection of an e-commerce platform. Background Art

[0002] The data maintenance of an e-commerce platform is mainly responsible by store operation. For example, the inventory information, expiration date information, price information, online / offline operations, coupon information, etc. of products all need to be maintained. However, during the daily operation and maintenance process, human misoperations are very likely to occur, resulting in operation accidents. Therefore, it is usually necessary to rely on a quality inspection platform to avoid the above problems, that is, to compare the e-commerce platform data with reference data, and check whether the difference data conforms to the quality inspection rules. If not, it is considered that there is a risk in the e-commerce platform; otherwise, there is no risk.

[0003] Currently, for the implementation of the quality inspection function, it is usually carried out in a single-machine environment, and a timing task is defined based on the annotation (@Scheduled annotation) provided by a lightweight open-source framework (Spring framework) for timing task control. As Figure 1 shown, the entire quality inspection process first requires the store operation to create the quality inspection rules of the specified store in the specified quality inspection items in the quality inspection system first, and initiate a store quality inspection task in the quality inspection system. Among them, after the quality inspection task is successfully created, the quality inspection system notifies the store operation, and polls the task table regularly to obtain new quality inspection tasks, and sends a data fetching request to the data fetching system according to the quality inspection items corresponding to the tasks, so that the data fetching system fetches data from the e-commerce platform. Further, after the data fetching system completes fetching data from the e-commerce platform, it carries the fetching result, that is, the e-commerce platform data, and calls back to the quality inspection system to evoke the next process of the quality inspection system, that is, the quality inspection system compares the e-commerce platform data with the quality inspection rule data, and synchronizes the obtained risk data to the store operation.

[0004] However, in the above quality inspection process, each type of quality inspection item is developed independently. There are as many polling timing tasks as there are types of quality inspection items. When encountering the peak period of quality inspection in a single-machine environment, the store operation will frequently create quality inspection tasks. At this time, a large number of threads will be created by the machine in a short period, resulting in large resource consumption. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide a method and system for data quality inspection of an e-commerce platform to solve the above problems.

[0006] In a first aspect, the embodiments of the present application provide a method for data quality inspection of an e-commerce platform, including the following steps:

[0007] Receiving and executing a first general timing task, and sending a data fetching request to the data fetching system for the data fetching system to fetch data from the e-commerce platform;

[0008] Receive and execute the second general timing task, obtain the data fetching results stored in the data fetching system, and store them in the corresponding table accordingly;

[0009] Receive and execute the third general timing task, perform quality inspection on the data fetching results with reference to the pre-configured quality inspection rules, and obtain the quality inspection results;

[0010] Among them, the first general timing task, the second general timing task, and the third general timing task are configured based on XXL-JOB and all have independent thread pools.

[0011] In some embodiments, the receiving and executing the first general timing task and sending a data fetching request to the data fetching system for the data fetching system to fetch data from the e-commerce platform includes:

[0012] Receive the first general timing task and obtain a first task list that simultaneously meets the first requirements of the quality inspection task table and the data fetching registration table;

[0013] Use the corresponding store information for sharding filtering so that each server filters out the quality inspection tasks that can be correspondingly executed from the first task list and stores them in the first blocking queue;

[0014] After using the threads in the first thread pool to obtain the corresponding quality inspection tasks, obtain the first distributed lock;

[0015] Judge whether the quality inspection task meets the corresponding data fetching conditions, and when it meets, send the data fetching request to the data fetching system according to the quality inspection item type for the data fetching system to fetch data from the e-commerce platform and create a data fetching record;

[0016] After storing the data fetching record in the data fetching registration table, release the first distributed lock.

[0017] In some embodiments, the first requirements are that the task status in the quality inspection task table is to be executed and the total number of data fetches is greater than 0; and the corresponding task ID in the data fetching registration table is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same data fetching current serial number as the quality inspection task table.

[0018] In some embodiments, the receiving and executing the second general timing task, obtaining the data fetching results stored in the data fetching system, and storing them in the corresponding table accordingly includes:

[0019] Receive the second general timing task and obtain a second task list that simultaneously meets the second requirements of the quality inspection task table and the data fetching registration table;

[0020] Perform sharding filtering using the corresponding store information, so that each server filters out the quality inspection tasks that can be correspondingly executed from the second task list and stores them in the second blocking queue;

[0021] After obtaining the corresponding quality inspection task by a thread in the second thread pool, obtain the second distributed lock;

[0022] Determine whether the quality inspection task meets the corresponding data fetching conditions, and when it meets, initiate a data fetching result acquisition request according to the data fetching return ID to obtain the data fetching result stored in the data fetching system;

[0023] After storing the data fetching result in the corresponding data fetching platform data table, update the data fetching registration form and release the second distributed lock.

[0024] In some embodiments, the second requirement is that the task status in the quality inspection task table is to be executed and the total number of data fetches is greater than 0; and the corresponding task ID and the current data fetching sequence number in the data fetching registration form are the same as the corresponding task ID and the current data fetching sequence number in the quality inspection task table, and the data fetching status is completed, and the data fetching result is not stored.

[0025] In some embodiments, receiving and executing the third general timing task, and performing quality inspection on the data fetching result with reference to the pre-configured quality inspection rules to obtain a quality inspection result, including:

[0026] Receive the third general timing task and obtain a third task list that meets the third requirement of the quality inspection task table;

[0027] Perform sharding filtering using the corresponding store information, so that each server filters out the quality inspection tasks that can be correspondingly executed from the third task list and stores them in the third blocking queue;

[0028] After obtaining the corresponding quality inspection task by a thread in the third thread pool, obtain the third distributed lock;

[0029] Determine whether the quality inspection task meets the corresponding data fetching conditions, and when it meets, store the snapshot data in the snapshot table and traverse the snapshot table to obtain the ID of the unprocessed quality inspection dimension data;

[0030] Obtain the quality inspection rule data and e-commerce platform data corresponding to the ID of the quality inspection dimension data, and perform quality inspection.

[0031] In some embodiments, the third requirement is that the task status in the quality inspection task table is in execution.

[0032] In some embodiments, receiving and executing the third general timing task, and performing quality inspection on the data fetching result with reference to the pre-configured quality inspection rules to obtain a quality inspection result, further includes:

[0033] Determine whether it is necessary to request the internal system, and when necessary, obtain the reference data of the internal system and store it in the internal system reference data table as the reference data during the quality inspection process.

[0034] In some embodiments, after obtaining the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and performing quality inspection, it further includes:

[0035] Store the quality inspection result in the quality inspection result table or the quality inspection result failure table, and update the snapshot table.

[0036] In a second aspect, an e-commerce platform data quality inspection system provided by an embodiment of the present application includes:

[0037] A first unit, configured to receive and execute a first general timing task, and send a data fetching request to the data fetching system for the data fetching system to fetch data from the e-commerce platform;

[0038] A second unit, configured to receive and execute a second general timing task, obtain the data fetching result stored in the data fetching system, and store it in a corresponding table;

[0039] A third unit, configured to receive and execute a third general timing task, perform quality inspection on the data fetching result with reference to a pre-set quality inspection rule to obtain a quality inspection result;

[0040] Wherein, the first general timing task, the second general timing task, and the third general timing task are configured based on XXL-JOB and each has an independent thread pool.

[0041] Technical effects of the present invention: The present invention changes from the original polling of each type of quality inspection item by initiating a timing task to three fixed general timing tasks, saving thread expenses. The purpose of sending a data fetching request, storing the data fetching result, and performing quality inspection on the data fetching result is achieved by sequentially executing the first general timing task, the second general timing task, and the third general timing task. Moreover, each of the three timing tasks has an independent thread pool. On the one hand, the number of threads corresponding to the thread pool is controllable manually, reducing resource consumption; on the other hand, the independent thread pool means that the timing tasks will not be blocked due to thread resources, effectively improving the processing ability and response speed of the system. And the data information of the key steps is stored in a table, enabling the quality inspection tasks that have not reached the final state to continue to execute after the project is restarted.

[0042] According to the following detailed description of the exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become clear. Description of the Drawings

[0043] The accompanying drawings that are included in and form a part of the specification illustrate exemplary embodiments, features, and aspects of the present disclosure, and are used to explain the principles of the present disclosure together with the specification.

[0044] Figure 1 Shown as the existing data quality inspection flow chart;

[0045] Figure 2 Shown as a schematic flow chart of the e-commerce platform data quality inspection method according to an embodiment of the present application;

[0046] Figure 3 Shown as a schematic flow chart of the first general timing task according to an embodiment of the present application;

[0047] Figure 4 Shown as a schematic flow chart of the second general timing task according to an embodiment of the present application;

[0048] Figure 5 Shown as a schematic flow chart of the third general timing task according to an embodiment of the present application;

[0049] Figure 6 Shown as a schematic structural diagram of the quality inspection rule table according to an embodiment of the present application;

[0050] Figure 7 Shown as a schematic structural diagram of the quality inspection task table, data extraction registration table, and data extraction platform data table according to an embodiment of the present application;

[0051] Figure 8 Shown as a schematic structural diagram of the internal system parameter data table according to an embodiment of the present application;

[0052] Figure 9 Shown as a schematic structural diagram of the snapshot table according to an embodiment of the present application;

[0053] Figure 10 Shown as a schematic structural diagram of the quality inspection result table according to an embodiment of the present application;

[0054] Figure 11 Shown as a schematic structural diagram of the quality inspection result failure table according to an embodiment of the present application;

[0055] Figure 12 Shown as a schematic implementation flow chart of the e-commerce platform data quality inspection method according to an embodiment of the present application. Detailed Description of the Specific Embodiment

[0056] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0057] As used herein, the term "exemplary" means "serving as an example, embodiment, or illustration". Any embodiment described as "exemplary" herein need not be construed as superior or better than other embodiments.

[0058] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0059] In the related art, when performing data quality inspection on an e-commerce data platform, each type of quality inspection item is usually developed independently, without separating the common quality inspection process, that is, each type of quality inspection item is coded separately. This means that for different quality inspection items, developers may need to repeatedly write a lot of similar codes, which not only increases the development workload but also correspondingly increases the cost of later management and maintenance. Moreover, there are as many polling timing tasks as there are types of quality inspection items. In a single-machine environment, when encountering a peak period of quality inspection and frequent creation of quality inspection tasks by store operations, the server will create corresponding threads for each timing task to process the tasks. The creation of a large number of threads in a short period will consume a large amount of system resources, such as CPU, memory, and the overhead of thread context switching, etc., resulting in tight resource pressure.

[0060] Therefore, the present application provides an e-commerce platform data quality inspection method. By extracting the common quality inspection processes of different quality inspection items and uniformly maintaining them, the basic processes can be implemented in a general framework, and each type of quality inspection item only needs to supplement the differential content, which can effectively avoid repetitive labor.

[0061] Embodiment 1

[0062] The embodiment of the present application provides an e-commerce platform data quality inspection method. Figure 2 It is a flowchart of the e-commerce platform data quality inspection method according to the embodiment of the present application, as Figure 2 shown. The method includes the following steps:

[0063] S100. Receive and execute a first general timing task, and send a data fetching request to the data fetching system for the data fetching system to fetch data from the e-commerce platform;

[0064] In this step, the main function of the first general timed task is to enable the quality inspection system to send a data fetching request to the data fetching system. The first general timed task has an independent thread pool, namely the first thread pool. Here, the quality inspection system is the system that executes the quality inspection process, and the data fetching system is the system that interfaces with the e-commerce platform to obtain platform data. It should be noted that during the data quality inspection process in the quality inspection system, different quality inspection items may send data fetching requests simultaneously. The independent thread pool can dynamically adjust the number of threads according to the task load, avoiding resource waste. That is, in the case of high concurrency, the first thread pool can quickly respond to the data fetching requests of multiple quality inspection items, improving the throughput of the system.

[0065] S200. Receive and execute the second general timed task, obtain the data fetching results stored in the data fetching system, and store them in the corresponding table.

[0066] In this step, the main function of the second general timed task is to store the data fetching results in the corresponding table, and the second general timed task also has an independent thread pool, namely the second thread pool. It should be noted that in this step, by storing the data fetching results in the corresponding table, the data persistence can be effectively achieved, avoiding data loss caused by system restart or failure, enabling the quality inspection tasks that have not reached the final state after the project restarts to continue execution without having to fetch data again. Moreover, since the data fetching results may come from different quality inspection items and the quantity is large, in this step, by setting up an independent second thread pool, the data can be stored in parallel, and the number of threads can also be dynamically adjusted according to the storage load, ensuring the performance and stability of the system.

[0067] S300. Receive and execute the third general timed task, perform quality inspection on the data fetching results with reference to the pre-configured quality inspection rules, and obtain the quality inspection results.

[0068] In this step, the main function of the third general timed task is to perform quality inspection on the data fetching results according to the quality inspection rules. The third general timed task also has an independent thread pool, namely the third thread pool, which can ensure the efficient execution of the quality inspection task and quickly process a large amount of quality inspection data. At the same time, the number of threads can also be dynamically adjusted according to the complexity and data volume of the quality inspection task, improving the performance and response speed of the system.

[0069] It can be seen from this that after the user creates a quality inspection task, the subsequent link is mainly triggered by three general timed tasks. Here, it should be noted that "general" means that the timed task can control all categories of quality inspection items, such as "quality inspection for the removal of products on the home page of the store" and "quality inspection for product expiration dates". Among them, the quality inspection item of "quality inspection for the removal of products on the home page of the store" means inspecting whether the status of the products on the home page of the store platform has been removed. If so, it is considered that the products in the store are risky data. The quality inspection item of "quality inspection for product expiration dates" means inspecting whether the expiration dates configured for the products on the platform conform to the actual expiration dates in the warehouse management system (WMS system). For example, if the worst expiration date corresponding to a product in the WMS system is January 1, 2025, then if the expiration date of this product on the platform is configured as February 1, 2025, it is considered that there is a risk for this product in this store.

[0070] That is to say, the method of this application can complete the quality inspection task for all categories of quality inspection items through the first general timed task, the second general timed task, and the third general timed task in sequence. That is, extract the common quality inspection process and maintain it uniformly. Each category of quality inspection item only needs to supplement the differential content.

[0071] Among them, the first general timed task is responsible for issuing a data extraction request, which is the basic step of quality inspection. Whether it is "quality inspection for the removal of products on the home page of the store" or "quality inspection for product expiration dates", data needs to be obtained from the corresponding data sources to perform subsequent quality inspections. In this embodiment, a unified data extraction request can be initiated for different quality inspection items based on the first general timed task, making the data extraction process more unified and standardized, centrally managing the data extraction requests, and improving the reliability of data extraction. The data ranges corresponding to different quality inspection items are different. For example, the data range issued to the data extraction system for "quality inspection for the removal of products on the home page of the store" is the products on the home page of the store platform, and the data range issued to the data extraction system for "quality inspection for product expiration dates" is all the products that can be sold in the store. Just adjust according to the above differential content.

[0072] Furthermore, the second general timed task of this embodiment is to store the data of the data extraction results in a table, which is also applicable to all quality inspection items. Because no matter what the specific content of the quality inspection item is, the obtained data needs to be properly stored for subsequent quality inspection. After the data extraction system completes the data extraction, it will call back the quality inspection system in the form of a file. The data structures of the file contents corresponding to different quality inspection items are different, and the data information is also different. Therefore, each quality inspection item needs to be customized to parse the corresponding file and store the parsed file in the data table of its own data extraction platform. That is, except for a few fixed attributes, the remaining attributes of the data tables of the data extraction platforms corresponding to different quality inspection items are different. For example, the file content returned by "quality inspection for the removal of products on the home page of the store" reflects the product ID, product name, and product status, while "quality inspection for product expiration dates" will reflect the product ID, product name, product expiration date, etc. Then these two quality inspection items will be stored in 2 data tables of the data extraction platform respectively.

[0073] Furthermore, the third general scheduled task of this embodiment is responsible for quality inspection of the acquired data collection result and quality inspection rules. All quality inspection items have their specific quality inspection rules, that is, the expected rules of the quality inspection items. For example, when "quality inspection of product removal on the homepage", the quality inspection rule is: the product status cannot be removed from the shelves; when "quality inspection of product expiration date", the quality inspection rule is: the expiration date of the products maintained on the default platform must be 1 month earlier than the worst expiration date of the actual warehouse, and the products of special categories must be 1 week earlier than the worst expiration date of the warehouse. The data collection results can be quality inspected through pre-configured quality inspection rules to determine whether the risk is within the acceptable range of store operations. That is, the content of quality inspection for different quality inspection items is different. The former quality checks whether the status of the product on the homepage is removed from the shelves, and the latter verifies whether the expiration date of the product is normal.

[0074] It should also be noted that the first general scheduled task, the second general scheduled task and the third general scheduled task of this embodiment are based on XXL-JOB, that is, the open source task scheduling center configures the routing strategy and execution time of the task to ensure that the entire quality inspection process is started on time and the reliable execution of the task is guaranteed.

[0075] If Figure 3 As shown, in some embodiments, receiving and executing the first general scheduled task, sending a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform, includes:

[0076] Receive the first general scheduled task, and obtain the first task list that satisfies the first requirement of the quality inspection task table and the data acquisition registration table;

[0077] Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the first task list and store them in the first blocking queue;

[0078] After using the threads in the first thread pool to obtain the corresponding quality inspection task, obtain the first distributed lock;

[0079] Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if it does, send a data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create a data acquisition record;

[0080] After storing the data acquisition record in the data acquisition registration table, the first distributed lock is released.

[0081] In this embodiment, the routing strategy of XXL-JOB is shard broadcast. When the first general scheduled task is triggered, the shard information is passed to the first general scheduled task. Here, it should be noted that the shard information means that if a total of 3 machines are registered to XXL-JOB, and the current machine is the first connected, then the returned shard information can be the current machine serial number 0 and the total number of machines 3. When a certain value is sharded, the remainder can be obtained by dividing the value by the total number of machines to determine whether the remainder is equal to the current machine serial number. For example, if the value to be sharded is 3, then 3 / 3 has a remainder of 0, and it can be executed by the current machine. If the value of the shard is 4, then 4 / 3 has a remainder of 1, and it cannot be executed by the current machine.

[0082] The following will explain the first general timing task in detail.

[0083] 1. After receiving the first general scheduled task, the quality inspection system initially obtains the first task list, where the tasks in the first task list need to meet the first requirements of the quality inspection task list and the data acquisition registration form at the same time.

[0084] 2. Use store information, such as store ID, as the shard value for sharding to ensure that each machine only processes the quality inspection task ID that meets the sharding result and carries the quality inspection item type. That is, only the quality inspection task list that can be executed by the machine is filtered out, so that tasks can be assigned to different machines according to the business attributes of the store to achieve parallel processing of tasks.

[0085] 3. Put the filtered quality inspection task data into the first blocking queue. It should be noted that the blocking queue of this embodiment provides a buffer area for task processing, which can play a buffering and coordination role when the task generation and processing speed do not match.

[0086] 4. The thread of the first thread pool corresponding to the first general scheduled task obtains the quality inspection task from the blocking queue to realize concurrent processing of tasks.

[0087] 5. Obtain the first distributed lock. It should be emphasized that in order to avoid the same quality inspection task being executed by multiple threads, this embodiment obtains the first distributed lock, thereby ensuring that the same quality inspection task can only be executed by one thread at the same time. Among them, the first distributed lock of this embodiment is set based on the task ID and the quality inspection item type.

[0088] 6. Further determine whether the quality inspection task meets the data collection conditions;

[0089] If the quality inspection task meets the data acquisition conditions, the quality inspection system will initiate a data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create corresponding data acquisition records;

[0090] If the quality inspection task does not meet the data acquisition conditions, the quality inspection task will be skipped.

[0091] Here, it should be noted that in order to avoid the task not meeting the data extraction conditions due to changes such as the task status, it is necessary to judge again whether the quality inspection task meets the data extraction conditions to ensure that the task still meets the execution requirements at the current moment. The specific logic is as follows: Judge again whether the task ID meets the conditions corresponding to step 1 above.

[0092] 7. Register the data extraction records of the data extraction system and store them in the data extraction registration form. At this time, the data extraction status is the initial state.

[0093] 8. Release the first distributed lock.

[0094] It should be noted that before executing the first general timed task, the quality inspection system needs to first receive the quality inspection items and corresponding quality inspection rules configured by the user, create a quality inspection task, and notify the user that the quality inspection task creation is completed.

[0095] Among them, as Figure 6 shown, the content of the quality inspection rule table includes the primary key ID, store ID, and specific rule content, which specifically depends on the quality inspection item;

[0096] As Figure 7 shown, the content of the quality inspection task table includes the task creator, task creation time, task completion time, task ID, quality inspection item type, store, task status, total number of data extractions, current data extraction serial number, number of successes, number of failures, and number of risks; among them, the task status includes to be executed, executing, completed, and failed;

[0097] The content of the data extraction registration form includes the task creation time, primary key ID, task ID, data extraction return ID, current data extraction serial number, whether the data extraction result has been stored, data extraction result, data extraction status, and data extraction callback time; among them, the data extraction status includes the initial state, completed, and failed, and if the status is failed, the data extraction result represents the failure content.

[0098] In some embodiments, the first requirement is that the task status in the quality inspection task table is to be executed and the total number of data extractions is greater than 0; and the corresponding task ID in the data extraction registration form is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same current data extraction serial number as the quality inspection task table.

[0099] In this embodiment, when initially obtaining the first task list, the requirements of the quality inspection task table and the data extraction registration form need to be considered simultaneously, that is, the tasks in the first task list need to meet the first requirement. Specifically, it is required that the task status in the quality inspection task table is to be executed and the number of data extractions is greater than 0, ensuring that only the tasks that need to perform data extraction will be included in the scope to be processed, and filtering out information such as task IDs, quality inspection item types, stores, and the current data extraction sequence numbers that meet the conditions. Subsequently, it is compared with the data extraction registration form to ensure that the task IDs in the data extraction registration form are the same as those in the quality inspection task table, and there are no records with the same current data extraction sequence number for these tasks in the data extraction registration form, ensuring that each task will not be processed repeatedly under a specific data extraction sequence number.

[0100] As Figure 4 shown, in some embodiments, a second general timing task is received and executed, the data extraction results stored in the data extraction system are obtained, and they are stored in a corresponding table, including:

[0101] Receiving the second general timing task and obtaining a second task list that simultaneously meets the corresponding second requirements of the quality inspection task table and the data extraction registration form;

[0102] Using the corresponding store information for sharding filtering, so that each server filters out the quality inspection tasks that can be correspondingly executed from the second task list and stores them in the second blocking queue;

[0103] After obtaining the corresponding quality inspection task by using the threads in the second thread pool, obtain the second distributed lock;

[0104] Judge whether the quality inspection task meets the corresponding data extraction conditions, and when it meets, initiate a data extraction result acquisition request according to the data extraction return ID to obtain the data extraction results stored in the data extraction system;

[0105] After storing the data extraction results in the corresponding data extraction platform data table, update the data extraction registration form and release the second distributed lock.

[0106] In this embodiment, by storing the data extraction results in a table, on the one hand, the original data can be saved for traceability and review when needed, and on the other hand, it also provides a fast data access method for subsequent quality inspection steps.

[0107] Next, a detailed description will be given for the second general timing task.

[0108] 1. After the quality inspection system receives the second general timing task, it obtains the second task list, where the tasks in the second task list need to simultaneously meet the second requirements of the quality inspection task table and the data extraction registration form.

[0109] It should be noted that when triggering the second general timing task, the sharding information is also passed to the second general timing task.

[0110] 2. Use store information, such as the store ID, as the sharding value for sharding to ensure that each machine only processes the quality inspection task IDs that match the sharding results and carry the quality inspection item types.

[0111] 3. Put the filtered quality inspection task data into the second blocking queue.

[0112] 4. Threads in the second thread pool corresponding to the second general timing task obtain quality inspection tasks from the blocking queue.

[0113] 5. Obtain the second distributed lock.

[0114] 6. Further determine whether the quality inspection task meets the data fetching conditions.

[0115] If the quality inspection task meets the data fetching conditions, initiate a data fetching result acquisition request to the data fetching system according to the data fetching return ID to obtain the data fetching result in the data fetching system.

[0116] If the quality inspection task does not meet the data fetching conditions, skip the quality inspection task.

[0117] 7. Store the obtained data fetching result in the data fetching platform data table.

[0118] 8. Update the data fetching registration form. Specifically, change whether the data fetching result in the data fetching registration form has been stored to yes.

[0119] 9. Determine whether the quality inspection task still needs to fetch data. Specifically, it is necessary to determine whether the current data fetching sequence number of the task is the same as the number of data fetching times in the quality inspection task table. If they are not the same, it is considered that the task still needs to continue fetching data. At this time, it is necessary to update the current data fetching sequence number in the quality inspection task table by 1. If they are the same, update the task status in the quality inspection task table to in execution.

[0120] 10. Release the second distributed lock.

[0121] In some embodiments, the second requirement is that the task status in the quality inspection task table is to be executed and the total number of data fetches is greater than 0; and the corresponding task ID and the current data fetching sequence number in the data fetching registration form are the same as the corresponding task ID and the current data fetching sequence number in the quality inspection task table, and the data fetching status is completed and the data fetching result has not been stored.

[0122] In this embodiment, for the determination of tasks in the second task list, the second requirements corresponding to both the quality inspection task list and the data extraction registration form need to be satisfied simultaneously. Specifically, for the quality inspection task list, it is required that the task status in the quality inspection task list is to be executed and the total number of data extractions is greater than 0. When the above conditions are met, the corresponding task ID, quality inspection item type, store, and current data extraction serial number can be obtained. Among them, the task ID is the unique identifier of each quality inspection task, used to track and manage tasks in the system, and the quality inspection item type determines the specific content and method of quality inspection. Also, in this embodiment, it is required that the task ID and the current data extraction serial number in the data extraction registration form are the same as those in the quality inspection task list to ensure that the obtained tasks are correct. At the same time, the data extraction status is limited to completed and the data extraction result has not been stored, ensuring that only those tasks that have completed the data extraction operation but have not stored the results are obtained. Thus, unnecessary tasks are avoided from being processed or completed tasks from being processed repeatedly, effectively improving the system efficiency and ensuring the accuracy of the quality inspection results.

[0123] As Figure 5 shown, in some embodiments, a third general timing task is received and executed, and the data extraction result is quality inspected with reference to pre-configured quality inspection rules to obtain a quality inspection result, including:

[0124] Receiving a third general timing task and obtaining a third task list that meets the third requirements corresponding to the quality inspection task list;

[0125] Using the corresponding store information for sharding and filtering, so that each server filters out the quality inspection tasks that can be correspondingly executed from the third task list and stores them in a third blocking queue;

[0126] After obtaining the corresponding quality inspection task using a thread in the third thread pool, obtaining a third distributed lock;

[0127] Judging whether the quality inspection task meets the corresponding data extraction conditions, and when it meets, storing the snapshot data in the snapshot table and traversing the snapshot table to obtain the unprocessed quality inspection dimension data IDs;

[0128] Obtaining the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID for quality inspection.

[0129] In this embodiment, quality inspecting the data extraction result in combination with the quality inspection rules is a key link to ensure data quality. If the quality inspection result is risky, the risk result needs to be promptly fed back to the user so that measures can be taken to improve data quality. At the same time, relying on the introduced snapshot table, for some quality inspection items with a relatively long quality inspection time, after the project restarts, the unprocessed quality inspection data can continue to be executed downward, avoiding the loss of the quality inspection progress of the quality inspection tasks.

[0130] In some embodiments, the third requirement is that the task status in the quality inspection task list is in execution.

[0131] In this embodiment, when executing the third general timing task, among the obtained third task list, the tasks need to meet the requirement that the task status in the quality inspection task table is in execution. When the task changes from the to-be-executed state to the in-execution state, it means that the first general timing task and the second general timing task have been processed, ensuring that each task can proceed according to the preset process and avoiding the situation of task omission or duplicate processing.

[0132] In some embodiments, after obtaining the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and performing quality inspection, it further includes: storing the quality inspection result in the quality inspection result table or the quality inspection result failure table, and updating the snapshot table.

[0133] In this embodiment, as Figure 9 shown, the content of the snapshot table includes the primary key ID, task ID, whether it has been processed, and quality inspection dimension data ID;

[0134] As Figure 10 shown, the content of the quality inspection result table includes the creation time, and the primary key ID, task ID, quality inspection dimension ID, the remaining data of the quality inspection dimension, and whether there is a risk, rule data, e-commerce platform data, and internal system reference data;

[0135] As Figure 11 shown, the content of the quality inspection result failure table includes the creation time, and the primary key ID, task ID, quality inspection dimension data ID, and the reason for failure;

[0136] The above-mentioned quality inspection dimension data ID, such as the commodity ID, and the specific fields depend on the quality inspection items. Other data of the quality inspection dimension, such as the commodity name, code, connection, etc., and the specific fields depend on the quality inspection items.

[0137] Next, a detailed description will be given of the third general timing task.

[0138] 1. After the quality inspection system receives the third general timing task, it obtains the third task list. Among them, the tasks in the third task list need to meet the third requirement of the quality inspection task table; that is, the tasks need to meet the requirement that the task status in the quality inspection task table is in execution, and obtain the corresponding task ID, quality inspection item type, and store;

[0139] It should be noted that when triggering the third general timing task, the sharding information is also passed to the third general timing task.

[0140] 2. Use the store information, such as the store ID, as the sharding value for sharding processing to ensure that each machine only processes the quality inspection task IDs that meet the sharding results and carries the quality inspection item type.

[0141] 3. Put the filtered quality inspection task data into the third blocking queue.

[0142] 4. Threads in the third thread pool corresponding to the third general timed task obtain quality inspection tasks from the blocking queue.

[0143] 5. Obtain the third distributed lock.

[0144] 6. Further determine whether the quality inspection task meets the data fetching conditions.

[0145] If the quality inspection task meets the data fetching conditions, store the snapshot data.

[0146] If the quality inspection task does not meet the data fetching conditions, skip this quality inspection task.

[0147] Here, it should be noted that the snapshot data table stores the quality inspection dimension data ID. For example, for the "product expiration quality inspection" quality inspection dimension, the quality inspection dimension is the "product dimension", and the quality inspection dimension data ID is the product ID. The significance of the snapshot table is that the quality inspection time for some quality inspection items is relatively long, to avoid repeated quality inspection of data with the same quality inspection dimension when restarting.

[0148] It should also be noted that the quality inspection dimension data ID in this embodiment can be directly obtained from the quality inspection rules or jointly determined based on e-commerce platform data and quality inspection rules, depending on the specific quality inspection item. And if the snapshot table already has the data corresponding to this task ID, it is considered that the task was stored during the previous execution.

[0149] 7. Traverse the snapshot table to obtain the unprocessed quality inspection dimension data IDs.

[0150] 8. Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID for quality inspection.

[0151] 9. Store the quality inspection results in the quality inspection result table or the quality inspection result failure table, and update the snapshot table and the quality inspection task table.

[0152] Among them, after a single piece of quality inspection dimension data is completed with quality inspection, the quality inspection result will be stored in the quality inspection result table. If it fails, it will be stored in the quality inspection result failure table. At the same time, update the status recorded in the snapshot table to processed. When all data in the snapshot table is processed, update information such as the number of successes, failures, and risks in the quality inspection task table.

[0153] 10. Notify the store operation of the quality inspection results.

[0154] 11. Release the third distributed lock.

[0155] In some of these embodiments, a third general timing task is received and executed to perform quality inspection on the data fetching result with reference to pre-configured quality inspection rules to obtain a quality inspection result. It further includes: determining whether to request the internal system, and when necessary, obtaining reference data from the internal system and storing it in the internal system reference data table as reference data during the quality inspection process.

[0156] In this embodiment, it can be determined whether to request the internal system according to the actual situation to obtain reference data in the internal system. The specific reason is that for some simple quality inspection items, reference data intervention in the quality inspection is not required, that is, this part of the data does not need to be obtained by requesting the internal system. For another part of the data, there may be a situation where the project is restarted, that is, the reference data has been obtained during the previous execution of this quality inspection task. Therefore, it is also not necessary to request the internal system, and it can be judged by checking the internal system reference data table. Here, it should be noted that the internal system includes a warehouse management system (WMS system), an order management system (OMS system), and a commodity information maintenance system. The internal system reference data refers to data that can be obtained within the company. For example, the above-mentioned WMS system, OMS system, and commodity information maintenance system.

[0157] Among them, the internal system reference data tables and data fetching platform data tables corresponding to different quality inspection items may be different. As Figure 8 shown, the internal system reference data table includes a first internal system reference data table and a second internal system reference data table. That is, the number of such tables is not fixed, but the content includes a primary key ID, a task ID, and other fields depending on the quality inspection item. The content of the data fetching platform data table includes a primary key ID, a task ID, and other fields depending on the data fetched back. That is, there are common attributes, namely the task ID. For example, the data structures of "quality inspection for product off-shelf on the home page" and "quality inspection for product expiration date" are different. Therefore, the data fetched back will be respectively stored in their respective data fetching platform data tables, such as a first data fetching platform data table and a second data fetching platform data table. And because the "quality inspection for product expiration date" needs to be compared with the expiration date in the warehouse, warehouse data will also be obtained from the WMS system and stored in the internal system reference data table.

[0158] Specifically, after determining whether the quality inspection task meets the data fetching conditions, if it does, it is further determined whether to request the internal system. If so, the quality inspection system obtains reference data from the internal system and saves the reference data, that is, stores it in the internal system reference data table, and further executes the step of storing snapshot data; if not, the step of storing snapshot data is directly executed. And during subsequent quality inspections, quality inspection rules data, e-commerce platform data, and internal system reference data corresponding to the quality inspection dimension data ID can be obtained for quality inspection, and the quality inspection results are stored in the quality inspection result table or the quality inspection failure table.

[0159] It should be noted that after obtaining the data table of the data extraction platform and the reference data table of the internal system in this embodiment, the reason for storing the data of the e-commerce platform and the fields of the reference data table of the internal system in the quality inspection result table is that storing them in the data table of the data extraction platform and the reference data table of the internal system is on the one hand for data traceability in the quality inspection system, serving as two basic data from different sources participating in quality inspection, and on the other hand, in the actual use process, there may be a demand for the store operation to export such basic data based on the quality inspection task dimension. And the two fields stored in the quality inspection result table are for the convenience of the operation to view the quality inspection results based on the quality inspection task and the quality inspection data ID.

[0160] As Figure 12 shown, the following will provide a complete embodiment for the data quality inspection process and elaborate in detail.

[0161] 1. Receive the quality inspection items and quality inspection rules configured by the user;

[0162] 2. Receive the quality inspection task created by the user;

[0163] 3. Return the result of the completed quality inspection task creation to the user;

[0164] 4. Receive and execute the general timed task A of XXL-JOB, that is, the first general timed task, to obtain the task list to be executed and for which no data extraction request has been issued this time;

[0165] 5. Initiate a data extraction task to the data extraction system;

[0166] 6. Register the current data extraction request;

[0167] 7. The data extraction system extracts data from the e-commerce platform;

[0168] 8. The data extraction system stores the data extraction result;

[0169] 9. Update the current data extraction status to completed;

[0170] 10. Receive and execute the general timed task B of XXL-JOB, that is, the second general timed task, to obtain the task list to be executed and for which the data extraction has been completed this time but the result has not been stored;

[0171] 11. Obtain the corresponding data extraction result from the data extraction system;

[0172] 12. Store the data extraction result in the table;

[0173] 13. Determine whether the task needs to continue data extraction. If not, transition to in execution; if so, wait for the trigger of task A;

[0174] 14. Receive and execute the general timed task C, that is, the third general timed task, to obtain the task list in execution;

[0175] 15. Obtain reference data from the internal system;

[0176] 16. Store the reference data in a table;

[0177] 17. Conduct data quality inspection;

[0178] 18. Notify users of the early warning risk results.

[0179] Thus, the method of this application changes from each type of quality inspection item initiating a timed task polling separately to three fixed general timed tasks, saving thread expenses. Moreover, each of the three timed tasks has an independent thread pool. On the one hand, the number of threads corresponding to the thread pool can be controlled manually, reducing resource consumption; on the other hand, the independent thread pool means that the timed tasks will not be blocked due to thread resources, effectively improving the processing capacity and response speed of the system. At the same time, dispersing the quality inspection task data of multiple stores to different machines for execution can relieve the execution pressure on a single machine and improve the execution speed. And storing the data information of key steps in a table enables the quality inspection tasks that have not reached the final state to continue execution after the project restarts.

[0180] Embodiment 2

[0181] Second, the embodiment of this application provides an e-commerce platform data quality inspection system, including:

[0182] The first unit is configured to receive and execute the first general timed task, and send a data fetching request to the data fetching system for the data fetching system to fetch data from the e-commerce platform;

[0183] The second unit is configured to receive and execute the second general timed task, obtain the data fetching result stored in the data fetching system, and store it in a corresponding table;

[0184] The third unit is configured to receive and execute the third general timed task, conduct quality inspection on the data fetching result with reference to the pre-set quality inspection rules, and obtain the quality inspection result;

[0185] Among them, the first general timed task, the second general timed task, and the third general timed task are configured based on XXL-JOB and all have independent thread pools.

[0186] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiment and optional implementation manners, and will not be repeated here.

[0187] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of these technical features do not conflict, they should all be considered as the scope recorded in this specification.

[0188] The embodiments described above merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

[0189] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those of ordinary skill in the technical field to which this application belongs. The words such as "a", "one", "the" and the like involved in this application do not indicate a quantity limitation and may represent a singular or plural number. The terms "comprising", "including", "having" and any variations thereof involved in this application are intended to cover non-exclusive inclusion. The words such as "connected", "linked", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more, and the "and / or" describes the association relationship of the associated objects, indicating that three relationships can exist. The character " / " generally represents an "or" relationship between the front and rear associated objects. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific order for the objects.

[0190] The above embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications or improvements can still be made, and these all fall within the protection scope of this application. Therefore, the protection scope of this application patent shall be subject to the appended claims.

Claims

1. A data quality inspection method for an e-commerce platform, characterized in that: The steps include: Receive and execute a first general scheduled task, and send a data acquisition request to a data acquisition system, so that the data acquisition system can acquire data from an e-commerce platform; wherein the first general scheduled task is a data acquisition request uniformly initiated by different quality inspection items; Receive and execute the second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in the corresponding table; Receive and execute the third general scheduled task, perform quality inspection on the data acquisition result with reference to the pre-configured quality inspection rules, and obtain the quality inspection result; and all the quality inspection items have specific quality inspection rules; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

2. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the first general scheduled task, and sending a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform, includes: Receive the first general scheduled task, and obtain a first task list that satisfies the first requirement corresponding to the quality inspection task table and the data acquisition registration table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the first task list and store them in the first blocking queue; After using the thread in the first thread pool to obtain the corresponding quality inspection task, obtain the first distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, send the data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create a data acquisition record; After storing the data retrieval record in the data retrieval registration table, the first distributed lock is released.

3. The e-commerce platform data quality inspection method according to claim 2 is characterized in that: The first requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisitions is greater than 0; and the corresponding task ID in the data acquisition registration table is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same current serial number as the quality inspection task table.

4. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the second general timing task, obtaining the data acquisition result stored in the data acquisition system, and storing it in the corresponding table, includes: Receive the second general scheduled task, and obtain a second task list that satisfies the second requirement corresponding to the quality inspection task table and the data acquisition registration table; Using the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed accordingly from the second task list and store them in the second blocking queue; After using the thread in the second thread pool to obtain the corresponding quality inspection task, obtain the second distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, initiate a data acquisition result acquisition request according to the data acquisition return ID to obtain the data acquisition results stored in the data acquisition system; After storing the data acquisition result in the corresponding data acquisition platform data table, the data acquisition registration table is updated and the second distributed lock is released.

5. The e-commerce platform data quality inspection method according to claim 4 is characterized in that: The second requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisition is greater than 0; and the corresponding task ID and current number of data acquisition in the data acquisition registration table are the same as the corresponding task ID and current number of data acquisition in the quality inspection task table, and the data acquisition status is completed, and the data acquisition result is not stored.

6. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the third general scheduled task, and performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, includes: Receive the third general scheduled task, and obtain a third task list that meets the third requirement corresponding to the quality inspection task table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the third task list and store them in the third blocking queue; After using the threads in the third thread pool to obtain the corresponding quality inspection task, obtain the third distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, store the snapshot data in the snapshot table, and traverse the snapshot table to obtain the unprocessed quality inspection dimension data ID; Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and perform quality inspection.

7. The e-commerce platform data quality inspection method according to claim 6, characterized in that: The third requirement is that the task status in the quality inspection task table is in execution.

8. The e-commerce platform data quality inspection method according to claim 6, characterized in that: The receiving and executing the third general scheduled task, and performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, further includes: Determine whether it is necessary to request the internal system, and obtain reference data of the internal system when necessary, and store it in the internal system reference data table as reference data in the quality inspection process.

9. The e-commerce platform data quality inspection method according to any one of claims 6 to 8, characterized in that: The obtaining of the quality inspection rule data and the e-commerce platform data corresponding to the quality inspection dimension data ID and performing quality inspection further includes: The quality inspection result is stored in a quality inspection result table or a quality inspection result failure table, and the snapshot table is updated.

10. An e-commerce platform data quality inspection system, characterized in that: include: The first unit is configured to receive and execute a first general scheduled task, and send a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform; wherein the first general scheduled task is a data acquisition request uniformly initiated by different quality inspection items; The second unit is configured to receive and execute a second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in a corresponding table; The third unit is configured to receive and execute a third general timed task, perform quality inspection on the data acquisition result with reference to a preset quality inspection rule, and obtain a quality inspection result; and all the quality inspection items have specific quality inspection rules; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

Images