E-commerce platform data quality inspection method and system

Abstract

The invention relates to an e-commerce platform data quality inspection method and system, and the method comprises the steps: receiving and executing a first universal timed task, and transmitting an access request to an access system, so as to enable the access system to access data from an e-commerce platform; receiving and executing a second universal timed task, obtaining an access result stored in the access system, and storing the access result in a corresponding falling table; receiving and executing a third universal timed task, and carrying out quality inspection on the access result by referring to a pre-configured quality inspection rule to obtain a quality inspection result; wherein the first universal timed task, the second universal timed task and the third universal timed task are obtained based on XXL-JOB configuration and are all provided with independent thread pools. The original mode that each class of quality inspection items initiate timed task polling is changed into the mode that three fixed universal timed tasks are used, and thread expenditure is saved. And the independent thread pool means that the timed tasks are not blocked due to thread resources, so that the processing capacity and the response speed of the system are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality detection, and in particular to an e-commerce platform data quality detection method and system. Background Art

[0002] The data maintenance of e-commerce platforms is mainly the responsibility of store operations. For example, product inventory information, expiration information, price information, on-shelf and off-shelf operations, or coupon information all need to be maintained. However, in the daily operation and maintenance process, human errors are likely to occur, causing operational accidents. Therefore, it is usually necessary to use a quality inspection platform to avoid the above problems, that is, to compare the e-commerce platform data with the reference data to check whether the difference data meets the quality inspection rules. If not, it is considered that the e-commerce platform is at risk, otherwise, there is no risk.

[0003] At present, the implementation of quality inspection functions is usually carried out in a single-machine environment, and the scheduled tasks are defined based on the annotation (@Scheduled annotation) for scheduled task control provided by the lightweight open source framework (Spring framework). Figure 1 As shown, the entire quality inspection process first requires the store operation to create the quality inspection rules for the specified store in the specified quality inspection items before the quality inspection system, and initiate the store quality inspection task in the quality inspection system. Among them, the quality inspection system notifies the store operation after the quality inspection task is successfully created, and regularly polls the task table to obtain new quality inspection tasks, and initiates a data acquisition request to the data acquisition system according to the quality inspection items corresponding to the task, so that the data acquisition system can acquire data from the e-commerce platform. Furthermore, after the data acquisition system completes the data acquisition from the e-commerce platform, it carries the data acquisition result, that is, the e-commerce platform data calls back to the quality inspection system, and calls up the next step of the quality inspection system, that is, the quality inspection system compares the e-commerce platform data with the quality inspection rule data, and synchronizes the obtained risk data to the store operation.

[0004] However, in the above quality inspection process, each type of quality inspection item is developed independently. There are as many polling timed tasks as there are types of quality inspection items. When encountering the peak period of quality inspection in a single-machine environment, store operations will frequently create quality inspection tasks. At this time, the machine will create a large number of threads in a short period of time, resulting in large resource consumption. Summary of the invention

[0005] In view of this, an embodiment of the present application provides an e-commerce platform data quality inspection method and system to solve the above problems.

[0006] In a first aspect, an embodiment of the present application provides an e-commerce platform data quality inspection method, comprising the following steps: Receive and execute the first general scheduled task, and send a data acquisition request to the data acquisition system so that the data acquisition system can acquire data from the e-commerce platform; Receive and execute the second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in the corresponding table; Receive and execute the third general scheduled task, perform quality inspection on the data acquisition result according to the pre-configured quality inspection rules, and obtain the quality inspection result; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

[0007] In some embodiments, the receiving and executing the first general scheduled task, and sending a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform, includes: Receive the first general scheduled task, and obtain a first task list that satisfies the first requirement corresponding to the quality inspection task table and the data acquisition registration table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the first task list and store them in the first blocking queue; After using the thread in the first thread pool to obtain the corresponding quality inspection task, obtain the first distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, send the data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create a data acquisition record; After storing the data retrieval record in the data retrieval registration table, the first distributed lock is released.

[0008] In some embodiments, the first requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisitions is greater than 0; and the corresponding task ID in the data acquisition registration table is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same current serial number of data acquisition as in the quality inspection task table.

[0009] In some embodiments, the receiving and executing the second general timing task, obtaining the data acquisition result stored in the data acquisition system, and storing it in the corresponding table includes: Receive the second general scheduled task, and obtain a second task list that satisfies the second requirement corresponding to the quality inspection task table and the data acquisition registration table; Using the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed accordingly from the second task list and store them in the second blocking queue; After using the thread in the second thread pool to obtain the corresponding quality inspection task, obtain the second distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, initiate a data acquisition result acquisition request according to the data acquisition return ID to obtain the data acquisition results stored in the data acquisition system; After storing the data acquisition result in the corresponding data acquisition platform data table, the data acquisition registration table is updated and the second distributed lock is released.

[0010] In some embodiments, the second requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisition is greater than 0; and the corresponding task ID and current number of data acquisition in the data acquisition registration table are the same as the corresponding task ID and current number of data acquisition in the quality inspection task table, and the data acquisition status is completed, and the data acquisition result is not stored.

[0011] In some embodiments, the receiving and executing the third general scheduled task, and performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, include: Receive the third general scheduled task, and obtain a third task list that meets the third requirement corresponding to the quality inspection task table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the third task list and store them in the third blocking queue; After using the threads in the third thread pool to obtain the corresponding quality inspection task, obtain the third distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, store the snapshot data in the snapshot table, and traverse the snapshot table to obtain the unprocessed quality inspection dimension data ID; Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and perform quality inspection.

[0012] In some embodiments, the third requirement is that the task status in the quality inspection task table is in execution.

[0013] In some embodiments, the receiving and executing the third general scheduled task, performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, further includes: Determine whether it is necessary to request the internal system, and obtain reference data of the internal system when necessary, and store it in the internal system reference data table as reference data in the quality inspection process.

[0014] In some embodiments, the obtaining of the quality inspection rule data and the e-commerce platform data corresponding to the quality inspection dimension data ID and the quality inspection further includes: The quality inspection result is stored in a quality inspection result table or a quality inspection result failure table, and the snapshot table is updated.

[0015] In a second aspect, an embodiment of the present application provides an e-commerce platform data quality inspection system, including: The first unit is configured to receive and execute a first general scheduled task, and send a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform; The second unit is configured to receive and execute a second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in a corresponding table; The third unit is configured to receive and execute a third general scheduled task, perform quality inspection on the data acquisition result with reference to a preset quality inspection rule, and obtain a quality inspection result; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

[0016] Technical effect of the present invention: The present invention changes the original scheduled task polling initiated by each type of quality inspection item to three fixed general scheduled tasks, which saves thread expenses. The purpose of issuing data acquisition requests, storing data acquisition results, and performing quality inspection on data acquisition results is achieved by executing the first general scheduled task, the second general scheduled task, and the third general scheduled task in sequence. In addition, each of the three scheduled tasks has an independent thread pool. On the one hand, the number of threads corresponding to the thread pool can be controlled manually, reducing resource consumption; on the other hand, the independent thread pool means that the scheduled tasks will not be blocked due to thread resources, effectively improving the processing capacity and response speed of the system. In addition, the data information of the key steps is stored in the table, so that the quality inspection tasks that have not reached the final state can continue to be executed after the project is restarted.

[0017] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0019] Figure 1 Shown is the existing data quality inspection flow chart; Figure 2 A schematic diagram showing a flow chart of an e-commerce platform data quality inspection method according to an embodiment of the present application is shown; Figure 3 A schematic diagram showing a process flow of a first general timing task according to an embodiment of the present application is shown; Figure 4 A schematic diagram showing a flow chart of a second general timing task according to an embodiment of the present application is shown; Figure 5 A schematic diagram showing a process flow of a third general timing task according to an embodiment of the present application is shown; Figure 6 A schematic diagram showing the structure of a quality inspection rule table according to an embodiment of the present application is shown; Figure 7 A schematic diagram showing the structure of a quality inspection task table, a data acquisition registration table, and a data acquisition platform data table in an embodiment of the present application is shown; Figure 8 A schematic diagram showing the structure of an internal system parameter data table according to an embodiment of the present application is shown; Fig. 9 A schematic diagram showing the structure of a snapshot table according to an embodiment of the present application is shown; Fig.10 A schematic diagram showing the structure of a quality inspection result table of an embodiment of the present application is shown; Fig.11 A schematic diagram showing the structure of a quality inspection result failure table according to an embodiment of the present application is shown; Fig.12 The following is a schematic diagram of the implementation process of the e-commerce platform data quality inspection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0021] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0022] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0023] In the related art, when conducting data quality inspection on e-commerce data platforms, each type of quality inspection item is usually developed independently without separating out the common quality inspection process, that is, each type of quality inspection item is coded separately. This means that for different quality inspection items, developers may need to repeatedly write a lot of similar codes, which not only increases the development workload, but also increases the cost of later management and maintenance. In addition, there are as many polling timed tasks as there are types of quality inspection items. In a single-machine environment, if there is a peak period of quality inspection, when store operations frequently create quality inspection tasks, the server will create corresponding threads for each timed task to handle the tasks. The creation of a large number of threads in a short period of time will consume a lot of system resources, such as CPU, memory, and thread context switching overhead, which will lead to tight resource pressure.

[0024] Therefore, this application provides an e-commerce platform data quality inspection method, which extracts common quality inspection processes for different quality inspection items and maintains them in a unified manner. The basic process can be implemented in a common framework, and each type of quality inspection item only needs to be supplemented with differentiated content, which can effectively avoid duplication of work.

[0025] Embodiment 1 The present application embodiment provides a method for quality inspection of e-commerce platform data. Figure 2 is a flow chart of the e-commerce platform data quality inspection method according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps: S100, receiving and executing a first general scheduled task, and sending a data acquisition request to a data acquisition system, so that the data acquisition system can acquire data from an e-commerce platform; In this step, the main function of the first general scheduled task is to enable the quality inspection system to send a data retrieval request to the data retrieval system, and the first general scheduled task has an independent thread pool, namely the first thread pool. Among them, the quality inspection system is a system that executes the quality inspection process, and the data retrieval system is a system that connects to the e-commerce platform to obtain platform data. It should be noted that during the data quality inspection process of the quality inspection system, different quality inspection items may send data retrieval requests at the same time. The independent thread pool can dynamically adjust the number of threads according to the task load to avoid waste of resources. That is, in the case of high concurrency, the first thread pool can quickly respond to data retrieval requests for multiple quality inspection items and improve the throughput of the system.

[0026] S200, receiving and executing a second general timing task, obtaining the data acquisition result stored in the data acquisition system, and storing it in a corresponding table; In this step, the second general scheduled task is mainly used to store the data of the number acquisition results in a table, and the second general scheduled task also has an independent thread pool, namely the second thread pool. It should be noted that in this step, by storing the number acquisition results in a table, the data persistence can be effectively achieved, avoiding data loss caused by system restart or failure, so that the quality inspection tasks that have not reached the final state after the project restart can continue to be executed without re-acquiring data. Not only that, since the number acquisition result data may come from different quality inspection items and the number is large, this step can store the data in parallel by setting an independent second thread pool, and can also dynamically adjust the number of threads according to the storage load to ensure the performance and stability of the system.

[0027] S300, receiving and executing the third general scheduled task, performing quality inspection on the data acquisition result according to the pre-configured quality inspection rules, and obtaining the quality inspection result; In this step, the main function of the third general scheduled task is to perform quality inspection on the data acquisition results according to the quality inspection rules, and the third general scheduled task also has an independent thread pool, namely the third thread pool, which can ensure the efficient execution of the quality inspection task and quickly process a large amount of quality inspection data. At the same time, the number of threads can be dynamically adjusted according to the complexity of the quality inspection task and the amount of data, so as to improve the performance and response speed of the system.

[0028] It can be seen that after the user creates the quality inspection task, the subsequent links of the method of this application are mainly triggered by three general scheduled tasks. Here, it should be noted that general means that the scheduled task can control all categories of quality inspection items, such as "quality inspection of product removal from the homepage", "quality inspection of product expiration date", etc. Among them, the quality inspection item "quality inspection of product removal from the homepage" refers to whether the status of the product on the homepage of the quality inspection store platform has been removed from the shelves. If so, the product of the store is considered to be risky data; the quality inspection item "quality inspection of product expiration date" refers to whether the expiration date configured for the product on the quality inspection platform is consistent with the actual expiration date of the warehouse in the warehouse management system (WMS system). For example, the worst expiration date corresponding to the medium-quality product in the WMS system is 2025-01-01. If the expiration date of the product on the platform is configured as 2025-02-01, it is considered that the product in the store is at risk.

[0029] That is to say, the method of the present application can complete the quality inspection tasks for all categories of quality inspection items through the first general scheduled task, the second general scheduled task and the third general scheduled task in sequence. That is, the common quality inspection process is extracted and maintained uniformly, and each type of quality inspection item only needs to supplement the differentiated content.

[0030] Among them, the first general scheduled task is responsible for issuing data retrieval requests, which is the basic step of quality inspection. Whether it is "quality inspection of product removal from the homepage" or "quality inspection of product expiration date", data needs to be obtained from the corresponding data source before subsequent quality inspection can be carried out. This embodiment can be based on the first general scheduled task to uniformly initiate data retrieval requests for different quality inspection items, making the data retrieval process more unified and standardized, centrally managing data retrieval requests, and improving data retrieval reliability. Different quality inspection items correspond to different data ranges. For example, the data range of "quality inspection of product removal from the homepage" sent to the data retrieval system is the product on the homepage of the store platform, and the data range of "quality inspection of product expiration date" sent to the data retrieval system is all the products that can be sold in the store. Corresponding adjustments can be made to the above differentiated content.

[0031] Furthermore, the second general timing task of this embodiment is to store the data of the data acquisition results in a table, which is also applicable to all quality inspection items, because no matter what the specific content of the quality inspection item is, the acquired data needs to be properly stored for subsequent quality inspection. Among them, after completing the data acquisition, the data acquisition system will call back the data acquisition content in the form of a file to the quality inspection system. The file content data structure corresponding to different quality inspection items is different, and the data information is also different. Therefore, each quality inspection item needs to be customized to parse the corresponding file, and the stored parsed file falls into the respective data acquisition platform data table, that is, the data acquisition platform data table corresponding to different quality inspection items has different attributes except for a few fixed attributes. For example, the file content returned by "Homepage Product Off-Shelf Quality Inspection" reflects the product ID, product name, and product status, while "Product Expiration Quality Inspection" will reflect the product ID, product name, product expiration date, etc., then these two quality inspection items will fall into 2 copies of the data acquisition platform data table respectively.

[0032] Furthermore, the third general scheduled task of this embodiment is responsible for quality inspection of the obtained data collection result and quality inspection rules. All quality inspection items have their specific quality inspection rules, that is, the expected rules of the quality inspection items. For example, when "quality inspection of product removal from the homepage", the quality inspection rule is: the product status cannot be removed from the shelves; when "quality inspection of product expiration date", the quality inspection rule is: the expiration date of the products maintained on the default platform must be 1 month earlier than the worst expiration date of the actual warehouse, and the products of special categories must be 1 week earlier than the worst expiration date of the warehouse. The data collection results can be quality inspected through pre-configured quality inspection rules to determine whether the risk is within the acceptable range of store operations. That is, the content of quality inspection for different quality inspection items is different. The former quality checks whether the status of the products on the homepage is removed from the shelves, and the latter verifies whether the expiration date of the products is normal.

[0033] It should also be noted that the first general scheduled task, the second general scheduled task and the third general scheduled task in this embodiment are based on XXL-JOB, that is, the open source task scheduling center configures the routing strategy and execution time of the task, ensures the on-time start of the entire quality inspection process, and guarantees the reliable execution of the task.

[0034] like Figure 3 As shown, in some embodiments, receiving and executing the first general scheduled task, sending a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform, includes: Receive a first general scheduled task, and obtain a first task list that satisfies both the quality inspection task list and the data acquisition registration list corresponding to the first requirement; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the first task list and store them in the first blocking queue; After using the threads in the first thread pool to obtain the corresponding quality inspection task, the first distributed lock is obtained; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, send a data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create a data acquisition record; After storing the data retrieval record in the data retrieval registration table, the first distributed lock is released.

[0035] In this embodiment, the routing strategy of XXL-JOB is shard broadcast. When the first general scheduled task is triggered, the shard information is passed to the first general scheduled task. Here, it should be noted that the shard information means that if a total of 3 machines are registered to XXL-JOB, and the current machine is the first connected, then the returned shard information can be the current machine serial number 0, the total number of machines 3, and when a certain value is sharded, the remainder can be obtained by dividing the value / the total number of machines to determine whether the remainder is equal to the current machine serial number. For example, if the value to be sharded is 3, then 3 / 3 is 0, and it can be executed by the current machine. If the value of the shard is 4, then 4 / 3 is 1, and it cannot be executed by the current machine.

[0036] The following will describe the first general timing task in detail.

[0037] 1. After receiving the first general scheduled task, the quality inspection system preliminarily obtains the first task list, wherein the tasks in the first task list need to meet the first requirements of the quality inspection task list and the data acquisition registration table at the same time.

[0038] 2. Use store information, such as store ID, as the shard value for sharding to ensure that each machine only processes the quality inspection task ID that meets the sharding result and carries the quality inspection item type. That is, only the quality inspection task list that can be executed by the machine is filtered out, so that tasks can be assigned to different machines according to the business attributes of the store to achieve parallel processing of tasks.

[0039] 3. Put the filtered quality inspection task data into the first blocking queue. It should be noted that the blocking queue of this embodiment provides a buffer area for task processing, which can play a buffering and coordination role when the task generation and processing speeds do not match.

[0040] 4. The thread of the first thread pool corresponding to the first general scheduled task obtains the quality inspection task from the blocking queue to realize concurrent processing of the task.

[0041] 5. Obtain the first distributed lock. It should be emphasized that in order to avoid the same quality inspection task being executed by multiple threads, this embodiment obtains the first distributed lock, thereby ensuring that the same quality inspection task can only be executed by one thread at the same time. Among them, the first distributed lock of this embodiment is set based on the task ID and the quality inspection item type.

[0042] 6. Further determine whether the quality inspection task meets the data collection conditions; If the quality inspection task meets the data acquisition conditions, the quality inspection system will initiate a data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create corresponding data acquisition records; If the quality inspection task does not meet the data acquisition conditions, the quality inspection task will be skipped.

[0043] Here, it should be noted that in order to avoid the task not meeting the data acquisition conditions due to changes in task status, it is necessary to judge again whether the quality inspection task meets the data acquisition conditions to ensure that the task still meets the execution requirements at the current moment. The specific logic is: judge again whether the task ID meets the conditions corresponding to step 1 above.

[0044] 7. Register the data acquisition records of the data acquisition system and store them in the data acquisition registration table. At this time, the data acquisition state is the initial state.

[0045] 8. Release the first distributed lock.

[0046] It should be noted that before executing the first general scheduled task, the quality inspection system needs to first receive the quality inspection items and corresponding quality inspection rules configured by the user, create a quality inspection task, and notify the user to complete the quality inspection task creation.

[0047] Among them, Figure 6 As shown, the contents of the quality inspection rule table include primary key ID, store ID, and specific rule contents, which depend on the quality inspection items; like Figure 7 As shown, the content of the quality inspection task table includes the task creator, task creation time, task completion time, task ID, quality inspection item type, store, task status, total number of data acquisitions, current data acquisition sequence number, and success number, failure number, and risk number; among them, the task status includes pending execution, executing, completed, and failed; The contents of the data acquisition registration form include the task creation time, primary key ID, task ID, data acquisition return ID, current data acquisition sequence number, whether the data acquisition result has been stored, data acquisition result, data acquisition status and data acquisition callback time; among them, the data acquisition status includes initial state, completed, and failed, and if the status is failed, the data acquisition result indicates the failure content.

[0048] In some of the embodiments, the first requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisitions is greater than 0; and the corresponding task ID in the data acquisition registration table is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same current serial number as the quality inspection task table.

[0049] In this embodiment, when initially acquiring the first task list, the requirements of the quality inspection task table and the data acquisition registration table must be considered at the same time, that is, the tasks in the first task list must meet the first requirement. Specifically, the task status in the quality inspection task table must be pending and the number of data acquisitions must be greater than 0, ensuring that only tasks that require data acquisition are included in the scope of processing, and screening out qualified task IDs, quality inspection item types, stores, and current data acquisition serial numbers. Subsequently, it is compared with the data acquisition registration table to ensure that the task ID in the data acquisition registration table is the same as the task ID in the quality inspection task table, and that these tasks do not have records of the same current data acquisition serial number in the data acquisition registration table, ensuring that each task will not be processed repeatedly under a specific data acquisition serial number.

[0050] like Figure 4 As shown, in some embodiments, receiving and executing the second general timing task, obtaining the data acquisition result stored in the data acquisition system, and storing it in the corresponding table include: Receive a second general scheduled task, and obtain a second task list that satisfies both the quality inspection task list and the data acquisition registration table corresponding to the second requirement; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed accordingly from the second task list and store them in the second blocking queue; After using the threads in the second thread pool to obtain the corresponding quality inspection task, the second distributed lock is obtained; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, initiate a data acquisition result acquisition request based on the data acquisition return ID to obtain the data acquisition results stored in the data acquisition system; After storing the data acquisition result in the corresponding data acquisition platform data table, the data acquisition registration table is updated and the second distributed lock is released.

[0051] In this embodiment, by storing the data acquisition results in a table, on the one hand, the original data can be saved for tracing and review when necessary, and on the other hand, a fast data access method is provided for subsequent quality inspection steps.

[0052] The second general timing task will be described in detail below.

[0053] 1. After receiving the second general scheduled task, the quality inspection system obtains a second task list, wherein the tasks in the second task list need to meet the second requirements of the quality inspection task list and the data acquisition registration table at the same time.

[0054] It should be noted that when the second general scheduled task is triggered, the shard information is also passed to the second general scheduled task.

[0055] 2. Use store information, such as store ID, as the shard value for sharding to ensure that each machine only processes the quality inspection task ID that meets the sharding results and carries the quality inspection item type.

[0056] 3. Put the filtered quality inspection task data into the second blocking queue.

[0057] 4. The thread of the second thread pool corresponding to the second general scheduled task obtains the quality inspection task from the blocking queue.

[0058] 5. Acquire the second distributed lock.

[0059] 6. Further determine whether the quality inspection task meets the data collection conditions.

[0060] If the quality inspection task meets the data acquisition conditions, a data acquisition result acquisition request is initiated to the data acquisition system according to the data acquisition return ID to obtain the data acquisition result in the data acquisition system; If the quality inspection task does not meet the data acquisition conditions, the quality inspection task will be skipped.

[0061] 7. Store the acquired data acquisition results in the data acquisition platform data table.

[0062] 8. Update the data acquisition registration form. Specifically, change "Whether the data acquisition results have been stored" in the data acquisition registration form to "Yes".

[0063] 9. Determine whether the quality inspection task still needs to fetch data. Specifically, it is necessary to determine whether the current serial number of the task is consistent with the number of fetches in the quality inspection task table. If not, it is considered that the task still needs to fetch data. At this time, it is necessary to update the current serial number of the fetch in the quality inspection task table plus 1; if they are consistent, the task status in the quality inspection task table is updated to be in execution.

[0064] 10. Release the second distributed lock.

[0065] In some of the embodiments, the second requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisitions is greater than 0; and the corresponding task ID and current number of data acquisitions in the data acquisition registration table are the same as the corresponding task ID and current number of data acquisitions in the quality inspection task table, and the data acquisition status is completed, and the data acquisition results are not stored.

[0066] In this embodiment, for the determination of the tasks in the second task list, the second requirement corresponding to both the quality inspection task table and the data acquisition registration table must be met at the same time. Specifically, for the quality inspection task table, the task status in the quality inspection task table is required to be executed and the total number of data acquisition is greater than 0. When the above conditions are met, the corresponding task ID, quality inspection item type, store and current serial number of data acquisition can be obtained. Among them, the task ID is the unique identifier of each quality inspection task, which is used to track and manage tasks in the system, and the quality inspection item type determines the specific content and method of quality inspection. In addition, in this embodiment, the task ID and the current serial number of data acquisition in the data acquisition registration table are required to be the same as the quality inspection task table to ensure that the acquired tasks are correct. At the same time, the data acquisition status is limited to completed and the data acquisition results are not stored, ensuring that only those tasks that have completed the data acquisition operation but have not stored the results are acquired. Thus, it is avoided to process those unnecessary tasks or repeat the completed tasks, effectively improve the system efficiency, and ensure the accuracy of the quality inspection results.

[0067] like Figure 5 As shown, in some embodiments, receiving and executing the third general scheduled task, performing quality inspection on the data acquisition result with reference to the pre-configured quality inspection rules, and obtaining the quality inspection result includes: Receive a third general scheduled task, and obtain a third task list that meets the third requirement corresponding to the quality inspection task table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the third task list and store them in the third blocking queue; After using the threads in the third thread pool to obtain the corresponding quality inspection task, the third distributed lock is obtained; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, store the snapshot data in the snapshot table, and traverse the snapshot table to obtain the unprocessed quality inspection dimension data ID; Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID for quality inspection.

[0068] In this embodiment, quality inspection of the data acquisition results in combination with quality inspection rules is a key link to ensure data quality. If the quality inspection result shows that there is a risk, the risk result needs to be fed back to the user in a timely manner so that measures can be taken to improve data quality. At the same time, relying on the introduced snapshot table, some quality inspection items with a long quality inspection time can continue to execute unprocessed quality inspection data after the project is restarted, avoiding the loss of quality inspection progress of quality inspection tasks.

[0069] In some embodiments, the third requirement is that the task status in the quality inspection task table is in execution.

[0070] In this embodiment, when executing the third general scheduled task, in the third task list obtained, the task must satisfy the task status of being executed in the quality inspection task table. When the task changes from the pending status to the executing status, it means that the first general scheduled task and the second general scheduled task have been processed, ensuring that each task can be carried out according to the preset process to avoid missing tasks or repeated processing.

[0071] In some of the embodiments, quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID are obtained, and after quality inspection, the quality inspection results are stored in a quality inspection result table or a quality inspection result failure table, and the snapshot table is updated.

[0072] In this embodiment, Fig. 9 As shown, the contents of the snapshot table include primary key ID, task ID, whether it has been processed, and quality inspection dimension data ID; like Fig.10 As shown, the contents of the quality inspection result table include the creation time, primary key ID, task ID, quality inspection dimension ID, other data of the quality inspection dimension, whether there is risk, rule data, e-commerce platform data, and internal system reference data; like Fig.11 As shown, the contents of the quality inspection result failure table include the creation time, primary key ID, task ID, quality inspection dimension data ID, and failure reason; The specific fields of the above-mentioned quality inspection dimension data ID, such as product ID, depend on the quality inspection item. The specific fields of other data in the quality inspection dimension, such as product name, code, connection, etc., depend on the quality inspection item.

[0073] The third general timing task will be described in detail below.

[0074] 1. After receiving the third general scheduled task, the quality inspection system obtains the third task list, wherein the tasks in the third task list need to meet the third requirement of the quality inspection task list; that is, the tasks need to meet the task status of being executed in the quality inspection task list, and obtain the corresponding task ID, quality inspection item type and store; It should be noted that when the third general scheduled task is triggered, the shard information is also passed to the third general scheduled task.

[0075] 2. Use store information, such as store ID, as the shard value for sharding to ensure that each machine only processes the quality inspection task ID that meets the sharding results and carries the quality inspection item type.

[0076] 3. Put the filtered quality inspection task data into the third blocking queue.

[0077] 4. The thread of the third thread pool corresponding to the third general scheduled task obtains the quality inspection task from the blocking queue.

[0078] 5. Acquire the third distributed lock.

[0079] 6. Further determine whether the quality inspection task meets the data collection conditions.

[0080] If the quality inspection task meets the data acquisition conditions, the snapshot data is stored; If the quality inspection task does not meet the data acquisition conditions, the quality inspection task will be skipped.

[0081] Here, it should be noted that the snapshot data table contains the quality inspection dimension data ID. For example, the quality inspection dimension of "product expiration date quality inspection" is the "product dimension", and the quality inspection dimension data ID is the product ID. The significance of the snapshot table is that the quality inspection of some quality inspection items takes a long time, so as to avoid repeated quality inspection of the same quality inspection dimension data when restarting.

[0082] It should also be noted that the quality inspection dimension data ID in this embodiment can be directly obtained from the quality inspection rules, or determined based on the e-commerce platform data and the quality inspection rules, depending on the quality inspection items. In addition, if the snapshot table already has data corresponding to the task ID, it is considered that the task has been stored when it was last executed.

[0083] 7. Traverse the snapshot table to obtain the unprocessed quality inspection dimension data ID.

[0084] 8. Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and conduct quality inspection.

[0085] 9. Store the quality inspection results in the quality inspection result table or the quality inspection result failure table, and update the snapshot table and quality inspection task table.

[0086] After a single quality inspection dimension data is inspected, the inspection result will be stored in the quality inspection result table. If it fails, it will be stored in the quality inspection result failure table, and the status of the record in the snapshot table will be updated to processed. When all data in the snapshot table is processed, the information such as the number of successes, failures, and risks in the quality inspection task table will be updated.

[0087] 10. Notify store operations of quality inspection results.

[0088] 11. Release the third distributed lock.

[0089] In some of the embodiments, a third general scheduled task is received and executed, and the data acquisition results are quality inspected according to pre-configured quality inspection rules to obtain quality inspection results. It also includes: determining whether it is necessary to request an internal system, and obtaining reference data of the internal system when necessary, and storing it in the internal system reference data table as reference data in the quality inspection process.

[0090] In this embodiment, it is possible to determine whether it is necessary to request the internal system and obtain the reference data in the internal system based on the actual situation. The specific reason is that some simple quality inspection items do not require reference data to be involved in quality inspection, that is, this part of the data does not need to be requested from the internal system, while another part of the data may be restarted, that is, the reference data has been obtained during the last execution of this quality inspection task, so there is no need to request the internal system, and it can be determined by checking the internal system reference data table. Here, it should be noted that the internal system includes the warehouse management system (WMS system), the order management system (OMS system) and the product information maintenance system, and the internal system reference data refers to the data that can be obtained within the company. For example, the above-mentioned WMS system, OMS system, and product information maintenance system.

[0091] Among them, the internal system reference data table and data acquisition platform data table corresponding to different quality inspection items may be different, such as Figure 8 As shown, the internal system reference data table includes the No. 1 internal system reference data table and the No. 2 internal system reference data table, that is, the number of such tables is not fixed, but the content includes the primary key ID, task ID and other fields depending on the quality inspection items. The content of the data acquisition platform data table includes the primary key ID, task ID and other fields, depending on the data returned. That is, there is a common attribute, namely the task ID. For example, the data structures of "Homepage Product Delisting Quality Inspection" and "Product Expiration Quality Inspection" are different, so the data returned by the data acquisition will be placed in their respective data acquisition platform data tables, such as the No. 1 data acquisition platform data table and the No. 2 data acquisition platform data table. And because the "Product Expiration Quality Inspection" needs to be compared with the expiration date in the warehouse, the warehouse data will also be obtained from the WMS system and placed in the internal system reference data table.

[0092] Specifically, after determining whether the quality inspection task meets the data acquisition conditions, if it does, further determine whether it is necessary to request the internal system. If so, the quality inspection system obtains reference data from the internal system and saves the reference data, that is, stores it in the internal system reference data table, and further executes the step of storing snapshot data; if not, directly execute the step of storing snapshot data. In the subsequent quality inspection, the quality inspection rule data, e-commerce platform data, and internal system reference data corresponding to the quality inspection dimension data ID can be obtained for quality inspection, and the quality inspection results are stored in the quality inspection result table or the quality inspection result failure table.

[0093] It should be noted that the reason why this embodiment stores the e-commerce platform data and the internal system reference data table fields in the quality inspection result table after the data acquisition platform data table and the internal system reference data table is that the data is stored in the data acquisition platform data table and the internal system reference data table. On the one hand, it is to leave a trace of the data in the quality inspection system as two basic data from different sources participating in the quality inspection. On the other hand, in the actual use process, the store operation may have the demand to export such basic data based on the quality inspection task dimension. The two fields stored in the quality inspection result table are for the convenience of the operation to view the quality inspection results based on the quality inspection task and the quality inspection data ID.

[0094] like Fig.12 As shown, a complete embodiment of the data quality inspection process will be provided below for detailed description.

[0095] 1. Receive quality inspection items and quality inspection rules configured by users; 2. Receive quality inspection tasks created by users; 3. Return the creation result of the quality inspection task to the user; 4. Receive and execute XXL-JOB's general scheduled task A, i.e., the first general scheduled task, and obtain the task list to be executed and for which no data acquisition request has been issued this time; 5. Initiate data acquisition task to the data acquisition system; 6. Register the data retrieval request; 7. The data acquisition system acquires data from the e-commerce platform; 8. The data acquisition system stores the data acquisition results; 9. Update the data acquisition status to completed; 10. Receive and execute the general scheduled task B of XXL-JOB, i.e., the second general scheduled task, and obtain a list of tasks to be executed and for which data has been obtained but the results have not been stored; 11. Obtain the corresponding data acquisition results from the data acquisition system; 12. Store the data acquisition results in a table; 13. Determine whether the task needs to continue to fetch data. If not, transition to execution; if yes, wait for the triggering of task A; 14. Receive and execute general scheduled task C, i.e., the third general scheduled task, and obtain the list of tasks being executed; 15. Obtain reference data from internal systems; 16. Put the reference data into the table; 17. Conduct data quality inspection; 18. Notify users of warning risk results.

[0096] Therefore, the method of the present application is changed from the original scheduled task polling initiated by each type of quality inspection item to three fixed general scheduled tasks, which saves thread expenses. In addition, each of the three scheduled tasks has an independent thread pool. On the one hand, the number of threads corresponding to the thread pool can be controlled manually, reducing resource consumption; on the other hand, the independent thread pool means that the scheduled tasks will not be blocked due to thread resources, effectively improving the processing power and response speed of the system. At the same time, distributing the quality inspection task data of multiple stores to different machines for execution can alleviate the execution pressure of a single machine and improve the execution speed. In addition, the data information of the key steps is stored in the table, so that the quality inspection tasks that have not reached the final state can continue to be executed after the project is restarted.

[0097] Embodiment 2 In a second aspect, an embodiment of the present application provides an e-commerce platform data quality inspection system, including: The first unit is configured to receive and execute a first general scheduled task, and send a data acquisition request to a data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform; The second unit is configured to receive and execute the second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in the corresponding table; The third unit is configured to receive and execute a third general scheduled task, perform quality inspection on the data acquisition result by referring to a preset quality inspection rule, and obtain a quality inspection result; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are configured based on XXL-JOB, and each has an independent thread pool.

[0098] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0099] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

[0101] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "one", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more, and "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0102] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but the limitation of the scope of the invention patent cannot be understood as a result. It should be pointed out that, for a person of ordinary skill in the art, several modifications or improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data quality inspection method for an e-commerce platform, characterized in that: The steps include: Receive and execute the first general scheduled task, and send a data acquisition request to the data acquisition system so that the data acquisition system can acquire data from the e-commerce platform; Receive and execute the second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in the corresponding table; Receive and execute the third general scheduled task, perform quality inspection on the data acquisition result according to the pre-configured quality inspection rules, and obtain the quality inspection result; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

2. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the first general scheduled task, and sending a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform, includes: Receive the first general scheduled task, and obtain a first task list that satisfies the first requirement corresponding to the quality inspection task table and the data acquisition registration table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the first task list and store them in the first blocking queue; After using the thread in the first thread pool to obtain the corresponding quality inspection task, obtain the first distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, send the data acquisition request to the data acquisition system according to the quality inspection item type, so that the data acquisition system can acquire data from the e-commerce platform and create a data acquisition record; After storing the data retrieval record in the data retrieval registration table, the first distributed lock is released.

3. The e-commerce platform data quality inspection method according to claim 2 is characterized in that: The first requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisitions is greater than 0; and the corresponding task ID in the data acquisition registration table is the same as the corresponding task ID in the quality inspection task table, and there is no record with the same current serial number as the quality inspection task table.

4. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the second general timing task, obtaining the data acquisition result stored in the data acquisition system, and storing it in the corresponding table, includes: Receive the second general scheduled task, and obtain a second task list that satisfies the second requirement corresponding to the quality inspection task table and the data acquisition registration table; Using the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed accordingly from the second task list and store them in the second blocking queue; After using the thread in the second thread pool to obtain the corresponding quality inspection task, obtain the second distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, initiate a data acquisition result acquisition request according to the data acquisition return ID to obtain the data acquisition results stored in the data acquisition system; After storing the data acquisition result in the corresponding data acquisition platform data table, the data acquisition registration table is updated and the second distributed lock is released.

5. The e-commerce platform data quality inspection method according to claim 4 is characterized in that: The second requirement is that the task status in the quality inspection task table is to be executed and the total number of data acquisition is greater than 0; and the corresponding task ID and current number of data acquisition in the data acquisition registration table are the same as the corresponding task ID and current number of data acquisition in the quality inspection task table, and the data acquisition status is completed, and the data acquisition result is not stored.

6. The e-commerce platform data quality inspection method according to claim 1, characterized in that: The receiving and executing the third general scheduled task, and performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, includes: Receive the third general scheduled task, and obtain a third task list that meets the third requirement corresponding to the quality inspection task table; Use the corresponding store information to perform shard filtering, so that each server can filter out the quality inspection tasks that can be executed from the third task list and store them in the third blocking queue; After using the threads in the third thread pool to obtain the corresponding quality inspection task, obtain the third distributed lock; Determine whether the quality inspection task meets the corresponding data acquisition conditions, and if so, store the snapshot data in the snapshot table, and traverse the snapshot table to obtain the unprocessed quality inspection dimension data ID; Obtain the quality inspection rule data and e-commerce platform data corresponding to the quality inspection dimension data ID and perform quality inspection.

7. The e-commerce platform data quality inspection method according to claim 6, characterized in that: The third requirement is that the task status in the quality inspection task table is in execution.

8. The e-commerce platform data quality inspection method according to claim 6, characterized in that: The receiving and executing the third general scheduled task, and performing quality inspection on the data acquisition result with reference to a pre-configured quality inspection rule to obtain a quality inspection result, further includes: Determine whether it is necessary to request the internal system, and obtain reference data of the internal system when necessary, and store it in the internal system reference data table as reference data in the quality inspection process.

9. The e-commerce platform data quality inspection method according to any one of claims 6 to 8, characterized in that: The obtaining of the quality inspection rule data and the e-commerce platform data corresponding to the quality inspection dimension data ID and performing quality inspection further includes: The quality inspection result is stored in a quality inspection result table or a quality inspection result failure table, and the snapshot table is updated.

10. An e-commerce platform data quality inspection system, characterized in that: include: The first unit is configured to receive and execute a first general scheduled task, and send a data acquisition request to the data acquisition system, so that the data acquisition system can acquire data from the e-commerce platform; The second unit is configured to receive and execute a second general timing task, obtain the data acquisition result stored in the data acquisition system, and store it in a corresponding table; The third unit is configured to receive and execute a third general scheduled task, perform quality inspection on the data acquisition result with reference to a preset quality inspection rule, and obtain a quality inspection result; Among them, the first general scheduled task, the second general scheduled task and the third general scheduled task are obtained based on XXL-JOB configuration, and each has an independent thread pool.

Images