Job task management method and system for model verification

Abstract

The invention belongs to the technical field of task management, and provides a job task management method and system for model verification. The method comprises the steps of data receiving, request verification, message packaging, task record data construction, task preheating and queue loading, task distribution, result data receiving and file association, event capture and downstream push completion, data synchronization and report query. According to the method, through the set receiving and locking processes, the probability of repeated distribution and task loss is reduced, and task circulation can still be kept consistent in a high-concurrency receiving scene; through acceptance records, the processing progress and abnormal reasons can be quickly positioned, and idempotence and traceability of task access are realized; through a recoverable delivery and duplicate removal mechanism, the completion state result can still be finally delivered under the conditions that the downstream is temporarily unavailable, the network jitters and the like, the delivery reliability of the completion result is improved, and the risk of missed delivery is reduced.

Description

Technical Field

[0001] This invention relates to the field of task management technology, and in particular to a method and system for managing job tasks for model validation. Background Technology

[0002] With the development of digital and intelligent business operations, business systems for model / algorithm verification, data annotation, and intelligent operations typically need to support multi-terminal access, high-concurrency task access and distribution, task status flow, data storage and statistical analysis, as well as end-to-end security and auditing capabilities. Such systems often serve both internal users and external systems, requiring them to ensure both access and processing efficiency, as well as data consistency, traceability, and stability.

[0003] Currently available task access, processing, and distribution platforms typically suffer from the following shortcomings in practical applications: insufficient reliability on the access side, limited peak shaving and queuing capabilities under high concurrency, weak consistency and recoverability of task status, high troubleshooting costs due to complex data links, risk of discrepancies between statistical analysis and business data, and insufficient granularity or fragmented implementation of security and access control. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a job task management method and system for model verification, which solves the consistency problem under high-concurrency task distribution, the problem of duplicate data entry caused by repeated pushes from external systems / network retries, the reliability problem of push of completed results, the stability and maintainability problem of data synchronization to the analysis database, and the online risks and maintenance costs caused by multi-component deployment.

[0005] To achieve the above objectives, the present invention provides the following solution: A job task management method for model validation includes: Receive task push request data from an external backend and obtain the received data; The received data is verified, and requests that pass the verification are filtered to obtain task access request data; The task access request data is encapsulated into task access and message data to obtain task message data; The task message data is cleaned, deduplicated, and stored in the database to obtain task record data; The task record data is preheated and loaded into a queue to obtain preheated task records; Based on the preheated task record, the task access request data is retrieved, locked, and distributed to obtain the locked task record. Based on the locked task record, the task result data submitted by the front end is received, and the task result data is completed, stored in the database, and associated with files to obtain the completed task record. Based on the locked task records and the completed task records, the completion event is captured, statistically updated, and pushed downstream to obtain statistical count data, completed event data, and push data. The statistical count data and the completed event data are analyzed, synchronized, and queryed to obtain the analysis data table and BI report results.

[0006] Preferably, the received data is verified, and requests that pass the verification are filtered to obtain task access request data, including: A health check is performed on the received data. If the check result indicates that the upstream is unavailable, a 502 status code is returned. Cross-domain control and cookie attribute enhancement are applied to the received data, and the frequency of the received data is limited according to the rate limiting configuration; The user request data is parsed using Cookie to obtain a JWTToken. The user identification information in the JWTToken is extracted, and the permission management interface that matches the user identification information is called according to the permission verification configuration to obtain the permission status. The X-API-Key of the task push request data is verified, and the verified task push request data and the user identification information with the permission status are integrated into the task access request data.

[0007] Preferably, the task access request data is encapsulated into task access and messaging data to obtain task message data, including: The task access request data is subjected to multi-dimensional verification and structure generation to obtain the task data structure; the multi-dimensional verification items include: field integrity, type validity, and maximum number of batch records; Extract the external ID or idempotent key of each task access request data according to the task data structure, and encapsulate the verified task access request data into a task message to be processed according to the external ID or the idempotent key. The task message to be processed is written to a specified topic in Kafka to obtain the task message data.

[0008] Preferably, the task message data is cleaned, deduplicated, and stored in a database to obtain task record data, including: The task message data is parsed, and the task message data is deduplicated or idempotently filtered according to the external ID obtained from the parsing to obtain filtered data. The filtered data is cleaned to obtain task records that can be stored in the database; the data cleaning includes: field normalization, missing value completion, time format unification, and necessary business mapping. Write the status of the task record that can be added to the database in the same transaction to the initialization status and create an audit field to obtain the task record data.

[0009] Preferably, the task record data is preheated and loaded into a queue to obtain preheated task records, including: Periodically read the queue parameters of the pending zset in Redis, and perform status queries on the task record data according to the queue parameters and the preheating strategy to obtain a set of task IDs to be preheated; Write the set of task IDs to be preheated into a Redis zset queue and set an expiration time. Write the task content corresponding to the set of task IDs to be preheated into a Redis cache. Update the status of the set of task IDs to be preheated to the preheated status to obtain the preheated task record.

[0010] Preferably, the task access request data is processed, locked, and distributed based on the pre-warmed task record to obtain a locked task record, including: Retrieve the task ID from the Redis zset queue; Based on the preheated task record corresponding to the task ID, generate the target task content to be returned to the front end, and update the task status corresponding to the target task content to the locked status; Record the information on who received the target task content; the information includes: the recipient and the time of receipt; By integrating the target task content, the locked status, and the claim information, the locked task record is obtained.

[0011] Preferably, the task result data submitted by the front end is received according to the locked task record, and the task result data is completed, stored in the database, and associated with files to obtain the completed task record, including: The task result data and task status are verified; The task result data that passes the verification is written into a MySQL transaction, and the task status corresponding to the task result data is updated to the completed status to obtain the completed task record.

[0012] Preferably, based on the locked task records and the completed task records, completion event capture, statistical update, and downstream push are performed to obtain statistical count data, completed event data, and push data, including: The event that captures the change of task status from locked or in progress to completed is used to obtain change information; the change information includes: task ID, completion time, and completer; The statistical incremental data is updated using atomic counting to obtain the statistical count data and the completed event data. The completed task events corresponding to the change information are written to a specified topic in Kafka. The completed task events are assembled according to the data format set by the downstream system to obtain the push data, and the push data is pushed to the downstream system.

[0013] Preferably, a job task management system for model validation includes: a presentation layer, a security layer, a gateway layer, a service layer, a persistence layer, an engineering layer, a monitoring and alarm layer, and a log tracing layer; the presentation layer embeds a Web frontend, an H5 frontend, an Electron client, and an externally accessible API; the security layer embeds firewall rules and access control policy components; the gateway layer embeds gateway services, a health check module, a cross-domain control module, a rate limiting module, and JWT. The system includes a token parsing module, a permission verification module, an API token verification module, a logging module, and a load balancing module. The service layer embeds an Auth service, a Validation API service, a scheduled module, and a data synchronization module. The persistence layer embeds a Kafka message queue, a Redis cache, a MySQL business database, a ClickHouse analytics database, and a COS object storage. The engineering layer embeds a Docker container and a Swagger API documentation tool. The monitoring and alerting layer embeds Exporter, Prometheus, Grafana, an alerting rule engine, and an alerting robot. The log tracing layer embeds a Logstash log collector, a Kafka message queue, an Elasticsearch log storage and retrieval cluster, and a Kibana log visualization and analysis platform. The gateway layer is connected to the presentation layer, the security layer, and the service layer respectively; the service layer is connected to the persistence layer and the engineering layer respectively; the monitoring and alarm layer is connected to the gateway layer, the service layer, the persistence layer, and the engineering layer respectively; the log tracing layer is connected to the presentation layer, the security layer, the gateway layer, the service layer, the persistence layer, the engineering layer, and the monitoring and alarm layer respectively.

[0014] The present invention discloses the following technical effects: This invention provides a job task management method and system for model validation. Through a defined retrieval and locking process, it solves the consistency problem in high-concurrency task distribution, ensuring consistent task flow in high-concurrency retrieval scenarios. By using acceptance records, it addresses the issue of duplicate data entry caused by repeated external system pushes / network retries, ensuring that the same external business data is generated / accepted only once even when retries or duplicate submissions occur. Through recoverable delivery and failure retry / deduplication mechanisms, it solves the reliability problem of completed result pushes, ensuring that completed results can be delivered even when downstream is temporarily unavailable or experiencing network fluctuations. By governing and constraining schema change and deletion semantics, it addresses the stability and maintainability issues of data synchronization to the analysis database, reducing the probability of synchronization anomalies. Through standardized deployment and configuration governance, link observable fields, and a monitoring and alarm system, it addresses the deployment risks and maintenance costs associated with multi-component deployment, reducing deployment and maintenance costs. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a job task management process for model validation provided in an embodiment of the present invention; Figure 2 A system architecture diagram provided for embodiments of the present invention; Figure 3 A data flow diagram provided for embodiments of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The purpose of this invention is to provide a job task management method and system for model validation, which solves the consistency problem under high-concurrency task distribution, the problem of duplicate data entry caused by repeated pushes from external systems / network retries, the reliability problem of push of completed results, the stability and maintainability problem of data synchronization to the analysis database, and the online risk and maintenance cost problem brought about by multi-component deployment.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 This is a schematic diagram of a job task management process for model validation provided in an embodiment of the present invention, such as... Figure 1 As shown, the present invention provides a job task management method for model validation, comprising: Step 100: Receive task push request data from the external backend and obtain the received data; Step 200: Verify the received data and filter out the requests that pass the verification to obtain task access request data; Step 300: Perform task access and message encapsulation on the task access request data to obtain task message data; Step 400: Perform task cleaning, deduplication, and database entry on the task message data to obtain task record data; Step 500: Perform task preheating and queue loading on the task record data to obtain preheated task records; Step 600: Based on the preheated task record, perform task retrieval, locking, and distribution on the task access request data to obtain the locked task record; Step 700: Receive the task result data submitted by the front end according to the locked task record, and perform completion storage and file association on the task result data to obtain the completed task record; Step 800: Based on the locked task records and the completed task records, perform completion event capture, statistical update, and downstream push to obtain statistical count data, completed event data, and push data; Step 900: Perform analysis data synchronization and report query on the statistical count data and the completed event data to obtain the analysis data table and BI report results.

[0021] refer to Figure 2 This embodiment relates to a system for model validation / intelligent job task access, processing, distribution, and statistical analysis, which can be divided into the following levels (from top to bottom): 1) The presentation and access layer includes: a Web / H5 / Electron frontend and an externally accessible API for batch pushing task data to external backends. This layer provides page operation entry points for internal users and standardized HTTP interface entry points for external systems.

[0022] 2) The security and gateway layer includes: security policy components, a custom gateway service based on OpenResty, and Nginx for hosting front-end static resources. It implements unified entry point, authentication, cross-domain control, rate limiting, token parsing, permission verification, APIKey verification, load balancing, logging, and health checks.

[0023] 3) The business service layer includes: The Auth service is used for interfaces related to authentication, login, and permission management. The ValidationAPI service is used for task-related interfaces; Scheduled task modules (such as daily task generation, SKU mapping synchronization, image quantity synchronization, etc.).

[0024] It carries the core business logic and external interfaces, and completes task lifecycle management.

[0025] 4) The message and caching layer includes: Kafka message queues, Redis cache and queues. This message and caching layer enables asynchronous decoupling, peak shaving and valley filling, task buffering and fast distribution, hot data caching, atomic counting statistics, etc.

[0026] 5) The persistence and file storage layer includes a MySQL business database. It ensures reliable data and status persistence, transaction consistency, and supports large file storage.

[0027] 6) The data synchronization and analysis layer includes: Canal, KafkaConnect, ClickHouse analytics library, and BI platform. This network layer incrementally / fully synchronizes business data from MySQL to the analytics library, supporting report analysis and query performance.

[0028] 7) Monitoring, alerting, and engineering support layers include: Exporter, Prometheus, Grafana, GitLab, Jenkins, SonarQube, Docker, Swagger, etc. These provide the system with operational monitoring, alerting, continuous integration and quality management, containerized deployment, and API documentation capabilities.

[0029] Furthermore, the connection relationships (structural connections / functional relationships) between modules at each layer can be summarized as follows (where → indicates the direction of data / request flow, (Indicates two-way interaction) 1) User / External Backend Gateway service. Users accessing front-end resources via browser or client: User → Nginx / Front-end static resource service; Users / external backends calling business interfaces: User / external backend → Gateway service.

[0030] 2) Gateway service Auth service. Login / authentication related requests: Gateway → Auth service; Permission verification: The gateway queries the Auth service for permissions based on the user information (403 is returned if no permission is granted).

[0031] 3) Gateway Service → ValidationAPI Service. Business requests (pushing tasks, querying tasks, claiming tasks, submitting tasks, etc.) are forwarded to the ValidationAPI via the gateway.

[0032] 4) ValidationAPI service Kafka / MySQL / Redis / COS. After task access, data is written to Kafka; after cleaning and storage, it is written to MySQL, and then written to / read from Redis when needed; file-based data is uploaded to / referenced in COS.

[0033] 5) Canal monitors MySQL → Redis / Kafka / ClickHouse. Canal monitors MySQL binlog: MySQL → Canal; Canal uses changes for front-end statistics and counting: Canal → Redis; Canal retrieves completed tasks and pushes them to Kafka: Canal → Kafka; or via Debezium + KafkaConnect + ClickHouseSink: MySQL → Debezium → Kafka → ClickHouse.

[0034] 6) ClickHouse → BI Platform → Users. To query ClickHouse on the BI platform: BI → ClickHouse; to display reports to users: BI → Users.

[0035] 7) Monitoring and alarm connection: Exporter / Service → Prometheus; Prometheus → Grafana (display); Grafana / Alarm Rules → Alarm Notification.

[0036] Furthermore, the following section uses task data as the main thread to trace the data transmission paths of key system modules and the functions of each module: 1) Presentation and Access Layer → Gateway Layer: User operation data is generated in the Web / H5 / Electron frontend. The frontend outputs HTTP request data and transmits it to the gateway service. The frontend provides interactive interfaces for viewing, claiming, and submitting tasks; it also connects to backend interfaces. External backend task data is generated in the external backend system. The external backend outputs HTTP request data carrying the X-API-Key and transmits it to the gateway service. The external backend pushes tasks / data to the platform in batches, realizing inter-system integration.

[0037] 2) Gateway Layer Internal Processing → Business Service Layer: After HTTP request data enters the gateway service, the gateway executes the following in sequence: Health Check Module: Checks the availability of the backend service according to the upstream configuration, outputting a 502 if unavailable; Cross-Origin Control Module: Allows or restricts cross-origin requests according to the allow_cors configuration; Rate Limiting Module: Controls the request frequency according to the rate limiting policy and outputs the rate limiting result; JWTToken Parsing Module: Parses the JWT in the cookie, outputs the user identity information and writes it to the request header; Permission Verification Module: Outputs the user identity information and passes it to the Auth service permission management interface for verification; if no permission is granted, a 403 is output directly; APIToken Verification Module: Verifies the X-API-Key (or signature) in the external backend request header, allowing the request only if the verification passes. The gateway ultimately outputs the verified forwarded request data and passes it to the ValidationAPI service or Auth service. The gateway is used for unified entry point governance (authentication, rate limiting, cross-origin, permissions, logging, load balancing), reducing the complexity of backend services. Login / authentication request data is output by the gateway and passed to the Auth service, which generates / refreshes the Auth service. Credential data (such as JWT / session information) The credentials are then output to the gateway, which writes them to a cookie and returns them to the frontend. The Auth service is used to provide login, user authentication, permission management, and internal interfaces (such as / sso / me, / auth / permission, etc.).

[0038] 3) ValidationAPI: Task Inbound → Kafka (Asynchronous Buffer): After the task inbound request data arrives at the ValidationAPI task inbound module, this module performs parameter validation, external ID parsing, and other processing on the request, generates standardized task message data, and outputs this task message data to Kafka (for peak smoothing and asynchronous processing), while simultaneously outputting... Write successful receipt data (external ID / acceptance result). The result is returned to the caller. The task access module is used for unified access, standardized task data, rapid response to the caller, and asynchronous processing of subsequent tasks.

[0039] 4) Kafka → Task Cleaning and Ingestion Module → MySQL (Transactional Ingestion): After task message data is temporarily stored in Kafka, it is retrieved in batches by the task cleaning and ingestion module. This module cleans the messages, removes duplicates (e.g., filtering existing external IDs), and normalizes fields to form task data ready for ingestion. This ready-to-ingest task data is then written to MySQL. Upon successful writing, the corresponding consumption offset is output and committed. The task cleaning and ingestion module implements batch consumption, data cleaning, and idempotent filtering, ensuring both ingestion quality and throughput. After receiving the ready-to-ingest task data, MySQL completes the task record writing and initial state setting (e.g., "initialization") in a transactional manner, and outputs persistent task data for subsequent modules to query / update. MySQL carries the task master data and state machine transitions, providing transactional consistency.

[0040] 5) Task Preheating Module: MySQL → Redis (Queues and Caches): Initialization status task data is read from MySQL by the task preheating module (scheduled tasks). This module filters tasks based on queue length / strategy, generates task IDs to be distributed, and outputs them to a Redis zset queue. Simultaneously, task details are output to a Redis cache, and the task status in MySQL is updated to "preheated." The task preheating module preloads "distributable tasks" into a high-performance queue, improving retrieval efficiency and smoothing the load. After receiving the task IDs to be distributed, Redis forms a sortable queue in the zset and caches the task details, outputting quickly readable queue / cache data for the distribution module. Redis provides high-performance queues and caches to support fast retrieval and access to hot data.

[0041] 6) Task Distribution Module: Redis → MySQL (Locking / Status Transition) → Frontend: After the task request data arrives at the task distribution module of the ValidationAPI, this module atomically retrieves the task ID from the Redis zset via Lua script, forming the task ID (request result data). It then outputs this task ID and passes it to MySQL, updates the task status to "Locked / In Progress," and removes the corresponding task ID from the Redis queue. Finally, it outputs the task details data back to the frontend. The task distribution module implements atomic retrieval, prevents concurrent duplicate retrieval, and drives state machine transitions. Task submission data, after being generated at the frontend, is passed to the task distribution / submission interface module. This module outputs the submission result data to MySQL (updating the status to "Completed," etc.) and outputs the completion event data required for subsequent pushes as needed. The task submission module receives user results, stores them in the database, and completes the status loop.

[0042] 7) Task Monitoring and Downstream Push: MySQL → Canal → Redis / Kafka → Downstream: After task status change data is generated in MySQL, the Canal module listens to the binlog to obtain change events. Canal outputs two types of data: statistical change data is output and passed to the Redis statistics and counting module for atomic count updates of front-end statistics such as task pool and today's completions; completed task event data is output and passed to Kafka for subsequent push or direct consumption by downstream systems. Canal uses the binlog as the standard to capture data changes, reducing the polling pressure on the business database and supporting real-time statistics and push. After obtaining completed task event data from Kafka, the task push module generates downstream push data and outputs it to the downstream system (or the downstream system directly consumes the Kafka topic). The task push module reliably notifies the downstream of the platform's processing results, forming an end-to-end closed loop.

[0043] 8) Data Analysis Chain: MySQL → (Debezium / Connect) → Kafka → ClickHouse → BI: After business data changes are generated in MySQL, they can be captured by DebeziumMySQLConnector and output as Kafka change event data, which is then written to Kafka. ClickHouseSinkConnector then consumes this data and outputs it to ClickHouse, forming analytical data tables. Debezium / Connect standardizes the CDC (Change Data Capture) chain, supporting full and incremental synchronization and schema evolution. The BI platform sends query requests to ClickHouse, ClickHouse outputs report query results to the BI platform, and the BI platform outputs visualized reports to the user. ClickHouse provides high-performance analytical queries; BI provides report building and visualization.

[0044] 9) File Data Link: Frontend / Service → COS: Image / video and other file data are generated by the frontend or business service, output and passed to COS object storage for storage; COS returns file address / key data to the business service, which associates the address / key with the task record and writes it to MySQL. COS is used to handle large file storage, reducing the pressure on the business database and improving access efficiency.

[0045] 10) Monitoring and Alarming Chain: Service / Component → Exporter → Prometheus → Grafana: Operational metric data is generated on each service / component side, collected by the Exporter, and then transmitted to Prometheus for storage. Prometheus then outputs the data to Grafana for display. When a metric triggers a threshold rule, an alarm notification is sent to the alarm channel. The monitoring layer ensures the observability and stable operation of the system.

[0046] Specifically, 1) The presentation layer is the top layer of the system, facing end users and external systems. Users access the system through Web / H5 / Electron clients and initiate user requests for data to the gateway layer. External system backends initiate task push requests for data to the gateway layer through open API access terminals; the presentation layer internally embeds a Web frontend, H5 frontend, Electron client, and an externally open API access terminal. 2) The security layer works in conjunction with the gateway layer to provide the system with unified security policies and access control. The security layer executes security policies on requests received from the presentation layer and then forwards them to the gateway layer for further processing; the security layer embeds firewall rules and access control policy components. 3) As the unified entry point of the system, the gateway layer is responsible for routing, authenticating, and rate limiting user / external backend request data received from the security layer or directly from the presentation layer, and then forwarding it to the Auth service or ValidationAPI service in the service layer. Simultaneously, the gateway layer also undertakes the function of returning login / authentication response data to the frontend user. The gateway layer embeds gateway services, health check modules, cross-domain control modules, rate limiting modules, JWT token parsing modules, permission verification modules, API token verification modules, logging modules, and load balancing modules. 4) The service layer is the core business logic layer of the system, and its modules are connected to the gateway layer, persistence layer, engineering layer, and log tracing layer, respectively. The Auth service interacts with the gateway layer, handling login / authentication requests and returning JWT credentials to the gateway. The ValidationAPI service receives task access requests from the gateway layer, writes task messages to the persistent layer's Kafka, reads and distributes tasks from the persistent layer's Redis queue, updates the task status to the persistent layer's MySQL, and retrieves or stores files from the persistent layer's COS. The data synchronization module monitors changes in the persistent layer's MySQL and synchronizes data to the persistent layer's ClickHouse. In addition, the service layer also writes runtime logs to the log tracing layer. The internal embedded modules of the service layer include: Auth service (for interfaces related to authentication, login, and access control); ValidationAPI service (including task access module, task cleaning and database entry module, task distribution module, task status flow module, task completion feedback / push module, etc.); Timed scheduling module (e.g., for daily task generation, SKU mapping synchronization, image quantity synchronization, and other timed tasks); Data synchronization module (e.g., based on Canal to monitor MySQL binlog).

[0047] 5) The persistence layer provides data storage and message queue services for the service layer. Kafka, as a message broker, receives task messages written by the ValidationAPI service and provides them for consumption by the task cleaning and database module. Redis, as a cache and queue, interacts bidirectionally with the ValidationAPI service to store the queue to be distributed (e.g., zset), task details, and statistical counts. MySQL, as a relational database, interacts bidirectionally with the ValidationAPI service to store and query structured data such as task records and task status. ClickHouse, as an analytical database, receives data synchronized from MySQL by the data synchronization module and provides it for querying by the BI platform. COS, as an object store, interacts bidirectionally with the ValidationAPI service to store and retrieve large files (e.g., images, videos) related to tasks. The persistence layer embeds a Kafka message queue, a Redis cache (including the task queue zset), a MySQL business database, a ClickHouse analytical database, and a COS object store. 6) The engineering layer provides support for system deployment, integration, and interface documentation. It collaborates with the monitoring and alerting layer to configure and apply monitoring and alerting rules. It collaborates with the service layer to provide a containerized deployment environment for the service and manages the code building, testing, and release of the service layer through continuous integration tools. It also provides API interface documentation. The engineering layer embeds Docker containers and Swagger interface documentation tools. 7) The monitoring and alerting layer collects runtime metric data from the service layer, persistence layer, gateway layer, and engineering layer. After collecting data from each layer through the Exporter, it aggregates it to Prometheus and visualizes it on Grafana. According to pre-defined alerting rules, Prometheus or Grafana triggers alerts when abnormal metrics are detected, and sends notifications through an alert bot. The monitoring and alerting layer embeds Exporter, Prometheus, Grafana, an alerting rule engine, and an alert bot. 8) The log tracing layer serves as the system's unified log management plane, implemented using the ELK technology stack based on Elasticsearch, Logstash, and Kibana. This layer is connected to the presentation layer, security layer, gateway layer, service layer, persistence layer, engineering layer, and monitoring and alerting layer. Log data generated by each layer and module is uniformly aggregated into the log tracking layer for centralized collection, storage, analysis, and retrieval.The presentation layer (Web / H5 / Electron clients), security layer (VPN components), gateway layer (Nginx / gateway service), service layer (Auth and Validation API services), persistence layer (Kafka / Redis / MySQL / ClickHouse), engineering layer (Docker containers), and monitoring and alerting layer (Exporter / Prometheus) all output various log data, including runtime logs, access logs, error logs, and audit logs, to the log tracing layer in real time. When system failures or anomalies occur, operations and maintenance personnel can quickly locate the cause of the problem through the log tracing layer, achieving end-to-end tracing and auditing. The log tracing layer embeds the Logstash log collector, Kafka message queue, Elasticsearch log storage and retrieval cluster, and Kibana log visualization and analysis platform. refer to Figure 3 The access request and security verification process (executed at the gateway layer, associated with the Auth service) is as follows: It receives user request data from the frontend (containing JWT or login status information in cookies) or task push request data from an external backend (the header carries X-API-Key / X-API-Token or signature information, and the task payload includes fields such as scene / sku). Subsequent processing is based on the gateway service (Openresty / Kong), and the Auth service is called when necessary. The gateway first performs a health check on the upstream service. If the upstream is unavailable, it returns a 502 error to prevent the request from entering an unavailable link. The gateway performs cross-origin control and cookie attribute enhancement (supporting both same-origin and cross-origin cookies) on the request and limits the request frequency according to the rate limiting configuration. If the limit is exceeded, a rate limiting response is returned. For front-end user requests: the gateway parses the JWTToken in the cookie, extracts the user identification information, and writes it into the request header. Then, it calls the authorization management interface of the Auth service according to the authorization verification configuration. If there is no authorization, a 403 error is returned. If there is authorization, the user ID and other information are written into the header and forwarded. For external backend requests: the gateway verifies the X-API-Key (or X-API-Token / signature) in the header. If the verification fails, the request is rejected; if the verification passes, forwarding is allowed. Output data: standardized forwarding request data that passes the verification (carrying user identity / authorization information or APIKey verification result). Requests that pass the verification are forwarded to the ValidationAPI service.

[0048] Furthermore, the task access module (HTTP interface) based on the ValidationAPI service performs task access and message encapsulation (executed by the business service layer and written to Kafka). First, task access request data (potentially batch scene / sku data) is input from the gateway. The task access module validates the request parameters (field integrity, type validity, maximum batch count, etc.) and generates an internally unified task data structure. It generates or extracts an "external ID / idempotent key" (e.g., a unique ID from an external system) for each task and encapsulates the task data into a task message (containing external ID, task type scene / sku, necessary business fields, creation time, etc.). The task message is written to a specified topic in Kafka for asynchronous peak smoothing. During writing, the write result is returned according to the system configuration (example: acks=1, but can be configured for a higher reliability level). The task access module returns an "acceptance result" to the caller, containing at least a list of successfully written external IDs or acceptance numbers for subsequent reconciliation / retry by the caller. The task messages in Kafka are consumed and stored in the database by the task cleaning and storage module.

[0049] Preferably, the task cleaning and storage module (Kafka consumer, deployed on the ValidationAPI service side or independent consumer service) is used to clean, deduplicate and store tasks (message and persistence layer collaboration, Kafka→MySQL). Consumers pull task messages from Kafka in batches, parsing them to obtain external IDs, task types, business fields, etc.; perform deduplication / idempotency filtering: querying MySQL using the external ID as the key or checking for existence using a unique index constraint; marking messages with existing external IDs as duplicates and skipping them from the database (or updating necessary fields according to business rules); performing data cleaning on tasks not yet entered into the database: field normalization, filling in default values, unifying time formats, and necessary business mappings (such as SKU mapping fields), forming task records that can be entered into the database; writing to MySQL in batches, and initializing the task status to "initialized" in the same transaction, while recording audit fields such as creation time and source; consumers only manually commit the corresponding Kafka offset after successful MySQL write; if the write fails, the offset is not committed so that message retry processing can be performed, thereby ensuring "at least once" processing semantics and avoiding data loss; outputting task record data in MySQL (status = initialized) and the committed Kafka consumer position.

[0050] Furthermore, the task preheating and queue loading process (business service layer scheduled task, MySQL→Redis) is as follows: Scheduled tasks are triggered periodically (e.g., at a fixed time each day or dynamically based on queue length), reading the length or threshold parameter of the pending zset queue in Redis; Based on the queue capacity and preheating strategy, query the "initialization" status tasks from MySQL according to conditions (sortable by creation time, priority, task type, etc.) to obtain the set of task IDs to be preheated. Write the task IDs to be preheated into a Redis zset queue (as a queue to be distributed) according to the sorting rules, and write the task details (or necessary fields) into the Redis cache, and set the expiration time (according to business configuration); Update the status of the corresponding task in MySQL to "preheated" to distinguish whether it has entered the distributable pool and avoid duplicate loading; Output the preheated task records, including the task ID data in the Redis zset queue, the task details data cached in Redis, and the status in MySQL.

[0051] Furthermore, after receiving the task request data from the front end (carrying user identity information from the gateway header) and Redis zset queue / cache data, the task distribution module (HTTP interface) of the ValidationAPI service and the RedisLua script execution unit perform task retrieval, locking, and execution distribution (business service layer, Redis→MySQL→front end). After receiving a request, the task distribution module calls a RedisLua script to atomically retrieve a task ID from the zset queue (the most suitable task can be selected based on score / time / priority), ensuring that the same task is not retrieved concurrently. After retrieving the task ID, the module reads the Redis cache or queries MySQL to obtain the task details and generates the task content to be returned to the front end. The task status in MySQL is synchronously updated to "locked," and information such as the claimant and retrieval time is recorded. After a successful update, the task ID is deleted from the Redispending zset to prevent it from being retrieved again. If the MySQL status update fails, the retrieval result on the Redis side is rolled back (e.g., the task ID is put back into the queue or marked as retryable) to maintain the consistency between the queue and the status. The locked task records and the task IDs removed from the Redis queue are output, where the locked task records include the task details data returned to the front end and the status in MySQL.

[0052] Specifically, the task submission module (HTTP interface) of the ValidationAPI service can optionally call the COS object storage to submit task results, complete the database entry, and associate files (business service layer, frontend / COS→MySQL). It retrieves the task result data submitted by the frontend (e.g., annotation results, validation results, conclusion fields, etc.) and optional image / video files or their reference information. If the submission includes files: the frontend or server uploads the files to COS and obtains the file key / URL; this key / URL is written as a file reference field into the submission data; the task submission module verifies the submission data and task status (must be locked / in progress, or a state where submission is allowed), and verifies whether the submitter is the recipient or has submission permissions; it writes the task result fields in a MySQL transaction and updates the task status to "completed," while recording the completion time, completer, result summary, and other information; if the write fails, an error is returned and the original status is maintained; if the write succeeds, a submission success response is returned.

[0053] Furthermore, it completes event capture, statistical updates, and downstream push (Canal monitors MySQL, MySQL → Redis / Kafka → downstream): 1) Canal continuously monitors the MySQL binlog, captures events where the task status changes from "locked / in progress" to "completed", and parses the change information such as task ID, completion time, and completer.

[0054] 2) For statistical data: Canal outputs incremental statistical data to Redis and uses atomic counting to update metrics such as "task pool quantity" and "today's completed quantity" for personal workbench / front-end statistical display.

[0055] 3) Pushing downstream: Canal writes the "Completed task event" output to the specified topic in Kafka.

[0056] 4) The task push module consumes completed events from Kafka, assembles the data format required by the downstream system, and pushes it to the downstream system; or the downstream system can directly subscribe to and consume Kafka topics to obtain completed task data.

[0057] 5) Output data: Statistical count data in Redis, completed event data in Kafka, and push data received downstream.

[0058] Preferably, Debezium captures MySQL changes and writes them to Kafka (Avro can be used and integrated with SchemaRegistry to support schema evolution); ClickHouseSinkConnector consumes Kafka change events and writes them to ClickHouse analysis tables, and can be configured with parameters such as automatic table creation, schema evolution, and batch flush to improve synchronization efficiency; the BI platform initiates queries on ClickHouse and displays reports; and outputs analysis data tables and BI report results from ClickHouse.

[0059] The beneficial effects of this invention are as follows: (1) This invention significantly reduces the probability of duplicate distribution and task loss by using state machine constraints and atomic retrieval / locking processes. It can maintain the consistency of task flow even in high-concurrency retrieval scenarios, thereby improving the correctness of task distribution and concurrency capabilities.

[0060] (2) The present invention achieves idempotency and traceability of task access. When external push occurs in the event of retry or repeated submission, it can still ensure that the same external business data is generated / accepted only once, and the processing progress and abnormal reasons can be quickly located through the acceptance record.

[0061] (3) This invention enables the completed result to be delivered in the event of a brief unavailability of the downstream or network jitter through a recoverable delivery and failure retry / deduplication mechanism, and supports tracking and auditing, thereby improving the reliability of the completed result delivery and reducing the risk of missed delivery.

[0062] (4) By governing and constraining the semantics of schema changes and deletions, this invention reduces the probability of synchronization anomalies, reduces data corruption caused by configuration or changes, improves the credibility of report data, and enhances the stability of the data analysis link and the consistency of statistical standards.

[0063] (5) This invention enables faster fault location, more controllable recovery, and reduced overall operation and maintenance costs through standardized deployment and configuration governance, link observable fields and monitoring and alarm system.

[0064] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0065] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A job task management method for model validation, characterized in that, include: Receive task push request data from an external backend and obtain the received data; The received data is verified, and requests that pass the verification are filtered to obtain task access request data; The task access request data is encapsulated into task access and message data to obtain task message data; The task message data is cleaned, deduplicated, and stored in the database to obtain task record data; The task record data is preheated and loaded into a queue to obtain preheated task records; Based on the preheated task record, the task access request data is retrieved, locked, and distributed to obtain the locked task record. Based on the locked task record, the task result data submitted by the front end is received, and the task result data is completed, stored in the database, and associated with files to obtain the completed task record. Based on the locked task records and the completed task records, the completion event is captured, statistically updated, and pushed downstream to obtain statistical count data, completed event data, and push data. The statistical count data and the completed event data are analyzed, synchronized, and queryed to obtain the analysis data table and BI report results.

2. The job task management method for model validation according to claim 1, characterized in that, The received data is verified, and requests that pass the verification are filtered to obtain task access request data, including: A health check is performed on the received data. If the check result indicates that the upstream is unavailable, a 502 status code is returned. Cross-domain control and cookie attribute enhancement are applied to the received data, and the frequency of the received data is limited according to the rate limiting configuration; The user request data is parsed using Cookie to obtain a JWTToken. The user identification information in the JWTToken is extracted, and the permission management interface that matches the user identification information is called according to the permission verification configuration to obtain the permission status. The X-API-Key of the task push request data is verified, and the verified task push request data and the user identification information with the permission status are integrated into the task access request data.

3. The job task management method for model validation according to claim 1, characterized in that, The task access request data is encapsulated into task access and messaging data to obtain task message data, including: The task access request data is subjected to multi-dimensional verification and structure generation to obtain the task data structure; the multi-dimensional verification items include: field integrity, type validity, and maximum number of batch records; Extract the external ID or idempotent key of each task access request data according to the task data structure, and encapsulate the verified task access request data into a task message to be processed according to the external ID or the idempotent key. The task message to be processed is written to a specified topic in Kafka to obtain the task message data.

4. The job task management method for model validation according to claim 1, characterized in that, The task message data is cleaned, deduplicated, and stored in the database to obtain task record data, including: The task message data is parsed, and the task message data is deduplicated or idempotently filtered according to the external ID obtained from the parsing to obtain filtered data. The filtered data is cleaned to obtain task records that can be stored in the database; the data cleaning includes: field normalization, missing value completion, time format unification, and necessary business mapping. Write the status of the task record that can be added to the database in the same transaction to the initialization status and create an audit field to obtain the task record data.

5. The job task management method for model validation according to claim 1, characterized in that, The task record data is preheated and loaded into a queue to obtain preheated task records, including: Periodically read the queue parameters of the pending zset in Redis, and perform status queries on the task record data according to the queue parameters and the preheating strategy to obtain a set of task IDs to be preheated; Write the set of task IDs to be preheated into a Redis zset queue and set an expiration time. Write the task content corresponding to the set of task IDs to be preheated into a Redis cache. Update the status of the set of task IDs to be preheated to the preheated status to obtain the preheated task record.

6. The job task management method for model validation according to claim 1, characterized in that, Based on the pre-warmed task records, the task access request data is processed to retrieve, lock, and distribute tasks, resulting in locked task records, including: Retrieve the task ID from the Redis zset queue; Based on the preheated task record corresponding to the task ID, generate the target task content to be returned to the front end, and update the task status corresponding to the target task content to the locked status; Record the information on who received the target task content; the information includes: the recipient and the time of receipt; By integrating the target task content, the locked status, and the claim information, the locked task record is obtained.

7. The job task management method for model validation according to claim 1, characterized in that, Based on the locked task records, the system receives task result data submitted by the front end, performs completion storage and file association on the task result data, and obtains completed task records, including: The task result data and task status are verified; The task result data that passes the verification is written into a MySQL transaction, and the task status corresponding to the task result data is updated to the completed status to obtain the completed task record.

8. The job task management method for model validation according to claim 1, characterized in that, Based on the locked task records and the completed task records, completion events are captured, statistically updated, and pushed downstream to obtain statistical count data, completed event data, and push data, including: The event that captures the change of task status from locked or in progress to completed is used to obtain change information; the change information includes: task ID, completion time, and completer; The statistical incremental data is updated using atomic counting to obtain the statistical count data and the completed event data. The completed task events corresponding to the change information are written to a specified topic in Kafka. The completed task events are assembled according to the data format set by the downstream system to obtain the push data, and the push data is pushed to the downstream system.

9. A job task management system for model validation, characterized in that, To implement the job task management method for model validation as described in claim 1, the system comprises: a presentation layer, a security layer, a gateway layer, a service layer, a persistence layer, an engineering layer, a monitoring and alarm layer, and a log tracing layer; the presentation layer embeds a Web frontend, an H5 frontend, an Electron client, and an externally accessible API; the security layer embeds firewall rules and access control policy components; the gateway layer embeds gateway services, a health check module, a cross-domain control module, a rate limiting module, and JWT. The system includes a token parsing module, a permission verification module, an API token verification module, a logging module, and a load balancing module. The service layer embeds an Auth service, a Validation API service, a scheduled module, and a data synchronization module. The persistence layer embeds a Kafka message queue, a Redis cache, a MySQL business database, a ClickHouse analytics database, and a COS object storage. The engineering layer embeds a Docker container and a Swagger API documentation tool. The monitoring and alerting layer embeds Exporter, Prometheus, Grafana, an alerting rule engine, and an alerting robot. The log tracing layer embeds a Logstash log collector, a Kafka message queue, an Elasticsearch log storage and retrieval cluster, and a Kibana log visualization and analysis platform. The gateway layer is connected to the presentation layer, the security layer, and the service layer respectively; the service layer is connected to the persistence layer and the engineering layer respectively; the monitoring and alarm layer is connected to the gateway layer, the service layer, the persistence layer, and the engineering layer respectively; the log tracing layer is connected to the presentation layer, the security layer, the gateway layer, the service layer, the persistence layer, the engineering layer, and the monitoring and alarm layer respectively.

Images