Data processing method and device for image understanding and visual question and answer tasks

Abstract

The invention discloses a data processing method and device for image understanding and visual question and answer tasks, which are used for solving the technical problems that tasks are easy to lose or accumulate for a long time, peak resources are congested or analysis is repeated in the prior art. The method comprises the following steps: when it is detected that a new file is written, screening a completely written picture file through stability identification; performing image metadata analysis on the picture file, and determining an event source type and an image path; according to the event source type and the image path, the picture file is distributed to a multi-channel AI queue for consumption processing in combination with a distribution strategy; and when flow limiting or QPS overrun occurs in the AI reasoning process, executing a flow limiting and persistent retry mechanism based on the queue consumption container until AI reasoning is completed and a reasoning result is returned.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method for image understanding and visual question answering tasks, a data processing device for image understanding and visual question answering tasks, an electronic device, and a storage medium. Background Technology

[0002] In the engineering implementation of computer vision tasks, a common practice is to write images captured by image acquisition devices (such as cameras) to a file directory or object storage, and then use message queues or stream processing frameworks (such as Kafka, RabbitMQ, and RedisStreams) to achieve concurrent consumption and backend analysis. Consumers typically use multi-threading to process the images, store the recognition results in the database, and trigger downstream actions (alarms, notifications, work orders, etc.).

[0003] In related processing, calls to external AI services (such as Visual Question Answering (VQA) and Image Retrieval and Generation (IRAG)) commonly use simple retry strategies (such as exponential backoff, delayed queues, or dead-letter queues). In high-concurrency or rate-limited scenarios, this approach can easily lead to task loss or long-term backlogs. Regarding multi-priority and time-planning control, a uniform rate or coarse-grained priority is typically used, which can easily cause peak resource congestion or duplicate analysis. Summary of the Invention

[0004] This invention provides a data processing method, a data processing device, an electronic device, and a storage medium for image understanding and visual question answering tasks, which are used to solve or partially solve the technical problems in related technologies, such as tasks being easily lost or accumulating for a long time, peak resource congestion, or repetitive analysis.

[0005] This invention provides a data processing method for image understanding and visual question answering tasks, the method comprising:

[0006] When a new file is detected being written, image files that have been completely written are selected based on stability identification.

[0007] The image file is parsed for image metadata to determine the event source type and image path;

[0008] Based on the event source type and image path, and combined with the distribution strategy, the image file is distributed to a multi-channel AI queue for consumption processing;

[0009] When rate limiting or QPS exceeding the limit occurs during AI inference, a rate limiting and persistent retry mechanism based on queue consumption containers is executed until AI inference is completed and the inference result is returned.

[0010] Optionally, the step of distributing the image file to a multi-channel AI queue for consumption processing based on the event source type and image path, combined with a distribution strategy, includes:

[0011] Based on the type of event source, determine the multi-channel AI queue into which the image file enters;

[0012] Acquire inputs including deployment status, weekly schedule, and system on / off status;

[0013] Based on the deployment status, weekly schedule, and system switch, the image file is retrieved from the image path and distributed to at least one AI channel queue in the multi-channel AI queue for consumption processing. After the consumption processing is completed, the queue tasks in the AI ​​channel queue are cleared.

[0014] Optionally, when the event source type is an internal location source, the multi-channel AI queue includes a visual question-answering AI channel queue and an image understanding AI channel queue, and the visual question-answering AI channel queue includes a first priority queue and a second priority queue; when the event source type is a third-party source, the multi-channel AI queue includes an image understanding AI channel queue.

[0015] Optionally, each AI channel of the multi-channel AI queue is configured with a corresponding queue consumption container; when rate limiting or QPS exceeding the limit occurs during AI inference, a rate limiting and persistent retry mechanism based on the queue consumption container is executed until AI inference is completed and the inference result is returned, including:

[0016] When rate limiting or QPS exceeding the limit occurs during AI inference, the target AI channel queue where rate limiting or QPS exceeding the limit occurs is identified, and the consumption processing of the queue consumption container corresponding to the target AI channel queue is suspended.

[0017] Obtain the task parameters of the currently processed queue task in the target AI channel queue, and serialize the task parameters into a ZSet for ordered retries. At the same time, start a lightweight deduplication lock mechanism to avoid concurrent duplicate consumption.

[0018] Periodically scan the expired tasks in the ZSet in batches;

[0019] If the retry of the currently expired task is successful, delete the retry record of the currently expired task, clean up temporary files, and restore the rate limiting state;

[0020] If the retry of the currently expired task fails and the error type is a non-rate-limiting error, then the file of the currently expired task will be cleaned up and the retry will be terminated.

[0021] When all the due tasks of this scan are completed, the queue consumption container corresponding to the target AI channel queue resumes normal processing and automatically continues to execute the consumption processing of historical compensation and new message queues.

[0022] When each AI channel of the multi-channel AI queue has successfully completed the consumption process, the AI ​​inference is completed and the inference result is returned.

[0023] Optionally, the method further includes:

[0024] During AI inference, when the thread pool reaches its maximum load, a rejection protection mechanism is activated for the current processing task entering the thread pool, and the task parameters of the current processing task are extracted.

[0025] The task parameters of the currently processed task are persisted to a ZSet, and the corresponding task parameters of the currently processed task are automatically deleted from the original queue.

[0026] Optionally, when the event source type of the image file is an internal location source, the method further includes:

[0027] After completing AI inference and successfully returning the inference result, event-triggered actions are executed in sequence to form a complete closed-loop event processing chain. The event-triggered actions include event rule matching, deduplication and event storage, increasing the hit count, linked outbound calls, multi-level image archiving, generating fixed-size thumbnails, and cleaning up temporary files.

[0028] Optionally, the method further includes:

[0029] The message read / write latency, queue consumption container running status, and queue backlog of the multi-channel AI queue are periodically detected.

[0030] When an anomaly is detected, automatic recovery actions are performed sequentially on the queue consumption containers that experienced the anomaly; wherein, the automatic recovery actions include stopping, rebuilding, starting, and restoring historical compensation.

[0031] The present invention also provides a data processing apparatus for image understanding and visual question answering tasks, the apparatus comprising:

[0032] The image file detection unit is used to filter fully written image files by stability identification when a new file is detected being written.

[0033] The metadata parsing unit is used to parse the image metadata of the image file to determine the event source type and image path;

[0034] The queue distribution unit is used to distribute the image file to a multi-channel AI queue for consumption processing based on the event source type and image path, combined with a distribution strategy.

[0035] The consumption retry unit is used to execute a rate limiting and persistent retry mechanism based on the queue consumption container when rate limiting or QPS exceeding occurs during AI inference, until the AI ​​inference is completed and the inference result is returned.

[0036] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0037] The memory is used to store program code and transmit the program code to the processor;

[0038] The processor is configured to execute, according to instructions in the program code, the data processing method for image understanding and visual question answering tasks as described above.

[0039] The present invention also provides a computer-readable storage medium for storing program code for performing the data processing method for image understanding and visual question answering tasks as described in any of the preceding claims.

[0040] As can be seen from the above technical solutions, the present invention has the following advantages:

[0041] This paper presents a data processing method for image understanding and visual question answering tasks. When a new file is detected being written, stability identification is used to filter fully written image files to ensure the integrity and validity of the input images. Image metadata is parsed to determine the event source type and image path for subsequent multi-channel AI queue distribution. Based on the event source type and image path, and combined with a distribution strategy, the image files are distributed to multi-channel AI queues for consumption processing. This multi-channel distribution and consumption processing alleviates peak resource congestion. When rate limiting or QPS exceeding limits occurs during AI inference, a rate limiting and persistent retry mechanism based on the queue consumption container is executed until AI inference is completed and the inference result is returned. This rate limiting and persistent retry mechanism based on the queue consumption container automatically pauses the corresponding streaming consumption when the external AI service experiences rate limiting or failure, and persists the task parameters of the rate-limited or failed task to an ordered retry queue for ordered retries. This avoids task loss and prevents long-term backlog and redundant analysis. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating the steps of a data processing method for image understanding and visual question answering tasks;

[0044] Figure 2 This is a schematic diagram of the overall process of a data processing method for image understanding and visual question answering tasks.

[0045] Figure 3 This is a block diagram of a data processing device for image understanding and visual question answering tasks. Detailed Implementation

[0046] This invention provides a data processing method, a data processing device, an electronic device, and a storage medium for image understanding and visual question answering tasks, which are used to solve or partially solve the technical problems in related technologies, such as tasks being easily lost or accumulating for a long time, peak resource congestion, or repetitive analysis.

[0047] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0048] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present invention, some of the technical features involved in the solutions are briefly described first:

[0049] Consumer Group: A logical concept used to describe the collaborative relationship of a group of consumer instances, primarily used for load balancing and fault tolerance.

[0050] Consumer Container: A physical or logical runtime environment used to host consumer instances, and is related to containerization technology.

[0051] Historical compensation refers to compensating for signals lost or damaged during past transmissions. A historical compensation channel is a dedicated channel used to store or transmit historical compensation information.

[0052] ZSet: A data structure in the Redis database, suitable for scenarios requiring ordered and unique elements. It combines the uniqueness of sets with the efficiency of sorting, enabling fast processing of leaderboards, priority queues, and other similar requirements.

[0053] As an example, in the engineering implementation of computer vision tasks, a common practice is to write images captured by image acquisition devices (such as cameras) to a file directory or object storage, and then use message queues or stream processing frameworks (such as Kafka, RabbitMQ, and Redis Streams) to achieve concurrent consumption and backend analysis. Consumers typically use multi-threading to process the images, store the recognition results in the database, and trigger downstream actions (alarms, notifications, work orders, etc.).

[0054] In related processing, calls to external AI services (such as image question answering (VQA) and image understanding / embedded IRAG) commonly use simple retry strategies (such as exponential backoff, delayed queues, or dead-letter queues). In high-concurrency or rate-limited scenarios, this approach can easily lead to task loss or long-term backlogs. Regarding multi-priority and time-planning control, a uniform rate or coarse-grained priority is typically used, which can easily cause peak resource congestion or duplicate analysis.

[0055] Further analysis reveals that current data processing methods for image question answering and image understanding still have the following shortcomings:

[0056] Current practices for container health and self-healing primarily rely on general orchestrators or manual intervention. The engineering loop for online health checks, synchronous restarts in case of anomalies, and automatic compensation for historical messages in "queue consumption containers" is incomplete. The lack of online health checks and synchronous restart mechanisms after connection or operational anomalies in queue consumption containers makes container health invisible, hinders the automatic compensation and continuation of historical message consumption, and results in untimely failure recovery.

[0057] In terms of event management and evidence retention, the deduplication and data entry of identification results, linked invocation, hierarchical image storage, and thumbnail generation lack a unified and rollbackable engineering loop, resulting in a poor auditing and retrieval experience. There is still room for improvement in rollback and recovery of failed paths and in the consistency of audit retrieval.

[0058] For images from third-party sources (with incomplete metadata and no internal location rules), conventional solutions are prone to misjudgment or processing failure, lacking compatible branch paths. Forcibly applying internal rules to external third-party images during data processing easily leads to failure or misjudgment. Currently, there is a lack of a compatible branch for image understanding / embedding only.

[0059] In rate-limiting scenarios, current methods mainly rely on memory queues and simple retries, lacking an integrated mechanism of "pausing streaming consumption + disk sorting queue + breakpoint resume". This makes it impossible to guarantee the stable progress and orderly recovery of tasks under high-pressure rate limiting, and tasks are prone to being lost or piling up for a long time.

[0060] Meanwhile, there is a lack of persistent retries and concurrent deduplication locks. Specifically, when external services return rate-limiting codes or transient failures, task parameters cannot be completely written to disk and processed in an orderly manner according to time scores, and lightweight locks are not set in concurrent scenarios to prevent duplicate execution.

[0061] The current thread pool rejection policy is not geared towards business semantics. Common Abort / Discard / CallerRuns policies are prone to issues such as lost tasks and blocked calling threads, failing to perform "recoverable business-level rollback" based on task parameters upon rejection.

[0062] Furthermore, the current priority and time / interval control are coarse, making it impossible to throttle resources based on the idempotency judgment of "weekly time plan + atomic interval bond", and making it difficult to avoid redundant analysis and resource waste when dynamically changing intervals.

[0063] Therefore, one of the core inventive points of this invention is to provide a queued data processing method for image understanding and visual question answering tasks. Through a rate-limiting and persistent retry mechanism based on a queued consumption container, when an external AI service experiences rate limiting or failure, the corresponding streaming consumption is automatically paused. Combined with a lightweight deduplication lock mechanism and timed batch processing, the parameters of the rate-limited or failed tasks are persistently written into an ordered retry queue for ordered retries, thus avoiding task loss and preventing long-term backlog and repetitive analysis. When the thread pool reaches its maximum load, the parameterized rejection and rollback mechanism of the thread pool can prevent task loss and alleviate peak resource congestion. Online health monitoring and synchronous restart of the consumer container are performed, and historical message processing is automatically restored. At the business level, fine-grained throttling and idempotency judgment based on priority and weekly time plans are supported, providing event-based management and closed-loop linkage, as well as compatibility processing for images from third-party sources, thereby maintaining stable throughput, improving availability, and engineering reliability under high concurrency and rate-limiting conditions.

[0064] Reference Figure 1The diagram illustrates a flowchart of a data processing method for image understanding and visual question answering tasks provided by an embodiment of the present invention, which may specifically include the following steps:

[0065] Step 101: When a new file is detected being written, the image files that have been completely written are filtered out using stability identification.

[0066] In practical applications, the first step is to perform image input acquisition and stability assessment. This step can be achieved by setting up a file monitoring module. Within this module, a file system listener (WatchService / inotify) monitors newly captured images or image files input through other means; and a file lock detection and stability polling algorithm is used to perform write stability assessment and type filtering to ensure the integrity and validity of the input image.

[0067] In the specific implementation, the first step is file monitoring triggering. The system monitors newly added image files (jpg / jpeg / png) in a specified directory. When a new image file is detected being written, the monitoring process is automatically initiated. The second step is file write stability identification. Methods such as polling for file size changes and checking for read-only locks are used to identify when an image file has been completely written, while partially written or incomplete files are filtered out to ensure the stability of subsequent AI inference and analysis.

[0068] Step 102: Parse the image metadata of the image file to determine the event source type and image path;

[0069] This step primarily involves parsing image metadata from the complete image files selected in the preceding steps. This step can be implemented by setting up a metadata building module. Within this module, firstly, based on string templates, the timestamp, pointId, and event source of the image file are parsed according to the naming template. Then, based on the parsed information, a standardized payload is constructed using regular expressions. Specifically, information such as timestamp, pointId, eventId, and endpoint is parsed from the filename to construct a standardized payload containing point attributes, event source type, and image path.

[0070] Step 103: Based on the event source type and image path, and combined with the distribution strategy, distribute the image file to a multi-channel AI queue for consumption processing;

[0071] This step primarily involves distributing the images to be inferred to multi-channel AI queues and consuming the corresponding queues. The queue distribution step can be implemented by setting up a distribution module. Within this module, a distribution strategy engine and I / O file copying are combined to distribute the images to the Visual Question Answering (VQA) / Image Understanding (IRAG) queues according to the strategy and perform file copying.

[0072] The previous step used metadata parsing to determine the event source type of the image file. Based on the event source type, we can distinguish whether the image file originates from an internal location or a third-party source. An internal location refers to an image uploaded via video frames captured through web page operations using RTSP (Real-Time Streaming Protocol) or ONVIF protocols. A third-party source refers to images uploaded by other developers through APIs (Application Programming Interfaces) provided by the backend service.

[0073] Furthermore, the video streams acquired through the protocol at the monitoring points can continuously generate frame-by-frame images of the same scene, which can then be used for deployment analysis using visual question answering. Therefore, image files from internal monitoring points can be used for both Visual Question Answering (VQA) and Image Recognition and Analysis (IRAG) AI channels.

[0074] Third-party image files are not necessarily continuous visuals; they may be images generated by alarms or images from a specific event (one of the two), thus failing to demonstrate deployment capabilities. Therefore, third-party image files only enter the image understanding channel and do not undergo event closure or image archiving.

[0075] In this embodiment of the invention, according to the deployment priority (high / low), the Visual Question Answering (VQA) AI channel queue is set up with two queues: a high-priority queue (defined as the first priority queue for easy distinction) and a low-priority queue (defined as the second priority queue), while the Image Understanding (IRAG) AI channel queue is set up with only one queue.

[0076] The principle behind the above queue setup is that the results of Visual Question Answering (VQA) are used to report event records for newly deployed controls. These event records need to be reported in real time. Based on the priority (high / low) of the controls, the order is: Deployment (High) > Deployment (Low) > Image Understanding (IRAG). During data processing, all GPU (Graphics Processing Unit) resources should be prioritized for deployment (corresponding to the visual question answering task), with the remaining image understanding resources used for text retrieval. This indicates that image understanding does not have strict real-time requirements and can be completed asynchronously.

[0077] The execution of the multi-channel AI queue distribution strategy is related to the deployment status, weekly time plan, and system on / off status input by the user through the interface. The deployment status is controlled by the true / false buttons on the interface. The weekly time plan is a time control that allows users to select a seven-day period by sliding the mouse. Deployment is only effective if the current time falls within the weekly time plan. The system on / off status includes the selection of visual question-answering priority (high / low) and whether to enable image understanding. Users can choose these options themselves, and different conditions and tasks will be executed based on the true / false status of the on / off status in the business logic.

[0078] For multi-channel AI queue distribution, the system determines whether to distribute image files from internal locations to the Visual Question Answering (VQA) high-priority / low-priority queue or the Image Understanding (IRAG) queue, based on deployment status, weekly schedule, and system on / off status. For image files from third-party sources, the system determines whether to distribute them to the Image Understanding (IRAG) queue.

[0079] Since third-party image files can only be processed through the Image Understanding (IRAG) channel, they are generally assigned to the IRAG queue for consumption by default. However, in another scenario, if the user deems it unnecessary to perform inference processing on a third-party image file, they can manually disable image understanding via a system switch, and the system will directly ignore further inference processing for that image file.

[0080] After the multi-channel AI queue has finished processing, the source file is cleaned up to avoid duplicate consumption.

[0081] In this embodiment of the invention, an AI consumption container processing and event closed-loop mechanism is set up. It mainly includes two parts: first, independent queues and container scheduling; second, event-based closed loop.

[0082] Independent queues and container scheduling can be implemented through a streaming consumption module. Using message queues (Redis Stream / Kafka), consumer group mechanisms, and containerized execution frameworks (threaded containers / lightweight Actors), independent consumption containers are provided for high-priority / low-priority queues for Visual Question Answering (VQA) and Image Understanding (IRAG) queues. By creating independent historical compensation channels for each queue, real-time reception of new messages and historical message compensation can be supported. Specifically, independent consumer groups, consumption containers, and historical compensation channels are created for both high-priority / low-priority queues for VQA and Image Understanding (IRAG) to support continuous streaming processing under high concurrency.

[0083] Event-driven closed-loop processing refers to a complete closed-loop processing chain formed by sequentially executing a series of event-triggered actions after the model successfully calls the image path and completes AI inference processing. This step can be achieved by setting up an event-driven and linkage module. Specifically, after completing AI inference and successfully returning the inference result, event rule matching is performed based on an event rule engine; deduplication and event entry are performed based on a relational database; the hit count is incremented based on a counter; linked outbound calls (extension / IVR calls) are implemented based on IVR (Interactive Voice Response) / Extension SDK (Software Development Kit); multi-level image archiving (location / year / month / day) and fixed-size thumbnail generation are completed based on image processing (Thumbnailator / Scalr) and file archiving systems; and temporary files are cleaned up using data cache cleanup methods. By sequentially executing the above series of event-triggered actions, a complete event-driven closed-loop processing chain is formed.

[0084] It should be noted that, for ease of understanding, the above steps present a complete closed-loop event processing chain. In practical applications, the execution / skipping of a certain action can be selected according to different scenarios. For example, for notification-type events, such as when a control event is triggered, it is necessary to notify specific personnel or the front desk, which requires the use of linked outbound calls (extension / IVR calls). For non-notification-type events, only the event is reported, and the notification action is not executed (i.e., the linked outbound call action is skipped). It is understood that this invention does not impose any limitations on this.

[0085] For the final image archiving and management, successfully processed images are stored in layers according to "location / year / month / day" for easy management and retrieval. Fixed-size (e.g., 200×200) thumbnails are generated uniformly for quick front-end display.

[0086] Based on the above, the specific implementation process of distributing image files to multi-channel AI queues for consumption processing according to the event source type and image path, combined with the distribution strategy, can include: determining the multi-channel AI queue into which the image file enters based on the event source type; obtaining the input deployment status, weekly time plan, and system switch; based on the deployment status, weekly time plan, and system switch, calling the image file from the image path to distribute it to at least one AI channel queue in the multi-channel AI queue for consumption processing, and clearing the queue tasks in the AI ​​channel queue after consumption processing is completed.

[0087] Specifically, when the event source type is internal location, the multi-channel AI queue includes a visual question-answering AI channel queue and an image understanding AI channel queue. The visual question-answering AI channel queue includes a first-priority queue and a second-priority queue. When the event source type is third-party, the multi-channel AI queue includes an image understanding AI channel queue.

[0088] Furthermore, when the event source type of the image file is an internal location source, after completing AI inference and successfully returning the inference result, the event triggering actions are executed sequentially to form a complete closed-loop event processing chain. Among them, the sequentially executed event triggering actions mainly include event rule matching, deduplication judgment and event storage, increasing the hit count, linked outbound calls, multi-level image archiving, generating fixed-size thumbnails, and cleaning up temporary files.

[0089] Step 104: When rate limiting or QPS exceeding the limit occurs during AI inference, execute the rate limiting and persistent retry mechanism based on the queue consumption container until the AI ​​inference is completed and the inference result is returned.

[0090] This step primarily involves implementing a rate-limiting and persistent retry mechanism when rate limiting or QPS (Queries Per Second) limits are exceeded during AI inference. This step can be implemented by setting up a rate-limiting and persistent retry module. The principle is as follows: firstly, when the AI ​​model encounters rate limiting or QPS exceeding issues during inference, container consumption is automatically paused; secondly, based on Redis ZSet, serialization tools (Jackson / FST), and a lightweight distributed lock (SET NX EX), the task is serialized and written to the ZSet, while a lightweight lock is attached for deduplication; furthermore, a scheduled task timer is used to periodically batch-process retry tasks, and upon successful retry, the rate-limiting state is cleaned up and reset.

[0091] Specifically, the rate limiting and persistent retry mechanism based on queue consumption containers mainly includes the following:

[0092] Rate limiting detection and container pause mechanism: When the AI ​​model returns a rate limit or QPS exceeds the limit, the consumption of the corresponding queue container is immediately paused to prevent backlog.

[0093] Task serialization write-back mechanism: The task parameters (messages, files, DTOs (Data Transfer Objects), and context) of the currently processed task in the queue are serialized and written to a ZSet. The ZSet score is equal to the current timestamp to enable ordered retries.

[0094] Lightweight deduplication lock mechanism: Uses an atomic lock of "queue key + score combination" to avoid concurrent duplicate consumption.

[0095] Scheduled batch processing retry mechanism: The scheduler scans expired tasks in the ZSet in batches and performs corresponding follow-up processing actions according to different situations.

[0096] Scenario 1: The current expired task successfully retried, the retry record was deleted, temporary files were cleaned up, and the rate limiting state was restored. The reason for restoring the rate limiting state is that there are still expired tasks in the ZSet that haven't been processed. "Expired" in the term "expired task" refers to a countdown. For example, if a scan is performed every 5 seconds and a task is found to be written to the ZSet, the retry process is executed. The current record is processed based on the retry result, and then the rate limiting state is restored. The purpose is to prevent the previous distribution strategy from continuing to process image logic. Instead, the distributed messages are written to the ZSet first, and then consumed gradually until no more rate limiting errors are returned, at which point the rate limiting state is restored.

[0097] Scenario 2: If the current expired task fails to be retried but is not due to a rate limiting error, then perform a complete cleanup and terminate the retry.

[0098] Scenario 3: After all the due tasks of this scan are completed, the queue consumption container resumes normal processing, and the container automatically continues to consume historical compensation and new messages.

[0099] In the specific implementation, based on the preceding discussion, each AI channel of the multi-channel AI queue is configured with a corresponding queue consumption container. When rate limiting or QPS exceeding the limit occurs during AI inference, the implementation process, which executes a rate limiting and persistent retry mechanism based on the queue consumption container until AI inference is completed and the inference result is returned, can include: when rate limiting or QPS exceeding the limit occurs during AI inference, identifying the target AI channel queue where the rate limiting or QPS exceeding the limit occurred, and pausing the consumption processing of the queue consumption container corresponding to the target AI channel queue; obtaining the task parameters of the currently processed queue tasks of the target AI channel queue, serializing the task parameters and writing them into a ZSet for ordered retries, while simultaneously initiating a lightweight deduplication lock mechanism to avoid concurrent duplicate consumption; and periodically scanning the expired tasks in the ZSet in batches.

[0100] In the first case, if the retry of the currently expired task is successful, the retry record of the currently expired task is deleted, temporary files are cleaned up, and the rate limiting state is restored.

[0101] In the second case, if the retry of the currently expired task fails and the error type is a non-rate-limiting error, then the file of the currently expired task is cleaned up and the retry is terminated.

[0102] In the third case, when all the due tasks of this scan have been completed, the queue consumption container corresponding to the target AI channel queue resumes normal processing and automatically continues to process historical compensation and new message queue consumption.

[0103] When each AI channel in the multi-channel AI queue has successfully completed consumption processing, the AI ​​inference is completed and the inference result is returned.

[0104] In some embodiments, this invention also designs a thread pool rejection protection and task rollback mechanism by constructing a thread pool execution and rejection rollback module. The principle is as follows: image analysis tasks are executed under high concurrency using Java ThreadPoolExecutor; based on ThreadPoolExecutor, the default rejection policy is extended through the RejectedExecutionHandler interface to capture thread pool rejection events; and combined with DTO parameter serialization and ZSet persistent write-back, task parameters are rolled back to the retry queue and the original stream message is deleted.

[0105] Specifically, when the thread pool reaches its maximum load, a rejection protection mechanism is triggered, and the task parameters of failed tasks at the time of rejection are collected, including message content, file path, DTO parameters, and runtime context. The complete and persistent task parameters of the failed tasks are written to a ZSet, and the original stream messages (i.e., the task parameters of the failed tasks in the original queue) are actively deleted to ensure that tasks are not lost, do not block the queue, and can be automatically compensated in subsequent rounds.

[0106] In the specific implementation, during the AI ​​inference process, when the thread pool reaches its maximum load, a rejection protection mechanism is enabled for the current processing task entering the thread pool, and the task parameters of the current processing task are extracted; the task parameters of the current processing task are completely persisted to ZSet, and the corresponding task parameters of the current processing task in the original queue are automatically deleted.

[0107] In some embodiments, this invention also designs a health monitoring and self-healing mechanism by constructing a container health monitoring and self-healing module. The principle is to continuously and periodically monitor the message system's read / write latency, queue backlog, and container running status through heartbeat detection and indicator collection (queue latency / backlog); based on automatic recovery scripts and container lifecycle management, when an anomaly is detected, a "stop → rebuild → start" process is executed on the abnormal queue consumption container, and historical compensation is restored to ensure the system continues to run stably.

[0108] In the specific implementation, during the data processing, the message read / write latency of the multi-channel AI queue, the running status of the queue consumption container, and the queue backlog are periodically detected. When an anomaly is detected, the abnormal queue consumption container is executed with automatic anomaly recovery actions in sequence. The automatic anomaly recovery actions include stopping, rebuilding, starting, and restoring historical compensation.

[0109] In some embodiments, this invention also designs a weekly time plan and atomic interval throttling mechanism by constructing a time plan and atomic interval throttling module. The principle is that, based on weekly plan configuration parsing, the weekly time plan controls the image analysis window, supporting multi-time period, multi-day plans, and precise analysis window constraints; based on Redis atomic keys (GETSET / SET EX) and a concurrency idempotency determination algorithm, atomic keys + expiration time are used to achieve concurrently safe interval control, supporting real-time dynamic changes to interval values ​​in the background to avoid repeated triggering of analysis at the same location within a short period.

[0110] This invention provides a queued data processing method for image understanding and visual question answering tasks. By employing a rate-limiting and persistent retry mechanism based on a queued consumption container, when an external AI service experiences rate limiting or failure, the corresponding streaming consumption is automatically paused. Combined with a lightweight deduplication lock mechanism and timed batch processing, the parameters of the rate-limited or failed tasks are persistently written to an ordered retry queue for ordered retries, preventing task loss and avoiding long-term backlogs and redundant analysis. When the thread pool reaches its maximum load, the parameterized rejection and rollback mechanism of the thread pool can prevent task loss and alleviate peak resource congestion. Online health monitoring and synchronous restart of the consumer container are performed, and historical message processing is automatically restored. At the business level, fine-grained throttling and idempotency determination based on priority and weekly schedules are supported, providing event-based management and closed-loop linkage, as well as compatibility processing for images from third-party sources, thereby maintaining stable throughput, improving availability, and engineering reliability under high concurrency and rate-limiting conditions.

[0111] For better illustration, refer to Figure 2 This diagram illustrates the overall flow of a data processing method for image understanding and visual question answering tasks according to an embodiment of the present invention. It should be noted that this embodiment only provides a brief description of the general flow of data processing for image understanding and visual question answering tasks. The specific implementation process of each step can be understood by referring to the relevant content in the foregoing embodiments, and will not be elaborated here. It is understood that the present invention does not impose any limitations on this.

[0112] Step 201: When a new file is detected being written, the image files that have been completely written are filtered through stability identification, and the image metadata of the image files is parsed to determine the event source type and image path;

[0113] Step 202: Based on the event source type, determine the multi-channel AI queue into which the image file enters. Based on the input deployment status, weekly time plan, and system switch, call the image file from the image path and distribute it to at least one AI channel queue in the multi-channel AI queue for consumption processing. After the consumption processing is completed, clean up the queue tasks in the AI ​​channel queue.

[0114] Step 203: When rate limiting or QPS exceeding the limit occurs during AI inference, execute the rate limiting and persistent retry mechanism based on the queue consumption container until the AI ​​inference is completed and the inference result is returned;

[0115] Step 204: When the thread pool reaches its maximum load during AI inference, enable the rejection protection mechanism for the current processing task entering the thread pool, extract the task parameters of the current processing task and persist them completely to ZSet, and automatically delete the corresponding task parameters of the current processing task from the original queue.

[0116] Step 205: When the event source type is internal location source, after completing AI inference and successfully returning the inference result, the following actions are executed in sequence: event rule matching, deduplication judgment and event entry into the database, increasing the number of hits, linking outbound calls, multi-level archiving of images, generating fixed-size thumbnails, and cleaning up temporary files, so as to form a complete closed-loop event processing chain.

[0117] Reference Figure 3 The diagram illustrates a structural block diagram of a data processing device for image understanding and visual question answering tasks provided by an embodiment of the present invention, which may specifically include:

[0118] The image file detection unit 301 is used to filter the completely written image files by stability identification when a new file is detected being written.

[0119] Metadata parsing unit 302 is used to parse image metadata of the image file to determine the event source type and image path;

[0120] The queue distribution unit 303 is used to distribute the image file to a multi-channel AI queue for consumption processing based on the event source type and image path, combined with a distribution strategy.

[0121] The consumption retry unit 304 is used to execute a rate limiting and persistence retry mechanism based on the queue consumption container when rate limiting or QPS exceeding occurs during AI inference, until the AI ​​inference is completed and the inference result is returned.

[0122] In one alternative embodiment, the queue distribution unit 303 includes:

[0123] A multi-channel AI queue determination unit is used to determine the multi-channel AI queue into which the image file enters based on the event source type.

[0124] The input information acquisition unit is used to acquire the input deployment status, weekly schedule, and system on / off status.

[0125] The queue distribution subunit is used to call the image file from the image path and distribute it to at least one AI channel queue in the multi-channel AI queue for consumption processing according to the deployment status, weekly time plan and system switch, and to clean up the queue tasks in the AI ​​channel queue after the consumption processing is completed.

[0126] In one optional embodiment, when the event source type is an internal location source, the multi-channel AI queue includes a visual question-answering AI channel queue and an image understanding AI channel queue, and the visual question-answering AI channel queue includes a first priority queue and a second priority queue; when the event source type is a third-party source, the multi-channel AI queue includes an image understanding AI channel queue.

[0127] In one optional embodiment, each AI channel of the multi-channel AI queue is provided with a corresponding queue consumption container; the consumption retry unit 304 includes:

[0128] The consumption processing pause unit is used to identify the target AI channel queue that has experienced rate limiting or QPS exceeding the limit when rate limiting or QPS exceeding the limit occurs during AI inference, and to pause the consumption processing of the consumption container corresponding to the target AI channel queue.

[0129] The ordered retry unit is used to obtain the task parameters of the current processing queue task of the target AI channel queue, and serialize the task parameters into ZSet for ordered retries. At the same time, a lightweight deduplication lock mechanism is started to avoid concurrent duplicate consumption.

[0130] The due task scanning unit is used to periodically scan the due tasks in the ZSet in batches.

[0131] The first execution unit for expired tasks is used to delete the retry record of the current expired task, clean up temporary files, and restore the rate limiting state when the current expired task is successfully retried.

[0132] The second execution unit for the due task is used to clean up the file of the current due task and terminate the retry when the retry execution of the current due task fails and the error type is a non-rate-limiting error.

[0133] The consumption recovery unit is used to restore the consumption container of the target AI channel queue to normal processing when all the due tasks of this scan have been completed, and to automatically continue to execute the consumption processing of historical compensation and new message queues.

[0134] The AI ​​inference completion unit is used to complete AI inference and return the inference result when each AI channel of the multi-channel AI queue has successfully completed the consumption process.

[0135] In one alternative embodiment, the device further includes:

[0136] The denial protection startup unit is used to enable the denial protection mechanism for the current processing task entering the thread pool when the thread pool reaches its maximum load during the AI ​​inference process, and to extract the task parameters of the current processing task.

[0137] The task rollback unit is used to persist the complete task parameters of the currently processed task to a ZSet and automatically delete the corresponding task parameters of the currently processed task from the original queue.

[0138] In one alternative embodiment, the device further includes:

[0139] The event-triggered action execution unit is used to execute event-triggered actions in sequence after completing AI inference and successfully returning the inference result, so as to form a complete event closed-loop processing link. The event-triggered actions include event rule matching, deduplication judgment and event storage, increasing the hit count, linked outbound calls, multi-level image archiving, generating fixed-size thumbnails, and cleaning up temporary files.

[0140] In one alternative embodiment, the device further includes:

[0141] The periodic detection unit is used to periodically detect the message read / write latency, queue consumption container running status, and queue backlog of the multi-channel AI queue.

[0142] An automatic recovery action execution unit is used to execute automatic recovery actions in sequence on the queue consumption containers that have detected the abnormality when an abnormality is detected; wherein, the automatic recovery actions include stopping, rebuilding, starting, and restoring historical compensation.

[0143] As the device embodiment is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment above.

[0144] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, the embodiments of the present invention use "first" and "second" to distinguish and describe some technical features. "First" and "second" are only used to distinguish data and have no other special meaning. It is understood that the present invention does not impose any limitations on them.

[0145] This invention also provides an electronic device, which includes a processor and a memory:

[0146] The memory is used to store program code and transfer the program code to the processor;

[0147] The processor is used to execute, according to instructions in the program code, a data processing method for image understanding and visual question answering tasks according to any embodiment of the present invention.

[0148] This invention also provides a computer-readable storage medium for storing program code for executing the data processing method for image understanding and visual question answering tasks according to any embodiment of this invention.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0151] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method for image understanding and visual question answering tasks, characterized in that, include: When a new file is detected being written, image files that have been completely written are selected based on stability identification. The image file is parsed for image metadata to determine the event source type and image path; Based on the event source type and image path, and combined with the distribution strategy, the image file is distributed to a multi-channel AI queue for consumption processing; When rate limiting or QPS exceeding the limit occurs during AI inference, a rate limiting and persistent retry mechanism based on queue consumption containers is executed until AI inference is completed and the inference result is returned.

2. The data processing method for image understanding and visual question answering tasks according to claim 1, characterized in that, The step of distributing the image file to a multi-channel AI queue for consumption processing based on the event source type and image path, combined with a distribution strategy, includes: Based on the type of event source, determine the multi-channel AI queue into which the image file enters; Acquire inputs including deployment status, weekly schedule, and system on / off status; Based on the deployment status, weekly schedule, and system switch, the image file is retrieved from the image path and distributed to at least one AI channel queue in the multi-channel AI queue for consumption processing. After the consumption processing is completed, the queue tasks in the AI ​​channel queue are cleared.

3. The data processing method for image understanding and visual question answering tasks according to claim 2, characterized in that, When the event source type is an internal location source, the multi-channel AI queue includes a visual question answering AI channel queue and an image understanding AI channel queue, and the visual question answering AI channel queue includes a first priority queue and a second priority queue; when the event source type is a third-party source, the multi-channel AI queue includes an image understanding AI channel queue.

4. The data processing method for image understanding and visual question answering tasks according to claim 1, characterized in that, Each AI channel of the multi-channel AI queue is configured with a corresponding queue consumption container. When rate limiting or QPS exceeding the limit occurs during AI inference, a rate limiting and persistent retry mechanism based on the queue consumption container is executed until AI inference is completed and the inference result is returned, including: When rate limiting or QPS exceeding the limit occurs during AI inference, the target AI channel queue where rate limiting or QPS exceeding the limit occurs is identified, and the consumption processing of the queue consumption container corresponding to the target AI channel queue is suspended. Obtain the task parameters of the currently processed queue task in the target AI channel queue, and serialize the task parameters into a ZSet for ordered retries. At the same time, start a lightweight deduplication lock mechanism to avoid concurrent duplicate consumption. Periodically scan the expired tasks in the ZSet in batches; If the retry of the currently expired task is successful, delete the retry record of the currently expired task, clean up temporary files, and restore the rate limiting state; If the retry of the currently expired task fails and the error type is a non-rate-limiting error, then the file of the currently expired task will be cleaned up and the retry will be terminated. When all the due tasks of this scan are completed, the queue consumption container corresponding to the target AI channel queue resumes normal processing and automatically continues to execute the consumption processing of historical compensation and new message queues. When each AI channel of the multi-channel AI queue has successfully completed the consumption process, the AI ​​inference is completed and the inference result is returned.

5. The data processing method for image understanding and visual question answering tasks according to claim 1, characterized in that, Also includes: During AI inference, when the thread pool reaches its maximum load, a rejection protection mechanism is activated for the current processing task entering the thread pool, and the task parameters of the current processing task are extracted. The task parameters of the currently processed task are persisted to a ZSet, and the corresponding task parameters of the currently processed task are automatically deleted from the original queue.

6. The data processing method for image understanding and visual question answering tasks according to any one of claims 1 to 5, characterized in that, When the event source type of the image file is internal location source, the method further includes: After completing AI inference and successfully returning the inference result, event-triggered actions are executed in sequence to form a complete closed-loop event processing chain. The event-triggered actions include event rule matching, deduplication and event storage, increasing the hit count, linked outbound calls, multi-level image archiving, generating fixed-size thumbnails, and cleaning up temporary files.

7. The data processing method for image understanding and visual question answering tasks according to claim 6, characterized in that, Also includes: The message read / write latency, queue consumption container running status, and queue backlog of the multi-channel AI queue are periodically detected. When an anomaly is detected, automatic recovery actions are performed sequentially on the queue consumption containers that experienced the anomaly; wherein, the automatic recovery actions include stopping, rebuilding, starting, and restoring historical compensation.

8. A data processing device for image understanding and visual question answering tasks, characterized in that, include: The image file detection unit is used to filter fully written image files by stability identification when a new file is detected being written. The metadata parsing unit is used to parse the image metadata of the image file to determine the event source type and image path; The queue distribution unit is used to distribute the image file to a multi-channel AI queue for consumption processing based on the event source type and image path, combined with a distribution strategy. The consumption retry unit is used to execute a rate limiting and persistent retry mechanism based on the queue consumption container when rate limiting or QPS exceeding occurs during AI inference, until the AI ​​inference is completed and the inference result is returned.

9. An electronic device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the data processing method for image understanding and visual question answering tasks as described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the data processing method for image understanding and visual question answering tasks as described in any one of claims 1-7.

Images