Low cost interactive multi-task image processing system and method

Abstract

The application discloses a low-cost interactive multi-task image processing system and method, which comprises an image acquisition module, a storage module, a display interaction module, a serial communication module and a processor module; the image acquisition module is used for acquiring external scene images and outputting image frames; the storage module is used for storing firmware programs, task configuration parameters and color configuration parameters, and storing artificial intelligence model files; the display interaction module is used for displaying an interactive interface, to-be-displayed image frames and superimposed drawing results, and generating local interaction instructions according to user operations; the serial communication module is used for receiving serial byte streams sent by an external control device, and sending protocol frames containing structured task results or system state information to the external control device; and the processor module is used for uniformly executing image acquisition control, task scheduling, target image processing, display control and communication control. The application realizes unified scheduling, stable switching, display drawing and result feedback of multiple visual tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a low-cost interactive multi-task image processing system and method. Background Technology

[0002] Embedded vision modules are widely used in educational development, robot control, smart terminals, and industrial inspection. In these applications, in addition to image acquisition and target recognition, the system also needs to handle parameter configuration, task switching, result feedback, and human-computer interaction.

[0003] In the prior art, Chinese invention patent application CN101447075A discloses an FPGA and DSP embedded multi-value target threshold classification and tracking device based on a wide-angle lens. The FPGA performs image preprocessing, while the DSP performs target classification and tracking. This solution requires multiple processing units, memory, and supporting interface circuits, resulting in complex hardware-software coordination and system integration. Furthermore, it primarily focuses on color threshold classification and tracking, failing to establish a unified scheduling and dynamic switching mechanism for different visual tasks such as color detection, label recognition, line following, and AI target detection.

[0004] While general-purpose high-performance processing platforms can run complex vision algorithms or artificial intelligence models, they typically suffer from high power consumption, size, material costs, and maintenance complexity, making them unsuitable for widespread adoption in low-cost AI development boards, lightweight robot nodes, and resource-constrained embedded terminals.

[0005] Existing low-cost microcontroller vision solutions are still limited by computing power and storage resources. When image acquisition, real-time processing, display interaction, and communication feedback lack unified scheduling, or when frame buffers, queues, and algorithm resources are not reclaimed in an orderly manner during task switching, issues such as misaligned display results, unreclaimed frame buffers, queue residues, task switching failures, or watchdog timer resets can easily occur.

[0006] Therefore, there is a need for an image processing system and method that uses a single low-cost microcontroller to uniformly perform image acquisition, image processing, display interaction, and communication feedback, supports stable operation and dynamic switching of multiple vision tasks, and can output structured task results. Summary of the Invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention aims to provide a low-cost interactive multi-task image processing system and method, which realizes unified scheduling, stable switching, display rendering and result feedback of multiple visual tasks through a single microcontroller, independent task context and queue-based processing pipeline.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A low-cost interactive multi-task image processing system includes an image acquisition module, a storage module, a display interaction module, a serial communication module, and a processor module; The image acquisition module includes a camera, used to acquire images of the external scene and output image frames; The storage module includes a non-volatile memory and a memory card interface for connecting a memory card. The non-volatile memory is used to store firmware programs, task configuration parameters, and color configuration parameters, and the memory card is used to store artificial intelligence model files. The display interaction module includes an SPI interface LCD screen and a key input unit, which are used to display the interactive interface, the image frame to be displayed and the overlay drawing result, and to generate local interactive commands based on user operations, respectively. The serial communication module is used to receive serial byte streams sent by external control devices and send protocol frames containing structured task results or system status information to the external control devices. The processor module is a single microcontroller, which is connected to the image acquisition module, storage module, display interaction module and serial communication module respectively, and is used to uniformly execute image acquisition control, task scheduling, target image processing, display control and communication control.

[0009] The processor module runs a multi-task scheduler, which includes at least a scheduling task, an image acquisition task, a display task, a serial port receiving task, a serial port sending task, and multiple optional target image processing tasks corresponding to different visual tasks. Each task runs in an independent task context. Within the same runtime, the scheduling task determines a target image processing task as the current target image processing task.

[0010] The processor module establishes a task scheduling command queue, a serial port transmission event queue, an input frame queue Qin, a display frame queue Qout, and an overlay queue Qov in the accessible storage space; The task scheduling command queue is used to transmit task commands between serial port receiving tasks and scheduling tasks. The serial port sending event queue is used to send image processing result events or system status events to the serial port. The input frame queue Qin, the display frame queue Qout, and the overlay queue Qov are used to transmit image frames, image frames to be displayed, and overlay drawing requests, respectively.

[0011] The scheduling task is used to determine the target task, maintain the system's operating status, and control the establishment, stopping, and reconstruction of the processing pipeline; the image acquisition task is used to acquire image frames and write them into the input frame queue Qin or the display frame queue Qout according to the operating status; the current target image processing task is used to execute the algorithm corresponding to the target task and generate structured task results; the display task is used to uniformly access the display bus; the serial port receiving task and the serial port sending task are used for protocol parsing and protocol encoding and sending, respectively.

[0012] The task scheduling command queue and serial port transmission event queue are created during system initialization and remain operational during task switching. The input frame queue Qin, display frame queue Qout, and overlay queue Qov are processing pipeline queues, created, reused, cleaned up, or released by the scheduling task based on the target task, image format, and queue capacity. The system operating states include at least the idle state, preview state, running state, switching state, stopped state, and abnormal state.

[0013] The scheduling task determines the target image processing task to be started and its task parameters based on the local interactive instructions generated by the display interaction module, or the task commands written into the task scheduling command queue after parsing the protocol frame by the serial port receiving task. The scheduling task configures the corresponding image acquisition parameters, processing pipeline queue and algorithm resources, first creates or starts the target image processing task to be started, and makes it run as the current target image processing task, and then starts the image acquisition task.

[0014] In operation, the image acquisition task sends image frames into the input frame queue Qin; the current target image processing task reads image frames from the input frame queue Qin, generates the image frame to be displayed, the structured task result, and the overlay drawing request, sends the image frame to be displayed into the display frame queue Qout, sends the overlay drawing request into the overlay queue Qov, and writes the image processing result event into the serial port transmission event queue through the result submission interface; the display task uniformly completes the base map refresh and overlay drawing, and the serial port transmission task asynchronously completes the protocol encoding and transmission.

[0015] The processor module assigns a task instance identifier to the current target image processing task each time it is started, and assigns an image frame identifier to the image frame. The image frames to be displayed, structured task results, image processing result events, and overlay drawing result requests carry the same task instance identifier and image frame identifier; when establishing a preview pipeline, a preview instance identifier is assigned, and the task instance identifier and preview instance identifier are collectively referred to as the running instance identifier; the display task and optional Wi-Fi image transmission process only process image frames that match the currently valid running instance.

[0016] Each image frame's corresponding frame description information includes at least the image frame identifier, running instance identifier, image frame cache address, image size, pixel format, frame pool identifier, and usage count. The right to use an image frame is transferred between tasks and queues. When an image frame is used by multiple processes simultaneously, the usage count is increased. After each user completes the use, the usage count is decreased, and when the usage count drops to zero, the image frame is returned to the corresponding frame pool.

[0017] When the image acquisition task obtains an image frame from the camera, the count is initialized to one; after the image frame is successfully written into the input frame queue Qin, the right to use it is transferred from the image acquisition task to the input frame queue Qin. When the current target image processing task retrieves an image frame from the input frame queue Qin, the right to use it is transferred back to the current target image processing task. When the image frame to be displayed shares the same buffer as the input image frame, the right to use it is transferred to the display frame queue Qout. If the processing task still needs to continue using it or the image frame is shared by other processes, the usage count is increased for the new user.

[0018] The display task retrieves the image frame to be displayed from the display frame queue Qout and holds the right to display it; when the image frame is submitted to the Wi-Fi image transmission process at the same time, an image transmission usage count is added; after the display or image transmission is completed, the corresponding usage right is released respectively; when clearing the input frame queue Qin or the display frame queue Qout, the scheduling task only releases the usage right held by the queue, thereby avoiding the same image frame being returned repeatedly or being returned prematurely while still in use.

[0019] The image processing result event includes at least a transmission event type, task type, task instance identifier, image frame identifier, result validity flag, and structured task result payload; the system status event includes at least a transmission event type, system status type, and system status payload, and is not bound to a specific task instance. The serial port transmission task discards image processing result events belonging to failed task instances, but does not perform task instance filtering on system status events.

[0020] The processor module is configured to perform the following steps: Step (1), Instruction reception and task determination: The display interaction module generates local interaction instructions; or the serial port receiving task receives the serial port byte stream, calls the serial port protocol processing program to perform protocol parsing and verification, and generates a task command after the verification is passed; the scheduling task determines the target task and task parameters according to the local interaction instructions or task commands. Step (2), pipeline configuration and task startup: The scheduling task determines the image acquisition parameters, queue configuration parameters and target image processing method according to the target task and task parameters, creates or reuses the input frame queue, display frame queue and overlay queue, initializes algorithm resources, starts the target image processing task first, and then starts the image acquisition task; Step (3), image acquisition, processing, display and result feedback: the image acquisition task submits image frames to the input frame queue, the current target image processing task generates the image frame to be displayed, the structured task result and the overlay drawing request, the display task completes the corresponding display of the base map and the overlay content according to the image frame identifier, and the serial port sending task completes the protocol encoding and sending of the structured task result or system status information according to the sending event type. Step (4), task stop, resource reclamation and pipeline switching: When a task switching or stop control is received in the running state, the scheduling task notifies the current target image processing task and image acquisition task to stop, and notifies the display task to pause; after receiving the corresponding confirmation, the original task instance is invalidated, the image frames in the input frame queue and display frame queue are returned frame by frame, the overlay queue is cleaned up, the algorithm resources are released, and the processing pipeline queue is reused or recreated according to the resource requirements of the new target task, preview state or idle state.

[0021] The current target image processing task stops reading new frames from the input frame queue, completes the current processing cycle or exits at a safe stop point, and stops submitting new display data and result events; the image acquisition task stops acquiring and writing new frames, and returns the image frames that have been acquired but not yet enqueued to the frame pool; the display task stops reading new data from the original task instance, completes the current display and the optional image transmission that has started, and returns to pause confirmation.

[0022] After receiving all confirmations, the scheduling task cyclically retrieves the remaining image frames from the input frame queue and the display frame queue and releases the usage rights held by the queue. It also clears the overlay drawing requests and their dynamic data in the overlay queue and releases the algorithm object, model resources, temporary cache, and task context of the original target task. The corresponding queue is only reused when the queue has been cleared, no task continues to hold references, and the element type, capacity, and image format are compatible.

[0023] When the system is in preview mode, the image acquisition task works in preview acquisition mode, sending image frames directly into the display frame queue without starting the target image processing task. When starting the target task from preview mode, preview acquisition and preview display are stopped first. After receiving confirmation, the preview instance is invalidated and the remaining preview frames are recycled before the target processing pipeline is established. When the stop or pause confirmation times out, or when queue creation, algorithm resource initialization, or target image processing task creation fails, the scheduled task enters an abnormal state, releasing only resources that can be confirmed not to be held by the task or process, and generating system status events corresponding to task stop timeout, preview pipeline stop timeout, task switching failure, or task start failure.

[0024] The system status events include task stop timeout, preview pipeline stop timeout, task switching failure, and task start failure. When a reset is required, the serial port sending task returns a sending completion confirmation after the corresponding protocol frame is sent. The scheduling task executes a software module reset or system soft reset after receiving the confirmation or reaching the preset abnormal state sending waiting time. The target tasks include color learning tasks, color detection tasks, AprilTag recognition tasks, line following tasks, and artificial intelligence target detection tasks; the task parameters include task type, image acquisition parameters, stopping mode, and at least one of color number, detection threshold, tag dictionary number, model number, confidence threshold, display control parameters, and result feedback parameters. The serial communication module adopts a frame-based communication protocol with verification. The protocol frame includes at least a start identifier field, a device identifier field, a message type field, a payload length field, a payload field, and a verification field. The verification field is a Fletcher verification value calculated based on the start identifier field to the payload field. The color learning task is used to calculate the mean values ​​of the red, green, and blue components within a preset sampling window of an RGB565 image frame, and write them into a non-volatile memory according to the color number; the color detection task generates a binary target region mask based on the target color parameters and the color difference threshold, and calculates the area, area ratio, centroid, and bounding box of the target region; when the number of effective pixels or the target area does not meet the preset conditions, the corresponding results are marked as invalid; The AprilTag recognition task is used to perform tag detection on grayscale image frames, select the candidate tag with the largest area as the output target, and estimate the distance based on the tag's physical size, the tag's average pixel side length, and the camera's equivalent focal length; the line-following task calculates the lateral offset and heading angle error based on the line area mask, and determines the intersection type based on row coverage, column coverage, continuous segment length, and confirmation results across multiple frames. The AI-powered object detection task loads an object detection model via a memory card interface, performs preprocessing and model inference on the image, filters candidate detection boxes based on a confidence threshold, performs non-maximum suppression using an intersection-over-union threshold, and outputs the detection box category, confidence level, and coordinates.

[0025] Each optional target image processing task does not directly access the display bus. Instead, it generates image frames to be displayed and overlay drawing requests, with the display task uniformly handling the base map refresh and overlay drawing. The system may also include a Wi-Fi communication module, used to send single-frame or continuous image data upon receiving image transmission control information. When the same image frame is used for both display and image transmission, each user is managed using a counter, and the corresponding frame is returned to the pool after the usage count drops to zero.

[0026] A low-cost interactive multi-task image processing method, applied to the above system, includes: Step 1: Initialize the image acquisition module, storage module, display interaction module, and serial communication module; establish a multi-task operating environment; and create a task scheduling command queue and a serial port transmission event queue. Step 2: Receive local interaction instructions, or generate task commands after performing protocol parsing and verification on the serial port byte stream, and determine the target image processing task to be started and the task parameters according to the local interaction instructions or task commands. Step 3: Create or reuse the input frame queue, display frame queue, and overlay queue; initialize the corresponding algorithm resources; start the target image processing task first, then start the image acquisition task; the image acquisition task, the current target image processing task, the display task, and the serial port transmission task respectively complete image acquisition, target processing, unified display, and asynchronous result feedback. Step 4: When switching or stopping a task, send a stop or pause notification and wait for confirmation to invalidate the original running instance, reclaim image frames, clean up the queue, and release algorithm resources; when switching tasks, establish a processing pipeline for the new target task; when a task stops, enter the preview state or idle state; and generate system status events when there is an exception.

[0027] In preview mode, the image acquisition task directly sends image frames with preview instance identifiers to the display frame queue. When exiting preview mode, the scheduling task invalidates the preview instance and reclaims the remaining preview frames after receiving confirmation of preview acquisition stop and preview display pause. If not all confirmations are received or the creation of a new pipeline fails, it enters an abnormal state and sends back the corresponding system status event.

[0028] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0029] The beneficial effects of this invention are: 1. A single microcontroller is used to uniformly control image acquisition, target processing, display interaction, and communication feedback, reducing the need for FPGA, DSP, or external vision coprocessors and their interface circuits, thereby reducing hardware costs and system integration complexity.

[0030] 2. Scheduling, acquisition, target processing, display, and serial port transmission and reception tasks run in independent task contexts and transmit control information and data through five types of queues, thereby decoupling visual processing, display refresh, and serial port transmission and reducing the blocking of time-consuming operations on interaction and acquisition.

[0031] 3. When switching tasks, first stop or pause the task and wait for confirmation. Then, based on the task instance, image frame usage count, and queue status, reclaim image frames, clean up overlay requests, and release algorithm resources to avoid duplicate returns, dangling access, old results mixed into new tasks, and queue residue.

[0032] 4. The image frames to be displayed, the structured task results, and the overlay drawing requests correspond to the task instance identifier and the image frame identifier. The display task accesses the display bus in a unified manner, reducing bus contention and misalignment of overlay results.

[0033] 5. Image processing result events and system status events are asynchronously submitted to the serial port transmission event queue and sent using a frame protocol with Fletcher verification. This allows for filtering of expired results during task switching and ensures that abnormal status information can be transmitted back.

[0034] 6. Color, AprilTag, line following, and AI object detection tasks are all integrated into a unified pipeline and output structured results, taking into account both traditional vision algorithms and model inference; local interaction, serial port control, preview pipeline, and optional Wi-Fi image transmission facilitate on-site debugging and external system integration. Attached Figure Description

[0035] Figure 1 This is a hardware structure block diagram of the low-cost interactive multi-task image processing system of the present invention.

[0036] Figure 2 This is a block diagram of the software architecture and multi-task scheduling of the present invention.

[0037] Figure 3 This is a schematic diagram of the serial communication protocol frame structure of the present invention.

[0038] Figure 4 This is a flowchart of the multi-task image processing method of the present invention.

[0039] Figure 5 This is a schematic diagram of the color learning and color detection process of the present invention.

[0040] Figure 6 This is a schematic diagram of the AprilTag recognition and distance estimation process of the present invention.

[0041] Figure 7 This is a schematic diagram of the line inspection and intersection identification process of the present invention.

[0042] Figure 8 This is a schematic diagram of the artificial intelligence target detection process and result feedback structure of the present invention.

[0043] Figure 9 This is a schematic diagram of the optional Wi-Fi image transmission process of the present invention. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] This invention discloses a low-cost interactive multi-task image processing system and method. The following implementation uses a low-cost artificial intelligence development board as an example to illustrate the system composition, task scheduling, resource recycling, and workflow of each target task. Specific parameters and synchronization interfaces can be configured according to the microcontroller and real-time operating system, and do not constitute a limitation on the scope of protection; Define the image width as W, the image height as H, and the pixel coordinates as... The color learning window is The number of effective pixels within the window is The target color parameter corresponding to color number k is: , and The overall color difference is Δ(x,y); the centroid of the target region is (xc,yc); the physical side length of AprilTag is... The average pixel side length is The camera's equivalent focal length is Attitude angle is The distance is D; the lateral deviation of the line is... The heading angle error is The original confidence level and smoothed confidence level of the intersection in the nth frame are respectively... and Candidate detection boxes are and .

[0046] Step 1, as follows Figure 1 As shown, the hardware system is constructed. The image acquisition module uses a camera module, the display and interaction module includes an SPI interface LCD screen and a button input unit, the storage module includes non-volatile memory and a memory card interface, the serial communication module uses a UART interface, the processor module uses a single microcontroller, and an optional Wi-Fi communication module is available for wireless image transmission.

[0047] Step 2: Perform system initialization. After power-on, the system sequentially initializes the clock, memory, non-volatile memory, serial port, display, camera, memory card, and optional Wi-Fi module. Then, a multi-task runtime environment is established, creating scheduling tasks, display tasks, serial port receiving tasks, serial port sending tasks, a task scheduling command queue, and a serial port sending event queue. The scheduling task determines the target image processing task to be started based on task control information, creates or starts the task after the processing pipeline is configured, and makes it run as the current target image processing task.

[0048] Step 3, as follows Figure 2As shown, a task scheduling framework is established. The producer and consumer of the task scheduling command queue are the serial port receive task and the scheduling task, respectively; the producer of the serial port send event queue includes the current target image processing task and the scheduling task, and the consumer is the serial port send task. In the running state, the image acquisition task provides image frames to the current target image processing task through the input frame queue Qin, and the current target image processing task provides the image frames to be displayed and the overlay drawing request to the display task through the display frame queue Qout and the overlay queue Qov; in the preview state, the image acquisition task directly submits the image frames to be displayed to Qout.

[0049] In runtime, Qin's producer is the image acquisition task, and its consumer is the current target image processing task; Qout and Qov's producers are the current target image processing tasks, and their consumers are the display tasks. In preview mode, the image acquisition task operating in preview acquisition mode is the producer of Qout, and the display task is the consumer; Qin does not participate in preview image transfer. When preview auxiliary markers are needed, the display task completes the drawing according to the interface configuration.

[0050] Image processing result events must include at least the event type, task type, task instance identifier, image frame identifier, result validity flag, and result payload; system status events must include at least the event type, system status type, and status payload. The serial port transmission task discards expired image processing result events based on the task instance identifier, but continues to transmit system status events that are not bound to a specific task instance.

[0051] Each image frame's frame description information includes at least the image frame identifier, runtime instance identifier, cache address, image size, pixel format, frame pool identifier, and usage count. When an image frame is successfully written to the queue or retrieved by a task, its usage right is transferred between the task and the queue. When the same image frame is used by both the display task and the Wi-Fi image transmission process, the usage count is increased for each new user, decreased after each user completes their task, and returned to the frame pool when the usage count reaches zero.

[0052] Step 4, as follows Figure 3 As shown, a task control and result transmission mechanism is established. The serial port receive task sequentially executes start identifier detection, device identifier parsing, message type parsing, payload length parsing, payload reception, and Fletcher verification. After successful verification, the task control callback function generates a task command and writes it to the task scheduling command queue. The current target image processing task generates an image processing result event through the result submission interface; the scheduling task generates a system status event in case of an exception. The serial port send task selects the encoding method and encapsulates the protocol frame according to the event type. The scheduling task does not handle result relay, and the target image processing task does not directly call the underlying UART driver.

[0053] Step 5, as follows Figure 4As shown, the processing pipeline is configured and executed. The scheduling task configures the image pixel format, resolution, Qin, Qout, and Qov, as well as algorithm resources. It first creates or starts the target image processing task to be started, making it the current target image processing task, and then starts the image acquisition task. The current target image processing task generates the image frame to be displayed, the structured task result, the image processing result event, and the overlay drawing request based on the same input image frame, assigning them the same task instance identifier and image frame identifier. The display task completes the corresponding display accordingly, and the serial port sending task asynchronously sends the results.

[0054] When a task switches or stops, the scheduling task sends a stop or pause notification to the current target image processing task, image acquisition task, and display task, respectively. The current target image processing task stops reading new frames and completes the current processing cycle; the image acquisition task stops fetching frames and enqueuing them; and the display task completes the current display and any optional Wi-Fi image transmission that has already started. Each task releases its own image frame usage rights and returns an acknowledgment.

[0055] After receiving a stop notification, the current target image processing task will stop reading new frames from Qin and release the image frames it holds and the local objects of the current processing cycle after completing the processing of the current frame or reaching the preset safe stop point; after receiving a stop notification, the image acquisition task will stop acquiring new frames from the camera, and will dequeue and return the acquired but not yet queued image frames to the frame pool.

[0056] After receiving the pause notification, the display task stops reading new data from the original task instance, completes the current base map refresh and overlay drawing; when the Wi-Fi image transmission still occupies the current frame, it waits for the current transmission to complete, or sends a cancellation notification after the preset image transmission stop time is reached, and returns to the display pause confirmation after the image transmission process releases the right to use.

[0057] After receiving all acknowledgments within a preset waiting time, the scheduled task invalidates the original task instance, retrieves the remaining image frames from Qin and Qout, releases the usage rights held by the queues, clears the overlay rendering requests and their dynamic data in Qov, and releases the original task's algorithm object, model resources, temporary cache, task parameter copies, and task workspace. The task scheduling command queue and serial port transmission event queue remain running during the switchover. The serial port transmission task completes the protocol frames it has already started transmitting and discards any untransmitted result events corresponding to the invalidated task instance.

[0058] Algorithm resources are released according to task type: color learning tasks release color statistics cache, color detection tasks release target mask and connected region processing cache, AprilTag recognition tasks release grayscale cache, detector objects and corner results, line following tasks release morphology cache and cross-frame state, and artificial intelligence object detection tasks release model handle, inference working area, input and output tensors and nonmaximum suppression cache.

[0059] The serial port transmit event queue remains running during task switching; protocol frames that have already started encoding or transmitting are allowed to complete transmission, image processing result events that have not yet been transmitted and belong to failed task instances are discarded and dynamic payloads are released, and system status events do not perform task instance filtering.

[0060] Qin, Qout, and Qov are reused only when they have been cleared, no task continues to hold references, the element type and size are consistent, the capacity meets the requirements, and the image format and frame description structure are compatible; otherwise, they are recreated. During task switching, camera and algorithm resources are reconfigured according to the new target task; when a task stops, it enters either preview or idle state depending on the stop mode. In preview state, the image acquisition task directly submits image frames to Qout; when exiting preview state, it first waits for confirmation of preview acquisition stop and preview display pause before invalidating the preview instance and reclaiming the remaining preview frames.

[0061] When entering preview mode, the scheduling task creates or reuses Qout and assigns a preview instance identifier. The image acquisition task writes the preview instance identifier into the frame description information and then directly submits the image frame to Qout. When entering idle mode, the image acquisition task remains stopped and a preset idle interface is displayed. When starting a target task from preview mode, preview acquisition and preview display must be stopped first. After receiving full confirmation, the preview instance is invalidated, the remaining preview frames in Qout are recycled, and the auxiliary drawing state is cleared.

[0062] If the creation of queues, algorithm resources, or target image processing tasks in a new pipeline fails, the image acquisition task will not be started, and resources that have been confirmed to be safe during the creation process will be released; the preview pipeline will be re-established according to the preset rollback method, enter an idle state, or remain in an abnormal state.

[0063] When the stop or pause confirmation times out, or when the creation of queues, algorithm resources, or target image processing tasks fails, the scheduling task enters an abnormal state, releases only resources that can be confirmed as safe, and submits the corresponding system status event; when a reset is required, the serial port sending task returns an acknowledgment after the system status protocol frame is sent, and the scheduling task performs a software module reset or system soft reset after receiving the acknowledgment or reaching the preset waiting time.

[0064] Step 6, as follows Figure 5 As shown, color learning and color detection are performed. The color learning task extracts and expands the red, green, and blue components from RGB565 pixels within the sampling window. Internally calculate the target color parameters corresponding to color number k: In the formula, , and These are the red, green, and blue components extracted from RGB565 pixels and expanded to the same numerical range, respectively. The color learning task will... , and Write the color number to non-volatile memory and submit a structured task result containing a result validity flag, color number, and three mean values; if the number of valid pixels is insufficient, keep the original color parameters unchanged and output an invalid flag.

[0065] The color detection task reads the target color parameters, calculates the overall color difference, and generates a binary mask. In the formula, A is the pixel area of ​​the target region, η is the area ratio, and the bounding box is determined by the minimum and maximum coordinates of the target pixels. If A is less than the preset minimum area threshold, an invalid flag is output; otherwise, the centroid, area ratio, and bounding box are calculated, the structured task results are submitted, and a request for overlaying the bounding box and center point is generated.

[0066] Step 7, as follows Figure 6 As shown, AprilTag recognition and distance estimation are performed. The four corner points of the candidate tags are arranged in order, the area of ​​the quadrilateral is calculated, and the candidate tag with the largest area is selected. In the formula, D is the average pixel length of the four sides of the label, where D is the physical side length of the label. Camera equivalent focal length and Calculate the attitude angles Calculated from the label edge direction. If no valid label is detected or the area is less than the threshold, an invalid flag is output; otherwise, the label number, center position, area percentage, attitude angle, and distance are submitted, and a label outline and center point overlay drawing request is generated.

[0067] Step 8, as follows Figure 7As shown, line following and intersection recognition are performed. The line following task generates a line region mask according to the pixel classification method of color detection. After morphological processing, in the region of interest Internal calculation of line center, lateral deviation, and heading angle error: In the formula, and These are row coverage and column coverage, respectively. The maximum length of a continuous segment. This is the indicative function. The line-following task distinguishes between T-junctions and crossroads based on coverage, continuous segment length, and branch connectivity, and uses smooth confidence and continuous frame confirmation to suppress single-frame misjudgments; when the effective line pixels are insufficient, an invalid flag is output.

[0068] Step 9, as follows Figure 8 As shown, artificial intelligence object detection is performed. The model is loaded from the memory card according to the model number, and the input size, class mapping, and inference workspace are configured. Image size transformation, color format transformation, and normalization are performed. After inference, candidate detection boxes are selected based on a confidence threshold. In the formula, Candidate detection boxes Confidence level, This is the non-maximum suppression threshold. Detection boxes are retained according to confidence level from high to low, and candidate boxes with an intersection-union ratio (IU) greater than the threshold with the selected detection boxes of the same category are deleted, resulting in a set of valid detection boxes. The result includes the number of valid detection boxes and the category, confidence level, and coordinates of each detection box. When the set is empty, the number of detection boxes is set to zero.

[0069] Step 10: Perform unified display and optional Wi-Fi image transmission. Each target image processing task does not directly access the display bus. The display task first refreshes the base image in Qout, and then draws the bounding box, label outline, center point, line-following marker, or detection box according to the Qov request corresponding to the same task instance identifier and image frame identifier. The Wi-Fi image transmission process only receives image frames from currently active instances. When display and image transmission share the same frame, they each hold usage rights. Receiving new frames from old instances stops when switching tasks, stopping, or exiting preview mode. Timeout transmission is cancelled if necessary, and the frame pool is returned after all users release the frame.

[0070] Step 11: Execute unified loop control. The scheduling task continuously processes task initiation, task switching, task stopping, task parameter configuration, and image transmission control information; when a new initiation or switching control is received during a switching period, the target task to be initiated and its parameters are updated, and a processing pipeline is established according to the latest valid task control information after the current resource reclamation is completed.

[0071] Through the above implementation methods, the present invention forms a multi-task image processing mechanism on a single microcontroller platform, with scheduling tasks as the control center, five types of queues as data channels, running instance identifiers as data validity boundaries, and image frame usage counts as the basis for resource return. It also incorporates color learning, color detection, AprilTag recognition, line following and intersection recognition, and artificial intelligence target detection into a unified display and structured result feedback framework.

Claims

1. A low-cost interactive multi-task image processing system, characterized in that, It includes an image acquisition module, a storage module, a display and interaction module, a serial communication module, and a processor module; The image acquisition module includes a camera, used to acquire images of the external scene and output image frames; The storage module includes a non-volatile memory and a memory card interface for connecting a memory card. The non-volatile memory is used to store firmware programs, task configuration parameters, and color configuration parameters, and the memory card is used to store artificial intelligence model files. The display interaction module includes an SPI interface LCD screen and a key input unit, which are used to display the interactive interface, the image frame to be displayed and the overlay drawing result, and to generate local interactive commands based on user operations, respectively. The serial communication module is used to receive serial byte streams sent by external control devices and send protocol frames containing structured task results or system status information to the external control devices. The processor module is a single microcontroller, which is connected to the image acquisition module, storage module, display interaction module and serial communication module respectively, and is used to uniformly execute image acquisition control, task scheduling, target image processing, display control and communication control.

2. The low-cost interactive multi-task image processing system according to claim 1, characterized in that, The processor module runs a multi-task scheduler, which includes at least a scheduling task, an image acquisition task, a display task, a serial port receiving task, a serial port sending task, and multiple optional target image processing tasks corresponding to different visual tasks. Each task runs in an independent task context. Within the same runtime period, the scheduling task determines a target image processing task as the current target image processing task. The processor module establishes a task scheduling command queue, a serial port transmission event queue, an input frame queue Qin, a display frame queue Qout, and an overlay queue Qov in the accessible storage space; The task scheduling command queue is used to transmit task commands between serial port receiving tasks and scheduling tasks. The serial port sending event queue is used to send image processing result events or system status events to the serial port. The input frame queue Qin, the display frame queue Qout, and the overlay queue Qov are used to transmit image frames, image frames to be displayed, and overlay drawing requests, respectively. The scheduling task is used to determine the target task, maintain the system's operating status, and control the establishment, stopping, and reconstruction of the processing pipeline; the image acquisition task is used to acquire image frames and write them into the input frame queue Qin or the display frame queue Qout according to the operating status; the current target image processing task is used to execute the algorithm corresponding to the target task and generate structured task results; The display task is used for unified access to the display bus; the serial port receive task and the serial port send task are used for protocol parsing and protocol encoding and transmission, respectively.

3. The low-cost interactive multi-task image processing system according to claim 2, characterized in that, The task scheduling command queue and serial port transmission event queue are created during the system initialization phase and remain running during task switching; the input frame queue Qin, the display frame queue Qout, and the overlay queue Qov are processing pipeline queues, which are created, reused, cleaned up, or released by the scheduling task according to the target task, image format, and queue capacity. The system's operating status includes at least the following states: idle, preview, running, switching, stopped, and abnormal. The scheduling task determines the target image processing task to be started and its task parameters based on the local interactive instructions generated by the display interaction module, or the task commands written into the task scheduling command queue after parsing the protocol frame by the serial port receiving task. The scheduling task configures the corresponding image acquisition parameters, processing pipeline queue and algorithm resources, first creates or starts the target image processing task to be started, and makes it run as the current target image processing task, and then starts the image acquisition task. During operation, the image acquisition task sends image frames into the input frame queue Qin; The current target image processing task reads image frames from the input frame queue Qin, generates image frames to be displayed, structured task results, and overlay drawing requests, sends the image frames to be displayed to the display frame queue Qout, sends the overlay drawing requests to the overlay queue Qov, and writes the image processing result event to the serial port transmission event queue through the result submission interface; the display task uniformly completes the base map refresh and overlay drawing, and the serial port transmission task asynchronously completes the protocol encoding and transmission.

4. The low-cost interactive multi-task image processing system according to claim 3, characterized in that, The processor module assigns a task instance identifier to the current target image processing task each time it is started, and assigns an image frame identifier to the image frame. The image frames to be displayed, structured task results, image processing result events, and overlay rendering result requests carry the same task instance identifier and image frame identifier; when establishing a preview pipeline, a preview instance identifier is assigned, and the task instance identifier and preview instance identifier are collectively referred to as the running instance identifier; display tasks and optional Wi-Fi image transmission processes only process image frames that match the currently valid running instance; Each image frame's corresponding frame description information includes at least the image frame identifier, runtime instance identifier, image frame cache address, image size, pixel format, frame pool identifier, and usage count; the right to use an image frame is transferred between tasks and queues; when an image frame is used by multiple processes simultaneously, the usage count is increased; after each user completes the use, the usage count is decreased; and when the usage count drops to zero, the image frame is returned to the corresponding frame pool. When the image acquisition task obtains an image frame from the camera, the count is initialized to one; after the image frame is successfully written into the input frame queue Qin, the right to use it is transferred from the image acquisition task to the input frame queue Qin. When the current target image processing task retrieves an image frame from the input frame queue Qin, the right to use it is transferred back to the current target image processing task. When the image frame to be displayed shares the buffer with the input image frame, the right to use it is transferred to the display frame queue Qout by writing to the display frame queue Qout; when the processing task still needs to continue to use it or the image frame is shared by other processes, the usage count is increased for the new user. The display task holds the right to display the image frame after retrieving it from the display frame queue Qout; when the image frame is submitted to the Wi-Fi image transmission process at the same time, an image transmission usage count is added; after the display or image transmission is completed, the corresponding usage right is released respectively; when clearing the input frame queue Qin or the display frame queue Qout, the scheduling task only releases the usage right held by the queue, thereby avoiding the same image frame being returned repeatedly or being returned prematurely while still in use; The image processing result event includes at least the event type, task type, task instance identifier, image frame identifier, result validity flag, and structured task result payload.

5. A low-cost interactive multi-task image processing system according to claim 4, characterized in that, The system status event includes at least the event type, system status type, and system status payload, and is not bound to a specific task instance; the serial port sending task discards image processing result events belonging to failed task instances, but does not perform task instance filtering on the system status event.

6. The low-cost interactive multi-task image processing system according to claim 4, characterized in that, The processor module is configured to perform the following steps: Step (1): The display interaction module generates local interaction instructions; or the serial port receiving task receives the serial port byte stream, calls the serial port protocol processing program to perform protocol parsing and verification, and generates task commands after the verification is passed. The scheduling task determines the target task and task parameters based on the local interactive instructions or task commands. Step (2): The scheduling task determines the image acquisition parameters, queue configuration parameters and target image processing method according to the target task and task parameters, creates or reuses the input frame queue, display frame queue and overlay queue, initializes algorithm resources, starts the target image processing task first, and then starts the image acquisition task. Step (3): The image acquisition task submits the image frame to the input frame queue. The current target image processing task generates the image frame to be displayed, the structured task result and the overlay drawing request. The display task completes the corresponding display of the base map and the overlay content according to the image frame identifier. The serial port sending task completes the protocol encoding and sending of the structured task result or system status information according to the sending event type. Step (4): When a task switching or stop control is received during the running state, the scheduling task notifies the current target image processing task and image acquisition task to stop, and notifies the display task to pause. Upon receiving the corresponding confirmation, the original task instance is invalidated, the image frames in the input frame queue and display frame queue are returned frame by frame, the overlay queue is cleared, the algorithm resources are released, and the processing pipeline queue is reused or recreated according to the resource requirements of the new target task, preview state, or idle state.

7. A low-cost interactive multi-task image processing system according to claim 6, characterized in that, The current target image processing task stops reading new frames from the input frame queue, completes the current processing cycle or exits at a safe stopping point, and stops submitting new display data and result events; the image acquisition task stops acquiring and writing new frames, and returns the image frames that have been acquired but not yet enqueued to the frame pool. The display task stops reading new data from the original task instance, and returns to pause confirmation after completing the current display and the optional image transmission that has already started; After receiving all confirmations, the scheduling task will cyclically retrieve the remaining image frames from the input frame queue and the display frame queue and release the usage rights held by the queues. It will also clear the overlay drawing requests and their dynamic data in the overlay queue and release the algorithm object, model resources, temporary cache, and task context of the original target task. The corresponding queue will only be reused when the queue has been cleared, no task continues to hold references, and the element type, capacity, and image format are compatible. When the system is in preview mode, the image acquisition task works in preview acquisition mode, sending image frames directly into the display frame queue without starting the target image processing task. When starting the target task from preview mode, preview acquisition and preview display are stopped first. After receiving confirmation, the preview instance is invalidated and the remaining preview frames are recycled before the target processing pipeline is established. When the stop or pause confirmation times out, or when queue creation, algorithm resource initialization, or target image processing task creation fails, the scheduled task enters an abnormal state, releasing only resources that can be confirmed not to be held by the task or process, and generating system status events corresponding to task stop timeout, preview pipeline stop timeout, task switching failure, or task start failure.

8. A low-cost interactive multi-task image processing system according to claim 4, characterized in that, The system status events include task stop timeout, preview pipeline stop timeout, task switching failure, and task start failure. When a reset is required, the serial port sending task returns a sending completion confirmation after the corresponding protocol frame is sent. The scheduling task executes a software module reset or system soft reset after receiving the confirmation or reaching the preset abnormal state sending waiting time. The target tasks include color learning tasks, color detection tasks, AprilTag recognition tasks, line following tasks, and artificial intelligence target detection tasks; the task parameters include task type, image acquisition parameters, stopping mode, and at least one of color number, detection threshold, tag dictionary number, model number, confidence threshold, display control parameters, and result feedback parameters. The serial communication module adopts a frame-based communication protocol with verification. The protocol frame includes at least a start identifier field, a device identifier field, a message type field, a payload length field, a payload field, and a verification field. The verification field is a Fletcher verification value calculated based on the start identifier field to the payload field. The color learning task is used to calculate the mean values ​​of the red, green, and blue components within a preset sampling window of an RGB565 image frame, and write them into a non-volatile memory according to the color number. The color detection task generates a binary target region mask based on the target color parameters and color difference threshold, and calculates the area, area ratio, centroid, and bounding box of the target region; when the number of effective pixels or the target area does not meet the preset conditions, the corresponding result is marked as invalid. The AprilTag recognition task is used to perform tag detection on grayscale image frames, select the candidate tag with the largest area as the output target, and estimate the distance based on the tag's physical size, the tag's average pixel side length, and the camera's equivalent focal length; the line-following task calculates the lateral offset and heading angle error based on the line area mask, and determines the intersection type based on row coverage, column coverage, continuous segment length, and confirmation results across multiple frames. The AI-powered object detection task loads an object detection model through a memory card interface, performs preprocessing and model inference on the image, filters candidate detection boxes based on a confidence threshold, performs non-maximum suppression using an intersection-over-union threshold, and outputs the detection box category, confidence level, and coordinates.

9. A low-cost interactive multi-task image processing system according to claim 4, characterized in that, Each selectable target image processing task does not directly access the display bus, but instead generates the image frame to be displayed and the overlay drawing request. The display task uniformly completes the base map refresh and overlay drawing. The system also includes a Wi-Fi communication module, which is used to send single frame or continuous image data when it receives image transmission control information. When the same image frame is used for both display and image transmission, each user is managed by using a counter, and the corresponding frame is returned to the pool after the usage count drops to zero.

10. A low-cost interactive multi-task image processing method, applied to the system described in any one of claims 1-9, characterized in that, include: Step 1: Initialize the image acquisition module, storage module, display interaction module, and serial communication module; establish a multi-task operating environment; and create a task scheduling command queue and a serial port transmission event queue. Step 2: Receive local interaction instructions, or generate task commands after performing protocol parsing and verification on the serial port byte stream, and determine the target image processing task to be started and the task parameters according to the local interaction instructions or task commands. Step 3: Create or reuse the input frame queue, display frame queue, and overlay queue; initialize the corresponding algorithm resources; start the target image processing task first, then start the image acquisition task; the image acquisition task, the current target image processing task, the display task, and the serial port transmission task respectively complete image acquisition, target processing, unified display, and asynchronous result feedback. Step 4: When switching or stopping a task, send a stop or pause notification and wait for confirmation to invalidate the original running instance, reclaim image frames, clean up the queue and release algorithm resources; when switching tasks, establish a processing pipeline for the new target task; when a task stops, enter the preview state or idle state; and generate system status events when there is an exception. In preview mode, the image acquisition task directly sends image frames with preview instance identifiers to the display frame queue. When exiting preview mode, the scheduling task invalidates the preview instance and reclaims the remaining preview frames after receiving confirmation of preview acquisition stop and preview display pause. If not all confirmations are received or the creation of a new pipeline fails, it enters an abnormal state and sends back the corresponding system status event.

Images