Image acquisition methods

By dynamically switching acquisition modes and frequencies in industrial settings and combining them with deep learning inference, the overload and false alarm problems of existing image acquisition control mechanisms have been solved, achieving low-power and high-efficiency target detection.

CN123075263APending Publication Date: 2026-11-10BINZHOU WEIQIAO NATIONAL SCIENCE & TECHNOLOGY ADVANCED TECHNOLOGY RESEARCH INSTITUTE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611404992.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-10
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing image acquisition and control mechanisms in industrial settings suffer from problems such as equipment overload and overheating, wake-up delays leading to missed critical moments, and inability to cope with sudden changes in lighting and environmental interference, resulting in false alarms and deadlocks.

Method used

By employing a purely visual target detection mechanism based on observed and background images, the acquisition mode is dynamically switched. Utilizing the frequency difference between the first and second acquisition modes, combined with deep learning inference, the target workpiece is automatically identified and the acquisition frequency is switched to adapt to changes in lighting and environmental interference.

Benefits of technology

It reduces system hardware complexity and maintenance difficulty, saves computing power consumption, achieves a balance between low power consumption and fast response, adapts to different detection needs, and reduces false alarms and deadlocks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123075263A_ABST
    Figure CN123075263A_ABST
Patent Text Reader

Abstract

This invention provides an image acquisition method applicable to industrial automation and image processing technologies. The method includes: sending a first instruction to an image acquisition device installed at a predetermined position on a production line, causing the image acquisition device to acquire images of a target observation area based on a first acquisition mode; determining a background image of frame n based on the (n-1)th frame background image and the acquired nth frame observation image; and, if it is determined that a target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, sending a second instruction to the image acquisition device, causing the image acquisition device to switch from the first acquisition mode to a second acquisition mode to acquire images of the target observation area, wherein the acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of industrial automation technology and image processing technology, and specifically to an image acquisition method. Background Technology

[0002] In industrial settings, smart manufacturing, and edge machine vision inspection systems, industrial cameras are typically used to acquire images of targets on production lines and perform defect detection. However, existing image acquisition and control mechanisms have several issues. For example, to prevent missed detections, some systems are forced to operate in a 24 / 7 full-load "continuous acquisition and inference" mode, leading to prolonged CPU / GPU overload and overheating. Conversely, if a low-frequency sleep mode is used, the delay in waking up and loading when a target suddenly appears often results in missing the most critical image at the moment the event occurs. Furthermore, related technologies often cannot handle sudden changes in workshop lighting (such as switching lights on and off), conveyor belt reflections, or dust interference, easily generating high-frequency false alarms. Additionally, "background adhesion" can occur when the target is stationary, leading to deadlock. Summary of the Invention

[0003] In view of the above problems, the present invention provides an image acquisition method, comprising:

[0004] A first instruction is sent to an image acquisition device installed at a predetermined position on the production line, so that the image acquisition device performs image acquisition on the target observation area based on a first acquisition mode, wherein the production line is used to transport the target workpiece;

[0005] Based on the (n-1)th frame background image and the acquired nth frame observation image, the nth frame background image is determined, where n is a positive integer, and the first frame background image is the first frame observation image.

[0006] Based on the nth frame observation image and the nth frame background image, if it is determined that the target workpiece appears in the target observation area, a second instruction is sent to the image acquisition device to switch the image acquisition device from the first acquisition mode to the second acquisition mode to acquire images of the target observation area. The acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

[0007] A second aspect of the present invention provides an image acquisition device, comprising:

[0008] The first sending module is used to send a first instruction to an image acquisition device installed at a predetermined position on the production line, so that the image acquisition device can perform image acquisition on the target observation area based on a first acquisition mode, wherein the production line is used to transport the target workpiece;

[0009] The first determining module is used to determine the background image of the nth frame based on the background image of the (n-1)th frame and the acquired observation image of the nth frame, where n is a positive integer and the background image of the first frame is the observation image of the first frame;

[0010] The second sending module is used to send a second instruction to the image acquisition device when it is determined that the target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, so that the image acquisition device switches from the first acquisition mode to the second acquisition mode to perform image acquisition for the target observation area, wherein the acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

[0011] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0012] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0013] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0014] According to embodiments of the present invention, the pure visual target detection mechanism based on observed and background images can automatically identify the arrival of a target workpiece and trigger mode switching without an external hardware trigger source. This eliminates the deployment and wiring costs of external devices such as photoelectric switches and proximity sensors, reducing the hardware complexity and maintenance difficulty of the system. Furthermore, by switching between the first and second acquisition modes, CPU resources can be released and the system can operate with low power consumption when there is no target, avoiding the waste of computing power and overheating problems caused by 24-hour full-load acquisition. When a target appears, the system can switch from the first acquisition mode to the second acquisition mode, balancing the needs of low-power operation and fast response. It only operates at full speed when needed (i.e., when there is a target), which can save computing power consumption compared to traditional continuous acquisition schemes in production line conditions (where the interval between target workpieces is relatively long).

[0015] Furthermore, by determining the background image of the nth frame based on the (n-1)th frame background image and the acquired nth frame observation image, a background model with dynamic maintenance and adaptive updates is realized, which can adapt to changes in lighting in the industrial environment (such as workshop lights switching, changes in natural light, etc.) and resist the impact of environmental interference on target detection.

[0016] Furthermore, by decoupling the first acquisition mode and the second acquisition mode into independent decision logic, it can flexibly adapt to different detection needs (such as "normal parts are centered and archived" or "defective parts are alarmed immediately"). This decouples the underlying acquisition logic from the upper-level detection strategy, allowing the same underlying image acquisition program to be reused across different production lines and different detection tasks, and production line switching can be completed without modifying the underlying code. Attached Figure Description

[0017] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration schematically depicts an application scenario of the image acquisition method, apparatus, device, medium, and program product according to embodiments of the present invention;

[0019] Figure 2 A flowchart illustrating an image acquisition method according to an embodiment of the present invention is shown schematically;

[0020] Figure 3 A schematic diagram illustrating the structure of an image acquisition device according to an embodiment of the present invention is shown; and

[0021] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing an image acquisition method according to an embodiment of the present invention. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] Figure 1 The illustration schematically depicts an application scenario of the image acquisition method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0027] like Figure 1 As shown, application scenario 100 according to this embodiment may include an image acquisition device 101, a computing device 102, and a network 103. The network 103 serves as a medium for providing a communication link between the image acquisition device 101 and the computing device 102. The network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0028] Image acquisition device 101 can be an industrial camera, a smart camera, or any terminal device with image acquisition capabilities, such as a Charge-Coupled Device (CCD) camera, a Complementary Metal Oxide Semiconductor (CMOS) camera, an infrared camera, or an RGB-D (Red-Green-Blue-Depth) camera. In one embodiment, image acquisition device 101 has a global shutter or a rolling shutter and is capable of communicating via standard industrial camera protocols such as GigEVision, USB3 Vision, Camera Link, or CoaXPress. Image acquisition device 101 acquires images of a target observation area (e.g., a specific workstation on a production line) at different acquisition frequencies (e.g., low-frequency acquisition in a first acquisition mode and high-frequency acquisition in a second acquisition mode).

[0029] The computing device 102 can be an edge node with computing capabilities, such as an industrial personal computer (IPC), an embedded AI computing platform, a programmable logic controller (PLC), or a dedicated AI edge inference server. Deployed near the production site, the computing device 102 incorporates the image acquisition method of this embodiment, enabling real-time processing of images acquired by the image acquisition device 101. The computing device 102 performs target detection in the first acquisition mode using lightweight algorithms such as background modeling, pixel difference, and connected component analysis. Upon determining that a target workpiece appears in the target observation area, it automatically switches to the second acquisition mode, activating a deep learning inference engine to perform target detection and defect identification on the image.

[0030] It should be noted that the image acquisition method provided in the embodiments of the present invention can generally be executed by the computing device 102. Correspondingly, the image acquisition device provided in the embodiments of the present invention can generally be located in the computing device 102. The image acquisition method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from the computing device 102 and capable of communicating with the image acquisition device 101 and / or the computing device 102. Correspondingly, the image acquisition device provided in the embodiments of the present invention can also be located in a server or server cluster that is different from the computing device 102 and capable of communicating with the image acquisition device 101 and / or the computing device 102.

[0031] Figure 2 A flowchart illustrating an image acquisition method according to an embodiment of the present invention is shown schematically.

[0032] like Figure 2 As shown, the image acquisition method of this embodiment includes operations S201 to S203, and the image acquisition method can be applied to the computing device 102 as described above.

[0033] In operation S201, a first instruction is sent to an image acquisition device installed at a predetermined position on the production line, so that the image acquisition device performs image acquisition on the target observation area based on a first acquisition mode, wherein the production line is used to transport the target workpiece.

[0034] In operation S202, based on the background image of the (n-1)th frame and the acquired observation image of the nth frame, the background image of the nth frame is determined, where n is a positive integer, and the background image of the first frame is the observation image of the first frame;

[0035] In operation S203, when it is determined that the target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, a second instruction is sent to the image acquisition device to switch the image acquisition device from the first acquisition mode to the second acquisition mode to acquire images of the target observation area. The acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

[0036] According to an embodiment of the present invention, the image acquisition device can be an industrial camera, which is installed above or to the side of a production line conveyor belt, with a field of view covering a predetermined area (i.e., the target observation area) on the conveyor belt. In a first acquisition mode, the image acquisition device acquires images at a first frequency, for example, acquiring one frame per second. In this mode, the acquisition thread of the computing device does not run continuously, but is periodically woken up according to a timer. Specifically, the computing device maintains an atomic type timer trigger flag (timer_trigger_flag), which is set by the underlying timer at a predetermined period (e.g., 1 second). When the flag is set, the acquisition thread is woken up and performs one image acquisition; when the flag is not set and other wake-up conditions are not met, the acquisition thread suspends itself by calling the acquisition wake-up condition variable (cv_capture_wake), thereby releasing CPU resources. This method enables low-power operation in the absence of a target, avoiding the waste of computing power caused by continuous polling of the acquisition thread.

[0037] In one embodiment, the first acquisition mode performs only lightweight operations on the acquired images without loading heavyweight deep learning models. Specifically, it calls a computer vision operator library optimized with the underlying instruction set to process the images, including preprocessing operations such as grayscale conversion and Gaussian blur denoising. This processing method allows the system to consume only a small amount of computing resources even in the absence of a target.

[0038] According to an embodiment of the present invention, a computing device dynamically maintains a background image model to characterize the scene image of the target observation area in a targetless state. To adapt to environmental factors such as changes in lighting and conveyor belt vibration in industrial settings, this background model needs to be continuously updated. Specifically, the nth frame background image is determined by a weighted fusion of the (n-1)th frame background image (i.e., the historical background) and the currently acquired nth frame observation image. For example, assuming the currently acquired observation image is the 5th frame I5, B5 will be determined based on the 4th frame background image B4 and I5. For the 1st frame observation image, since there is no previous frame background image, the 1st frame observation image I1 is directly used as the initial background image B1.

[0039] Furthermore, if it is determined that the target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, a second instruction is sent to the image acquisition device to switch the image acquisition device from the first acquisition mode to the second acquisition mode for image acquisition of the target observation area, wherein the acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

[0040] According to embodiments of the present invention, whether a target workpiece appears in the target observation area is determined purely by visual means, without relying on external hardware triggering sources (such as photoelectric switches, proximity sensors, etc.). Specifically, a foreground mask image is obtained by calculating the pixel difference between the nth frame of the observation image and the nth frame of the background image. Subsequently, the area of ​​connected components in the foreground mask image is determined. If the area of ​​connected components is greater than a preset sensitivity threshold, it is determined that the target workpiece has appeared in the target observation area.

[0041] Once the target workpiece is determined to be within the target observation area, the system switches from the first acquisition mode to the second acquisition mode. Specifically, the computing device sets the atomic-type continuous acquisition enable flag (continue_enable_flag) to true. Upon detecting this flag change, the acquisition thread immediately switches from a suspended state to a continuously running state, and the image acquisition device begins continuous acquisition at a higher acquisition frequency (e.g., 30 frames per second). Simultaneously, the deep learning inference engine is activated, a pre-trained deep learning model is loaded, and target detection or defect identification is performed on each acquired frame. The acquisition frequency of the second acquisition mode (e.g., 30 frames / second) is higher than that of the first acquisition mode (e.g., 1 frame / second), thereby ensuring that the target workpiece can be continuously tracked as it moves within the field of view, without missing any critical moments.

[0042] According to embodiments of the present invention, the pure visual target detection mechanism based on observed and background images can automatically identify the arrival of a target workpiece and trigger mode switching without an external hardware trigger source. This eliminates the deployment and wiring costs of external devices such as photoelectric switches and proximity sensors, reducing the hardware complexity and maintenance difficulty of the system. Furthermore, by switching between the first and second acquisition modes, CPU resources can be released and the system can operate with low power consumption when there is no target, avoiding the waste of computing power and overheating problems caused by 24-hour full-load acquisition. When a target appears, the system can switch from the first acquisition mode to the second acquisition mode, balancing the needs of low-power operation and fast response. It only operates at full speed when needed (i.e., when there is a target), which can save computing power consumption compared to traditional continuous acquisition schemes in production line conditions (where the interval between target workpieces is relatively long).

[0043] Furthermore, by determining the background image of the nth frame based on the (n-1)th frame background image and the acquired nth frame observation image, a background model with dynamic maintenance and adaptive updates is realized, which can adapt to changes in lighting in the industrial environment (such as workshop lights switching, changes in natural light, etc.) and resist the impact of environmental interference on target detection.

[0044] Furthermore, by decoupling the first acquisition mode and the second acquisition mode into independent decision logic, it can flexibly adapt to different detection needs (such as "normal parts are centered and archived" or "defective parts are alarmed immediately"). This decouples the underlying acquisition logic from the upper-level detection strategy, allowing the same underlying image acquisition program to be reused across different production lines and different detection tasks, and production line switching can be completed without modifying the underlying code.

[0045] According to an embodiment of the present invention, in order to achieve low-lock conflict automatic switching of "pure visual perception-acquisition-publishing" in a multi-threaded environment, a set of atomic state variables and synchronization primitives are independently allocated and maintained for each camera.

[0046] Specifically, the atomic state variables include the continuous acquisition enable flag (continue_enable_flag), the burst request flag (burst_request_flag), and the timer_trigger_flag. The continuous acquisition enable flag indicates whether the acquisition thread needs to run continuously at full capacity without sleeping. In the first acquisition mode, this flag is false, and the acquisition thread is suspended via a condition variable when there are no tasks. When the target artifact is determined to appear in the target observation area by background difference, the flag is set to true, and the acquisition thread is then awakened and runs continuously, switching to the second acquisition mode.

[0047] The multi-frame observation images acquired by the image acquisition device are stored in a predetermined cache (such as the memory of a computing device). A burst request flag indicates whether the current frame and historical keyframes need to be extracted immediately for precise capture and distribution. When the deep learning model determines in the second acquisition mode that the current frame meets the capture conditions (e.g., the target is centered and the sharpness reaches a local peak), the flag is set to true. The distribution thread is then awakened, reads the corresponding image frame from the cache, packages it, and sends it to the external system. A timed trigger flag is set by an underlying timer at a predetermined period (e.g., 1 second) to achieve extremely low-frequency sampling in the first acquisition mode, enabling the acquisition thread to be periodically awakened and acquire one frame for background difference detection.

[0048] Synchronization primitives include the capture wake-up condition variable (cv_capture_wake), the publish wake-up condition variable (cv_publish_wake), and the ring buffer mutex (ring_buffer_mutex). The capture wake-up condition variable suspends the capture thread to sleep when there are no capture tasks, releasing CPU resources. It is triggered to wake the capture thread when the continuous capture enable flag is set to true or the timed trigger flag is set. The publish wake-up condition variable wakes up the publishing thread when a new image is acquired or a capture event occurs. The ring buffer mutex protects the consistency of concurrent reads and writes of image data in the buffer, ensuring that when the capture thread writes image data to the ring buffer, the publishing thread will not simultaneously read from the buffer, thus avoiding data races and inconsistent read results.

[0049] According to an embodiment of the present invention, the method for determining the background image of the nth frame based on the background image of the (n-1)th frame and the acquired observation image of the nth frame may include determining a learning rate based on the grayscale deviation between the background image of the (n-1)th frame and the observation image of the nth frame, and using the learning rate to perform weighted updates on the background image.

[0050] Specifically, firstly, based on the grayscale deviation between the (n-1)th frame background image and the nth frame observation image... Determine the learning rate α for the nth iteration; The calculation method is as follows: calculate the global average grayscale value for the (n-1)th frame background image and the nth frame observation image respectively, and then calculate the absolute difference between the two. This grayscale deviation reflects the current degree of change in ambient lighting relative to the background model.

[0051] When determining the learning rate for the nth time, an adaptive adjustment mechanism is used: if Below the steady-state threshold (e.g., 10), it indicates that the current ambient lighting is stable, and the difference between the background image and the observed image mainly comes from camera noise or extremely slow environmental changes. In this case, the learning rate α maintains its first value: the baseline minimum (e.g., 0.005), causing the background model to update at an extremely slow rate to maintain an accurate representation of the stable scene; if... Significantly increased and exceeding the transient threshold (e.g.) If the value reaches 50 or above, it indicates a sudden change in ambient light (such as switching workshop lights on and off, fluorescent lights flickering, or changes in direct sunlight). The learning rate α is automatically increased to a second value (e.g., increased to 0.1), prompting the background model to quickly absorb the changes in ambient light.

[0052] Then, based on the learning rate of the nth time, the observed image of the nth frame, and the background image of the (n-1)th frame, the background image of the nth frame is determined. The calculation method is based on the following formula (1).

[0053] (1).

[0054] in, This is the pixel matrix corresponding to the background image of the nth frame. This is the pixel matrix corresponding to the background image of the (n-1)th frame. This is the pixel matrix corresponding to the nth frame of the observed image. Let be the learning rate for the nth iteration. When taking the minimum value (e.g., 0.005), Mainly due to historical background The decision is that the weight of the current frame is very small, and the background model hardly changes; when α takes a larger value (such as 0.1), the weight of the current frame increases, and the background model can quickly move closer to the currently observed image.

[0055] It should be noted that the initial value B0 of the background model is the first frame image captured by the image acquisition device when the system is first started. In the initial stage of startup, there is no target workpiece in the field of view, and the first frame image acquired at this time is directly used as the initial value of the background model.

[0056] According to embodiments of this disclosure, in actual industrial scenarios, the aforementioned adaptive learning rate mechanism solves the problems of false alarms caused by sudden changes in workshop lighting and "ghosting" deadlocks caused by target stagnation. For example, when the workshop lights are suddenly turned on, the brightness of the entire image changes abruptly within a short period. If the learning rate of the background model remains fixed and is too small, the background model will not be able to keep up with the lighting changes in time, resulting in a large pixel difference between the current frame and the background model across the entire image. This can lead to misjudgments of a large foreground target, triggering false alarms or incorrect switching. However, by monitoring grayscale deviation in real time… When the illumination changes abruptly, the learning rate is automatically increased from 0.005 to 0.1, enabling the background model to adapt to the new illumination conditions within a few frames, thereby avoiding large-area differential false triggering caused by sudden changes in illumination.

[0057] According to an embodiment of the present invention, determining whether a target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image can be achieved by the following operations:

[0058] First, the pixel difference between the nth frame observation image and the nth frame background image is calculated, and the pixel difference is binarized according to a preset threshold to obtain the foreground mask image.

[0059] The process of binarizing the pixel difference image according to a preset global grayscale threshold includes classifying pixels with a difference greater than the threshold as foreground (e.g., assigning a value of 1) and pixels with a difference less than or equal to the threshold as background (e.g., assigning a value of 0), thereby obtaining a foreground mask image. In this foreground mask image, the region where the pixel value corresponds to the foreground indicates that the position has changed and may correspond to the target workpiece; the region where the pixel value corresponds to the background indicates that the position has not changed and belongs to the background scene.

[0060] Then, determine the area of ​​connected components in the foreground mask image.

[0061] Specifically, this can involve performing a connected component labeling algorithm on the foreground mask image to identify all interconnected foreground pixel regions, and calculating the number of pixels contained in each connected component as the area of ​​each connected component. Further, the area of ​​at least one connected component can be determined from the identified connected components, and the area of ​​the largest connected component can be used as the final connected component area for judgment.

[0062] The operation of performing connected component labeling algorithm on the foreground mask image and calculating the area of ​​connected components can be implemented by a region growing algorithm based on pixel neighborhood (such as 4-neighborhood or 8-neighborhood), including the following operations.

[0063] First, traverse the foreground mask image, using currently unlabeled foreground pixels as new seed points. Then, starting from this seed point, examine its surrounding neighboring pixels (e.g., 4-neighborhood, i.e., the four directions up, down, left, and right; or 8-neighborhood, i.e., the eight directions including the diagonal), grouping neighboring pixels that satisfy the connectivity condition (i.e., are both foreground pixels) into the same label set. Next, using the newly grouped pixels as new growth points, repeat the above neighborhood checking and merging process until no new connected pixels can be grouped into the set, thus forming a complete connected component label set. Finally, count the total number of pixels contained in this label set; this total number is the area of ​​the connected component.

[0064] Continue traversing the unlabeled foreground pixels in the foreground mask image, repeating the seed point selection, neighborhood growth, and area calculation processes described above, until all foreground pixels are labeled, thus obtaining the area corresponding to each connected component in the foreground mask image. Finally, compare the areas of all connected components, extract the maximum value as the maximum connected component area, and use it for subsequent comparison with the sensitivity threshold to determine whether the target workpiece appears in the target observation area.

[0065] Subsequently, if the area of ​​the connected region is greater than the sensitivity threshold, it is determined that the target workpiece appears in the target observation area; otherwise, if the area of ​​the connected region is less than or equal to the sensitivity threshold, it is determined that there is no target workpiece in the target observation area, and the first acquisition mode is maintained and continues to operate.

[0066] The sensitivity threshold is calculated based on the field of view of the image acquisition device and the size of the target workpiece.

[0067] Based on the camera's field of view (FOV) and the physical dimensions of the target workpiece, the expected pixel area of ​​the target workpiece in the image can be pre-calculated, and the sensitivity threshold can be set to a value slightly smaller than this expected pixel area. Specific calculation methods include:

[0068] (1) Obtain the horizontal pixel resolution of the image acquisition device screen. And its corresponding physical field of view on the production line. ;

[0069] (2) Calculate the pixel physical scale of the image. (Unit: pixels / mm);

[0070] (3) Calculate the physical projected area of ​​the target workpiece based on its actual physical geometric dimensions. (Unit: square millimeters, for example, the area of ​​a rectangle is length × width);

[0071] (4) According to the formula Calculate the expected pixel area of ​​the target workpiece in the image. It should be noted that this formula implicitly assumes that the vertical scale is the same as the horizontal scale; if the pixels of the image sensor are not square, then the scales in both the horizontal and vertical directions need to be calculated separately.

[0072] To obtain the expected pixel area Then, set the sensitivity threshold. Set to slightly less The value is determined so that the target workpiece can be detected in time when it appears, while avoiding false switching triggered by minor interference.

[0073] For example, in a gear detection application, if the actual diameter of the gear is 50 mm, within the physical area covered by the camera's 1920×1080 pixel field of view (e.g., 500 mm × 280 mm), the corresponding pixel size of the gear in the image is approximately 192 pixels, and the corresponding connected region area is approximately 28,953 square pixels. In this case, the user can set the sensitivity threshold to 20,000 square pixels. By setting the sensitivity threshold slightly smaller than the actual imaging area of ​​the target workpiece, it is possible to ensure that the target workpiece can be detected promptly after entering the field of view, while effectively avoiding false switching triggered by minor interference such as dust, flying insects, and conveyor belt reflection noise.

[0074] Furthermore, the sensitivity threshold is dynamically injected into the system via a JSON configuration file or an inter-process communication (IPC) interface. When the production line switches its inspection target, such as from inspecting gears to inspecting bearings, the user only needs to update the sensitivity threshold parameter in the JSON configuration file, without modifying the underlying code, thus achieving decoupled compatibility for inspection targets of different sizes.

[0075] According to an embodiment of the present invention, after obtaining the foreground mask image and before determining the area of ​​the connected components, the foreground mask image can be further processed to filter out discrete noise regions with an area smaller than a predetermined kernel size.

[0076] Based on this, determining the area of ​​connected components in the foreground mask image includes:

[0077] First, based on a predetermined kernel size, erosion and dilation operations are sequentially performed on the foreground mask image to obtain the target mask image. The erosion operation is used to remove discrete noise points in the foreground mask image with an area smaller than the predetermined kernel size, thereby obtaining the retained foreground region. The dilation operation is used to restore the retained foreground region to the original geometric size of the foreground mask image.

[0078] Then, the area of ​​the connected components corresponding to the observed target is extracted from the target mask image.

[0079] In the erosion operation, a structuring element of a predetermined kernel size slides across the foreground mask image. Only when the entire area covered by the structuring element consists of foreground pixels is the central pixel retained as foreground; otherwise, it is set to background. After erosion, small, discrete noise areas (such as dust particles or conveyor belt reflections) are removed, while the large foreground area corresponding to the real target is retained, although its edges are clipped. The dilation operation then applies to the foreground area retained after erosion, using a structuring element of the same or similar size to expand the retained foreground area, restoring it to a size close to its original geometry. Through this "erosion first, dilation later" opening operation, minute noise can be effectively filtered out while preserving as much of the geometric shape and size information of the real target workpiece as possible, resulting in a clean target mask image.

[0080] In one embodiment, the predetermined kernel size can be configured by the user according to the actual scenario, or it can be adaptively determined according to the resolution of the image acquisition device and the size of the target workpiece. For example, discrete noise generated by interference factors such as dust and conveyor belt surface texture usually has a much smaller area than the imaging area of ​​the target workpiece. By selecting an appropriate kernel size (e.g., 5×5 pixels or 7×7 pixels), these interferences can be filtered out.

[0081] After obtaining the target mask image, the area of ​​connected components corresponding to the observed target is extracted from the target mask image. Specifically, a connected component labeling algorithm is performed on the target mask image to identify all interconnected foreground pixel regions, and the number of pixels in each connected component is calculated as its area. The area of ​​the largest connected component is determined as the area of ​​the connected component corresponding to the observed target. This area of ​​the connected component is then compared with a sensitivity threshold to determine whether the target workpiece appears in the target observation area.

[0082] For example, in a gear inspection scenario, dust particles occasionally drift across the conveyor belt, potentially forming scattered foreground pixels in the foreground mask image. Directly extracting the connected component area of ​​these foreground pixels would result in small connected components, potentially leading to misidentification as the target workpiece. However, the aforementioned processing removes these dust particles, smaller than the kernel size, through erosion, preventing them from entering the subsequent connected component area extraction step. Simultaneously, the gear, as a large real target, retains its main body in its foreground area after erosion, and after expansion, restores its original size, thus being correctly extracted as a connected component and triggering mode switching. This method reduces the false trigger rate caused by environmental factors such as conveyor belt reflection and dust interference, improving robustness in complex industrial environments.

[0083] According to an embodiment of the present invention, the above method further includes: writing multiple frames of observation images acquired based on a first acquisition mode and / or a second acquisition mode into a predetermined cache, wherein the number of multiple frames of observation images stored in the predetermined cache is less than or equal to a predetermined number threshold, and each observation image is marked with a timestamp corresponding to the acquisition time.

[0084] Specifically, a fixed-capacity ring buffer is pre-allocated in memory to store recently acquired multiple frames of observation images. The maximum number of observation images that can be stored in this pre-allocated buffer is less than or equal to a predetermined threshold N (e.g., N = 15~30). Each frame of observation image in the buffer is marked with a timestamp corresponding to the acquisition time and a frame sequence number (frame_id) when it is written, so as to facilitate subsequent historical backtracking and image retrieval.

[0085] The method for writing any frame of the acquired observation image to be stored into a predetermined cache includes the following operations: if the number of multi-frame observation images already stored in the predetermined cache is less than a predetermined number threshold, the observation image to be stored is directly written into the predetermined cache; if the number of multi-frame observation images already stored in the predetermined cache is equal to the predetermined number threshold, the observation image to be stored overwrites the earliest observation image written among the multi-frame observation images already stored, based on the timestamp.

[0086] That is, if the number of multi-frame observation images stored in the predetermined buffer is less than a predetermined threshold N, it indicates that the buffer is not yet full. The observation image to be stored is directly written to the next free slot in the buffer, and the timestamp and frame sequence number of that frame image are recorded. If the number of multi-frame observation images stored in the predetermined buffer is equal to the predetermined threshold N, it indicates that the buffer is full. Based on the timestamps of each frame image, the earliest frame written among the stored multi-frame observation images is determined, and the observation image to be stored is overwritten and written to the slot containing that earliest frame. In this way, the circular buffer always maintains a fixed-size sliding time window, which always stores at most N most recently acquired observation images.

[0087] For example, assuming a predetermined threshold N=20, and observation images with frame numbers 1 to 20 have been acquired and written, the buffer is full. When an observation image with frame number 21 is acquired, frame 21 is overwritten in the position of frame 1; when frame 22 is acquired, frame 22 is overwritten in the position of frame 2, and so on. In this way, at any given time, the circular buffer stores the most recent 20 observation images along with their corresponding timestamps and metadata, while earlier images are automatically discarded.

[0088] In one embodiment, each cache unit in the circular cache stores, but is not limited to, the following: a pointer to raw image data (pointing to the original resolution image data), a timestamp, a frame sequence number, and perceptual label metadata (such as the class name, confidence score, and bounding box coordinates of the model output). The raw image data pointer points to the original resolution image (e.g., 1920×1080 pixels) directly captured by the image acquisition device, rather than a scaled-down low-resolution version, thus ensuring that high-quality original images can be obtained during subsequent historical backtracking captures.

[0089] In the second acquisition mode, deep learning inference is also performed on each acquired frame. It's important to note that the images written to the circular buffer are the original resolution images captured by the camera. When this image needs to be fed into the deep learning model for inference, a separate copy of the image data is made and scaled to the model's required input size (e.g., 640×640 pixels), while the original image data stored in the circular buffer remains unchanged. This design ensures that the AI ​​inference engine obtains images that meet its input requirements while also ensuring that high-resolution original images can be acquired during historical backtracking, balancing inference efficiency and image quality.

[0090] The aforementioned circular cache write mechanism, through its fixed-capacity circular overwrite design and original image preservation strategy, achieves at least the following technical advantages: First, because the cache capacity is fixed and has an upper limit (a predetermined threshold N), the memory space occupied is constant and predictable, avoiding memory overflow problems caused by the infinite growth of image data, making it suitable for deployment on edge computing devices with limited memory resources. Second, the fixed-capacity sliding time window characteristic allows access to the most recent continuous sequence of observed images at any given time. When the deep learning model determines that the target meets the capture conditions at a certain moment, it can extract multiple historical images from the circular cache before that moment, thus solving the problem of "missing the best capture moment" caused by algorithm processing delays. This is particularly important in defect detection scenarios. When the model completes its inference judgment only after the target has moved past the optimal position, it can still use the circular cache to backtrack to the historical frames when the target was in the optimal position, achieving accurate capture with "zero missed frames." The specific implementation method of historical backtracking capture will be described in detail below.

[0091] Furthermore, since the acquisition thread and the publishing thread may concurrently access the circular buffer (the acquisition thread continuously writes new frames, and the publishing thread reads historical frames when a snapshot is triggered), the above embodiment also protects the read and write operations of the circular buffer queue with a mutex lock (ring_buffer_mutex), ensuring that only one thread can access the buffer at any time, thereby avoiding data competition and inconsistent read results, and ensuring the integrity and consistency of the cached data.

[0092] According to an embodiment of the present invention, the method further includes: determining the capture time in the second acquisition mode; determining the target observation image corresponding to the capture time from the multi-frame observation images stored in the predetermined cache; and reading k+1 consecutive frames of observation images from the predetermined cache based on the timestamps of the stored multi-frame observation images and a predefined backtracking depth k, wherein the observation image with the latest acquisition time among the k+1 frames of observation images is the target observation image.

[0093] Specifically, in the second acquisition mode, the timing for capturing an image can be determined based on the recognition results of the deep learning model. When the current frame is determined to meet the preset capture conditions (e.g., the target workpiece is located in the center of the image and the clarity reaches its peak, or a defective target is detected), the atom-type burst request flag (burst_request_flag) is set to true to trigger the capture event. The specific method for determining the capture time will be described in detail below in conjunction with different working modes.

[0094] After a capture event is triggered, the target observation image corresponding to the current capture moment is determined from the multiple observation images stored in the pre-defined buffer (i.e., the circular buffer). Specifically, the current write pointer (write_index) of the circular buffer is read, which points to the most recently written observation image frame, i.e., the latest image frame acquired at the current moment.

[0095] Subsequently, based on the timestamps of the stored multi-frame observation images and the predefined backtracking depth k, k+1 consecutive frames of observation images acquired in a circular buffer are read from the buffer. Here, k is the predefined backtracking depth, representing the number of historical frames to be backtracked. During reading, starting from the latest frame pointed to by the current write pointer, k frames are reversed backward (i.e., in the historical direction), extracting all image frames from the kth frame before the write pointer to the current frame, for a total of k+1 frames. Among these k+1 frames, the frame with the latest acquisition time is the aforementioned target observation image (i.e., the latest frame), and the remaining k frames are historical frames acquired earlier than the target observation image.

[0096] According to embodiments of the present invention, the aforementioned backtracking depth k can be dynamically configured through an external strategy configuration file (e.g., a JSON file), and different detection target categories can correspond to different k values. For example, for center-captured archiving of normal workpieces, the k value can be set to 1, that is, only the current frame and the previous frame, a total of 2 frames, are extracted; while for defect detection alarm scenarios, the k value can be set to 5, that is, the current frame and the 5 frames before it, a total of 6 frames, are extracted to show the complete evolution process of the defect from its appearance to its confirmation.

[0097] For example, assuming the circular buffer capacity N=20, and the current write pointer points to frame number 105, when analyzing frame 105, if the deep learning model determines that the frame meets the capture conditions, and if the backtracking depth is 5, the system will reverse-engineer and extract frames 100, 101, 102, 103, 104, and 105 from the circular buffer—a total of 6 consecutively acquired observation images, with frame 105 being the latest acquired target observation image. In this way, it is possible to simultaneously acquire 5 historical images prior to the capture trigger moment, achieving a "time reversal" historical backtracking capture.

[0098] After extracting the observed images, the extracted multiple frames, along with corresponding metadata (such as target category, confidence score, bounding box coordinates, and other perceptual labels), are packaged and assembled into a complete capture event message. This event message is then pushed to an external non-real-time channel via a publishing thread for further processing by external systems (such as quality inspection platforms, alarm systems, or databases). After packaging and publishing, the burst request flag is reset to false to complete the processing of this capture event.

[0099] According to embodiments of the present invention, in related technologies, the common practice is "determination first, then capture," meaning that inference is performed on the current frame first, and the camera captures only after the condition is met. However, inference itself takes time (usually tens to hundreds of milliseconds). By the time the inference is completed and the capture command is issued, the target workpiece may have moved out of the optimal capture position, resulting in missing the most critical image frame. In industrial conveyor belt scenarios, the conveyor belt speed is usually 0.5 to 2 meters per second. Even with only a 100-millisecond delay, the target will move 5 to 20 centimeters, causing it to deviate from the center of the image or even move out of the field of view. The embodiments of the present invention, through a circular buffer mechanism, store each frame of image in the buffer when it is acquired. When inference and determination are completed in subsequent frames, the buffer can be directly used to backtrack to any historical frame before the determination time, thereby eliminating the contradiction between "determination delay" and "capture timing," achieving "zero-latency" event capture.

[0100] Furthermore, by decoupling the packaging and publishing of captured events from real-time acquisition, network transmission latency does not affect the continuous operation of the acquisition thread. Even if the external system's network becomes congested or temporarily unavailable, the acquisition thread can still continue to acquire and store data in a circular buffer, without frame loss due to publishing blockage. This "acquisition-publishing" separation architecture design enables stable and reliable operation even under high frame rate acquisition and complex network environments.

[0101] According to an embodiment of the present invention, in the second acquisition mode, the working mode can be determined based on the working scenario configuration information; the working scenario configuration information can pre-configure the working mode, for example, it can be configured to adopt a precise capture mode, a defect detection mode, a continuous sampling mode, etc. Different working modes correspond to different capture time determination logic to adapt to diverse industrial inspection needs.

[0102] For example, the precision capture mode is suitable for scenarios where a standard photo of the target workpiece in the exact center of the frame with the clearest focus is needed, such as archiving good-quality mechanical parts like gears and bearings. For example, the defect detection mode is suitable for detecting severe surface scratches, damage, and other quality defects. As soon as a defect appears in the field of view, even if it's still at the edge of the frame and not yet clear, it needs to be captured immediately and an alarm should be triggered; the target cannot wait to move to the center of the frame. For example, the continuous sampling mode is suitable for detecting particularly long target objects, such as continuous pieces of fabric, steel strips, or long pipes. These objects pass continuously on conveyor belts without a clear "center," making the "capture only when it moves to the center" logic of the precision capture mode unsuitable. Therefore, continuous sampling and detection are required throughout the object's entire passage.

[0103] Deep learning models adapted to the aforementioned working modes can be pre-configured. In the corresponding working mode, the appropriate deep learning model is invoked for image processing. Specifically, the working scene configuration information exists in the form of a strategy configuration file in JSON or YAML format, which defines the mapping relationship between the working modes and the deep learning models. For example, when the working mode is determined to be the precise capture mode, the object detection model is invoked to infer the observed image to identify the target workpiece and output the bounding box coordinates. When the working mode is determined to be the defect detection mode, an image classification model can be invoked to classify the entire observed image to determine whether the target workpiece is defective; alternatively, the object detection model can be invoked to identify whether a specific category of defective target exists in the image. When the working mode is determined to be the continuous sampling mode, the same model as in the precise capture mode is invoked to continuously detect the target object according to a predetermined sampling interval.

[0104] In one embodiment, the aforementioned work scenario configuration information is dynamically loaded into the configuration file during startup or operation, allowing for switching of work modes without restarting the system, thus decoupling the underlying acquisition logic from the upper-level detection strategy. Through the configuration of these multiple work modes, the same underlying image acquisition program can adapt to different business logics such as "centering and archiving normal parts," "instant alarm for serious defects," and "continuous sampling of long materials." Users only need to change the configuration file to achieve cross-production line function switching without modifying the underlying code.

[0105] The following sections describe in detail how to determine the capture time under these different working modes.

[0106] In one embodiment, when the working mode is determined to be the precision capture mode (corresponding to "center_capture" in the strategy configuration file), determining the capture time includes: identifying the target workpiece in the observed image and determining the bounding box of the target workpiece; determining the reference distance between the center of the observed image and the center of the bounding box; determining the sharpness of the observed image; and determining the capture time based on the reference distance and / or sharpness. That is, when the working mode is determined to be the precision capture mode, it is necessary to identify the target workpiece in the observed image and determine its bounding box, then calculate the reference distance between the target center and the center of the image, determine the sharpness of the observed image, and finally determine the capture time based on the above reference distance and sharpness.

[0107] Specifically, the system identifies target workpieces in the observed image by calling a target detection model and outputs the bounding box coordinates of the target workpieces. Assume the coordinates of the top-left corner of the bounding box are... The coordinates of the lower right corner are Then calculate the coordinates of the center of the bounding box: , .

[0108] Simultaneously, the coordinates of the center of the observed image (W / 2, H / 2) are obtained, where W and H are the width and height of the observed image, respectively. Then, the Euclidean distance D between the center of the bounding box and the center of the image is calculated: This distance D corresponds to the aforementioned reference distance and reflects the degree of deviation of the target workpiece from the center of the image.

[0109] To determine whether an image is blurred due to motion, the sharpness of the observed image is further assessed. Specifically, an image region within a bounding box is cropped from the observed image, the second derivative gradient of this region is extracted, the standard deviation of this gradient is calculated, and the square of this standard deviation is used as the Laplacian variance. The larger this value, the sharper the image edges and the clearer the focus; conversely, the smaller the value, the more blurred the image.

[0110] Then, the capture time can be determined based on the reference distance and / or sharpness.

[0111] For example, the capture time can be determined based on both reference distance and image sharpness. Specifically, when the reference distance D of the target workpiece is greater than the preset center tolerance radius R, it indicates that the target workpiece has not yet moved to the vicinity of the image center. At this time, even if the image sharpness meets the condition, capture is not triggered, and the tracking continues to the next frame. When the reference distance D is less than or equal to R, it indicates that the target workpiece has entered the center area of ​​the image. At this time, the sharpness condition is further judged. If the Laplacian variance is greater than the preset sharpness threshold (or reaches a local peak), the current moment is determined to be the capture time; if the sharpness does not meet the condition, tracking continues. Only when both the "centering position" and "image sharpness" conditions are met simultaneously is capture triggered, thereby ensuring that the captured image is both centered and sharp.

[0112] For example, the capture time can be determined based on a single factor, either reference distance or sharpness. For instance, in applications where the conveyor belt moves quickly and the target workpiece spends very little time in the center of the image, capture can be triggered immediately when the reference distance D is less than or equal to R, without waiting for sharpness determination, ensuring that the target is not missed due to waiting for sharpness to meet the standard. Similarly, in certain detection tasks with extremely high image sharpness requirements, capture can be triggered immediately when the Laplacian variance reaches its peak, without requiring the target to be strictly centered to capture the sharpest focusing moment.

[0113] Furthermore, in the method of determining the capture time based on both reference distance and sharpness, capture is only triggered when both conditions are met simultaneously: "centering position" (reference distance D is less than or equal to R) and "image sharpness" (Laplacian variance is greater than a preset sharpness threshold). If the Laplacian variance cannot be greater than the preset sharpness threshold within the time interval where reference distance D is less than or equal to R, then as a fallback, the capture time can be determined as the moment when the sharpness of the observed image reaches a local peak.

[0114] Specifically, in one embodiment, a combination of a fixed sharpness threshold and local peak determination is used to determine the capture time. When the reference distance D of the target workpiece of a specified category is less than or equal to the preset center tolerance radius R, and the Laplace variance is greater than the preset sharpness threshold (laplacian_clear_threshold), the current time is determined to be the capture time.

[0115] In another embodiment, for objects with sparse surface texture that cannot reach the absolute high variance threshold, a local peak determination method is used: the Laplacian variance of multiple consecutive adjacent observation images is recorded, and the acquisition time of the observation image with the highest sharpness corresponding to the Laplacian variance is determined as the capture time. For example, the Laplacian variance of three consecutive adjacent frames is recorded. , and When detected and When the image at time t-1 reaches a local peak of sharpness, it is determined that this frame is the sharpest among several adjacent frames. Taking the detection of a smooth metal bearing as an example, the surface of this object lacks texture features, and its absolute value of Laplacian variance is usually low (e.g., only 60~80), which cannot reach the sharpness threshold set by the fixed threshold method (e.g., 120). In this case, the fixed threshold method would determine that the object is never sharp enough, and thus would never trigger a capture. However, the local peak determination method, by comparing the relative changes between adjacent frames, can identify the relatively sharpest frame within the range of V values ​​of only 60~80 (e.g., ...). =78, and =65、 =72), thus adaptively determining the timing of the capture, improving adaptability to workpieces of different materials.

[0116] When the current frame is determined to meet the capture conditions, the circular buffer mechanism allows for direct backtracking to retrieve and publish historical frames from time t-1, resolving the issue of missing the clearest image due to delayed judgment. After determining the capture time, the burst request flag (burst_request_flag) is set to true to trigger subsequent historical backtracking capture and event publishing processes.

[0117] It should be noted that the aforementioned precise capture mode is suitable for scenarios where a standard photo of the target workpiece in the exact center of the frame with the clearest focus is required, such as archiving good-quality mechanical parts like gears and bearings. When users need to adapt to different production lines or different inspection tasks, they only need to adjust the above parameters (such as center tolerance radius R, sharpness threshold, or whether to enable local peak detection) through an external strategy configuration file, without modifying the underlying code to complete the configuration update.

[0118] According to an embodiment of the present invention, in one embodiment, when the working mode is determined to be the defect detection mode (corresponding to the action_type being "immediate_burst" in the strategy configuration file), determining the capture time includes: identifying the category of the observed image; and when the category of the observed image indicates that the target workpiece is a defective product, determining the current capture time as the capture time.

[0119] Specifically, the image classification model infers the probability distribution of the image belonging to each category. When the category of the entire image is determined to be a "key observation category" (such as "surface scratches", "dents", "cracks" and other defective product categories) as defined in the strategy file, and the confidence level of the category is higher than the preset confidence level threshold, the current shooting time is immediately determined as the capture time, and the capture event is triggered.

[0120] Unlike the aforementioned precise capture mode, the defect detection mode skips the calculation of the reference distance (i.e., the Euclidean distance D between the target center and the image center) and the determination of Laplacian sharpness. It does not require waiting for the target workpiece to move to the center of the image, nor does it require judging whether the image is sufficiently sharp. Capture is triggered instantly as long as the model output of the current frame contains a defective product and the confidence level is greater than the threshold. Simultaneously, this mode typically configures a large backtracking depth k to retrieve k historical images from the circular buffer prior to the current frame when capture is triggered.

[0121] The defect detection mode is suitable for detecting severe surface scratches, damage, and other quality defects. In such scenarios, the timeliness of detection takes precedence over the aesthetics of the image. As soon as a defect is revealed at the edge of the field of view, even if the defect is still at the edge of the image and the image is not yet clear, it is necessary to immediately capture the image and trigger an alarm. We cannot wait for the target to move to the center of the image. If a "capture only when the target is in the center" strategy is adopted, the system will not trigger a capture when the defect appears at the edge of the image. By the time the defect moves to the center of the image, due to the high-speed movement of the conveyor belt, the defect may have already expanded or changed, or it may even be difficult to clearly present due to changes in the target's posture. The instantaneous alarm mode triggers a capture the moment the defect first appears at the edge of the field of view, and obtains the complete evolution process of the defect from its appearance to the current frame through a circular buffer backtracking mechanism (e.g., a multi-frame sequence of a scratch gradually expanding from a small mark to a significant defect), providing complete data support for subsequent quality analysis and defect tracing.

[0122] According to an embodiment of the present invention, in one embodiment, when the working mode is determined to be continuous sampling mode (corresponding to action_type being "interval_sampling" in the strategy configuration file), determining the capture time includes: wherein determining the capture time includes: within the time interval from the appearance of the target workpiece to its departure from the target observation area, determining a plurality of capture times distributed at predetermined time intervals.

[0123] Specifically, once the target workpiece is determined to have appeared in the target observation area through background subtraction or a deep learning model in the second acquisition mode, a timed sampling mechanism is immediately initiated. While the target workpiece remains within the field of view (i.e., during the duration of the second acquisition mode), a snapshot is triggered at a pre-configured fixed frame interval (e.g., every 10 frames). Each time it is triggered, the latest frame and a backtracking frame are retrieved from the circular buffer and published. This sampling process continues until it is determined that the target workpiece has left the target observation area and the system reverts to the first acquisition mode.

[0124] It should be noted that in continuous sampling mode, the system does not wait for the target to move to the center of the frame, nor does it require the image to reach optimal clarity. Instead, it adopts a "pass-through, sample-through" strategy, continuously sampling frames during the entire passage of the target.

[0125] Continuous sampling mode is suitable for detecting long target objects, such as continuous pieces of fabric, steel strips, and long pipes. These objects pass continuously on conveyor belts, have a large aspect ratio, and lack a clear "center." Taking steel strip surface defect detection as an example, the steel strip travels continuously for several meters or even tens of meters on the conveyor belt, and defects may appear at any location on the steel strip. If a precise capture mode is used, the "center" position will never be determined, and a capture will not be triggered. However, using continuous sampling mode, an image is captured at fixed frame intervals throughout the entire passage of the steel strip, ensuring that the entire steel strip is uniformly covered for detection, and no defects are missed while waiting for a specific location. Through the above method, this embodiment can adapt to the detection needs of continuous long objects, ensuring comprehensive sampling coverage throughout the entire passage of the target object.

[0126] According to an embodiment of the present invention, the method further includes: in the second acquisition mode, when the rollback condition is met, sending a third instruction to the image acquisition device to cause the image acquisition device to switch back from the second acquisition mode to the first acquisition mode.

[0127] The backoff condition includes at least one of the following: the target workpiece is not detected in the target observation area based on M consecutive frames of observation images; the area of ​​the connected region is less than or equal to the sensitivity threshold.

[0128] To ensure reliable rollback under different deep learning model paradigms and to prevent deadlock caused by "illusions" (such as continuously identifying stains or shadows on an empty conveyor belt as targets) in complex environments, a dual verification mechanism of "training model semantic judgment" and "physical pixel verification" is adopted.

[0129] Regarding semantic determination in the training model, dynamic judgment is made based on the currently mounted model strategy. If the mounted model is an object detection model, it is determined that the object has left the field of view when no valid object bounding box is output for M consecutive frames, or when the geometric center of the object touches the edge region of the image. If the mounted model is an image classification model, it is determined that the object has left the field of view when the classification result output for M consecutive frames is the "background / empty" category predefined in the strategy file.

[0130] Regarding physical pixel verification, regardless of the semantic result output by the inference engine, the underlying lightweight background difference calculation always runs in parallel, independent of the AI ​​inference process. If the maximum connected region area of ​​the difference foreground drops below the sensitivity threshold (meaning there are no foreign objects physically within the field of view), it is determined that the target has left.

[0131] The above dual verification adopts the following approach: as long as either the "target is lost in the semantic determination of the training model" or the "the field of view is empty in the determination of the physical pixels" is met, the current detection event is determined to have completely ended. The continuous acquisition enable flag (continue_enable_flag) is reset to false, the computationally intensive inference engine is actively suspended to release GPU / CPU resources, smoothly return to the first acquisition mode, and resume the low-frequency timed sleep waiting state.

[0132] Through the aforementioned dual verification mechanism, even if the AI ​​model continuously outputs the erroneous result of "target exists" due to lighting interference, model illusion, or data distribution drift, it can still reliably reset as long as the physical pixel verification determines that there is no foreground target within the field of view. This avoids the situation where the system is permanently stuck in the second acquisition mode due to AI anomalies, significantly improving the long-term operational stability of the system in complex industrial environments. For example, in an empty conveyor belt scenario, there is an oil stain on the surface of the conveyor belt. The deep learning model may continuously misidentify the oil stain as a "gear" because its texture and color are similar to the target workpiece, causing it to be unable to revert. However, since the oil stain is stationary, the area of ​​the connected components detected by the underlying background difference will gradually decrease as the background model learns and eventually drop below the sensitivity threshold. At this point, a forced revert from the second acquisition mode to the first acquisition mode is made, thereby avoiding deadlock and high-power idle running caused by model illusion.

[0133] According to an embodiment of the present invention, the image acquisition method further includes an execution flow of a publishing thread. The publishing thread is decoupled from the acquisition thread and the perception decision layer, runs independently, and is in a blocked waiting state.

[0134] Specifically, the publishing thread is suspended when there are no tasks, using the publish wake-up condition variable (cv_publish_wake), thus not consuming CPU resources. Once the acquisition thread completes the acquisition of a frame and writes it to the circular buffer, it calls cv_publish_wake.notify_all() to wake up the publishing thread. After being woken up, the publishing thread performs the corresponding publishing operation based on the current system state.

[0135] When the publishing thread detects that the burst request flag (burst_request_flag) is true, it performs an event backtracking and packaging publishing operation. Specifically, the publishing thread acquires a read lock on the ring buffer mutex (ring_buffer_mutex), and counts backward K frames based on the current write pointer of the ring buffer. It assembles the K+1 frame images (i.e., all image frames from the Kth frame before the write pointer to the latest frame) along with the corresponding tag metadata (including perception results such as target category, confidence score, and bounding box coordinates) into a complete event message, and then pushes it to the external system through an external non-real-time channel (such as WebSocket or ZMQ). After publishing is complete, the publishing thread resets the burst request flag to false.

[0136] When the publishing thread detects that the burst request flag is false and the current system is running in the first acquisition mode, it performs a regular heartbeat sending operation. Specifically, the publishing thread pulls the latest observed image frame written from the circular buffer, assembles it into a heartbeat message along with device status information (such as system time, runtime, current acquisition mode, etc.), and sends it to the external system so that the external system can confirm the device's online status and perform remote image preview.

[0137] On the one hand, the publishing thread and the acquisition thread are independent of each other, and the latency caused by pushing event messages outward through the network will not block the continuous operation of the acquisition thread. Even if the external network is congested or temporarily unavailable, the acquisition thread can still continue to acquire images and write them to the circular buffer without frame loss due to publishing blockage. On the other hand, the publishing thread distinguishes between "regular heartbeats" and "event captures" by detecting burst request flags. When a capture event occurs, event publishing is prioritized, and heartbeats are sent to maintain the device's online status when there are no events, thus achieving on-demand adaptive scheduling of publishing tasks. Through the above mechanisms, the system can maintain stable and reliable operation even in high frame rate acquisition and complex network environments.

[0138] Based on the above image acquisition method, the present invention also provides an image acquisition device. The following will be combined with... Figure 3 The device is described in detail.

[0139] Figure 3 A schematic block diagram of an image acquisition device according to an embodiment of the present invention is shown.

[0140] like Figure 3 As shown, the image acquisition device 300 of this embodiment includes a first sending module 301, a first determining module 302, and a second sending module 303.

[0141] The first sending module 301 is used to send a first instruction to an image acquisition device installed at a predetermined position on the production line, so that the image acquisition device can perform image acquisition on the target observation area based on a first acquisition mode, wherein the production line is used to transport the target workpiece;

[0142] The first determining module 302 is used to determine the nth frame background image based on the (n-1)th frame background image and the acquired nth frame observation image, where n is a positive integer and the first frame background image is the first frame observation image.

[0143] The second sending module 303 is used to send a second instruction to the image acquisition device when it is determined that the target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, so that the image acquisition device switches from the first acquisition mode to the second acquisition mode to perform image acquisition for the target observation area, wherein the acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

[0144] According to embodiments of the present invention, any plurality of modules among the first transmitting module 301, the first determining module 302, and the second transmitting module 303 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the first transmitting module 301, the first determining module 302, and the second transmitting module 303 can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented by any other reasonable means of integrating or packaging the circuit, or implemented by any one of the three implementation methods of software, hardware, and firmware, or by any appropriate combination of any of them. Alternatively, at least one of the first transmitting module 301, the first determining module 302, and the second transmitting module 303 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0145] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing an image acquisition method according to an embodiment of the present invention.

[0146] like Figure 4As shown, an electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0147] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0148] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0149] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0150] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0151] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0152] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0153] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0154] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0158] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. An image acquisition method, comprising: A first instruction is sent to an image acquisition device installed at a predetermined position on the production line, so that the image acquisition device performs image acquisition on the target observation area based on a first acquisition mode, wherein the production line is used to transport the target workpiece; Based on the (n-1)th frame background image and the acquired nth frame observation image, the nth frame background image is determined, where n is a positive integer, and the first frame background image is the first frame observation image. When it is determined that the target workpiece appears in the target observation area based on the nth frame observation image and the nth frame background image, a second instruction is sent to the image acquisition device to switch the image acquisition device from the first acquisition mode to the second acquisition mode to acquire images of the target observation area, wherein the acquisition frequency corresponding to the second acquisition mode is higher than the acquisition frequency corresponding to the first acquisition mode.

2. The method according to claim 1, further comprising: Calculate the pixel difference between the nth frame observation image and the nth frame background image, and binarize the pixel difference according to a preset global grayscale threshold to obtain the foreground mask image; Determine the area of ​​connected components in the foreground mask image; If the area of ​​the connected region is greater than the sensitivity threshold, it is determined that the target workpiece appears in the target observation area; The sensitivity threshold is calculated based on the field of view of the image acquisition device and the size of the target workpiece.

3. The method according to claim 2, further comprising: In the second acquisition mode, when the rollback condition is met, a third instruction is sent to the image acquisition device to make the image acquisition device switch back from the second acquisition mode to the first acquisition mode; The rollback conditions include at least one of the following: The target workpiece was not detected in the target observation area based on M consecutive frames of observation images; The area of ​​the connected region is less than or equal to the sensitivity threshold.

4. The method according to claim 2, wherein, Determining the area of ​​connected components in a foreground mask image includes: Based on a predetermined kernel size, erosion and dilation operations are sequentially performed on the foreground mask image to obtain a target mask image. The erosion operation is used to remove discrete noise points in the foreground mask image with an area smaller than the predetermined kernel size, thereby obtaining the retained foreground region. The dilation operation is used to restore the retained foreground region to the original geometric size of the foreground mask image. Extract the area of ​​the connected components corresponding to the observed target from the target mask image.

5. The method according to claim 1, wherein, Based on the background image of frame n-1 and the acquired observation image of frame n, the background image of frame n is determined to include: The learning rate for the nth time is determined based on the grayscale difference between the background image of the (n-1)th frame and the observed image of the nth frame. The background image of the nth frame is determined based on the learning rate of the nth time, the observed image of the nth frame, and the background image of the (n-1)th frame.

6. The method according to claim 1, further comprising: Multiple frames of observation images acquired based on the first acquisition mode and / or the second acquisition mode are written into a predetermined cache, wherein the number of multiple frames of observation images stored in the predetermined cache is less than or equal to a predetermined number threshold, and each observation image is marked with a timestamp corresponding to the acquisition time. The method for writing any frame of the acquired observation image to be stored into the predetermined buffer includes the following operations: If the number of multi-frame observation images already stored in the predetermined cache is less than a predetermined threshold, the observation image to be stored will be directly written into the predetermined cache. If the number of multi-frame observation images already stored in the predetermined cache is equal to a predetermined threshold, the observation image to be stored will overwrite the earliest written observation image among the stored multi-frame observation images based on the timestamp.

7. The method according to claim 6, further comprising: In the second acquisition mode, the capture time is determined; The target observation image corresponding to the capture time is determined from the multiple frames of observation images stored in the predetermined cache; Based on the timestamps of the stored multi-frame observation images and the predefined backtracking depth k, k+1 consecutive frames of observation images with consecutive acquisition times are read from the predetermined cache, wherein the observation image with the latest acquisition time among the k+1 frames is the target observation image.

8. The method according to claim 7, wherein, Determining the capture time includes: In the second acquisition mode, the working mode is determined based on the working scenario configuration information; When the working mode is determined to be the precision capture mode, the target workpiece in the observed image is identified and the bounding box of the target workpiece is determined. Determine the reference distance between the center of the observed image and the position center of the bounding box; Determine the sharpness of the observed images; The capture time is determined based on the reference distance and / or the sharpness.

9. The method according to claim 7, wherein, Determining the capture time includes: In the second acquisition mode, the working mode is determined based on the working scenario configuration information; If the working mode is determined to be a defect detection mode, the category of the observed image is identified; If the category of the observed image indicates that the target workpiece is defective, the current shooting time is determined as the capture time.

10. The method according to claim 7, wherein, Determining the capture time includes: In the second acquisition mode, the working mode is determined based on the working scenario configuration information; When the working mode is determined to be continuous sampling mode, multiple capture times distributed at predetermined time intervals are determined within the time interval from the appearance of the target workpiece to its departure from the target observation area.