An event-triggered and abnormal area positioning-based coarse-fine collaborative visual automatic detection system and method

Abstract

The application discloses a kind of based on event trigger and abnormal area positioning coarse and thin coordination visual automatic detection system and method, belong to machine vision detection technical field.System is carried through two-dimensional motion mechanism variable-focus camera, and low-ratio acquisition covers the coarse monitoring image of equipment panel;After detecting the abnormal change of indicating component, target motion coordinates are generated by positioning abnormal area and perspective mapping, drive camera to move, zoom and focus, and collect local high-definition image.Indicating component label, text label and state are obtained by target detection and OCR identification, state meaning is obtained by matching equipment state database, and the result is output, then return coarse monitoring reference position and continue to monitor;When there are multiple abnormal areas, they are processed in priority order.The application triggers local fine monitoring with event, considers wide range coverage and detail recognition, reduces full-area high-ratio scanning, and improves equipment panel state detection efficiency and automation degree.

Description

Technical Field

[0001] This invention relates to the field of machine vision inspection technology, and in particular to a coarse-fine collaborative vision automatic inspection system and method for equipment panels, communication cabinets, industrial control cabinets or instrument panels, based on event triggering and abnormal area localization. Background Technology

[0002] Communication cabinets, industrial control cabinets, power distribution cabinets, and test instrument cabinets typically have power indicator lights, operation indicator lights, alarm indicator lights, digital displays, screen prompts, and text labels on their panels. Maintenance personnel can determine whether the equipment is powered on, operating normally, or experiencing alarms by observing these indicators, as well as the location and specific meaning of any abnormal conditions.

[0003] Currently, there are three main methods for monitoring these device panels: manual inspection, fixed camera monitoring, and periodic visual scanning with a motion mechanism. Manual inspection suffers from poor real-time performance, long inspection intervals, and the results are affected by factors such as personnel experience and attention, making it difficult to form continuous and objective records. While fixed cameras can remotely view the panels, installation space is often limited, and a single camera can hardly simultaneously achieve both large-area coverage and clear identification of small targets; with low-magnification or wide-angle lenses, the labels and text on small indicator lights are not clearly visible; and with high-magnification lenses, the field of view is too small, only able to see a very small area at a time.

[0004] Existing 2D motion scanning solutions can drive cameras to capture images of panel sections, but each scan involves a full-area high-magnification scan, resulting in long motion paths, a large number of images, and long processing times. For cabinet panels, the status is stable most of the time, and the area that truly requires high-magnification confirmation is usually only a small portion. Therefore, it is necessary to quickly detect status changes during the coarse monitoring phase, locate abnormal areas after triggering them, convert the image coordinates of the abnormal areas into the mechanism's motion coordinates, enable the camera to quickly reach the corresponding position for local high-definition identification, and automatically resume coarse monitoring after completing fine monitoring.

[0005] Furthermore, existing solutions largely operate as two independent processes for anomaly detection and detailed verification, lacking an effective linkage mechanism. While front-end images can identify localized brightness and color anomalies, a stable conversion relationship exists between image coordinates and the physical coordinates of the moving mechanism. After an anomaly is detected, manual judgment of the movement direction and manual control of the mechanism to the corresponding location for verification are still required. Although some monitoring solutions are equipped with corresponding moving mechanisms, these mechanisms can only perform periodic inspections along preset paths; the actual location of the anomaly does not change its trajectory, making it impossible to adjust the detection sequence in real time based on the actual anomaly location. For panels with densely distributed indicator lights, port labels, and screen prompts, if the label content cannot be accurately located and read after an anomaly is triggered, only the anomaly phenomenon can be detected, but the specific source of the anomaly cannot be identified. Simultaneously, most existing solutions do not support automatic switching between coarse and fine monitoring modes; after localized detailed verification, they cannot automatically switch back to large-scale monitoring mode. When multiple anomalies occur, they cannot be processed sequentially according to priority, leaving room for improvement in overall automation and detection efficiency. Summary of the Invention

[0006] The purpose of this invention is to solve the problems in existing device panel visual inspection, such as the difficulty in achieving both large-area coverage and local detail recognition, low efficiency of high-magnification scanning of the entire area, difficulty in quickly locating abnormal areas, and insufficient recognition of the status meaning of indicator components.

[0007] To achieve the above objectives, this invention provides a coarse-fine collaborative visual automatic detection system and method based on event triggering and abnormal area localization. The system determines the coarse monitoring area and coarse monitoring reference motion coordinates based on the equipment panel size, the low-magnification field of view of the camera, and the mechanism travel. It also establishes a perspective mapping relationship between the coarse monitoring image coordinates and the motion coordinates through at least four non-collinear calibration points. Taking a cabinet panel approximately 1m high and 0.5m wide as an example, when the panel height is greater than the effective low-magnification coarse monitoring height of a single camera, two sets of two-dimensional motion mechanisms and a variable-focus camera are installed at the front of the cabinet.

[0008] In coarse monitoring mode, the zoom camera is positioned at the coarse monitoring reference motion coordinates and maintains preset low magnification, focal length, and exposure parameters, continuously or periodically acquiring coarse monitoring images. The current image is compared with the baseline state library after position registration and brightness normalization; when the changes continuously meet the trigger conditions and do not conform to the normal flicker rules, a fine monitoring trigger event containing the location of the abnormal area is generated.

[0009] Upon triggering, the image processing module performs threshold segmentation, connected component analysis, and boundary correction on the changed area to determine the location point of the abnormal area. This location point is then input into the perspective mapping matrix to obtain the target motion coordinates of the two-dimensional motion mechanism. The motion control module acquires the current motion coordinates and generates motion control quantities based on the difference between the target motion coordinates and the current motion coordinates. This drives the two-dimensional motion mechanism to move the zoom camera to the target detection position.

[0010] The coordinate mapping is established using at least four dispersed, non-collinear calibration points, and the perspective mapping matrix from the coarse monitoring image coordinates to the motion coordinates is solved. When the coarse monitoring area is divided into multiple sub-regions, a local mapping is established for each sub-region. When an anomaly is located in the overlapping zone of adjacent sub-regions, the two sets of mapping results are fused according to the normalized weights of the inverse distance from the anomaly to the center of the two sub-regions. Motion control uses limit switch homing and stepper motor pulse counting for positioning. After outputting a pulse, the current motion coordinates are updated according to the displacement of each pulse.

[0011] Once the camera reaches the target detection position, it switches to high magnification mode, employs an autofocus method based on Laplacian variance to select the optimal focal length, and acquires a local high-resolution image. The image processing module performs indicator component status recognition and OCR text recognition, then matches the device status database based on the device model, installation location, and indicator component number to obtain the corresponding device operation meaning and outputs the result. After all abnormal areas have been processed, the camera returns to the coarse monitoring reference motion coordinates, restores the preset low-magnification imaging parameters, and continues to acquire images of the coarse monitoring area.

[0012] The fine monitoring magnification is determined based on the size of the boundary box of the abnormal region and the size of the target recognition pixels. If the target center in a local high-definition image deviates from the image center by more than a preset pixel threshold, a secondary fine adjustment is generated based on the local pixels and motion ratio coefficient under the fine monitoring magnification, driving the two-dimensional motion mechanism to make a small-range correction so that the target indicator component is located in the central region of the fine monitoring image.

[0013] This invention also provides a coarse-fine collaborative visual automatic detection system based on event triggering and abnormal region localization, including a two-dimensional motion mechanism, a variable-focus camera, a motion control module, an image processing module, and a status output module. The image processing module is used to perform change detection and event triggering, abnormal region localization, coordinate mapping, indicator part identification, and status meaning determination; the motion control module is used to drive the two-dimensional motion mechanism to complete zeroing, pulse counting localization, and target position fine-tuning; the status output module is used to output the detection results.

[0014] Compared with existing technologies, the present invention has the following advantages: First, it uses low-magnification coarse monitoring under normal conditions, and only performs high-magnification fine monitoring on abnormal areas after triggering, reducing invalid scanning and image processing workload; Second, it converts abnormal positions in coarse monitoring images into target coordinates of moving mechanisms through perspective mapping, realizing automatic connection from abnormality detection to local confirmation; Third, it obtains the label and status meaning of indicator components through target detection and OCR recognition, reducing misjudgments caused by direct judgment of low-magnification images; Fourth, it automatically restores the coarse monitoring state after fine monitoring is completed, and can process multiple abnormal areas sequentially according to trigger priority, improving detection efficiency and automation. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the two-dimensional motion mechanism of the camera in this invention.

[0016] Figure 2 This is a flowchart of the coarse-fine collaborative visual automatic detection method based on event triggering and abnormal region localization in this invention.

[0017] Figure 3 This is a functional block diagram of the coarse-fine collaborative vision automatic detection system based on event triggering and abnormal region localization in this invention.

[0018] The following are the labels in the diagram: 1. Vertical lead screw; 2. Vertical limit switch; 3. Vertical stepper motor mounting base; 4. Vertical guide optical axis fixing base; 5. Vertical guide optical axis; 6. Camera mounting base; 7. Variable zoom camera; 8. Horizontal guide optical axis; 9. Horizontal synchronous belt; 10. Horizontal stepper motor mounting base; 11. Horizontal limit switch. Detailed Implementation

[0019] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, the zoomable camera 7 is mounted on a camera mounting base 6, which can move along a horizontal guide optical axis 8. A horizontal stepper motor is mounted on a horizontal stepper motor mounting base 10 and drives the camera mounting base 6 to move horizontally via a horizontal synchronous belt 9. A vertical stepper motor is mounted on a vertical stepper motor mounting base 3 and drives the horizontal motion component to move vertically along a vertical guide optical axis 5 via a vertical lead screw 1. The vertical guide optical axis 5 is fixed by a vertical guide optical axis fixing base 4. Therefore, the zoomable camera 7 can be moved to a designated detection position in front of the monitored device panel.

[0021] In this embodiment, the monitored device panel is the front panel of a communication cabinet with a height of approximately 1m and a width of approximately 0.5m. Two sets of two-dimensional motion mechanisms are set at the front of the cabinet to monitor the upper and lower halves of the area, respectively. After the system starts, the horizontal and vertical mechanisms move towards the zero-point direction, triggering the corresponding limit switches and establishing the zero point of the motion coordinates. Each set of mechanisms saves the preset coarse monitoring reference motion coordinates M_r=(x_r,y_r). During coarse monitoring, the camera moves to M_r and restores the preset low magnification, focal length, and exposure parameters.

[0022] During calibration, the camera is first positioned at the coarse monitoring reference motion coordinates M_r and maintains preset low-magnification imaging parameters. At least four dispersed, non-collinear calibration points are selected within the coarse monitoring area. The coarse monitoring pixel coordinates P_i=(u_i,v_i) of the i-th calibration point are recorded. Then, the camera is moved to the corresponding fine monitoring position using jog control, and the calibration point is positioned at the center of the fine monitoring image. The motion coordinates M_i=(x_i,y_i) are recorded. The perspective mapping parameters h11 to h32 are solved using the least squares method from multiple sets of (P_i,M_i), satisfying: In this formula, the image coordinates (u,v) of the abnormal region's location point are substituted into the above formula to calculate the target motion coordinates (x_t,y_t) of the two-dimensional motion mechanism. h11 to h32 are perspective mapping parameters. When multiple sub-regions are used, the mapping is solved separately for each sub-region; the two sets of results M_1 and M_2 within the overlapping zone are fused according to M=λM_1+(1-λ)M_2. If the distance from the abnormal point to the center of the two sub-regions is d_1 and d_2, then λ=d_2 / (d_1+d_2).

[0023] Before coarse monitoring, multiple frames of images are captured to establish a baseline state library after the equipment is confirmed to be normal. The position, brightness, color, luminous area, displayed content, and normal flickering rules for each indicator component are also saved. During the coarse monitoring phase, camera 7 is located at M_r and captures images at 1 to 10 frames per second. The current image is first registered with the baseline image and its brightness is normalized, then compared with the baseline image, historical images, and state templates. An event is generated when a continuous change reaches a preset number of frames. For flickering changes, the frequency and duty cycle are calculated, and those conforming to the normal flickering rules are excluded. When the state of the same indicator component does not change again, the active event table suppresses repeated triggering.

[0024] Upon triggering, the anomaly region localization module performs threshold segmentation on the difference between the current image and the reference image, deleting noisy regions and merging adjacent regions using a connected component area threshold. Then, target detection obtains the indicator component detection box, verifies its position using a panel template, and corrects the candidate region boundaries. The center of the corrected detection box is used as the anomaly region localization point. The system substitutes the localization point coordinates (u,v) into the perspective mapping relationship to obtain the target motion coordinates (x_t,y_t), and calculates the motion increment based on the current pulse count coordinates (x_c,y_c).

[0025] For a vertical lead screw drive mechanism, if the lead of the vertical lead screw is L_y, the number of steps per revolution of the vertical stepper motor is N_y, and the microstepping factor is q_y, then the number of signed pulses in the vertical direction is:

[0026] For a horizontal synchronous belt drive mechanism, a horizontal stepper motor directly drives the driving synchronous pulley. If the number of steps per revolution of the horizontal stepper motor is N_x, the microstepping factor is q_x, the synchronous belt tooth pitch is p_b, and the number of teeth on the driving synchronous pulley is Z_b, then the horizontal belt pulse quantity is:

[0027] The motion control module determines the direction of motor rotation based on the signs of P_x and P_y, determines the actual number of output pulses based on abs(P_x) and abs(P_y), and outputs pulses using a trapezoidal acceleration and deceleration curve to drive the two-dimensional motion mechanism to move the variable-focus camera 7 to the target detection position.

[0028] Once the camera reaches the target detection position, the system first determines the framing area containing the abnormal indicator and its markings based on the relative positions of the indicator components and their numbers and text labels in the panel template. The system controls the zoom camera to switch to an appropriate magnification, ensuring that the indicator components and markings are clearly visible in the image and fully within the field of view. After zooming, the system automatically focuses within a preset focal length range, uses Laplacian variance to evaluate image sharpness, and selects the sharpest position to acquire a local high-resolution image. If the target deviates from the center of the field of view, the system needs to drive a two-dimensional motion mechanism to fine-tune its position and re-acquire the image based on pixel deviation, ensuring that the target indicator is in the center of the fine monitoring image or reaches the maximum number of corrections.

[0029] After acquiring localized high-definition images, the image processing module first locates the indicator area using a target detection algorithm, outputting the detection box, category, and confidence level. Then, it performs indicator status recognition and OCR text recognition: status recognition obtains the indicator's on / off state, color, and flashing status through color analysis and brightness determination; OCR recognition reads the indicator's number and text label. The system searches the status database by device model, installation location, and indicator number, matching normal status rules, abnormal status rules, and status meanings, and outputs the results. If the confidence level in any step falls below a threshold, it is marked as an uncertain state and prompts for manual review.

[0030] When multiple abnormal areas exist within the same coarse monitoring area, the system first sorts them according to the alarm priority in the database. For areas with the same priority, they are processed from closest to furthest from the current camera position. The processing results are written to the active event table, ensuring that events with the same state are not triggered repeatedly. The suppression is lifted once the state is restored. After all abnormal areas have been processed, the camera returns to the coarse monitoring reference motion coordinates M_r, reverts to the preset low-magnification imaging parameters, and continues to acquire images of the coarse monitoring area.

[0031] It should be noted that the above embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention. All equivalent substitutions, simple modifications, or improvements made within the principles of this invention should be included within the scope of protection of this invention.

Claims

1. A coarse-fine collaborative visual automatic detection system and method based on event triggering and abnormal region localization, characterized in that, The system includes a two-dimensional motion mechanism, a variable-focus camera, a motion control module, an image processing module, and a status output module. The variable-focus camera is mounted on the two-dimensional motion mechanism and is used to acquire coarse monitoring images at low magnification in coarse monitoring mode and acquire local high-definition images at high magnification in fine monitoring mode. The image processing module is used to complete change detection and event triggering, abnormal area localization, coordinate mapping, indicator component identification, and state meaning determination; the motion control module is used to drive the two-dimensional motion mechanism; the state output module is used to output the detection results; wherein, the two-dimensional motion mechanism includes a vertical lead screw, a vertical stepper motor and its mounting base, a vertical limit switch, a vertical guide optical axis fixing base, a vertical guide optical axis, a camera mounting base, a horizontal guide optical axis, a horizontal stepper motor and its mounting base, a horizontal limit switch, and a horizontal synchronous belt; the camera mounting base, the horizontal guide optical axis, the horizontal stepper motor and its mounting base, and the horizontal synchronous belt together constitute a horizontal motion component; a variable-focus camera is mounted on the camera mounting base; the horizontal stepper motor drives the camera mounting base to move horizontally along the horizontal guide optical axis through the horizontal synchronous belt; the vertical stepper motor drives the horizontal motion component to move vertically along the vertical guide optical axis through the vertical lead screw; the detection method of this system includes the following steps: S1. Monitoring area division and system calibration: Determine the coarse monitoring area and coarse monitoring reference motion coordinates based on the panel size of the monitored equipment, the low magnification field of view of the variable zoom camera, and the travel of the two-dimensional motion mechanism; Set calibration points when the variable zoom camera is located at the reference motion coordinates and maintains the preset low magnification imaging parameters, and establish the mapping relationship between the coarse monitoring image coordinates and the motion coordinates of the two-dimensional motion mechanism. S2. Coarse monitoring image acquisition and baseline state establishment: Control the variable zoom camera to be in a low magnification and wide field of view state, continuously or periodically acquire coarse monitoring images covering the coarse monitoring area; establish a baseline state library containing baseline images and baseline feature information of indicator components; S3. Change Detection and Event Triggering: The current coarse monitoring image is compared with the baseline state library to detect abnormal state changes of the indicator component; when the abnormal state change meets the preset triggering conditions, a fine monitoring triggering event is generated, which includes the location information of the abnormal area in the coarse monitoring image. S4. Abnormal area location and target motion coordinate generation: Determine the abnormal area location point based on the location information in the fine monitoring trigger event, and use the mapping relationship to convert the image coordinates of the abnormal area location point into the target motion coordinates of the two-dimensional motion mechanism. S5. Camera motion control and detailed monitoring image acquisition: Obtain the current motion coordinates of the zoom camera, generate motion control quantity based on the difference between the target motion coordinates and the current motion coordinates, drive the two-dimensional motion mechanism to move the zoom camera to the target detection position; switch the zoom camera to high magnification mode and perform automatic focusing to acquire local high-definition images containing the abnormal area; S6. Identification of Indicator Components and Determination of Status Meaning: Based on the identification of indicator component numbers, text labels, and indicator component status in the local high-definition image, the indicator component status is matched with preset status rules and the device status database to obtain the device operation meaning corresponding to the indicator component status; the abnormal area location, local high-definition image, indicator component number, indicator component status, and the device operation meaning are output. S7. Coarse monitoring recovery and next event processing: When there are multiple abnormal areas, the next abnormal area is processed in sequence according to the trigger priority; after all abnormal areas are processed, the zoom camera is driven to return to the coarse monitoring reference motion coordinates and restore the coarse monitoring image to the preset low magnification imaging parameters; trigger events are not repeatedly generated for continuous abnormalities whose state has not changed again.

2. The method according to claim 1, characterized in that, In step S1, when the height of the monitored device panel is greater than the effective coarse monitoring height of a single zoom camera at low magnification, the monitored device panel is divided into at least two coarse monitoring areas, and a two-dimensional motion mechanism and a zoom camera are set up for each coarse monitoring area.

3. The method according to claim 1, characterized in that, In step S3, the current coarse monitoring image is compared with the baseline image, historical image and preset state template in the baseline state library to obtain the change area; the change area is analyzed to identify abnormal state changes; For flickering, the flickering event is determined by multiple consecutive frames of images, and then normal state changes are excluded by combining the preset normal flickering rules.

4. The method according to claim 1, characterized in that, In step S4, the abnormal region localization includes: performing threshold segmentation and connected component analysis on the pixel region that triggers the change to obtain the change candidate region; correcting the boundary of the change candidate region according to the target detection box of the indicator component and the preset template, and selecting at least one of the following as the abnormal region localization point: the center point of the bounding box, the geometric center of the indicator component, and the key point output by the detection model.

5. The method according to claim 1, characterized in that, In step S1, the mapping relationship is established through no less than 4 non-collinear calibration points, and the perspective mapping matrix from coarse monitoring image coordinates to motion coordinates is solved based on the calibration points; when the coarse monitoring area is divided into multiple sub-regions, a local perspective mapping is established in each sub-region, and the mapping results of adjacent sub-regions are interpolated and corrected.

6. The method according to claim 1, characterized in that, In step S5, position control uses limit switch homing and stepper motor pulse counting for positioning: when the system starts, the limit switch is triggered to establish the zero point of the motion coordinates; when the target motion is executed, the number, direction and frequency of the stepper motor pulses are generated according to the difference between the target motion coordinates and the current motion coordinates, and the current motion coordinates are updated according to the number of output pulses.

7. The method according to claim 1, characterized in that, In step S5, the fine monitoring magnification is determined based on the size of the boundary box of the abnormal area and the size of the target recognition pixel; the autofocus adopts an image sharpness evaluation function based on Laplacian variance; in step S6, the device status database includes device model, installation location, indicator component number, name, normal status rules, abnormal status rules, and status meaning description.

8. The method according to claim 1, characterized in that, In step S5, after the zoom camera reaches the target detection position, when the center of the target in the local high-definition image deviates from the image center by more than a preset pixel threshold, a secondary fine-tuning amount is generated based on the local pixels and motion ratio coefficient under fine monitoring magnification, driving the two-dimensional motion mechanism to make a small-range correction so that the target indicator component is located in the center area of ​​the fine monitoring image.

Images