A multi-camera cooperative online detection method and system based on visual tracking

Abstract

This application belongs to the field of visual inspection technology and discloses a multi-camera collaborative online inspection method and system based on visual tracking. By introducing tracking cameras to detect the pose of the product under inspection in real time, and combining it with pre-determined reference information, it intelligently determines whether the product under inspection has reached the detection position of each surface to be inspected. When the product reaches the detection position, its current pose is recorded and the corresponding detection camera is triggered to acquire the actual image. Subsequently, the actual image is corrected according to the current pose, reference pose, and reference shooting distance to eliminate the viewing angle difference caused by the change in product pose. Finally, the corrected image is compared with the reference image to obtain the inspection result. This can improve inspection efficiency, reduce equipment costs, and simplify system installation and debugging.

Description

Technical Field

[0001] This application relates to the field of visual inspection technology, and more specifically, to a multi-camera collaborative online inspection method and system based on visual tracking. Background Technology

[0002] Product inspection is a crucial step in ensuring product quality. It is involved throughout the entire product lifecycle, including component inspection, in-process assembly inspection, and final product inspection. Conveyor belts, as the most common transportation tool in industrial manufacturing, move various parts or products at different stages of manufacturing from one workstation to the next. Therefore, performing quality inspection on conveyor belts is the most direct and efficient method.

[0003] With increasingly stringent product quality requirements, vision-based online inspection systems have become widely used in production line scenarios. However, these systems typically have high requirements for the position and orientation of the product under inspection, usually requiring the use of fixtures or positioners to constrain the product's orientation to ensure stable imaging of the product's surface during inspection. This strict constraint on product pose increases the system's complexity and cost.

[0004] On the other hand, with the diversification of product forms, it is often necessary to inspect multiple surfaces of the product, such as the top, left, right, front, back, and even the bottom. To address this, existing technologies typically involve setting up corresponding position sensors (such as infrared sensors), positioning fixtures, and inspection cameras at multiple different inspection stations. Each inspection camera corresponds to a surface to be inspected. Each time the product reaches an inspection station and triggers the position sensor, the positioning fixture positions the product, and then the corresponding inspection camera performs image inspection. This step-by-step inspection method not only requires multiple position sensors and positioning fixtures, resulting in high equipment costs, but also suffers from low inspection efficiency, complex system installation and debugging, and difficulty in quickly adapting to different product models.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this application is to provide a multi-camera collaborative online inspection method and system based on visual tracking, which aims to solve the technical problems of existing online visual inspection systems, such as high requirements for the pose of the inspected product, system complexity, high cost, poor flexibility, low efficiency of multi-surface inspection, and high equipment cost.

[0007] Firstly, this application provides a multi-camera collaborative online inspection method based on visual tracking. The method uses a tracking camera and multiple detection cameras to inspect products on a conveyor belt, with each camera having a fixed shooting pose. The tracking camera is located above the conveyor belt, and each detection camera is used to acquire images of various surfaces of the product to be inspected. The method includes the following steps:

[0008] A1. Obtain predetermined reference information for the product to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of the standard product captured by the inspection camera in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the inspection camera and the corresponding surface to be inspected in the reference pose of the standard product; the lateral trigger position is the lateral position of the trigger point at the top of the standard product in the tracking camera image coordinate system in the reference pose, and this lateral direction is parallel to the conveying direction, with each surface to be inspected corresponding to one of the trigger points; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose of the standard product;

[0009] A2. By tracking the camera to detect the position and pose of the product to be inspected in real time, and combining the trigger point position and the lateral trigger position, it is determined whether the product to be inspected has reached the detection position of each surface to be inspected;

[0010] A3. When the product to be inspected reaches a detection position, record the current position of the product to be inspected, and acquire the actual image of the corresponding surface to be inspected through the corresponding detection camera;

[0011] A4. Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image;

[0012] A5. Compare each of the corrected images with the corresponding reference images to obtain the detection results.

[0013] Preferably, the reference information for each surface to be inspected is predetermined through the following steps:

[0014] B1. Place the standard product to be inspected on the conveyor belt at a reference position corresponding to the current surface to be inspected; at this reference position, the entire top of the standard product is within the field of view of the tracking camera, and the corresponding detection camera is facing the current surface to be inspected on the standard product;

[0015] B2. Using the intersection of the optical axis of the tracking camera and the top of the standard product as the trigger point corresponding to the current surface to be inspected, the tracking camera is used to identify the lateral position of the trigger point in the tracking camera image coordinate system and its position relative to the center point of the top surface of the standard product, so as to obtain the lateral trigger position and the trigger point position corresponding to the current surface to be inspected.

[0016] B3. Use the corresponding detection camera to acquire an image of the current surface to be detected on the standard product, and obtain the reference image corresponding to the current surface to be detected;

[0017] B4. Detect the distance between the corresponding detection camera and the current surface to be detected on the standard product to obtain the reference shooting distance corresponding to the current surface to be detected.

[0018] Preferably, step B2 includes:

[0019] An image of the top of the standard product is captured using a tracking camera and recorded as the first tracking image;

[0020] Based on the first tracking image, identify the position of the top center point of the standard product in the tracking camera image coordinate system, and record it as the first center position;

[0021] The intersection of the optical axis of the tracking camera and the top of the standard product is used as the trigger point corresponding to the current surface to be detected. Based on the first tracking image, the position of the trigger point in the tracking camera image coordinate system is identified and recorded as the first position.

[0022] Extract the x-coordinate value of the first position to obtain the lateral trigger position corresponding to the current surface to be detected;

[0023] Calculate the relative position of the first position with respect to the first center position to obtain the trigger point position corresponding to the current surface to be detected.

[0024] Preferably, step A2 includes:

[0025] A201. Real-time images of the top of the product under inspection are captured by a tracking camera and recorded as real-time tracking images;

[0026] A202. Using an image recognition method, the pose of the product to be inspected is detected based on the real-time tracking image; the pose includes the position and rotation angle of the product to be inspected in the tracking camera image coordinate system.

[0027] A203. Based on the detected pose and the position of the trigger point, calculate the real-time position of each trigger point of the product under inspection, and record it as the real-time position of the trigger point;

[0028] A204. Compare the horizontal coordinate value of the real-time position of each trigger point with the corresponding horizontal trigger position. When the horizontal coordinate value of the real-time position of a trigger point reaches the corresponding horizontal trigger position, it is determined that the product to be inspected has reached the detection position of the corresponding surface to be inspected.

[0029] Preferably, step A3 includes:

[0030] A301. When the product to be inspected reaches a detection position, record the pose of the currently detected product to be inspected as the current pose;

[0031] A302. When the product to be inspected reaches a detection position, a trigger signal is generated and sent to the corresponding detection camera, so that the corresponding detection camera can capture the actual image of the corresponding surface to be inspected on the product to be inspected.

[0032] Preferably, step A3 further includes:

[0033] Obtain ambient illuminance;

[0034] If the ambient illuminance is lower than a preset illuminance threshold, then when the product to be inspected reaches or is about to reach a detection position, the supplementary light source corresponding to the corresponding surface to be inspected is turned on, and the supplementary light source is turned off after the corresponding detection camera completes the acquisition of the actual image.

[0035] Preferably, among the plurality of detection cameras, there is a front camera for acquiring images of the front surface of the product to be inspected and capable of vertical movement, and / or a rear camera for acquiring images of the rear surface of the product to be inspected and capable of vertical movement.

[0036] Step A3 also includes:

[0037] Obtain the height of the product to be inspected;

[0038] Obtain the height of the front camera in the shooting pose; and / or, obtain the height of the rear camera in the shooting pose;

[0039] If the height of the product under inspection is greater than or equal to the height of the rear camera in the shooting pose, then the rear camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the rear surface image of the product under inspection is acquired; and / or, if the height of the product under inspection is greater than or equal to the height of the front camera in the shooting pose, then the front camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the front surface image of the product under inspection is acquired.

[0040] Preferably, step A4 includes:

[0041] A401. Obtain the intrinsic parameters of the detection camera;

[0042] A402. Based on the intrinsic parameters of the detection camera, the current pose, the reference pose, and the reference shooting distance, calculate the transformation matrix from the viewpoint of the reference image to the viewpoint of the corresponding actual image using a coordinate transformation method.

[0043] A403. Calculate the inverse of the transformation matrix, and perform viewpoint transformation on the actual image based on the inverse matrix to obtain the corresponding corrected image.

[0044] Preferably, the reference information further includes preset test item information for each surface to be tested;

[0045] Step A5 includes:

[0046] A501. Based on the preset test item information of each surface to be tested, compare each of the calibration images and the corresponding reference images to obtain the test results representing the pass / fail status of each preset test item;

[0047] A502. By combining the test results of each item, a final test result indicating whether the product under test is qualified is obtained.

[0048] Secondly, this application provides a multi-camera collaborative online inspection system based on visual tracking for inspecting products to be inspected on a conveyor belt. The system includes a host computer, a tracking camera and multiple inspection cameras, and the shooting posture of each camera is fixed. The tracking camera is located above the conveyor belt, and each of the inspection cameras is used to collect images of each surface of the product to be inspected.

[0049] The host computer is used to execute:

[0050] Obtain predetermined reference information for the product to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of a standard product captured by a detection camera in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the detection camera and the corresponding surface to be inspected in the reference pose; the lateral trigger position is the lateral position of the trigger point on the top of the standard product in the tracking camera image coordinate system in the reference pose, and this lateral position is parallel to the transport direction, with each surface to be inspected corresponding to one trigger point; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose;

[0051] The position and pose of the product under inspection are detected in real time by a tracking camera, and the trigger point position and the lateral trigger position are combined to determine whether the product under inspection has reached the detection position of each surface to be inspected.

[0052] When the product to be inspected reaches a detection position, the current position and pose of the product to be inspected are recorded, and the actual image of the corresponding surface to be inspected is captured by the corresponding detection camera.

[0053] Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image;

[0054] The detection results are obtained by comparing each of the corrected images with the corresponding reference images.

[0055] Beneficial Effects: This application provides a multi-camera collaborative online inspection method and system based on vision tracking. By tracking the product pose in real time and performing image correction, this application eliminates the need for complex fixtures or positioners to precisely constrain the product pose, significantly reducing system complexity, manufacturing costs, and maintenance difficulty, while improving system flexibility, enabling it to quickly adapt to multi-variety, small-batch production modes. Furthermore, addressing the need for multi-surface inspection, this application employs multi-camera collaborative operation, achieving precise triggering and image acquisition of multiple surfaces to be inspected during product movement. This avoids the problems of traditional step-by-step inspection methods, which require numerous sensors and positioning devices, have lengthy inspection processes, and are inefficient. Therefore, it improves inspection efficiency, reduces equipment costs, and simplifies system installation and debugging. In summary, this application overcomes the shortcomings of existing technologies through innovative vision tracking and image correction technologies, providing a highly efficient, accurate, flexible, and cost-effective multi-camera collaborative online inspection solution. Attached Figure Description

[0056] Figure 1A flowchart of a multi-camera collaborative online detection method based on visual tracking provided in this application.

[0057] Figure 2 This is a schematic diagram of a multi-camera collaborative online detection system based on visual tracking, provided in this application.

[0058] Labeling Explanation: 1. Host computer; 2. Tracking camera; 3. Detection camera; 90. Conveyor belt; 91. Product to be inspected. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0060] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0061] Please refer to Figure 1 This application discloses a multi-camera collaborative online inspection method based on visual tracking in some embodiments. This method uses a tracking camera and multiple detection cameras to inspect products on a conveyor belt, with each camera having a fixed shooting pose. The tracking camera is located above the conveyor belt, and each detection camera is used to acquire images of various surfaces of the product to be inspected. The method includes the following steps:

[0062] A1. Obtain predetermined reference information for the product to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of the standard product captured by the inspection camera in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the inspection camera and the corresponding surface to be inspected in the reference pose of the standard product; the lateral trigger position is the lateral position of the trigger point at the top of the standard product in the tracking camera image coordinate system in the reference pose, and this lateral direction is parallel to the conveying direction, with each surface to be inspected corresponding to one of the trigger points; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose of the standard product;

[0063] A2. By tracking the camera to detect the position and pose of the product to be inspected in real time, and combining the trigger point position and the lateral trigger position, it is determined whether the product to be inspected has reached the detection position of each surface to be inspected;

[0064] A3. When the product to be inspected reaches a detection position, record the current position of the product to be inspected, and acquire the actual image of the corresponding surface to be inspected through the corresponding detection camera;

[0065] A4. Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image;

[0066] A5. Compare each of the corrected images with the corresponding reference images to obtain the detection results.

[0067] This application introduces visual tracking technology to achieve real-time detection of the pose of products to be inspected on a conveyor belt. Combined with preset reference information, it intelligently determines whether the product has reached the inspection position. Therefore, when the product reaches the inspection position, the inspection camera is precisely triggered to acquire an image, which is then corrected. Finally, the inspection result is obtained by comparing the image with the reference image. This method effectively solves the problems of high requirements for product pose and low efficiency in multi-surface inspection in traditional online visual inspection systems, significantly improving the flexibility, efficiency, and accuracy of the inspection system while reducing system cost and maintenance difficulty.

[0068] Here, "tracking camera" refers to a visual sensor capable of capturing and analyzing images in real time to determine the position and orientation of a target object. In this application, the tracking camera is mainly used to detect the pose of the product to be inspected on the conveyor belt in real time, that is, its position and rotation angle in the image coordinate system.

[0069] "Inspection camera" refers to a vision sensor used to acquire images of specific surfaces of a product to be inspected. Multiple inspection cameras work together, with each camera responsible for acquiring an image of one surface of the product to be inspected. These inspection cameras typically have high resolution and image quality to ensure the accuracy of the inspection results. Multiple inspection cameras are usually configured based on the number and location of the surfaces to be inspected (e.g., ...). Figure 2 The system includes a front camera (a), a rear camera (b), a left camera (c), a right camera (d), and a top camera (e), which are used to capture images of the front, rear, left, right, and top surfaces of the product to be inspected, respectively, for inspection purposes.

[0070] "Reference information" refers to pre-determined key data used to guide the detection process. It includes "reference image", "reference shooting distance", "lateral trigger position", and "trigger point position".

[0071] The "reference image" is an image captured by the detection camera of the standard product in a specific "reference pose," serving as a benchmark for subsequent comparison with actual images. The reference poses for each surface to be inspected may be the same or different. The "reference shooting distance" is the distance between the detection camera and the corresponding surface to be inspected in the reference pose. The "lateral trigger position" is the lateral position of the top "trigger point" of the standard product in the tracking camera's image coordinate system in the reference pose, parallel to the transport direction. The "trigger point position" is the position of the trigger point of the standard product in the tracking camera's image coordinate system relative to the center point of the top surface of the standard product in the reference pose. This reference information collectively provides the necessary data foundation for subsequent pose determination, image acquisition, and image correction.

[0072] "Pose" refers to the position and orientation of an object in three-dimensional space. In this application, pose specifically refers to the position and rotation angle of the product under inspection in the coordinate system of the tracking camera image.

[0073] "Detection position" refers to the location on the conveyor belt where the specific surface of the product to be inspected is in the optimal shooting state. When the product to be inspected reaches the detection position, the corresponding detection camera will be triggered to acquire an image.

[0074] The implementation environment of this application typically includes a conveyor belt, a tracking camera, and multiple inspection cameras. The tracking camera is usually mounted above the conveyor belt, providing a top-down view of the entire conveyor belt and monitoring the movement of the products to be inspected in real time. The multiple inspection cameras are mounted on the side or above the conveyor belt in fixed shooting positions, depending on the different surfaces of the products to be inspected, ensuring clear image capture of each surface. The entire system is controlled and coordinated by a host computer to achieve automated, online product inspection.

[0075] The core of the multi-camera collaborative online inspection method based on visual tracking proposed in this application lies in achieving accurate and efficient inspection of products to be inspected on a conveyor belt through a series of steps.

[0076] In step A1, it is necessary to obtain pre-determined reference information for the product to be inspected. This reference information forms the basis for subsequent inspection processes. Specifically, the reference information includes reference images of each surface to be inspected, reference shooting distances, lateral trigger positions, and trigger point positions. Reference images can be obtained manually by placing a standard product in a preset reference pose and then capturing images using a corresponding inspection camera. For example, for a product requiring inspection of its top surface and four sides, the standard product can be placed in five different reference poses, each corresponding to a surface to be inspected, and images can be captured by the corresponding inspection camera as reference images. The reference shooting distance can be measured using tools such as a laser rangefinder or a structured light sensor, measuring the distance between the inspection camera and the corresponding surface to be inspected when the standard product is in the reference pose. The lateral trigger positions and trigger point positions can be set manually or marked using image processing software. For example, an easily identifiable feature point on the top of the standard product can be selected as the trigger point, and then the lateral position of this trigger point in the tracking camera image coordinate system and its position relative to the center point of the top surface of the standard product can be recorded.

[0077] In step A2, the pose of the product under inspection is detected in real time by a tracking camera. Combined with the trigger point position and the lateral trigger position, it is determined whether the product has reached the detection position on each surface to be inspected. The tracking camera can employ image recognition algorithms based on feature matching and deep learning to analyze the acquired images in real time, thereby obtaining the real-time position and rotation angle of the product under inspection on the conveyor belt. For example, a deep learning model can be pre-trained to recognize the product under inspection and output its center point coordinates and rotation angle. Then, based on the preset trigger point position and lateral trigger position, the real-time position of each trigger point on the product under inspection is calculated. When the real-time lateral coordinate value of a trigger point matches the corresponding lateral trigger position, it can be determined that the product under inspection has reached the detection position on the corresponding surface to be inspected.

[0078] In step A3, each time the product to be inspected reaches a detection position, its current pose is recorded, and the corresponding detection camera acquires an actual image of the surface to be inspected. When the system determines that the product to be inspected has reached a certain detection position, it can immediately record the pose information of the product detected by the tracking camera at this time as the current pose. At the same time, a trigger signal is sent to the corresponding detection camera, enabling it to complete the image acquisition of the corresponding surface to be inspected in a very short time. For example, when the product to be inspected reaches the detection position for its top surface, the system records the current pose and triggers the top surface detection camera to acquire an image of the top surface.

[0079] In step A4, each actual image is corrected based on the current pose, reference pose, and reference shooting distance to obtain a corrected image. Since the actual pose of the product to be inspected on the conveyor belt may deviate from the reference pose, directly comparing the actual image and the reference image may lead to misjudgment. Therefore, it is necessary to correct the actual images. The correction process can utilize camera calibration parameters and the principles of three-dimensional geometric transformation. For example, the intrinsic parameters of the detection camera can be obtained first. Then, based on the intrinsic parameters of the detection camera, the current pose, the reference pose, and the reference shooting distance, a transformation matrix from the viewpoint of the reference image to the viewpoint of the corresponding actual image can be calculated using a coordinate transformation method. Next, the inverse matrix of this transformation matrix is ​​calculated, and the viewpoint of the actual image is transformed based on the inverse matrix to obtain a corrected image with a viewpoint consistent with the reference image.

[0080] In step A5, each calibrated image is compared with its corresponding reference image to obtain the detection result. After obtaining the calibrated image, various image processing and analysis algorithms can be used for comparison. For example, methods based on pixel differences, feature matching, texture analysis, or deep learning can be used to compare the differences between the calibrated image and the reference image. If the difference exceeds a preset threshold, the product under inspection is determined to have a defect; otherwise, it is determined to be qualified. For example, the mean square error between the calibrated image and the reference image can be calculated; if the error is greater than a certain threshold, it is considered to have a defect.

[0081] The multi-camera collaborative online inspection method based on visual tracking proposed in this application achieves automated and high-precision inspection of products to be inspected on a conveyor belt through the synergistic effect of the above steps.

[0082] Specifically, this application first provides a benchmark for the subsequent inspection process using pre-defined reference information. This reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions, detailing various parameters of the standard product under ideal inspection conditions. Traditional methods often require physical fixtures or complex robotic arms to force the product to reach this ideal state, while this application uses software definition to transform this ideal state into quantifiable data. Next, a tracking camera monitors the pose of the product under inspection in real time. This step is one of the core innovations of this application. Traditional methods typically rely on simple photoelectric sensors or limit switches to determine whether the product has reached the inspection position. This method cannot detect changes in the product's posture, nor can it cope with random displacements and rotations that may occur on the conveyor belt. This application uses a tracking camera to acquire the precise pose of the product under inspection in real time, greatly improving the system's adaptability to changes in product pose. Once the tracking camera detects the real-time pose of the product under inspection, combined with the preset trigger point and lateral trigger positions, the system can intelligently determine whether the product under inspection has reached the inspection position of each surface to be inspected. This judgment mechanism avoids missed or false detections caused by inaccurate product pose in traditional methods. Once the product to be inspected arrives at the inspection position, the system immediately records its current pose and triggers the corresponding inspection camera to capture the actual image. This process ensures the timing and accuracy of image acquisition. Subsequently, the acquired actual image is corrected based on the recorded current pose, the preset reference pose, and the reference shooting distance. The image correction step is another key technical contribution of this application. Since the actual pose of the product to be inspected on the conveyor belt may deviate from the reference pose, directly comparing the original image will lead to inaccurate detection results. Through precise image correction, the actual image is converted to the same viewpoint and scale as the reference image, thereby eliminating the influence of pose deviation on the detection results and ensuring the accuracy of the comparison. Finally, the system compares the corrected image with the corresponding reference image to obtain the detection result. This comparison based on the corrected image can accurately identify the differences between the product to be inspected and the standard product, thereby determining whether the product has defects.

[0083] In summary, this application, by introducing visual tracking technology, achieves real-time perception and compensation of product pose, eliminating reliance on physical positioning devices and significantly improving the flexibility of the inspection system. Simultaneously, through multi-camera collaborative operation and image correction technology, it solves the efficiency and accuracy issues of multi-surface inspection. Compared to traditional solutions requiring numerous physical positioning devices and step-by-step inspection, the method in this application not only reduces system cost and complexity but also improves inspection efficiency and adaptability, providing a more advanced and efficient solution for product quality control in modern industrial production.

[0084] Preferably, the reference information for each surface to be inspected can be determined in advance through the following steps:

[0085] B1. Place the standard product to be inspected on the conveyor belt at a reference position corresponding to the current surface to be inspected; at this reference position, the entire top of the standard product is within the field of view of the tracking camera, and the corresponding detection camera is facing the current surface to be inspected on the standard product;

[0086] B2. Using the intersection of the optical axis of the tracking camera and the top of the standard product as the trigger point corresponding to the current surface to be inspected, the tracking camera is used to identify the lateral position of the trigger point in the tracking camera image coordinate system and its position relative to the center point of the top surface of the standard product, so as to obtain the lateral trigger position and the trigger point position corresponding to the current surface to be inspected.

[0087] B3. Use the corresponding detection camera to acquire an image of the current surface to be detected on the standard product, and obtain the reference image corresponding to the current surface to be detected;

[0088] B4. Detect the distance between the corresponding detection camera and the current surface to be detected on the standard product to obtain the reference shooting distance corresponding to the current surface to be detected.

[0089] Specifically, during the pre-determination of reference information, a standard product of known size and shape is first placed at a predetermined reference position on the conveyor belt. This reference position is designed to ensure that key parts of the standard product, especially its top, can be fully captured by the tracking camera during subsequent image acquisition and data recognition, thus providing a foundation for accurate pose tracking. Simultaneously, the corresponding detection camera is initially adjusted so that its optical axis is aligned with the specific surface to be inspected on the standard product, ensuring that the subsequently acquired reference images have good viewing angle and clarity.

[0090] The trigger point is defined as the intersection of the optical axis of the tracking camera and the top of the standard product. This point serves as the reference for the image captured by the trigger camera during subsequent real-time inspection. By processing the image of the top of the standard product using the tracking camera, the precise lateral position of the trigger point in the tracking camera's image coordinate system can be identified. This lateral position will be used to determine whether the product under inspection has reached the detection position. Simultaneously, the position of the trigger point relative to the center point of the top surface of the standard product will also be identified. This relative position information is crucial for subsequent correction of pose deviations in the actual image.

[0091] In practical applications, after the standard product is in the reference position and the trigger point information is recorded, the corresponding detection camera is activated to acquire an image of the surface to be inspected on the standard product. This image serves as the reference image of the surface to be inspected, representing the standard appearance of the surface under the ideal reference pose, and will be used as a benchmark for comparison with the actual image in subsequent online inspections.

[0092] Furthermore, to ensure the accuracy of subsequent image correction, it is also necessary to accurately measure the distance between the corresponding detection camera and the surface to be inspected on the standard product. This distance is the reference shooting distance, which can be obtained through a laser rangefinder, structured light sensor, or other high-precision ranging equipment. This reference shooting distance, together with the reference pose, constitutes the basic data for correcting the actual image.

[0093] The solution in this application systematically predetermines the various reference information required for the inspection of the product under test through the aforementioned steps. Specifically, step B1 provides a standardized environment for subsequent data acquisition, ensuring the uniformity of initial conditions. Step B2, by accurately identifying the lateral and relative positions of the trigger points, provides key geometric parameters for triggering timing and pose correction in online inspection. Step B3 directly acquires a standard reference image of the surface to be inspected, providing a visual benchmark for defect detection. Step B4, by measuring the reference shooting distance, provides necessary depth information for subsequent image correction based on pose and distance changes. Thus, these predetermined reference information collectively construct a complete benchmark dataset, enabling subsequent real-time inspection, image acquisition, and correction to be performed within an accurate and quantifiable framework.

[0094] Through the above technical solution, this application can ensure that the acquired reference information has high accuracy and high reliability. The accurate acquisition of reference images, reference shooting distances, lateral trigger positions, and trigger point positions provides a solid foundation for subsequent pose tracking, image acquisition triggering, image correction, and final defect judgment in online inspection. This effectively avoids misjudgments or missed detections caused by inaccurate reference information, significantly improving the stability of the entire inspection system and the accuracy of the inspection results, thereby improving the efficiency and reliability of product quality control.

[0095] Preferably, step B2 may include:

[0096] An image of the top of the standard product is captured using a tracking camera and recorded as the first tracking image;

[0097] Based on the first tracking image, identify the position of the top center point of the standard product in the tracking camera image coordinate system, and record it as the first center position;

[0098] The intersection of the optical axis of the tracking camera and the top of the standard product is used as the trigger point corresponding to the current surface to be detected. Based on the first tracking image, the position of the trigger point in the tracking camera image coordinate system is identified and recorded as the first position.

[0099] Extract the x-coordinate value of the first position to obtain the lateral trigger position corresponding to the current surface to be detected;

[0100] Calculate the relative position of the first position with respect to the first center position to obtain the trigger point position corresponding to the current surface to be detected.

[0101] The first tracking image refers to an image captured by a tracking camera of the top of a standard product placed on a conveyor belt, with reference information predetermined. This image is used to subsequently identify key feature points on the top of the standard product. Specifically, the center point of the top of the standard product refers to the geometric center point of the top surface of the standard product, and its position in the tracking camera image coordinate system is identified and recorded as the first center position. This center point can serve as a reference point for the product in the image. The trigger point is defined as the intersection of the optical axis of the tracking camera and the top of the standard product. The position of this trigger point in the first tracking image is identified and recorded as the first position. This trigger point is a key reference point used in subsequent online inspection to determine whether the product has reached the inspection position. The lateral trigger position refers to the horizontal coordinate value of the first position, which is parallel to the conveying direction (the tracking camera image coordinate system includes an x-axis and a y-axis, with the x-axis parallel to the conveying direction and the y-axis horizontally parallel to the conveying direction, e.g., ...). Figure 2 As shown in this paper, the x-axis coordinate is referred to as the horizontal coordinate (or abscissa) and the y-axis coordinate as the ordinate (or ordinate). This coordinate is used to determine whether the product has reached a preset detection point by comparing the horizontal coordinates of its real-time position during online movement. The trigger point position refers to the relative position of the first position with respect to the first center position. For example, assuming the coordinates of the first center position are (x0, y0), the coordinates of the first position are (x1, y1), and the trigger point position is (xd, yd), then xd = x1 - x0, and yd = y1 - y0. By calculating the relative position, the influence of slight lateral deviations that may exist on the product on the conveyor belt on the trigger point determination can be eliminated, improving the accuracy of triggering.

[0102] The above technical solution enables the precise and repeatable pre-determination of the lateral trigger position and trigger point position of the product to be inspected. This method utilizes the visual recognition capability of a tracking camera, avoiding errors that may arise from traditional manual measurement, and improving the automation and accuracy of reference information determination. This provides a reliable basis for accurate triggering and pose determination in subsequent online inspections.

[0103] In some implementations, step A2 includes:

[0104] A201. Real-time images of the top of the product under inspection are captured by a tracking camera and recorded as real-time tracking images;

[0105] A202. Using an image recognition method, the pose of the product to be inspected is detected based on the real-time tracking image; the pose includes the position and rotation angle of the product to be inspected in the tracking camera image coordinate system.

[0106] A203. Based on the detected pose and the position of the trigger point, calculate the real-time position of each trigger point of the product under inspection, and record it as the real-time position of the trigger point;

[0107] A204. Compare the horizontal coordinate value of the real-time position of each trigger point with the corresponding horizontal trigger position. When the horizontal coordinate value of the real-time position of a trigger point reaches the corresponding horizontal trigger position, it is determined that the product to be inspected has reached the detection position of the corresponding surface to be inspected.

[0108] In step 201, the tracking camera continuously and periodically captures images of the top of the product to be inspected on the conveyor belt to obtain its real-time status during the conveying process. These images are recorded as real-time tracking images, providing a data basis for subsequent pose detection.

[0109] Further, in step 202, the real-time tracking image is processed using an image recognition method to accurately detect the real-time pose of the product under inspection. The pose can be understood as the spatial position of the product under inspection in the tracking camera image coordinate system (e.g., the coordinates of its top center point) and its rotation angle relative to a preset direction. The image recognition method can employ various techniques such as feature matching, deep learning models, or traditional image processing algorithms to ensure the accuracy and robustness of pose detection.

[0110] Based on this, in step 203, the real-time position of each trigger point on the product under inspection is calculated according to the detected real-time pose of the product under inspection and the pre-determined trigger point position. The trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system under the reference pose. By combining the detected real-time pose with the trigger point position, the precise image coordinates of the trigger point corresponding to each surface under inspection at the current moment can be calculated. These calculated real-time positions are recorded as the real-time trigger point positions. Specifically, the real-time trigger point positions can be calculated according to the following formulas: xt=xc+xd*cos(θ), yt=yc+yd*sin(θ), where xt and yt are the abscissa and ordinate of the real-time trigger point position, xc and yc are the abscissa and ordinate of the position in the detected pose of the product under inspection (e.g., the position of its top center point), xd and yd are the abscissa and ordinate of the trigger point position, and θ is the rotation angle in the detected pose of the product under inspection.

[0111] Finally, in step 204, the horizontal coordinate value of each trigger point's real-time position is compared with the corresponding lateral trigger position to determine whether the product under inspection has reached the corresponding detection position. The lateral trigger position is the lateral position of the trigger point on the top of the standard product in the tracking camera image coordinate system under the reference pose, and this lateral direction is parallel to the transmission direction. When the horizontal coordinate value of a trigger point's real-time position is equal to or reaches the corresponding lateral trigger position, or meets a preset trigger condition (e.g., the deviation between the two is within a preset range), it can be determined that the product under inspection has reached the detection position of the surface to be inspected, at which point subsequent image acquisition operations can be triggered.

[0112] The solution presented in this application achieves precise determination of the inspection position of products on a conveyor belt through the detailed steps described above. Specifically, by continuously acquiring real-time tracking images through a tracking camera, the motion state of the product under inspection can be dynamically captured. Analyzing these images using image recognition technology accurately obtains the real-time pose of the product under inspection, including its position and rotation angle in the image coordinate system. Based on this, combined with pre-calibrated trigger point positions, the real-time image coordinates of the trigger points corresponding to each surface to be inspected on the product can be calculated. Finally, by comparing the horizontal coordinates of these real-time trigger points with preset horizontal trigger positions, it is possible to accurately determine when the product under inspection has reached the optimal shooting position for each surface to be inspected. This mechanism based on real-time pose tracking and trigger point matching ensures the automation and high precision of the inspection process.

[0113] The above technical solution enables accurate and real-time determination of the detection position of products on the conveyor belt. This solution avoids the false or missed triggering problems that may occur due to changes in product pose in traditional fixed triggering methods, thus improving the accuracy and reliability of detection. Furthermore, by defining the steps of pose detection, real-time calculation of trigger point position, and comparison of lateral trigger positions in detail, the entire detection position determination process becomes clearer and more controllable, providing a solid foundation for subsequent image acquisition and defect detection, thereby improving the efficiency and performance of the entire multi-camera collaborative online detection method.

[0114] In some implementations, step A3 includes:

[0115] A301. When the product to be inspected reaches a detection position, record the pose of the currently detected product to be inspected as the current pose;

[0116] A302. When the product to be inspected reaches a detection position, a trigger signal is generated and sent to the corresponding detection camera, so that the corresponding detection camera can capture the actual image of the corresponding surface to be inspected on the product to be inspected.

[0117] Specifically, step A301 refers to the system immediately recording the precise pose of the product under inspection at a specific detection position on the surface once the tracking camera detects it in real time. This pose information is designated as the current pose for subsequent image correction. The current pose can be understood as the position and rotation angle of the product under inspection in the tracking camera's image coordinate system; the timeliness of its recording is crucial to ensuring the accuracy of subsequent image correction.

[0118] Further, step A302 refers to the system immediately generating a trigger signal at the same moment the product to be inspected arrives at the inspection position. This trigger signal is sent to the inspection camera corresponding to the current surface to be inspected, instructing the inspection camera to acquire an actual image of the corresponding surface on the product to be inspected at a precise moment. In this way, it can be ensured that the acquisition of the actual image is highly synchronized with the instant the product to be inspected arrives at the inspection position, thereby capturing an image of the product to be inspected in the optimal shooting pose.

[0119] The above technical solution achieves precise synchronization between the recording of the product's pose and the actual image acquisition. This ensures that the product's pose information is accurately captured when it is in the optimal inspection position, and the corresponding image of the inspected surface is acquired in a timely manner. This precise synchronization significantly improves the accuracy of subsequent image correction and reduces inspection errors caused by improper pose or image acquisition timing, thereby enhancing the overall efficiency and reliability of online inspection.

[0120] In industrial production environments, lighting conditions can vary due to factors such as time, weather, or production line layout, leading to insufficient ambient light at certain times. If this issue is not addressed, the actual images captured by the inspection cameras may exhibit insufficient brightness, low contrast, and increased noise when ambient light is insufficient. This can affect the accuracy and reliability of subsequent image correction and defect detection, potentially leading to missed detections or misjudgments, and ultimately reducing the overall performance and stability of the inspection system.

[0121] Therefore, in some preferred embodiments, step A3 may further include:

[0122] Obtain ambient illuminance;

[0123] If the ambient illuminance is lower than a preset illuminance threshold, then when the product to be inspected reaches or is about to reach a detection position, the supplementary light source corresponding to the corresponding surface to be inspected is turned on, and the supplementary light source is turned off after the corresponding detection camera completes the acquisition of the actual image.

[0124] Specifically, acquiring ambient illuminance refers to measuring the light intensity within the detection area in real time or periodically using a light sensor or other environmental monitoring equipment. This ambient illuminance can be understood as the intensity of ambient light received by the product under inspection at the detection location. A preset illuminance threshold is a light intensity value pre-set based on the actual application scenario and detection requirements, used to determine whether the current ambient light is sufficient to meet the image acquisition quality requirements of the inspection camera. For example, this threshold can be calibrated based on factors such as the sensitivity of the inspection camera, the surface characteristics of the product under inspection, and the required detection accuracy. When the ambient illuminance is lower than the preset illuminance threshold, it indicates insufficient ambient light, requiring supplemental lighting. Activating the supplemental lighting source corresponding to the surface under inspection means that when the product under inspection arrives at or is about to arrive at the detection location, the system controls the supplemental lighting source corresponding to the current surface under inspection to operate. The supplemental lighting source can be an LED light, a ring light, or other types of lighting equipment, the purpose of which is to provide sufficient and uniform illumination to the surface under inspection, ensuring that the inspection camera can acquire high-quality actual images. The supplementary light source is turned off after the corresponding detection camera has completed the acquisition of the actual image. This is to save energy, extend the service life of the supplementary light source, and avoid unnecessary light interference to subsequent detection or operation.

[0125] Through the above technical solution, this application can significantly improve the robustness and detection accuracy of the vision-tracking-based multi-camera collaborative online inspection method in complex or variable lighting environments. Because it can automatically activate the supplementary lighting source when lighting is insufficient, it ensures the quality of actual image acquisition and avoids problems such as image blurring, underexposure, or loss of detail caused by insufficient light, thereby effectively reducing the false detection rate and missed detection rate. Furthermore, the on-demand activation and deactivation mechanism of the supplementary lighting source not only achieves energy saving but also extends the equipment's lifespan and reduces operating costs. This solution enables the entire inspection system to adapt to a wider range of industrial production environments, improving the reliability and efficiency of online inspection.

[0126] In practical applications, when it is necessary to inspect the front and back surfaces of a product under inspection, there is a risk of interference with the product because the front and back cameras are positioned in the direction of the product's transmission. If this problem is not addressed, it could lead to blind spots or equipment damage. To address this, this application proposes an optimized solution: introducing a height-adjustable inspection camera and dynamically adjusting its shooting position according to the height of the product under inspection to ensure the effectiveness of image acquisition and the safety of the equipment.

[0127] At this time, among the multiple detection cameras, there is a front camera for capturing images of the front surface of the product to be inspected and capable of moving up and down, and / or a rear camera for capturing images of the rear surface of the product to be inspected and capable of moving up and down.

[0128] Step A3 also includes:

[0129] Obtain the height of the product to be inspected;

[0130] Obtain the height of the front camera in the shooting pose; and / or, obtain the height of the rear camera in the shooting pose;

[0131] If the height of the product under inspection is greater than or equal to the height of the rear camera in the shooting pose, then the rear camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the rear surface image of the product under inspection is acquired; and / or, if the height of the product under inspection is greater than or equal to the height of the front camera in the shooting pose, then the front camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the front surface image of the product under inspection is acquired.

[0132] Specifically, the aforementioned front and rear cameras are designed with height adjustment capabilities to accommodate products of varying heights, prevent physical collisions between the cameras and the products, and ensure image capture at the optimal shooting distance and angle. The front camera captures images of the front surface of the product under inspection, while the rear camera captures images of the rear surface. These cameras are typically controlled by an electrically operated lifting mechanism that allows for precise height adjustment.

[0133] The height of the product to be inspected can be obtained in various ways. For example, it can be measured in real time using laser rangefinders, ultrasonic sensors, or image processing technology from tracking cameras. Obtaining the height of the front camera and / or the rear camera in their shooting pose refers to acquiring the target height of the pre-calibrated cameras during image acquisition. This height is typically determined based on the position of the surface to be inspected and the optimal shooting distance.

[0134] In practical applications, when the height of the product to be inspected is detected to be greater than or equal to the height of the rear camera in its shooting pose, the rear camera will remain in an elevated state until the tracking camera detects the product to avoid collisions. Once the tracking camera detects the product and determines that it is about to reach the detection position, the rear camera will quickly descend to a preset shooting pose to acquire an image of the rear surface of the product. After image acquisition is complete, the rear camera will immediately return to its elevated state to make room for the next product to be inspected or even a taller product to pass through. Similarly, for the front camera, when the height of the product to be inspected is greater than or equal to the height of the front camera in its shooting pose, the front camera will also follow the same lifting logic to ensure effective acquisition of the front surface image.

[0135] Through the above technical solution, this application can significantly improve the adaptability of the multi-camera collaborative online inspection system to products of different heights. The height-adjustable inspection camera effectively avoids the risk of collision with tall products, thereby protecting the equipment and products and reducing the probability of production line downtime and maintenance. In addition, this solution also enhances the system's flexibility and robustness, enabling it to be widely used in production lines handling products of various sizes and shapes, thus expanding the application scope of the inspection method.

[0136] In some implementations, step A4 includes:

[0137] A401. Obtain the intrinsic parameters of the detection camera;

[0138] A402. Based on the intrinsic parameters of the detection camera, the current pose, the reference pose, and the reference shooting distance, calculate the transformation matrix from the viewpoint of the reference image to the viewpoint of the corresponding actual image using a coordinate transformation method.

[0139] A403. Calculate the inverse of the transformation matrix, and perform viewpoint transformation on the actual image based on the inverse matrix to obtain the corresponding corrected image.

[0140] Specifically, the intrinsic parameters of a detection camera refer to the set of parameters that describe the optical characteristics and imaging geometry of the camera, such as focal length, principal point coordinates, and distortion coefficients. These intrinsic parameters are the basis for accurate image correction and 3D reconstruction, and can be obtained through standard camera calibration methods, such as using a checkerboard or other calibration board for shooting and calculation.

[0141] In step A402, the coordinate transformation method can be understood as using geometric optics principles and linear algebra methods to mathematically model the relationship between the image coordinate system, the camera coordinate system, and the world coordinate system. By combining the intrinsic parameters of the detection camera, the current pose of the product under inspection, the preset reference pose, and the reference shooting distance between the detection camera and the surface under inspection, the transformation matrix that converts the reference image viewpoint into the actual image viewpoint can be accurately calculated. This transformation matrix typically includes rotation and translation components, used to describe the geometric deviation of the product under inspection relative to the reference pose during actual shooting.

[0142] For example, an intrinsic parameter matrix K can be established based on the intrinsic parameters: K=[[fx,0,cx],[0,fy,cy],[0,0,1]], fx=f / sx, fy=f / sy, where f is the focal length, fx is the horizontal pixel focal length, fy is the vertical pixel focal length, sx and sy are the horizontal and vertical pixel sizes respectively (sx and sy are generally equal), and cx and cy are the horizontal and vertical coordinates of the principal point coordinates (i.e., the coordinates of the image center point). Based on the current pose and the reference pose, the extrinsic parameters of the camera relative to the reference image are determined when acquiring the actual image. These extrinsic parameters include the rotation matrix R and the translation matrix t. The specific process is existing technology and will not be detailed here. Then, the transformation matrix H is calculated according to the following formula: H=K(R−t*n T / d)K −1 Where n is the unit normal vector of the surface to be detected in the camera coordinate system of the detection camera when the camera acquires the reference image (it can be calculated from the value of this unit normal vector in the product coordinate system of the standard product, using the coordinate transformation relationship between the product coordinate system and the camera coordinate system of the standard product in the reference pose, and this coordinate transformation relationship can be obtained through existing calibration methods), n T This represents the transpose of n, and d is the reference shooting distance.

[0143] Specifically, in step A403, the inverse of the transformation matrix is ​​used to perform a viewpoint transformation on the actual image. This is because the original transformation matrix describes the transformation from the reference viewpoint to the actual viewpoint, and its inverse transformation needs to be applied to correct the actual image to the reference viewpoint. By applying the inverse matrix to the actual image, image distortion and viewpoint differences caused by changes in the pose of the product under inspection or deviations in shooting distance can be eliminated, thereby ensuring that the corrected image is geometrically consistent with the corresponding reference image. When performing viewpoint transformation on the actual image based on the inverse matrix, existing transformation functions can be used, such as the `warpPerspectiv` function provided by the OpenCV library.

[0144] The solution presented in this application acquires the intrinsic parameters of the detection camera and, combined with the current pose of the product under inspection, the reference pose, and the reference shooting distance, can accurately calculate the viewpoint transformation relationship between the actual image and the reference image. It is precisely this precise geometric modeling that allows for the subsequent calculation of the inverse of the transformation matrix and its application to the actual image, effectively eliminating image distortion and viewpoint deviations caused by minute changes in the position or posture of the product under inspection on the conveyor belt. This ensures that the actual acquired image can be accurately corrected to the same viewpoint and scale as the reference image, providing a highly consistent input for subsequent image comparison and inspection.

[0145] The above technical solution enables precise geometric correction of the actual image, effectively compensating for the impact of potential pose deviations and shooting distance changes on image acquisition caused by the product under inspection on the conveyor belt. This precise viewpoint transformation and correction ensures a high geometric match between the corrected image and the reference image, greatly improving the accuracy and reliability of subsequent image comparison and detection. This reduces the false detection rate and the missed detection rate, thereby enhancing the overall performance and stability of the detection system.

[0146] In some embodiments, the reference information also includes preset test item information for each surface to be tested;

[0147] Step A5 includes:

[0148] A501. Based on the preset test item information of each surface to be tested, compare each of the calibration images and the corresponding reference images to obtain the test results representing the pass / fail status of each preset test item;

[0149] A502. By combining the test results of each item, a final test result indicating whether the product under test is qualified is obtained.

[0150] Specifically, pre-set inspection item information refers to the specific inspection standards or tasks pre-defined for each surface of the product to be inspected before product inspection, used to evaluate product quality. For example, for the top surface of a product, items such as "surface scratch inspection" and "color consistency inspection" can be pre-set; for the sides, items such as "edge integrity inspection" and "dimensional deviation inspection" can be pre-set. This information can include the name of each inspection item, inspection area, inspection algorithm parameters, and pass / fail threshold, etc., with the aim of breaking down the overall quality inspection task into multiple independently assessable sub-tasks.

[0151] In step A501, based on the preset inspection item information for each surface to be inspected, each calibration image and its corresponding reference image are compared. The purpose is to independently evaluate the pass / fail status of the product under inspection for each preset inspection item. For example, if a "surface scratch detection" item is preset, the system will call a dedicated scratch detection algorithm to analyze the calibration image and compare it with the reference image to determine whether scratches exist and whether the scratches exceed the pass / fail standard. Finally, the pass / fail status of the "surface scratch detection" item, i.e., the item inspection result, will be output.

[0152] In practical applications, step A502, which integrates the test results of various items, aims to summarize the overall quality status of the product from multiple detailed testing dimensions. For example, if a product has three testing items, namely Item 1, Item 2, and Item 3, and each item outputs a result of "qualified" or "unqualified", the final test result can be obtained through logical judgment (e.g., if all items are qualified, the product is qualified; otherwise, it is unqualified) or weighted scoring (e.g., each item is assigned a weight; when an item is qualified, its corresponding test result is represented by 1, otherwise by 0; then, the weighted sum of the test results of each item is calculated according to the weight; if the weighted sum is less than a preset scoring threshold, the product is determined to be unqualified; otherwise, it is determined to be qualified). This yields the final test result indicating whether the product under inspection is qualified.

[0153] Through the above technical solution, this application enables refined and multi-dimensional evaluation of product quality. Compared to solutions that only obtain a single test result, this application can clearly identify the specific reasons and types of defects in product non-conformity, such as surface scratches, color deviation, or dimensional discrepancies. This detailed test result helps the production line quickly locate problem areas, optimize production processes, and improve the efficiency and accuracy of product quality control. Furthermore, by comprehensively judging the pass / fail status of various preset test items, it can ensure that the product meets standards in all key quality attributes, thereby significantly improving the comprehensiveness and reliability of the overall testing.

[0154] refer to Figure 2 This application provides a multi-camera collaborative online inspection system based on visual tracking for inspecting products to be inspected on a conveyor belt. The system includes a host computer 1, a tracking camera 2 and multiple inspection cameras 3, and the shooting posture of each camera is fixed. The tracking camera 2 is located above the conveyor belt 90, and each of the inspection cameras 3 is used to collect images of each surface to be inspected of the product 91.

[0155] The host computer 1 is used to execute:

[0156] Obtain predetermined reference information for the product 91 to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of the standard product captured by the inspection camera 3 in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the inspection camera 3 and the corresponding surface to be inspected in the reference pose; the lateral trigger position is the lateral position of the trigger point at the top of the standard product in the tracking camera image coordinate system in the reference pose, and this lateral position is parallel to the conveying direction, with each surface to be inspected corresponding to one of the trigger points; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose (the specific process can be referred to step A1 above).

[0157] The position and pose of the product 91 to be inspected are detected in real time by the tracking camera 2, and the trigger point position and the lateral trigger position are combined to determine whether the product 91 to be inspected has reached the detection position of each surface to be inspected (the specific process can be referred to step A2 above).

[0158] When the product to be inspected 91 reaches a detection position, the current pose of the product to be inspected 91 is recorded, and the actual image of the corresponding surface to be inspected is acquired by the corresponding detection camera 3. For details, please refer to step A3 above.

[0159] Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image (for details, please refer to step A4 above).

[0160] Compare each of the corrected images with the corresponding reference images to obtain the detection results (for details, please refer to step A5 above).

[0161] In some embodiments, the plurality of inspection cameras 3 include a front camera for acquiring images of the front surface of the product 91 to be inspected and capable of vertical movement, and / or a rear camera for acquiring images of the rear surface of the product 91 to be inspected and capable of vertical movement.

[0162] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multi-camera collaborative online detection method based on visual tracking, characterized in that, The method involves inspecting products on a conveyor belt using a tracking camera and multiple detection cameras, with each camera positioned in a fixed orientation. The tracking camera is located above the conveyor belt, and each detection camera captures images of different surfaces of the product to be inspected. The method includes the following steps: A1. Obtain predetermined reference information for the product to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of the standard product captured by the inspection camera in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the inspection camera and the corresponding surface to be inspected in the reference pose of the standard product; the lateral trigger position is the lateral position of the trigger point at the top of the standard product in the tracking camera image coordinate system in the reference pose, the lateral direction being parallel to the transport direction, and each surface to be inspected corresponds to one of the trigger points, wherein the intersection of the optical axis of the tracking camera and the top of the standard product is taken as the trigger point corresponding to the current surface to be inspected; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose of the standard product; A2. By tracking the camera to detect the position and pose of the product to be inspected in real time, and combining the trigger point position and the lateral trigger position, it is determined whether the product to be inspected has reached the detection position of each surface to be inspected; A3. When the product to be inspected reaches a detection position, record the current position of the product to be inspected, and acquire the actual image of the corresponding surface to be inspected through the corresponding detection camera; A4. Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image; A5. Compare each of the corrected images with the corresponding reference images to obtain the detection results; Step A2 includes: A201. Real-time images of the top of the product under inspection are captured by a tracking camera and recorded as real-time tracking images; A202. Using an image recognition method, the pose of the product to be inspected is detected based on the real-time tracking image; the pose includes the position and rotation angle of the product to be inspected in the tracking camera image coordinate system. A203. Based on the detected pose and the position of the trigger point, calculate the real-time position of each trigger point of the product under inspection, and record it as the real-time position of the trigger point; A204. Compare the horizontal coordinate value of the real-time position of each trigger point with the corresponding horizontal trigger position. When the horizontal coordinate value of the real-time position of a trigger point reaches the corresponding horizontal trigger position, it is determined that the product to be inspected has reached the detection position of the corresponding surface to be inspected.

2. The multi-camera collaborative online detection method based on visual tracking according to claim 1, characterized in that, The reference information for each surface to be inspected is determined in advance through the following steps: B1. Place the standard product to be inspected on the conveyor belt at a reference position corresponding to the current surface to be inspected; at this reference position, the entire top of the standard product is within the field of view of the tracking camera, and the corresponding detection camera is facing the current surface to be inspected on the standard product; B2. Using the intersection of the optical axis of the tracking camera and the top of the standard product as the trigger point corresponding to the current surface to be inspected, the tracking camera is used to identify the lateral position of the trigger point in the tracking camera image coordinate system and its position relative to the center point of the top surface of the standard product, so as to obtain the lateral trigger position and the trigger point position corresponding to the current surface to be inspected. B3. Use the corresponding detection camera to acquire an image of the current surface to be detected on the standard product, and obtain the reference image corresponding to the current surface to be detected; B4. Detect the distance between the corresponding detection camera and the current surface to be detected on the standard product to obtain the reference shooting distance corresponding to the current surface to be detected.

3. The multi-camera collaborative online detection method based on visual tracking according to claim 2, characterized in that, Step B2 includes: An image of the top of the standard product is captured using a tracking camera and recorded as the first tracking image; Based on the first tracking image, identify the position of the top center point of the standard product in the tracking camera image coordinate system, and record it as the first center position; The intersection of the optical axis of the tracking camera and the top of the standard product is used as the trigger point corresponding to the current surface to be detected. Based on the first tracking image, the position of the trigger point in the tracking camera image coordinate system is identified and recorded as the first position. Extract the x-coordinate value of the first position to obtain the lateral trigger position corresponding to the current surface to be detected; Calculate the relative position of the first position with respect to the first center position to obtain the trigger point position corresponding to the current surface to be detected.

4. The multi-camera collaborative online detection method based on visual tracking according to claim 1, characterized in that, Step A3 includes: A301. When the product to be inspected reaches a detection position, record the pose of the currently detected product to be inspected as the current pose; A302. When the product to be inspected reaches a detection position, a trigger signal is generated and sent to the corresponding detection camera, so that the corresponding detection camera can capture the actual image of the corresponding surface to be inspected on the product to be inspected.

5. The multi-camera collaborative online detection method based on visual tracking according to claim 4, characterized in that, Step A3 also includes: Obtain ambient illuminance; If the ambient illuminance is lower than a preset illuminance threshold, then when the product to be inspected reaches or is about to reach a detection position, the supplementary light source corresponding to the corresponding surface to be inspected is turned on, and the supplementary light source is turned off after the corresponding detection camera completes the acquisition of the actual image.

6. The multi-camera collaborative online detection method based on visual tracking according to claim 4, characterized in that, Among the plurality of detection cameras, there is a front camera for capturing images of the front surface of the product to be inspected and capable of moving up and down, and / or a rear camera for capturing images of the rear surface of the product to be inspected and capable of moving up and down. Step A3 also includes: Obtain the height of the product to be inspected; Obtain the height of the front camera in the shooting pose; and / or, obtain the height of the rear camera in the shooting pose; If the height of the product under inspection is greater than or equal to the height of the rear camera in the shooting pose, then the rear camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the rear surface image of the product under inspection is acquired; and / or, if the height of the product under inspection is greater than or equal to the height of the front camera in the shooting pose, then the front camera remains in an elevated state until the tracking camera detects the product under inspection, and is lowered to the shooting pose when the tracking camera detects the product under inspection, and is restored to an elevated state after the front surface image of the product under inspection is acquired.

7. The multi-camera collaborative online detection method based on visual tracking according to claim 1, characterized in that, Step A4 includes: A401. Obtain the intrinsic parameters of the detection camera; A402. Based on the intrinsic parameters of the detection camera, the current pose, the reference pose, and the reference shooting distance, calculate the transformation matrix from the viewpoint of the reference image to the viewpoint of the corresponding actual image using a coordinate transformation method. A403. Calculate the inverse of the transformation matrix, and perform viewpoint transformation on the actual image based on the inverse matrix to obtain the corresponding corrected image.

8. The multi-camera collaborative online detection method based on visual tracking according to claim 1, characterized in that, The reference information also includes preset test item information for each surface to be tested; Step A5 includes: A501. Based on the preset test item information of each surface to be tested, compare each of the calibration images and the corresponding reference images to obtain the test results representing the pass / fail status of each preset test item; A502. By combining the test results of each item, a final test result indicating whether the product under test is qualified is obtained.

9. A multi-camera collaborative online inspection system based on vision tracking, used for inspecting products to be inspected on a conveyor belt, characterized in that, The system includes a host computer, a tracking camera, and multiple detection cameras. The shooting positions of each camera are fixed. The tracking camera is located above the conveyor belt, and each detection camera is used to collect images of each surface of the product to be inspected. The host computer is used to execute: Obtain predetermined reference information for the product to be inspected; the reference information includes reference images, reference shooting distances, lateral trigger positions, and trigger point positions for each surface to be inspected; the reference images are images of a standard product captured by a detection camera in a reference pose, and each surface to be inspected corresponds to one of the reference poses; the reference shooting distance is the distance between the detection camera and the corresponding surface to be inspected in the reference pose of the standard product; the lateral trigger position is the lateral position of the trigger point on the top of the standard product in the tracking camera image coordinate system in the reference pose, and this lateral direction is parallel to the transport direction, with each surface to be inspected corresponding to one of the trigger points, wherein the intersection of the optical axis of the tracking camera and the top of the standard product is taken as the trigger point corresponding to the current surface to be inspected; the trigger point position is the position of the trigger point relative to the center point of the top surface of the standard product in the tracking camera image coordinate system in the reference pose of the standard product; The position and pose of the product under inspection are detected in real time by a tracking camera, and the trigger point position and the lateral trigger position are combined to determine whether the product under inspection has reached the detection position of each surface to be inspected. When the product to be inspected reaches a detection position, the current position and posture of the product to be inspected are recorded, and the actual image of the corresponding surface to be inspected is captured by the corresponding detection camera. Based on the current pose, the reference pose, and the reference shooting distance, each of the actual images is corrected to obtain a corrected image; By comparing each of the corrected images with the corresponding reference images, the detection results are obtained; When the host computer detects the pose of the product under inspection in real time through the tracking camera, and combines the trigger point position and the lateral trigger position to determine whether the product under inspection has reached the detection position of each surface to be inspected, it executes the following: A201. Real-time images of the top of the product under inspection are captured by a tracking camera and recorded as real-time tracking images; A202. Using an image recognition method, the pose of the product to be inspected is detected based on the real-time tracking image; the pose includes the position and rotation angle of the product to be inspected in the tracking camera image coordinate system. A203. Based on the detected pose and the position of the trigger point, calculate the real-time position of each trigger point of the product under inspection, and record it as the real-time position of the trigger point; A204. Compare the horizontal coordinate value of the real-time position of each trigger point with the corresponding horizontal trigger position. When the horizontal coordinate value of the real-time position of a trigger point reaches the corresponding horizontal trigger position, it is determined that the product to be inspected has reached the detection position of the corresponding surface to be inspected.

Images