Visual tracking image acquisition method and device, electronic equipment and readable storage medium

Abstract

The application discloses a visual tracking image acquisition method and device, electronic equipment and readable storage medium, and relates to the technical field of robot inspection. The mapping relationship between the actual image coordinates and the camera coordinates is accurately established through a camera checkerboard calibration algorithm, and an intrinsic matrix and a distortion coefficient are obtained. A large-range image containing an inspection target is acquired by controlling a shooting holder, and a visual tracking algorithm is used to identify the target contour. The target magnification and the holder fine-tuning parameters are calculated based on the contour coordinate data, the intrinsic matrix and the distortion coefficient. The holder is rotated according to the fine-tuning parameters, the inspection target is accurately positioned to the center of the global image, and then the image magnification operation is performed according to the target magnification, so that the target image acquisition is completed. Through the cooperative cooperation of the calibration data support, the algorithm identification contour and the parameter-driven adjustment, the acquisition effect of the target image is not affected by the repeated positioning accuracy of the robot, and the acquired target image is both centered and clearly magnified, thereby providing a high-quality image basis for subsequent detection.

Description

Technical Field

[0001] This application relates to the field of robot inspection technology, and in particular to a visual tracking image acquisition method, device, electronic device, and readable storage medium. Background Technology

[0002] With the continuous development of robotic inspection technology, the application of operating robots in the field of power inspection is becoming increasingly widespread. Inspection robots use image recognition technology to automatically identify and detect power equipment, improving inspection efficiency and accuracy.

[0003] In related technologies, robots in complex factory areas mostly use pan-tilt cameras for image acquisition during inspections. This requires the robot to precisely reach the same target location each time, capturing images according to the pan-tilt parameters recorded by the robot. Commonly used pan-tilt parameters include position coordinates and magnification. However, the applicant recognizes that robot navigation has a high repetitive positioning accuracy, especially for legged inspection robots, where the repetitive positioning accuracy is on the order of tens of millimeters. This results in the target image not being captured or being incompletely captured, leading to low efficiency and accuracy in inspection and recognition. Summary of the Invention

[0004] In view of this, this application provides a visual tracking image acquisition method, device, electronic device and readable storage medium. The main purpose is to solve the problem that the repetitive positioning accuracy of robot navigation is relatively high, especially the repetitive positioning accuracy of legged inspection robots is on the order of tens of millimeters. This results in the target image not being captured or being captured incompletely, which leads to low efficiency and accuracy of inspection and recognition.

[0005] According to a first aspect of this application, a visual tracking image acquisition method is provided, the method comprising:

[0006] The inspection robot uses a camera pan-tilt unit to collect multiple chessboard calibration images. Based on the chessboard calibration algorithm, it calculates the intrinsic parameter matrix and distortion coefficients of the camera pan-tilt unit.

[0007] The inspection robot reaches the first target position of the inspection task and uses the shooting gimbal to acquire the first captured image. The first captured image is calculated using a visual tracking algorithm, the intrinsic parameter matrix of the shooting gimbal, and the distortion coefficient to obtain the target magnification and gimbal fine-tuning parameters.

[0008] The inspection robot adjusts the shooting gimbal using the gimbal fine-tuning parameters and the target magnification, acquires a second shooting image based on the adjusted shooting gimbal, and performs target detection on the second shooting image using a target detection algorithm to obtain a target image;

[0009] The inspection robot saves the target image, and upon reaching the next target position, readjusts the camera gimbal to perform image acquisition and target detection operations and saves the new target image, until the inspection task is completed.

[0010] Optionally, the checkerboard calibration algorithm is used to calculate the intrinsic parameter matrix and distortion coefficients of the shooting gimbal, including:

[0011] The inspection robot preprocesses the multiple chessboard calibration images to obtain multiple preprocessed images, and performs corner detection on each preprocessed image to obtain multiple corner pixel coordinates;

[0012] The inspection robot obtains the actual coordinates of the corner points corresponding to each chessboard grid calibration image, thus obtaining multiple actual coordinates of the corner points.

[0013] The inspection robot acquires an open-source computer vision library and uses the open-source computer vision library to calculate the pixel coordinates and actual coordinates of the multiple corner points to obtain the intrinsic parameter matrix and distortion coefficients of the shooting gimbal. The distortion parameters include a first radial distortion parameter, a second radial distortion parameter, a third radial distortion parameter, a first tangential distortion parameter, and a second tangential distortion parameter.

[0014] Optionally, before the inspection robot reaches the first target location of the inspection task and uses the pan-tilt-zoom (PTZ) to acquire the first image, the method further includes:

[0015] The inspection robot sequentially reaches each target location of the inspection task;

[0016] For each target location, the inspection robot adjusts the position parameters of the camera pan-tilt unit so that the inspection target is near the center of the captured image.

[0017] The inspection robot uses the position parameters of the camera gimbal as the initial coordinates of the gimbal corresponding to the target position.

[0018] Optionally, the inspection robot reaches the first target location of the inspection task and uses the pan-tilt-zoom (PTZ) to capture a first image, including:

[0019] When the inspection robot reaches the first target position of the inspection task, it obtains the initial coordinates of the gimbal corresponding to the first target position.

[0020] The inspection robot adjusts the camera pan-tilt according to the initial coordinates of the pan-tilt, and based on the adjusted camera pan-tilt, performs image acquisition on the inspection target corresponding to the first target position to obtain the first captured image.

[0021] Optionally, the step of using a visual tracking algorithm, the intrinsic parameter matrix of the shooting gimbal, and distortion coefficients to calculate the target magnification and gimbal fine-tuning parameters on the first captured image includes:

[0022] The inspection robot obtains image adjustment functions, color space conversion functions, and array transpose functions from an open-source computer vision library based on the visual tracking algorithm.

[0023] The inspection robot uses the image adjustment function to calculate the first captured image to obtain the adjusted first captured image, and uses the color space conversion function to calculate the adjusted first captured image to obtain an RGB image;

[0024] The inspection robot uses the array transpose function to calculate the RGB image, and performs mean normalization on the calculated RGB image to obtain a standard image;

[0025] The inspection robot uses the target detection algorithm to perform target recognition on the standard image to obtain a target detection image, which includes at least one target recognition box to be inspected.

[0026] When the target detection image includes a bounding box to be inspected, the inspection robot calculates the center coordinates of the bounding box and uses the center coordinates of the bounding box as the center coordinates of the target detection image.

[0027] The inspection robot obtains the set of outline pixel coordinates included in the inspection recognition box based on the visual tracking algorithm, and extracts the minimum value of the horizontal coordinate, the maximum value of the horizontal coordinate, the minimum value of the vertical coordinate, and the maximum value of the vertical coordinate from the set of outline pixel coordinates.

[0028] The inspection robot takes the difference between the maximum and minimum values ​​of the horizontal coordinate as the difference between the maximum and minimum values ​​of the horizontal coordinate, takes the difference between the maximum and minimum values ​​of the vertical coordinate as the difference between the maximum and minimum values ​​of the vertical coordinate, and takes the product of the difference between the maximum and minimum values ​​of the horizontal coordinate and the difference between the maximum and minimum values ​​of the vertical coordinate as the first area.

[0029] The inspection robot obtains the horizontal and vertical lengths of the target detection image based on the visual tracking algorithm, uses the product of the horizontal and vertical lengths as the second area, and uses the ratio of the second area to the first area as the theoretical magnification factor.

[0030] The inspection robot determines the target magnification using the theoretical magnification.

[0031] z1 = max(z2 - 0.5, 1),

[0032] Where z1 represents the target magnification and z2 represents the theoretical magnification;

[0033] The inspection robot uses the intrinsic parameter matrix and distortion coefficient of the shooting gimbal to calculate the center coordinates of the target detection image, and obtains the gimbal azimuth angle and the gimbal pitch angle, and uses the gimbal azimuth angle and the gimbal pitch angle as the gimbal fine-tuning parameters.

[0034] Optionally, after performing target recognition on the standard image using the target detection algorithm to obtain a target detection image, the method further includes:

[0035] When the target detection image includes multiple bounding boxes to be inspected, the inspection robot calculates the center coordinates of each bounding box to be inspected, obtains the center coordinates of the multiple bounding boxes to be inspected, calculates the average value of the center coordinates of the multiple bounding boxes to be inspected, obtains the average center coordinates, and uses the average center coordinates as the center coordinates of the target detection image.

[0036] The inspection robot obtains the set of contour pixel coordinates of each of the inspection recognition boxes based on the visual tracking algorithm, and obtains multiple sets of contour pixel coordinates. The minimum value of the horizontal coordinate, the maximum value of the horizontal coordinate, the minimum value of the vertical coordinate, and the maximum value of the vertical coordinate are extracted from the multiple sets of contour pixel coordinates.

[0037] Optionally, the inspection robot uses the intrinsic parameter matrix and distortion coefficients of the imaging gimbal to calculate the center coordinates of the target detection image, obtaining the gimbal azimuth angle and gimbal pitch angle, including:

[0038] The inspection robot uses the distortion parameters to correct the center coordinates of the target detection image, obtaining corrected coordinates in the image coordinate system. Then, it uses the intrinsic parameter matrix to perform a homogeneous coordinate transformation on the corrected coordinates to obtain the projected coordinates in the camera coordinate system.

[0039]

[0040] Where, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c The z-axis coordinate of the projected coordinates is represented by K, the intrinsic parameter matrix is ​​represented by u, the axis coordinate of the correction coordinates is represented by v, and z represents the distance between the inspection target and the camera pan-tilt unit.

[0041] The inspection robot, in the camera coordinate system, determines the angles with the xz plane and the yz plane based on the projected coordinates.

[0042]

[0043] Where, θ xz θ represents the angle with the xz plane. yz Indicates the angle with the yz plane, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c The z-axis coordinate represents the projection coordinates. The angle with the xz plane represents the angle between the projection of the line connecting the projection coordinates and the origin of the camera coordinate system on the xz plane and the z-axis. The angle with the yz plane represents the angle between the projection of the line connecting the projection coordinates and the origin of the camera coordinate system on the yz plane and the z-axis.

[0044] The inspection robot uses the angle with the xz plane to calculate the gimbal azimuth angle and the angle with the yz plane to calculate the gimbal pitch angle.

[0045]

[0046] Where angle[0] represents the azimuth angle of the gimbal, angle[1] represents the pitch angle of the gimbal, and θ xz θ represents the angle between the xz plane and the xz plane. yz This indicates the angle between the plane and the yz plane.

[0047] According to a second aspect of this application, a visual tracking image acquisition device is provided, the device comprising:

[0048] The calibration module is used by the inspection robot to acquire multiple chessboard calibration images using a camera gimbal, and to calculate the intrinsic parameter matrix and distortion coefficients of the camera gimbal based on the chessboard calibration algorithm.

[0049] The tracking module is used to allow the inspection robot to reach the first target position of the inspection task and acquire the first captured image using the shooting gimbal. The first captured image is calculated using a visual tracking algorithm, the intrinsic parameter matrix of the shooting gimbal, and the distortion coefficient to obtain the target magnification and gimbal fine-tuning parameters.

[0050] The adjustment module is used by the inspection robot to adjust the shooting gimbal using the gimbal fine-tuning parameters and the target magnification, acquire a second shooting image based on the adjusted shooting gimbal, and perform target detection on the second shooting image using a target detection algorithm to obtain a target image;

[0051] The storage module is used by the inspection robot to save the target image, and to readjust the camera pan-tilt head when reaching the next target position, perform image acquisition and target detection operations and save the new target image, until the inspection task is completed.

[0052] Optionally, the calibration module is used for the inspection robot to preprocess the multiple checkerboard calibration images to obtain multiple preprocessed images, and to perform corner detection on each preprocessed image to obtain multiple corner pixel coordinates; the inspection robot obtains the actual coordinates of the corner corresponding to each checkerboard calibration image to obtain multiple actual corner coordinates; the inspection robot obtains an open-source computer vision library, and uses the open-source computer vision library to calculate the multiple corner pixel coordinates and the multiple actual corner coordinates to obtain the intrinsic parameter matrix and distortion coefficients of the shooting gimbal, wherein the distortion parameters include a first radial distortion parameter, a second radial distortion parameter, a third radial distortion parameter, a first tangential distortion parameter, and a second tangential distortion parameter.

[0053] Optionally, the device further includes:

[0054] The acquisition module is used for the inspection robot to sequentially reach each target position of the inspection task; for each target position, the inspection robot adjusts the position parameters of the shooting gimbal so that the inspection target is near the center position of the captured image; the inspection robot uses the position parameters of the shooting gimbal as the initial coordinates of the gimbal corresponding to the target position.

[0055] Optionally, the tracking module is used to obtain the initial coordinates of the gimbal corresponding to the first target position when the inspection robot reaches the first target position of the inspection task; the inspection robot adjusts the shooting gimbal according to the initial coordinates of the gimbal, and based on the adjusted shooting gimbal, performs image acquisition on the inspection target corresponding to the first target position to obtain the first captured image.

[0056] Optionally, the tracking module is used for the inspection robot to obtain image adjustment functions, color space conversion functions, and array transpose functions from an open-source computer vision library based on the visual tracking algorithm; the inspection robot uses the image adjustment function to calculate the first captured image to obtain an adjusted first captured image, and uses the color space conversion function to calculate the adjusted first captured image to obtain an RGB image; the inspection robot uses the array transpose function to calculate the RGB image, and performs mean normalization processing on the calculated RGB image to obtain a standard image; the inspection robot uses the target detection algorithm to perform target recognition on the standard image to obtain a target detection image, the target detection image including at least one target recognition box to be inspected; when the target detection image includes a target recognition box to be inspected, the inspection robot calculates the center coordinates of the target recognition box to be inspected, and sets the target recognition box to a specified position. The center coordinates of the bounding box to be inspected are used as the center coordinates of the target detection image. The inspection robot obtains the set of contour pixel coordinates included in the bounding box based on the visual tracking algorithm, and extracts the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates from the set of contour pixel coordinates. The inspection robot uses the difference between the maximum and minimum values ​​of the horizontal coordinates as the difference between the maximum and minimum values ​​of the horizontal coordinates, and the difference between the maximum and minimum values ​​of the vertical coordinates as the difference between the maximum and minimum values ​​of the vertical coordinates. The product of the difference between the maximum and minimum values ​​of the horizontal and vertical coordinates is used as the first area. The inspection robot obtains the horizontal and vertical lengths of the target detection image based on the visual tracking algorithm, uses the product of the horizontal and vertical lengths as the second area, and uses the ratio of the second area to the first area as the theoretical magnification factor. The inspection robot uses the theoretical magnification factor to determine the target magnification factor.

[0057] z1 = max(z2 - 0.5, 1),

[0058] Where z1 represents the target magnification factor and z2 represents the theoretical magnification factor; the inspection robot uses the intrinsic parameter matrix and distortion coefficient of the shooting gimbal to calculate the center coordinates of the target detection image, and obtains the gimbal azimuth angle and the gimbal pitch angle, and uses the gimbal azimuth angle and the gimbal pitch angle as the gimbal fine-tuning parameters.

[0059] Optionally, the tracking module is configured to, when the target detection image includes multiple bounding boxes to be inspected, have the inspection robot calculate the center coordinates of each bounding box to be inspected, obtain the center coordinates of the multiple bounding boxes to be inspected, calculate the average value of the center coordinates of the multiple bounding boxes to be inspected, obtain the average center coordinates, and use the average center coordinates as the center coordinates of the target detection image; the inspection robot obtains the set of contour pixel coordinates included in each bounding box to be inspected based on the visual tracking algorithm, obtains multiple sets of contour pixel coordinates, and extracts the minimum value of the horizontal coordinate, the maximum value of the horizontal coordinate, the minimum value of the vertical coordinate, and the maximum value of the vertical coordinate from the multiple sets of contour pixel coordinates.

[0060] Optionally, the tracking module is used by the inspection robot to correct the center coordinates of the target detection image using the distortion parameters to obtain corrected coordinates in the image coordinate system, and then uses the intrinsic parameter matrix to perform homogeneous coordinate transformation on the corrected coordinates to obtain projected coordinates in the camera coordinate system.

[0061]

[0062] Where, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c Let Z represent the z-axis coordinate of the projected coordinates, K represent the intrinsic parameter matrix, u represent the z-axis coordinate of the calibration coordinates, v represent the z-axis coordinate of the calibration coordinates, and z represent the distance between the inspection target and the camera pan-tilt unit. The inspection robot, in the camera coordinate system, determines the angles with the xz plane and the yz plane based on the projected coordinates.

[0063]

[0064] Where, θ xz θ represents the angle with the xz plane. yz Indicates the angle with the yz plane, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c The z-axis coordinate represents the projected coordinates. The angle with the xz plane represents the angle between the projection of the line connecting the projected coordinates to the origin of the camera coordinate system on the xz plane and the z-axis. The angle with the yz plane represents the angle between the projection of the line connecting the projected coordinates to the origin of the camera coordinate system on the yz plane and the z-axis. The inspection robot uses the angle with the xz plane to calculate and determine the gimbal azimuth angle, and uses the angle with the yz plane to calculate and determine the gimbal pitch angle.

[0065]

[0066] Where angle[0] represents the azimuth angle of the gimbal, angle[1] represents the pitch angle of the gimbal, and θ xz θ represents the angle between the xz plane and the xz plane. yz This indicates the angle between the plane and the yz plane.

[0067] According to a third aspect of this application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0068] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0069] By employing the above-described technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0070] This application provides a visual tracking image acquisition method, apparatus, electronic device, and readable storage medium. The inspection robot uses a pan-tilt unit to acquire multiple checkerboard calibration images. Based on the checkerboard calibration algorithm, the intrinsic parameter matrix and distortion coefficients of the pan-tilt unit are calculated. Next, the inspection robot reaches the first target position of the inspection task and acquires a first image using the pan-tilt unit. The visual tracking algorithm, the intrinsic parameter matrix of the pan-tilt unit, and the distortion coefficients are used to calculate the target magnification and pan-tilt unit fine-tuning parameters for the first image. Subsequently, the inspection robot adjusts the pan-tilt unit using the fine-tuning parameters and the target magnification. Based on the adjusted pan-tilt unit, a second image is acquired, and a target detection algorithm is used to detect the target in the second image to obtain the target image. Then, the inspection robot saves the target image and, upon reaching the next target position, readjusts the pan-tilt unit, performs image acquisition and target detection operations, and saves a new target image until the inspection task is completed. A camera checkerboard calibration algorithm is used to accurately establish the mapping relationship between the actual image coordinates and the camera coordinates, obtaining the intrinsic parameter matrix and distortion coefficients. Next, the camera gimbal is controlled to acquire a large-area image containing the inspection target, and a visual tracking algorithm is used to identify the target contour. Based on the contour coordinate data, intrinsic parameter matrix, and distortion coefficients, the target magnification factor and gimbal fine-tuning parameters are calculated. Subsequently, the gimbal is rotated according to the fine-tuning parameters to accurately position the inspection target in the center of the global image, and then the image is magnified according to the target magnification factor to complete the target image acquisition. Through the coordinated efforts of calibration data support, algorithmic contour recognition, and parameter-driven adjustment, the target image acquisition effect is not affected by the robot's repetitive positioning accuracy, while ensuring that the acquired target image is both centered and clearly magnified, providing a high-quality image foundation for subsequent inspection.

[0071] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0072] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0073] Figure 1 This illustration shows a flowchart of a visual tracking image acquisition method provided in an embodiment of this application;

[0074] Figure 2A This invention provides a schematic flowchart of another visual tracking image acquisition method according to an embodiment of the present application.

[0075] Figure 2B This paper illustrates a schematic diagram of the visual tracking algorithm logic provided in an embodiment of this application.

[0076] Figure 3A This illustration shows a structural diagram of a visual tracking image acquisition method provided in an embodiment of this application;

[0077] Figure 3B This illustration shows a schematic diagram of another visual tracking image acquisition structure provided in an embodiment of this application;

[0078] Figure 4 A schematic diagram of the device structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0079] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0081] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present invention can be combined with each other.

[0082] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0083] In complex factory areas, robot inspections often utilize pan-tilt-zoom (PTZ) cameras for image acquisition. This requires the robot to precisely reach the same target location each time, and then acquire images according to the PTZ parameters recorded by the robot. Common PTZ parameters include position coordinates and magnification. Conventional techniques combine precise robot positioning with camera PTZ parameters to acquire target images. However, robot navigation repetitive positioning accuracy can be quite high, especially for legged inspection robots where the repetitive positioning accuracy is on the order of tens of millimeters. This can lead to the target image being missed or incompletely captured.

[0084] To address this issue, this application proposes a visual tracking image acquisition method. This method establishes the correspondence between actual image coordinates and camera coordinates through camera checkerboard calibration. Then, it captures a large-scale image containing the target object for contour recognition. A gimbal is rotated to position the target object in the center of the global image, followed by image magnification. This automatic and precise acquisition of the target image enables the inspection and recognition of the target object. The execution entity of this application is an inspection robot. The inspection robot relies on the computing power of a server to provide services to users. The server can be a standalone server or a server providing basic cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This ensures that the inspection robot can still acquire target images even when its repetitive positioning accuracy is low.

[0085] This application provides a visual tracking image acquisition method, such as... Figure 1 As shown, the method includes:

[0086] 101. The inspection robot uses a camera pan-tilt unit to collect multiple chessboard calibration images. Based on the chessboard calibration algorithm, it calculates the intrinsic parameter matrix and distortion coefficients of the camera pan-tilt unit.

[0087] In this embodiment, the inspection robot calculates the intrinsic parameter matrix and distortion coefficient of the shooting gimbal using a checkerboard calibration algorithm, providing accurate parameters for subsequent image coordinate transformation and distortion correction, ensuring the accuracy of image geometric information, and improving the reliability of subsequent target detection and analysis.

[0088] 102. The inspection robot arrives at the first target position of the inspection task and uses a camera pan-tilt to capture the first image. The first image is calculated using a visual tracking algorithm, the intrinsic parameter matrix of the camera pan-tilt, and the distortion coefficient to obtain the target magnification and the pan-tilt fine-tuning parameters.

[0089] In this embodiment, the inspection robot autonomously completes target location arrival and image acquisition, reducing human intervention and improving inspection efficiency, thus adapting to the needs of unmanned and automated industrial scenarios. Furthermore, the inspection robot combines visual tracking algorithms, intrinsic parameter matrices, and distortion coefficients to process images, comprehensively considering camera imaging characteristics and target visual features to ensure the accuracy of target magnification and gimbal fine-tuning parameters calculations, providing a reliable basis for subsequent operations.

[0090] 103. The inspection robot adjusts the gimbal's fine-tuning parameters and target magnification to capture a second image based on the adjusted gimbal. It then uses a target detection algorithm to detect the target in the second image to obtain the target image.

[0091] In this embodiment, the inspection robot adjusts the gimbal based on pre-calculated gimbal fine-tuning parameters and magnification to ensure that the acquired target image is both clearly magnified and centered, providing high-quality input images for subsequent target detection algorithms and ultimately improving the accuracy of target recognition in inspection tasks. Furthermore,

[0092] The inspection robot incorporates a target detection algorithm to process the second captured image, enabling professional identification of inspection targets in practical application scenarios and directly outputting target images that meet inspection requirements. This reduces subsequent processing costs and demonstrates the professionalism and practicality of visual tracking image acquisition methods in industrial applications.

[0093] 104. The inspection robot saves the target image, and when it reaches the next target position, it readjusts the shooting gimbal, performs image acquisition and target detection operations, and saves the new target image until the inspection task is completed.

[0094] In this embodiment, a closed-loop process of "image acquisition - parameter calculation - gimbal adjustment - image re-acquisition" is constructed, enabling the inspection robot to automatically execute the entire process. After the initial acquisition, the target magnification and gimbal fine-tuning parameters are calculated using a visual tracking algorithm combined with the intrinsic parameter matrix and distortion coefficients. After adjustment, the image is acquired again, reducing the problem of incomplete image acquisition caused by robot positioning errors or initial angle deviations, ensuring the integrity and clarity of the target image, thereby adapting to complex inspection scenarios and improving the execution efficiency of inspection tasks.

[0095] This application provides a visual tracking image acquisition method. Compared with the prior art, this invention uses an inspection robot with a camera gimbal to acquire multiple checkerboard calibration images. Based on the checkerboard calibration algorithm, the intrinsic parameter matrix and distortion coefficients of the camera gimbal are calculated from these images. Next, the inspection robot reaches the first target position of the inspection task and acquires a first image using the camera gimbal. The visual tracking algorithm, the intrinsic parameter matrix of the camera gimbal, and the distortion coefficients are used to calculate the target magnification and gimbal fine-tuning parameters from the first image. Subsequently, the inspection robot adjusts the camera gimbal using the gimbal fine-tuning parameters and the target magnification. Based on the adjusted camera gimbal, a second image is acquired, and a target detection algorithm is used to detect the target image from the second image. Then, the inspection robot saves the target image and, upon reaching the next target position, readjusts the camera gimbal, performs image acquisition and target detection operations, and saves a new target image, until the inspection task is completed. Through the camera checkerboard calibration algorithm, a precise mapping relationship between the actual image coordinates and the camera coordinates is established, and the intrinsic parameter matrix and distortion coefficients are obtained. Next, the camera pan-tilt unit is controlled to acquire a large-area image containing the inspection target, and a visual tracking algorithm is used to identify the target's contour. Based on the contour coordinate data, intrinsic parameter matrix, and distortion coefficient, the target magnification factor and pan-tilt unit fine-tuning parameters are calculated. Subsequently, the pan-tilt unit is rotated according to the fine-tuning parameters to accurately position the inspection target in the center of the global image, and then the image is magnified according to the target magnification factor to complete the target image acquisition. Through the coordinated efforts of calibration data support, algorithmic contour recognition, and parameter-driven adjustment, the target image acquisition effect is not affected by the robot's repetitive positioning accuracy, while ensuring that the acquired target image is both centered and clearly magnified, providing a high-quality image foundation for subsequent inspection.

[0096] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, this application provides another visual tracking image acquisition method, such as... Figure 2A As shown, the method includes:

[0097] 201. The inspection robot uses a camera pan-tilt unit to collect multiple chessboard calibration images. Based on the chessboard calibration algorithm, it calculates the intrinsic parameter matrix and distortion coefficients of the camera pan-tilt unit.

[0098] In this embodiment, the relationship between the actual image coordinate system and the camera coordinate system needs to be established first, i.e., the distortion coefficients and intrinsic parameter matrix of the camera pan-tilt unit need to be calculated. Specifically, a standard checkerboard pattern is fixed at the actual inspection point position, and the camera pan-tilt unit is controlled to take pictures from different directions, i.e., multiple checkerboard pattern calibration images. Considering that there are a total of 9 parameters for the intrinsic parameter matrix and distortion coefficients, at least 9 pictures need to be taken to solve for them. The inspection robot preprocesses the multiple checkerboard pattern calibration images to obtain multiple preprocessed images, including grayscale conversion and noise reduction.

[0099] Then, the inspection robot uses the `cv2.findChessboardCorners` function from the Open Source Computer Vision Library (OpenCV) to detect corners in each preprocessed image, obtaining multiple corner pixel coordinates. Subsequently, the inspection robot obtains the actual coordinates of the corners corresponding to each chessboard calibration image, resulting in multiple actual corner coordinates. It should be noted that the actual corner coordinates are real-world coordinates generated based on the physical dimensions of the chessboard.

[0100] Then, the inspection robot uses the `cv2.calibrateCamera` function from the open-source computer vision library to calculate the pixel coordinates of multiple corner points (2D pixel coordinates of all images) and the actual coordinates of multiple corner points (3D actual coordinates of all images), obtaining the intrinsic parameter matrix K and distortion coefficients of the shooting gimbal, providing clear parameter basis for subsequent image distortion correction. The distortion parameters include the first radial distortion parameter k1, the second radial distortion parameter k2, the third radial distortion parameter k3, the first tangential distortion parameter p1, and the second tangential distortion parameter p2.

[0101] 202. The inspection robot arrives at each target location of the inspection task in sequence and collects the initial coordinates of the gimbal corresponding to each target location.

[0102] In this embodiment, the inspection robot first sequentially reaches each target location of the inspection task. The inspection task refers to an image acquisition task involving multiple inspection targets, comprising multiple target locations. The inspection robot needs to acquire an image containing the corresponding inspection target at each target location. For each target location, the inspection robot adjusts the position parameters of the pan-tilt unit (PTU) to position the inspection target near the center of the captured image. Then, the inspection robot uses the PTU's position parameters as the initial coordinates of the PTU corresponding to the target location. The inspection robot arrives at each target location sequentially, automating the location arrival and parameter acquisition, ensuring the orderly progress of the inspection task, reducing manual intervention, and adapting to the needs of industrial automation scenarios. By collecting and recording the initial PTU coordinates, a benchmark reference is provided for subsequent PTU fine-tuning and parameter optimization, facilitating subsequent strategy adjustments using visual tracking algorithms to improve the PTU positioning accuracy for multi-target inspection.

[0103] 203. The inspection robot arrives at the first target location of the inspection task and uses a camera pan-tilt unit to capture the first image.

[0104] In this embodiment, when the inspection robot reaches the first target location of the inspection task, it obtains the initial coordinates of the gimbal corresponding to the first target location. Then, the inspection robot adjusts the camera gimbal according to the initial coordinates, and based on the adjusted camera gimbal, it acquires an image of the inspection target corresponding to the first target location, thus obtaining the first captured image.

[0105] 204. The inspection robot uses an open-source computer vision library to preprocess the first captured image to obtain a standard image.

[0106] In this embodiment, the inspection robot obtains image adjustment functions (cv2.resize), color space conversion functions (cv2.COLOR_BGR2RGB), and array transpose functions (np.transpose) from an open-source computer vision library based on a visual tracking algorithm. The inspection robot uses the image adjustment functions to expand and scale the first captured image, changing its size to obtain an adjusted first captured image. Then, it uses the color space conversion function to calculate and convert the BGR color image into an RGB image. Next, the inspection robot uses the array transpose function to calculate and change the image matrix structure of the RGB image. Finally, it performs mean normalization on the calculated RGB image to obtain a standard image. By leveraging the stability and efficiency of open-source tools, precise processing of image size, color, and structure is achieved, ensuring that the preprocessed image can seamlessly integrate with subsequent detection and tracking tasks, thus improving the overall system compatibility.

[0107] 205. The inspection robot uses a target detection algorithm to identify targets in a standard image to obtain a target detection image. The target detection image includes at least one target recognition box to be inspected. When the target detection image includes one target recognition box to be inspected, step 206 is executed. When the target detection image includes multiple target recognition boxes to be inspected, step 207 is executed.

[0108] In this embodiment, the inspection robot uses a target detection algorithm to identify targets in a standard image, obtaining a target detection image. The target detection algorithm is YOLOv3, and the target detection image includes at least one bounding box for a target to be inspected, which can be circular or square. When the target detection image includes one bounding box, it indicates that there is only one inspection target in the image, and step 206 is executed. When the target detection image includes multiple bounding boxes, it indicates that there are multiple inspection targets in the image, and step 207 is executed. Different processing is applied to different scenarios with single and multiple inspection targets in the target detection image, covering diverse inspection needs and improving the algorithm's adaptability to complex scenarios.

[0109] 206. When the target detection image includes a bounding box to be inspected, the inspection robot determines the center coordinates of the target detection image, as well as the minimum, maximum, minimum, and maximum values ​​of the horizontal coordinate, based on the bounding box.

[0110] In this embodiment, when the target detection image includes a bounding box to be inspected, the inspection robot calculates the center coordinates of the bounding box and uses these coordinates as the center coordinates of the target detection image. Next, the inspection robot obtains the set of contour pixel coordinates included in the bounding box based on a visual tracking algorithm, and extracts the minimum and maximum values ​​of the horizontal and vertical coordinates from this set.

[0111] 207. When the target detection image includes multiple bounding boxes to be inspected, the inspection robot determines the center coordinates of the target detection image, as well as the minimum, maximum, minimum, and maximum values ​​of the horizontal coordinate, based on the multiple bounding boxes.

[0112] In this embodiment, when the target detection image includes multiple bounding boxes to be inspected, the inspection robot calculates the center coordinates of each bounding box to obtain the center coordinates of the multiple bounding boxes. Then, the inspection robot calculates the average center coordinates of the multiple bounding boxes to obtain the average center coordinates, which are used as the center coordinates of the target detection image. Next, the inspection robot obtains the set of contour pixel coordinates included in each bounding box based on a visual tracking algorithm, obtaining multiple contour pixel coordinate sets. From these multiple contour pixel coordinate sets, the minimum and maximum values ​​of the horizontal coordinate, the minimum and maximum values ​​of the vertical coordinate are extracted.

[0113] 208. The inspection robot acquires the horizontal and vertical lengths of the target detection image and determines the target magnification factor based on the minimum and maximum values ​​of the horizontal and vertical coordinates, the minimum and maximum values ​​of the vertical coordinates, and the horizontal and vertical lengths of the target detection image.

[0114] In this embodiment, the inspection robot uses the difference between the maximum and minimum values ​​of the horizontal coordinate as the difference between the maximum and minimum values ​​of the horizontal coordinate, and the difference between the maximum and minimum values ​​of the vertical coordinate as the difference between the maximum and minimum values ​​of the vertical coordinate. The product of the differences between the maximum and minimum values ​​of the horizontal and vertical coordinates is used as the first area. Next, the inspection robot obtains the horizontal and vertical lengths of the target detection image based on a visual tracking algorithm, uses the product of the horizontal and vertical lengths as the second area, and uses the ratio of the second area to the first area as the theoretical magnification factor. To tolerate errors in the calculation of the theoretical magnification factor, the inspection robot uses the theoretical magnification factor to determine the target magnification factor, as shown in Formula 1 below:

[0115] Formula 1: z1=max(z2-0.5,1),

[0116] Where z1 represents the target magnification and z2 represents the theoretical magnification. By actively accommodating the calculation error of the theoretical magnification, unreasonable magnification caused by data fluctuations is avoided, thereby improving the reliability of the parameters.

[0117] 209. The inspection robot uses the intrinsic parameter matrix and distortion coefficient of the shooting gimbal to calculate the center coordinates of the target detection image, and obtains the gimbal azimuth angle and gimbal pitch angle. The gimbal azimuth angle and gimbal pitch angle are used as gimbal fine-tuning parameters.

[0118] In this embodiment, the inspection robot uses distortion parameters to correct the center coordinates of the target detection image, thereby obtaining the corrected coordinates in the image coordinate system.

[0119] Camera lenses exhibit two types of distortion: radial distortion and tangential distortion. Radial distortion is caused by the manufacturing process of the camera lens shape; the more severe the radial distortion, the more pronounced it becomes at the lens edge. In practice, the first few terms of the Taylor series expansion at r=0 are often used to approximate radial distortion. Tangential distortion is caused by errors in the mounting position of the lens and the CMOS or CCD sensor. Tangential distortion requires two additional distortion parameters to describe it. Therefore, radial and tangential distortions are corrected using the following formula 2.

[0120] Formula 2: x corrected =x(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0121] y corrected =y(1+k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 ),

[0122] Where, x corrected The x-coordinate of the correction coordinate, y corrected The vertical coordinate represents the correction coordinate, x represents the horizontal coordinate of the center coordinate, y represents the vertical coordinate of the center coordinate, k1 represents the first radial distortion parameter, k2 represents the second radial distortion parameter, k3 represents the third radial distortion parameter, p1 represents the first tangential distortion parameter, and p2 represents the second tangential distortion parameter.

[0123] Next, the inspection robot uses an intrinsic parameter matrix to perform homogeneous coordinate transformation on the correction coordinates to obtain the projected coordinates in the camera coordinate system, as shown in Formula 3 below:

[0124] Formula 3:

[0125] Where, x c The x-axis coordinates and y-axis coordinates represent the projected coordinates. c The y-axis coordinate and z-axis coordinate represent the projected coordinates. c The z-axis coordinate represents the projected coordinates, K represents the intrinsic parameter matrix, u represents the axis coordinate of the calibration coordinates, v represents the axis coordinate of the calibration coordinates, and z represents the distance between the inspection target and the camera pan-tilt unit, which is 1m.

[0126] The intrinsic parameter matrix is ​​shown in Formula 4 below:

[0127] Formula 4:

[0128] Where K represents the intrinsic parameter matrix, f represents the focal length of the shooting gimbal, and d x d represents the dimension of each pixel along the x-axis of the image coordinate system. y This represents the dimension of each pixel along the y-axis of the image coordinate system, where (u0, v0) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, and f x f represents the equivalent focal length along the x-axis in the image coordinate system. y This represents the equivalent focal length along the y-axis in the image coordinate system.

[0129] Subsequently, the inspection robot, in the camera coordinate system, determines the angles with the xz plane and the yz plane based on the projected coordinates, as shown in Formula 5 below:

[0130] Formula 5:

[0131]

[0132] Where, θ xz θ represents the angle with the xz plane. yz Indicates the angle with the yz plane, x c The x-axis coordinates and y-axis coordinates represent the projected coordinates. c The y-axis coordinate and z-axis coordinate represent the projected coordinates. c The z-axis coordinate represents the projection coordinates. The angle between the projection coordinates and the xz plane represents the angle between the projection of the line connecting the projection coordinates and the origin of the camera coordinate system onto the xz plane and the z-axis. The angle between the projection coordinates and the yz plane represents the angle between the projection of the line connecting the projection coordinates and the origin of the camera coordinate system onto the yz plane and the z-axis.

[0133] Then, the inspection robot calculates the gimbal azimuth angle using the angle with the xz plane and the gimbal pitch angle using the angle with the yz plane, as shown in Formula 6 below:

[0134] Formula 6:

[0135]

[0136] Where angle[0] represents the azimuth angle of the gimbal, angle[1] represents the pitch angle of the gimbal, and θ xz θ represents the angle with the xz plane. yz This represents the angle with the yz plane.

[0137] By employing a complete process of distortion correction, coordinate transformation, angle calculation, and gimbal parameter determination, the consistency and accuracy of visual tracking are ensured. Secondly, by clearly defining the distance (1m) between the inspection target and the gimbal, the intrinsic parameter matrix, combined with actual physical quantities such as pixel size and equivalent focal length, allows the calculation results (gimbal azimuth and pitch angles) to directly serve robot control, thus applying the theoretical model to industrial inspection scenarios and enhancing its practicality.

[0138] 210. The inspection robot adjusts the gimbal's fine-tuning parameters and target magnification to capture a second image based on the adjusted gimbal. It then uses a target detection algorithm to detect the target in the second image to obtain the target image.

[0139] In this embodiment, the inspection robot adjusts the gimbal using gimbal fine-tuning parameters and target magnification. The gimbal azimuth angle refers to the horizontal rotation angle, typically ranging from 0° to 360°, and the gimbal pitch angle refers to the vertical tilt angle, typically ranging from -90° to 90°. The current gimbal azimuth angle is azi, the current gimbal pitch angle is ele, angle[0] refers to the azimuth angle increment (horizontal adjustment), and angle[1] refers to the pitch angle increment (vertical adjustment). The specific adjustment process is as follows:

[0140] If the adjusted gimbal azi (azi+angle[0]) is greater than 0, then the gimbal azi is updated directly: azi=azi+angle[0], and the gimbal pitch is adjusted at the same time: ele=ele-angle[1];

[0141] If the adjusted gimbal azi (azi+angle[0]) is less than 0, then update the gimbal azi: azi=azi+angle[0]+360, and at the same time adjust the gimbal pitch angle: ele=ele-angle[1].

[0142] By using azimuth angle cyclic correction and pitch angle direct adjustment, the gimbal is ensured to be accurately positioned in the horizontal and vertical directions, so that the gimbal attitude meets the target tracking requirements of industrial inspection. It is suitable for scenarios such as robot inspection and monitoring that require dynamic target tracking.

[0143] Next, the inspection robot acquires a second image based on the adjusted pan-tilt head. By integrating the pan-tilt fine-tuning parameters with the target magnification, the pan-tilt head is optimized from two dimensions: device position control and imaging magnification effect. This makes the second image more accurately represent the inspection target, providing high-quality input data for the target detection algorithm. Finally, the inspection robot uses a target detection algorithm to detect targets in the second image, obtaining the target image. The target detection algorithm can be the YOLOv3 algorithm, which can automatically locate the inspection target in the second image. Through feature extraction and analysis, it accurately distinguishes the target from the background, reducing recognition errors in industrial inspections and ensuring the accuracy of the detection results.

[0144] 211. The inspection robot saves the target image, and when it reaches the next target position, it readjusts the shooting gimbal, performs image acquisition and target detection operations, and saves the new target image until the inspection task is completed.

[0145] In this embodiment, the inspection robot saves the target image to ensure complete archiving of inspection data. Then, it reaches the next target location, readjusts the pan-tilt head, performs image acquisition and target detection operations, and saves a new target image until the inspection task is completed. A unified closed-loop process of "image acquisition - parameter calculation - pan-tilt adjustment - image re-acquisition" is executed for each target location, avoiding deviations in detection results due to operational differences, improving the accuracy of multi-target inspection data, and seamlessly adapting to any number of targets in the inspection task. This demonstrates strong scalability for inspection tasks of different scales and complexities, meeting the diverse needs of industrial scenarios.

[0146] Based on the above process, the logical flow diagram of the visual tracking algorithm proposed in this application embodiment is as follows:

[0147] like Figure 2BAs shown, the process begins with initialization and image acquisition: After the robot arrives at the inspection point, it loads the pre-trained YOLOv3 model, initializes the gimbal parameters, adjusts the camera magnification to 1, and reads the camera image using VideoCapture. Next, image preprocessing is performed: cv2.resize is used to adjust the image size, cv2.COLOR_BGR2RGB is used to convert the BGR format to RGB, np.transpose is used to reconstruct the matrix structure, and mean normalization is performed to improve detection stability. Following this, target detection and segmentation are performed: Based on the YOLOv3 algorithm, the outlines of circular or square targets in the image are identified, the target region is segmented, and the bounding boxes and confidence ranking results are output. Then, coordinate calculation and mapping are performed: The geometric center coordinates of the bounding boxes corresponding to the inspected targets are calculated (if there are multiple inspected targets, the average is taken), and combined with the intrinsic parameter matrix (K matrix), the pixel coordinates are mapped to the 3D camera coordinate system at a distance of 1 meter, deriving the spatial projection angle of the inspected target in the xz and yz planes. Dynamic gimbal adjustment: The gimbal's azimuth (horizontal rotation) and pitch (vertical tilt) angles are calculated based on the projection angle. A closed-loop feedback mechanism (maximum two iterations) adjusts the gimbal's attitude to keep the target centered and saves a clear, magnified image. Based on high-precision calibration, dynamic gimbal control, and a fully automated process, this solution addresses the image acquisition challenge caused by insufficient robot positioning accuracy in complex factory environments. It features high precision, low cost, and easy deployment, making it particularly suitable for equipment inspection scenarios in the power and chemical industries.

[0148] This application provides a visual tracking image acquisition method. Compared with the prior art, this invention uses an inspection robot with a camera gimbal to acquire multiple checkerboard calibration images. Based on the checkerboard calibration algorithm, the intrinsic parameter matrix and distortion coefficients of the camera gimbal are calculated from these images. Next, the inspection robot reaches the first target position of the inspection task and acquires a first image using the camera gimbal. The visual tracking algorithm, the intrinsic parameter matrix of the camera gimbal, and the distortion coefficients are used to calculate the target magnification and gimbal fine-tuning parameters from the first image. Subsequently, the inspection robot adjusts the camera gimbal using the gimbal fine-tuning parameters and the target magnification. Based on the adjusted camera gimbal, a second image is acquired, and a target detection algorithm is used to detect the target image from the second image. Then, the inspection robot saves the target image and, upon reaching the next target position, readjusts the camera gimbal, performs image acquisition and target detection operations, and saves a new target image, until the inspection task is completed. Through the camera checkerboard calibration algorithm, a precise mapping relationship between the actual image coordinates and the camera coordinates is established, and the intrinsic parameter matrix and distortion coefficients are obtained. Next, the camera pan-tilt unit is controlled to acquire a large-area image containing the inspection target, and a visual tracking algorithm is used to identify the target's contour. Based on the contour coordinate data, intrinsic parameter matrix, and distortion coefficient, the target magnification factor and pan-tilt unit fine-tuning parameters are calculated. Subsequently, the pan-tilt unit is rotated according to the fine-tuning parameters to accurately position the inspection target in the center of the global image, and then the image is magnified according to the target magnification factor to complete the target image acquisition. Through the coordinated efforts of calibration data support, algorithmic contour recognition, and parameter-driven adjustment, the target image acquisition effect is not affected by the robot's repetitive positioning accuracy, while ensuring that the acquired target image is both centered and clearly magnified, providing a high-quality image foundation for subsequent inspection.

[0149] Furthermore, as Figure 1 To specifically implement the method, this application provides a visual tracking image acquisition device, such as... Figure 3A As shown, the device includes: a calibration module 301, a tracking module 302, an adjustment module 303, and a storage module 304.

[0150] The calibration module 301 is used by the inspection robot to acquire multiple chessboard calibration images using a camera gimbal, and to calculate the intrinsic parameter matrix and distortion coefficients of the camera gimbal based on the chessboard calibration algorithm.

[0151] The tracking module 302 is used to allow the inspection robot to reach the first target position of the inspection task and acquire a first captured image using the shooting gimbal. The first captured image is calculated using a visual tracking algorithm, the intrinsic parameter matrix of the shooting gimbal, and the distortion coefficient to obtain the target magnification and gimbal fine-tuning parameters.

[0152] The adjustment module 303 is used for the inspection robot to adjust the shooting gimbal using the gimbal fine-tuning parameters and the target magnification, to acquire a second shooting image based on the adjusted shooting gimbal, and to perform target detection on the second shooting image using a target detection algorithm to obtain a target image;

[0153] The storage module 304 is used by the inspection robot to save the target image, and to readjust the shooting gimbal when reaching the next target position, perform image acquisition and target detection operations and save the new target image, until the inspection task is completed.

[0154] In a specific application scenario, the calibration module 301 is used by the inspection robot to preprocess the multiple checkerboard calibration images to obtain multiple preprocessed images, and to perform corner detection on each preprocessed image to obtain multiple corner pixel coordinates; the inspection robot obtains the actual coordinates of the corner corresponding to each checkerboard calibration image to obtain multiple actual corner coordinates; the inspection robot obtains an open-source computer vision library, and uses the open-source computer vision library to calculate the multiple corner pixel coordinates and the multiple actual corner coordinates to obtain the intrinsic parameter matrix and distortion coefficients of the shooting gimbal, wherein the distortion parameters include a first radial distortion parameter, a second radial distortion parameter, a third radial distortion parameter, a first tangential distortion parameter, and a second tangential distortion parameter.

[0155] In specific application scenarios, such as Figure 3B As shown, the device also includes: a data acquisition module 305.

[0156] The acquisition module 305 is used for the inspection robot to sequentially reach each target position of the inspection task; for each target position, the inspection robot adjusts the position parameters of the shooting gimbal so that the inspection target is near the center position of the captured image; the inspection robot uses the position parameters of the shooting gimbal as the initial coordinates of the gimbal corresponding to the target position.

[0157] In a specific application scenario, the tracking module 302 is used to obtain the initial coordinates of the gimbal corresponding to the first target position when the inspection robot reaches the first target position of the inspection task; the inspection robot adjusts the shooting gimbal according to the initial coordinates of the gimbal, and based on the adjusted shooting gimbal, performs image acquisition on the inspection target corresponding to the first target position to obtain the first captured image.

[0158] In a specific application scenario, the tracking module 302 is used by the inspection robot to obtain image adjustment functions, color space conversion functions, and array transpose functions from an open-source computer vision library based on the visual tracking algorithm; the inspection robot uses the image adjustment function to calculate the first captured image to obtain an adjusted first captured image, and uses the color space conversion function to calculate the adjusted first captured image to obtain an RGB image; the inspection robot uses the array transpose function to calculate the RGB image, and performs mean normalization processing on the calculated RGB image to obtain a standard image; the inspection robot uses the target detection algorithm to perform target recognition on the standard image to obtain a target detection image, the target detection image including at least one target recognition box to be inspected; when the target detection image includes a target recognition box to be inspected, the inspection robot calculates the center coordinates of the target recognition box to be inspected. The inspection robot uses the center coordinates of the bounding box to be inspected as the center coordinates of the target detection image. Based on the visual tracking algorithm, the inspection robot obtains a set of contour pixel coordinates included in the bounding box, and extracts the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates from this set. The robot uses the difference between the maximum and minimum horizontal coordinates as the difference between the maximum and minimum horizontal coordinates, and the difference between the maximum and minimum vertical coordinates as the difference between the maximum and minimum vertical coordinates. The product of these two differences is used as the first area. The inspection robot also uses the visual tracking algorithm to obtain the horizontal and vertical lengths of the target detection image, and the product of these lengths is used as the second area. The ratio of the second area to the first area is used as the theoretical magnification factor. The inspection robot uses the theoretical magnification factor to determine the target magnification factor.

[0159] z1 = max(z2 - 0.5, 1),

[0160] Where z1 represents the target magnification factor and z2 represents the theoretical magnification factor; the inspection robot uses the intrinsic parameter matrix and distortion coefficient of the shooting gimbal to calculate the center coordinates of the target detection image, and obtains the gimbal azimuth angle and the gimbal pitch angle, and uses the gimbal azimuth angle and the gimbal pitch angle as the gimbal fine-tuning parameters.

[0161] In a specific application scenario, the tracking module 302 is used to, when the target detection image includes multiple inspection boxes, calculate the center coordinates of each of the inspection boxes to obtain the center coordinates of the multiple inspection boxes, calculate the average value of the center coordinates of the multiple inspection boxes to obtain the average center coordinates, and use the average center coordinates as the center coordinates of the target detection image; the inspection robot obtains the set of contour pixel coordinates included in each inspection box based on the visual tracking algorithm to obtain multiple contour pixel coordinate sets, and extracts the minimum value of the horizontal coordinate, the maximum value of the horizontal coordinate, the minimum value of the vertical coordinate, and the maximum value of the vertical coordinate from the multiple contour pixel coordinate sets.

[0162] In specific application scenarios, the tracking module 302 is used by the inspection robot to correct the center coordinates of the target detection image using the distortion parameters to obtain corrected coordinates in the image coordinate system, and then uses the intrinsic parameter matrix to perform homogeneous coordinate transformation on the corrected coordinates to obtain the projected coordinates in the camera coordinate system.

[0163]

[0164] Where, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c Let Z represent the z-axis coordinate of the projected coordinates, K represent the intrinsic parameter matrix, u represent the z-axis coordinate of the calibration coordinates, v represent the z-axis coordinate of the calibration coordinates, and z represent the distance between the inspection target and the camera pan-tilt unit. The inspection robot, in the camera coordinate system, determines the angles with the xz plane and the yz plane based on the projected coordinates.

[0165]

[0166] Where, θ xz θ represents the angle with the xz plane. yz Indicates the angle with the yz plane, x c The x-axis coordinates and y-axis coordinates of the projected coordinates represent the projection coordinates. c The y-axis coordinate and z-axis coordinate of the projected coordinates represent the projection coordinates. c The z-axis coordinate represents the projected coordinates. The angle with the xz plane represents the angle between the projection of the line connecting the projected coordinates to the origin of the camera coordinate system on the xz plane and the z-axis. The angle with the yz plane represents the angle between the projection of the line connecting the projected coordinates to the origin of the camera coordinate system on the yz plane and the z-axis. The inspection robot uses the angle with the xz plane to calculate and determine the gimbal azimuth angle, and uses the angle with the yz plane to calculate and determine the gimbal pitch angle.

[0167]

[0168] Where angle[0] represents the azimuth angle of the gimbal, angle[1] represents the pitch angle of the gimbal, and θ xz θ represents the angle between the xz plane and the xz plane. yz This indicates the angle between the plane and the yz plane.

[0169] This application provides a visual tracking image acquisition device. Compared with the prior art, this invention uses an inspection robot to acquire multiple checkerboard calibration images with a camera gimbal. Based on the checkerboard calibration algorithm, the intrinsic parameter matrix and distortion coefficients of the camera gimbal are calculated from these images. Next, the inspection robot reaches the first target position of the inspection task and acquires a first image using the camera gimbal. The visual tracking algorithm, the intrinsic parameter matrix of the camera gimbal, and the distortion coefficients are used to calculate the target magnification and gimbal fine-tuning parameters from the first image. Subsequently, the inspection robot adjusts the camera gimbal using the gimbal fine-tuning parameters and the target magnification. Based on the adjusted camera gimbal, a second image is acquired, and a target detection algorithm is used to detect the target image from the second image. The inspection robot then saves the target image and, upon reaching the next target position, readjusts the camera gimbal, performs image acquisition and target detection operations, and saves a new target image, until the inspection task is completed. Through the camera checkerboard calibration algorithm, a precise mapping relationship between the actual image coordinates and the camera coordinates is established, and the intrinsic parameter matrix and distortion coefficients are obtained. Next, the camera pan-tilt unit is controlled to acquire a large-area image containing the inspection target, and a visual tracking algorithm is used to identify the target's contour. Based on the contour coordinate data, intrinsic parameter matrix, and distortion coefficient, the target magnification factor and pan-tilt unit fine-tuning parameters are calculated. Subsequently, the pan-tilt unit is rotated according to the fine-tuning parameters to accurately position the inspection target in the center of the global image, and then the image is magnified according to the target magnification factor to complete the target image acquisition. Through the coordinated efforts of calibration data support, algorithmic contour recognition, and parameter-driven adjustment, the target image acquisition effect is not affected by the robot's repetitive positioning accuracy, while ensuring that the acquired target image is both centered and clearly magnified, providing a high-quality image foundation for subsequent inspection.

[0170] It should be noted that other corresponding descriptions of the functional units involved in the visual tracking image acquisition device provided in this application embodiment can be found by referring to... Figure 1 and Figures 2A to 2B The corresponding description in [the document] will not be repeated here.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0174] In an exemplary embodiment, see Figure 4 Furthermore, an electronic device is provided, comprising a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the visual tracking image acquisition method described in the above embodiments.

[0175] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the visual tracking image acquisition method.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0178] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0179] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0180] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A visual tracking image acquisition method, characterized in that, include: The inspection robot uses a camera pan-tilt unit to collect multiple chessboard calibration images. Based on the chessboard calibration algorithm, it calculates the intrinsic parameter matrix and distortion coefficients of the camera pan-tilt unit. The inspection robot reaches the first target location of the inspection task and uses the pan-tilt-zoom (PTZ) to acquire a first image. It then uses a visual tracking algorithm, the intrinsic parameter matrix of the PTG, and distortion coefficients to calculate the target magnification and PTG fine-tuning parameters. This includes: the inspection robot acquiring image adjustment functions, color space conversion functions, and array transpose functions from an open-source computer vision library based on the visual tracking algorithm; the inspection robot using the image adjustment functions to calculate the first image, obtaining an adjusted first image; and using the color space conversion functions to calculate the adjusted first image, obtaining an RGB image. The inspection robot uses the array transpose function to calculate the RGB image, and performs mean normalization on the calculated RGB image to obtain a standard image. The inspection robot then uses the target detection algorithm to perform target recognition on the standard image to obtain a target detection image, which includes at least one bounding box for a target to be inspected. When the target detection image includes a bounding box, the inspection robot calculates the center coordinates of the bounding box, uses these coordinates as the center coordinates of the target detection image, and obtains the set of contour pixel coordinates included in the bounding box based on the visual tracking algorithm. The minimum and maximum x-coordinates, minimum and maximum y-coordinates are extracted from the set of contour pixel coordinates. When the target detection image includes multiple bounding boxes to be inspected, the inspection robot calculates the center coordinates of each bounding box to obtain the center coordinates of the multiple bounding boxes. The average center coordinates of the multiple bounding boxes are calculated to obtain the average center coordinates. The average center coordinates are used as the center coordinates of the target detection image. Based on the visual tracking algorithm, the set of contour pixel coordinates included in each bounding box to be inspected is obtained, resulting in multiple sets of contour pixel coordinates. The minimum x-coordinate is extracted from the multiple sets of contour pixel coordinates. The inspection robot uses the difference between the maximum and minimum values ​​of the horizontal and vertical axes as the difference between the maximum and minimum values ​​of the horizontal axis, and the difference between the maximum and minimum values ​​of the vertical axis as the difference between the maximum and minimum values ​​of the vertical axis. The product of these two differences is used as the first area. Based on the visual tracking algorithm, the inspection robot obtains the horizontal and vertical lengths of the target detection image, uses the product of these lengths as the second area, and the ratio of the second area to the first area is used as the theoretical magnification factor. The inspection robot then uses this theoretical magnification factor to determine the target magnification factor. in, Indicates the target magnification factor. The theoretical magnification factor is represented here. The inspection robot uses the distortion coefficient to correct the center coordinates of the target detection image to obtain the corrected coordinates in the image coordinate system. The intrinsic parameter matrix is ​​used to perform homogeneous coordinate transformation on the corrected coordinates to obtain the projected coordinates in the camera coordinate system. In the camera coordinate system, the inspection robot determines the angle with the xz plane and the angle with the yz plane according to the projected coordinates. The angle with the xz plane is used to calculate the gimbal azimuth angle, and the angle with the yz plane is used to calculate the gimbal pitch angle. The gimbal azimuth angle and the gimbal pitch angle are used as the gimbal fine-tuning parameters. The inspection task includes multiple target positions. The inspection task refers to the inspection robot acquiring an image containing the corresponding inspection target at each target position. The inspection robot adjusts the shooting gimbal using the gimbal fine-tuning parameters and the target magnification, acquires a second shooting image based on the adjusted shooting gimbal, and performs target detection on the second shooting image using a target detection algorithm to obtain a target image; The inspection robot saves the target image, and upon reaching the next target position, readjusts the camera gimbal to perform image acquisition and target detection operations and saves the new target image, until the inspection task is completed.

2. The method according to claim 1, characterized in that, The checkerboard calibration algorithm is used to calculate the intrinsic parameter matrix and distortion coefficients of the shooting gimbal by calculating the multiple checkerboard calibration images, including: The inspection robot preprocesses the multiple chessboard calibration images to obtain multiple preprocessed images, and performs corner detection on each preprocessed image to obtain multiple corner pixel coordinates; The inspection robot obtains the actual coordinates of the corner points corresponding to each chessboard grid calibration image, thus obtaining multiple actual coordinates of the corner points. The inspection robot acquires an open-source computer vision library and uses the open-source computer vision library to calculate the pixel coordinates and actual coordinates of the multiple corner points to obtain the intrinsic parameter matrix and distortion coefficients of the shooting gimbal. The distortion coefficients include a first radial distortion parameter, a second radial distortion parameter, a third radial distortion parameter, a first tangential distortion parameter, and a second tangential distortion parameter.

3. The method according to claim 1, characterized in that, Before the inspection robot reaches the first target location of the inspection task and acquires the first image using the pan-tilt-zoom (PTZ) camera, the method further includes: The inspection robot sequentially reaches each target location of the inspection task; For each target location, the inspection robot adjusts the position parameters of the camera pan-tilt unit so that the inspection target is near the center of the captured image. The inspection robot uses the position parameters of the camera gimbal as the initial coordinates of the gimbal corresponding to the target position.

4. The method according to claim 1, characterized in that, The inspection robot reaches the first target location of the inspection task and uses the pan-tilt-zoom (PTZ) camera to capture the first image, including: When the inspection robot reaches the first target position of the inspection task, it obtains the initial coordinates of the gimbal corresponding to the first target position. The inspection robot adjusts the camera pan-tilt according to the initial coordinates of the pan-tilt, and based on the adjusted camera pan-tilt, performs image acquisition on the inspection target corresponding to the first target position to obtain the first captured image.

5. The method according to claim 1, characterized in that, The inspection robot uses the intrinsic parameter matrix and distortion coefficients of the imaging gimbal to calculate the center coordinates of the target detection image, obtaining the gimbal azimuth angle and gimbal pitch angle, including: The inspection robot uses the distortion coefficient to correct the center coordinates of the target detection image, obtaining corrected coordinates in the image coordinate system. Then, it uses the intrinsic parameter matrix to perform a homogeneous coordinate transformation on the corrected coordinates to obtain the projected coordinates in the camera coordinate system. in, Representing the projected coordinates Axis coordinates Representing the projected coordinates Axis coordinates Representing the projected coordinates Axis coordinates This represents the intrinsic parameter matrix. The axis coordinates representing the correction coordinates, The axis coordinates representing the correction coordinates, Indicates the distance between the inspection target and the camera pan / tilt unit; The inspection robot, in the camera coordinate system, determines the angles with the xz plane and the yz plane based on the projected coordinates. in, Indicates the angle with the xz plane. Indicates the angle with the yz plane. Representing the projected coordinates Axis coordinates Representing the projected coordinates Axis coordinates Representing the projected coordinates The angle between the projected coordinates and the xz plane represents the angle between the projection of the line connecting the projected coordinates and the origin of the camera coordinate system on the xz plane and the z-axis; the angle between the projected coordinates and the yz plane represents the angle between the projection of the line connecting the projected coordinates and the origin of the camera coordinate system on the yz plane and the z-axis. The inspection robot uses the angle with the xz plane to calculate the gimbal azimuth angle and the angle with the yz plane to calculate the gimbal pitch angle. in, This indicates the azimuth angle of the gimbal. This indicates the pitch angle of the gimbal. This indicates the angle between the xz plane and the xz plane. This indicates the angle between the plane and the yz plane.

6. A visual tracking image acquisition device, characterized in that, include: The calibration module is used by the inspection robot to acquire multiple chessboard calibration images using a camera gimbal, and to calculate the intrinsic parameter matrix and distortion coefficients of the camera gimbal based on the chessboard calibration algorithm. The tracking module is used when the inspection robot reaches the first target position of the inspection task and acquires a first captured image using the pan-tilt-zoom (PTZ) camera. It calculates the target magnification and PTN fine-tuning parameters using a visual tracking algorithm, the intrinsic parameter matrix of the PTN camera, and distortion coefficients. This includes: the inspection robot acquiring image adjustment functions, color space conversion functions, and array transpose functions from an open-source computer vision library based on the visual tracking algorithm; the inspection robot using the image adjustment functions to calculate the first captured image to obtain an adjusted first captured image; and using the color space conversion functions to calculate the adjusted first captured image to obtain... The inspection robot uses the array transpose function to calculate the RGB image, and performs mean normalization on the calculated RGB image to obtain a standard image. The inspection robot then uses the target detection algorithm to perform target recognition on the standard image to obtain a target detection image, which includes at least one bounding box for a target to be inspected. When the target detection image includes a bounding box, the inspection robot calculates the center coordinates of the bounding box and uses these coordinates as the center coordinates of the target detection image. Based on the visual tracking algorithm, it obtains the set of contour pixel coordinates included in the bounding box. The minimum and maximum values ​​of the horizontal and vertical coordinates are extracted from the set of contour pixel coordinates. When the target detection image includes multiple bounding boxes to be inspected, the inspection robot calculates the center coordinates of each bounding box to obtain the center coordinates of the multiple bounding boxes. The average center coordinates of the multiple bounding boxes are calculated to obtain the average center coordinates. The average center coordinates are used as the center coordinates of the target detection image. Based on the visual tracking algorithm, the set of contour pixel coordinates included in each bounding box to be inspected is obtained, resulting in multiple sets of contour pixel coordinates. The horizontal coordinates are extracted from the multiple sets of contour pixel coordinates. The inspection robot uses the difference between the maximum and minimum values ​​of the horizontal and vertical coordinates as the difference between the maximum and minimum values ​​of the horizontal coordinates, and the difference between the maximum and minimum values ​​of the vertical coordinates as the difference between the maximum and minimum values ​​of the vertical coordinates. The product of these two differences is used as the first area. Based on the visual tracking algorithm, the inspection robot obtains the horizontal and vertical lengths of the target detection image, uses the product of these lengths as the second area, and the ratio of the second area to the first area is used as the theoretical magnification factor. The inspection robot then uses this theoretical magnification factor to determine the target magnification factor. in, Indicates the target magnification factor. The theoretical magnification factor is represented here. The inspection robot uses the distortion coefficient to correct the center coordinates of the target detection image to obtain the corrected coordinates in the image coordinate system. The intrinsic parameter matrix is ​​used to perform homogeneous coordinate transformation on the corrected coordinates to obtain the projected coordinates in the camera coordinate system. In the camera coordinate system, the inspection robot determines the angle with the xz plane and the angle with the yz plane according to the projected coordinates. The angle with the xz plane is used to calculate the gimbal azimuth angle, and the angle with the yz plane is used to calculate the gimbal pitch angle. The gimbal azimuth angle and the gimbal pitch angle are used as the gimbal fine-tuning parameters. The inspection task includes multiple target positions. The inspection task refers to the inspection robot acquiring an image containing the corresponding inspection target at each target position. The adjustment module is used by the inspection robot to adjust the shooting gimbal using the gimbal fine-tuning parameters and the target magnification, acquire a second shooting image based on the adjusted shooting gimbal, and perform target detection on the second shooting image using a target detection algorithm to obtain a target image; The storage module is used by the inspection robot to save the target image, and to readjust the camera pan-tilt head when reaching the next target position, perform image acquisition and target detection operations and save the new target image, until the inspection task is completed.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images