Image processing method, device and computer equipment of inspection robot

Abstract

The application relates to an image processing method and device of an inspection robot, computer equipment, a storage medium and a computer program product. The method comprises the following steps: if the inspection robot reaches a predetermined position, the camera posture on the inspection robot is acquired; the inspection target is detected based on the camera posture on the inspection robot; the focal length of the current camera is updated according to the detection result, and a target focal length is obtained; the camera is controlled to shoot the inspection target at the target focal length, and an actual target image is obtained; the actual target image and an expected target image are input into a pre-trained image affine model, and the relative position relationship between the actual target image and the expected target image is obtained; and the actual target image is subjected to affine transformation according to the relative position relationship, and an inspection target image is obtained. According to the method, the position error and the angle error caused by the fact that the inspection robot does not reach the position or the camera posture is inaccurate can be eliminated, and the accuracy of a visual detection algorithm is improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product for an inspection robot. Background Technology

[0002] With social development, people's electricity demand is increasing, and the number and scale of power plants are also growing. Power plants contain all kinds of instruments and equipment. From the perspective of the safety and efficiency of the operators, the task of regularly inspecting the various instruments and equipment in the power plant needs to be performed by robots.

[0003] Inspection robots typically operate on fixed tracks. They can also be equipped with vision cameras to take pictures of instruments and equipment and identify whether they are operating normally. However, errors in the robot's positioning and control systems cannot guarantee that the robot will be in a fixed position when taking pictures, which in turn cannot guarantee the accuracy of the background visual detection algorithm.

[0004] There is an urgent need for a method to improve the accuracy of visual inspection algorithms for inspection robots. Summary of the Invention

[0005] Therefore, it is necessary to provide an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product for inspection robots that can improve the accuracy of vision algorithms, in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides an image processing method for an inspection robot, the method comprising:

[0007] If the inspection robot reaches the predetermined position, the camera pose on the inspection robot is obtained;

[0008] The inspection target is detected based on the camera posture on the inspection robot.

[0009] Based on the detection results, the current focal length of the camera is updated to obtain the target focal length;

[0010] The camera is controlled to capture images of the inspected target at the target focal length to obtain an actual target image;

[0011] The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0012] The inspection target image is obtained by performing an affine transformation on the actual target image based on the relative positional relationship.

[0013] In one embodiment, updating the current focal length of the camera based on the detection result to obtain the target focal length includes:

[0014] Based on the target category of the detection results, run the image detection algorithm;

[0015] If the image detection algorithm returns a qualified result, determine whether the allowable lower limit focal length is 0;

[0016] If the lower limit of the allowable focal length is 0, then the lower limit of the allowable focal length is set to the current focal length of the camera;

[0017] The target focal length is obtained by adding a preset focal length to the current focal length of the camera.

[0018] In one embodiment, the method further includes:

[0019] If the image detection algorithm returns an unqualified result, determine whether the previous image detection algorithm returned a qualified result.

[0020] If the previous image detection algorithm returned an unqualified result, then the step of increasing the current focal length of the camera by a preset focal length is performed to obtain the target focal length.

[0021] If the previous image detection algorithm returned a qualified result, the upper limit of the allowable focal length is set to the current focal length of the camera.

[0022] In one embodiment, the method further includes: calculating the upper limit focal length and the average of the upper limit focal length to obtain the target focal length.

[0023] In one embodiment, the relative positional relationship between the actual target image and the desired target image includes a positional relationship matrix between the actual target image and the desired target image. The step of inputting the actual target image and the desired target image into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image includes: acquiring inspection target information.

[0024] The desired target image is determined based on the inspection target information;

[0025] The actual target image and the desired target image are input into a pre-trained affine model to obtain the positional relationship matrix between the actual target image and the desired target image.

[0026] In one embodiment, the step of performing an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image includes:

[0027] The actual target image is transformed by performing an affine transformation based on the relative positional relationship to obtain the transformed actual target image; the transformed actual target image is then cropped based on the region of interest of the inspection target of the inspection robot to obtain the inspection target image.

[0028] Secondly, this application provides an image processing device for an inspection robot, the device comprising:

[0029] The acquisition module is used to acquire the camera pose on the inspection robot if the inspection robot reaches the predetermined position.

[0030] The detection module is used to detect the inspection target based on the camera posture on the inspection robot;

[0031] The preprocessing module is used to update the current focal length of the camera based on the detection results to obtain the target focal length;

[0032] The shooting module is used to control the camera to take pictures of the inspection target at the target focal length to obtain an actual target image;

[0033] The processing module is used to input the actual target image and the desired target image into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0034] The output module is used to perform an affine transformation on the actual target image according to the relative positional relationship to obtain the inspection target image.

[0035] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the following method steps:

[0036] If the inspection robot reaches the predetermined position, the camera pose on the inspection robot is obtained;

[0037] The inspection target is detected based on the camera posture on the inspection robot.

[0038] Based on the detection results, the current focal length of the camera is updated to obtain the target focal length;

[0039] The camera is controlled to capture images of the inspected target at the target focal length to obtain an actual target image;

[0040] The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0041] The inspection target image is obtained by performing an affine transformation on the actual target image based on the relative positional relationship.

[0042] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the following method steps:

[0043] If the inspection robot reaches the predetermined position, the camera pose on the inspection robot is obtained;

[0044] The inspection target is detected based on the camera posture on the inspection robot.

[0045] Based on the detection results, the current focal length of the camera is updated to obtain the target focal length;

[0046] The camera is controlled to capture images of the inspected target at the target focal length to obtain an actual target image;

[0047] The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0048] The inspection target image is obtained by performing an affine transformation on the actual target image based on the relative positional relationship.

[0049] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the following method steps:

[0050] If the inspection robot reaches the predetermined position, the camera pose on the inspection robot is obtained;

[0051] The inspection target is detected based on the camera posture on the inspection robot.

[0052] Based on the detection results, the current focal length of the camera is updated to obtain the target focal length;

[0053] The camera is controlled to capture images of the inspected target at the target focal length to obtain an actual target image;

[0054] The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0055] The inspection target image is obtained by performing an affine transformation on the actual target image based on the relative positional relationship.

[0056] The aforementioned image processing method, apparatus, computer equipment, storage medium, and computer program product for the inspection robot detects target objects based on the posture of the camera on the inspection robot, obtains detection results, updates the camera's focal length according to the detection results to obtain a more suitable focal length, making the captured target object as clear as possible, and then controls the camera to capture the target object with the more suitable focal length to obtain an actual target image. An affine transformation is then performed on the actual target image according to the relative position relationship to obtain a transformed target image, thereby aligning the actual target image with the desired target image. This method can eliminate positional and angular errors caused by the inspection robot not arriving or inaccurate camera posture, improving the accuracy of the visual detection algorithm. Attached Figure Description

[0057] Figure 1 This is an application environment diagram of the image processing method for an inspection robot in one embodiment;

[0058] Figure 2 This is a flowchart illustrating the image processing method for an inspection robot in one embodiment;

[0059] Figure 3 This is a schematic diagram illustrating the longitudinal distance error between the inspection robot and the inspection target in one embodiment;

[0060] Figure 4 This is a schematic diagram illustrating the horizontal distance error between the inspection robot and the inspection target in one embodiment;

[0061] Figure 5 This is a schematic diagram illustrating the angular error between the inspection robot and the inspection target in one embodiment;

[0062] Figure 6 This is a flowchart illustrating a method for generating the target focal length in one embodiment;

[0063] Figure 7 This is a flowchart illustrating the method for generating the target focal length in another embodiment;

[0064] Figure 8 This is a flowchart illustrating a method for improving the visual inspection accuracy of a mobile inspection robot in one embodiment.

[0065] Figure 9 This is a structural block diagram of the image processing device for an inspection robot in one embodiment;

[0066] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] During the inspection process, the mobile inspection robot needs to move autonomously within a certain area according to the predefined ledger information and complete the preset tasks. Specifically, it takes pictures of objects and uses certain image algorithms to identify them to obtain the detection and recognition results.

[0069] The ledger contains the robot's required postures, including its position and orientation. The information to be detected at that position includes electrical components, electrical equipment, indicator lights of equipment in the computer room, switch status, information on items left behind, and the status of door handles.

[0070] During the inspection process, the robot may fail to reach the limit position, or the robot's camera may be misaligned, resulting in the robot's processor being unable to recognize the images captured by the camera.

[0071] Existing methods often involve manually adjusting the camera's focus and manually selecting the images captured by the camera, with the selected images then used as input for the image algorithm model. This approach relies entirely on manual work by deployment personnel, which is time-consuming and labor-intensive.

[0072] In view of this, the image processing method for the inspection robot provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, a camera 104 is fixedly positioned on the inspection robot 102. When the inspection robot 102 reaches a predetermined position, it acquires the pose of the camera 104. Based on the pose of the camera 104, it detects the inspection target. According to the detection results, the inspection robot 102 updates the focal length of the current camera 104 to obtain the target focal length. The inspection robot 102 controls the camera 104 to capture an image of the inspection target at the target focal length, obtaining an actual target image. The inspection robot 102 inputs the actual target image and the desired target image into a pre-trained affine model to obtain the relative positional relationship between the actual target image and the desired target image. The inspection robot 102 performs an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image.

[0073] The inspection robot can be an intelligent track inspection robot or other intelligent robots with inspection functions, and the camera can be an industrial camera with zoom function. The inspection robot and the camera can be connected by a telescopic rod.

[0074] In one embodiment, such as Figure 2 As shown, an image processing method for an inspection robot is provided, which can be applied to... Figure 1 Taking the inspection robot in the example, the following steps are included:

[0075] S202, if the inspection robot reaches the predetermined position, the camera posture on the inspection robot is obtained.

[0076] Among them, the inspection robot moves autonomously within a certain area and completes the preset tasks according to the predefined ledger information.

[0077] The logbook includes information such as the types of inspection targets and the posture of the camera on the inspection robot. Specifically, the types of inspection targets include electrical components, electrical equipment, equipment indicator lights and switch statuses of computer room equipment, information on items left behind, and door handle statuses. The posture of the camera on the inspection robot includes the camera's horizontal position, vertical position, angle information, and focal length.

[0078] Specifically, the inspection robot can be a tracked inspection robot or a trackless inspection robot. For a tracked inspection robot, when the inspection robot reaches the track limit position, that is, when the inspection robot reaches the predetermined position, the camera attitude on the inspection robot is obtained. The limit position can be an actual limit structure or a virtual limit position set by sensors such as limit switches and photoelectric switches.

[0079] For trackless inspection robots, once the inspection robot receives position information from the distance sensor or positioning module, that is, when the inspection robot reaches the predetermined position, it acquires the camera posture on the inspection robot.

[0080] Among them, the camera can be an industrial camera with zoom function. Taking the camera lens as a PTZ camera as an example, P - Pan means horizontal, corresponding to the camera's rotation and movement in the direction of water; T - Tilt means tilt, corresponding to the camera's pitch rotation in the vertical direction; Z - Zoom means the lens's zoom function.

[0081] Specifically, the camera's orientation includes the camera's yaw angle (horizontal information of the PTZ camera), the camera's pitch angle (tilt information of the PTZ camera), the camera's focal length and magnification (zoom information of the PTZ camera), and the height of the camera relative to the lifting rod of the inspection robot body.

[0082] Zoom information includes optical zoom and digital zoom. Optical zoom does not change the image size or resolution, and the image description remains unchanged. Digital zoom uses image processing software to enlarge the image area, but this process occurs within the digital camera. It uses interpolation to enlarge a portion of the pixels on the original image sensor, thus changing the image size.

[0083] S204 detects inspection targets based on the camera posture on the inspection robot.

[0084] The inspection targets can be important instruments, equipment, and components in the working environment of the inspection robot, such as electrical components, electrical equipment, equipment indicator lights and switches in the computer room, items belonging to operators, door handles, etc.

[0085] Specifically, the inspection target can be detected using image detection algorithms to obtain the detection results.

[0086] The detection results can include the location and status information of the inspected target, such as the location, type, and whether the electrical components are illuminated, or whether the door handle is in the normally closed position.

[0087] The detection result can also be the state output of the image detection algorithm itself. For example, the state output of the image detection algorithm itself is "No valid inspection target was identified", or the state output of the image detection algorithm itself is "A valid inspection target has been identified".

[0088] S206. Based on the detection results, update the current camera's focal length to obtain the target focal length.

[0089] It should be noted that the image of the inspected target captured by the camera may be blurry due to the inspection robot not reaching the limit position or the inspection robot not being in a proper position with the camera, making further image recognition impossible. In this case, the current focal length of the camera can be updated based on the detection results to obtain the target focal length.

[0090] Specifically, if the detection result is that the image detection algorithm itself outputs "no valid inspection target identified", the detection of the inspection target is repeated by increasing the camera focal length to obtain a new detection result. If the detection result is that the image detection algorithm itself outputs "valid inspection target identified", the detection result is output.

[0091] Among them, the inspection robot failing to reach the limit position or the inspection robot and the camera causing positional errors include longitudinal distance errors between the inspection robot and the inspection target, horizontal distance errors between the inspection robot and the inspection target, and angular errors between the inspection robot and the inspection target.

[0092] Specifically, such as Figure 3 The diagram illustrates the longitudinal distance error between the inspection robot and the inspection target. When the inspection robot and the inspection target have a longitudinal distance error, it means that the actual camera shooting distance d deviates from the expected shooting distance D in the ledger by Δd. If this deviation Δd exceeds the camera's depth of field (DOF), it will cause the photo to be blurry, which in turn will cause the vision algorithm to malfunction.

[0093] Figure 4 The diagram illustrates the longitudinal and horizontal errors that occur between the inspection robot and the inspection target. When this error occurs, it means the target has shifted within the camera's field of view (FOV), causing a discrepancy between the target's position in the actual captured image and the desired image in the ledger. If this discrepancy is significant, it can also prevent the vision algorithm from functioning correctly.

[0094] Figure 5 The diagram shows the angle error between the inspection robot and the inspection target. When the inspection robot and the inspection target have an angle error, it means that the posture of the target in the actual captured image is deflected and deformed.

[0095] In the actual daily inspection process of mobile robots, these positional and orientation errors exist simultaneously. Therefore, this method of inspection according to a predetermined log often fails to clearly and accurately capture the expected targets (instruments, indicator lights, switches, etc.) in the log, resulting in the visual algorithm failing to accurately detect the targets and further leading to inspection errors.

[0096] S208 controls the camera to capture images of the inspection target at the target focal length, thereby obtaining an image of the actual target.

[0097] In this process, the inspection target is photographed based on the current camera posture and target focal length to obtain the actual target image.

[0098] The actual target image can be a RAW image file.

[0099] It should be noted that RAW image files can be processed to obtain inspection target images.

[0100] S210: Input the actual target image and the desired target image into the pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image.

[0101] The desired target image is a standard image predefined in the ledger. For each type of inspection target, the image of the inspection target is taken in advance to obtain the desired target image. The desired target image generally does not have distortion, blur, or other issues that image detection algorithms cannot recognize.

[0102] Among them, the image affine model is the application of convolutional neural networks in the field of image affine transformation. The image affine model can be trained based on images of various inspection targets and the manually labeled results of images of various inspection targets, such as the location, type, and state of the inspection targets.

[0103] In geometry, an affine transformation refers to a linear transformation of a vector space followed by a translation, transforming it into another vector space. An affine transformation is also a transformation between two vector spaces.

[0104] Specifically, the relative positional relationship between the actual target image and the desired target image can be the positional relationship between the two images in their respective vector spaces.

[0105] Specifically, the actual target image IMG-raw and the desired target image IMG-des are input into a pre-trained image affine model TNet, which outputs a first image img1 and a second image img2. The transformation matrix of the first image img1 and the second image img2 is calculated, and the first image img1 is affinely transformed into the second image img2 according to the transformation matrix. During the affine transformation process, the relative positional relationship between the actual target image and the desired target image is obtained.

[0106] S212, Perform an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image.

[0107] Specifically, after obtaining the relative positional relationship between the actual target image and the desired target image, an affine transformation can be performed on the actual target image IMG-raw based on the relative positional relationship to obtain the transformed actual target image IMG-rect. The transformed target image IMG-rect is then cropped to inspect the target image.

[0108] In the image processing method for the aforementioned inspection robot, the target object is detected based on the camera's pose, and the detection result is obtained. The camera's focal length is then updated based on the detection result to obtain a more suitable focal length, ensuring the captured target object is as clear as possible. The camera is then controlled to capture the target object again at the more suitable focal length, obtaining an actual target image. An affine transformation is then performed on the actual target image based on the relative positional relationship to obtain a transformed target image. This process aligns the actual target image with the desired target image. This method can eliminate positional and angular errors caused by the inspection robot not arriving or inaccurate camera pose, improving the accuracy of the visual detection algorithm.

[0109] In one embodiment, the current camera focal length is updated based on the detection results to obtain the target focal length, such as... Figure 6 The flowchart illustrating the method for generating the target focal length includes:

[0110] S602, run the image detection algorithm according to the target type of the detection results.

[0111] The target categories for the test results include electrical components, electrical equipment, equipment indicator lights and switches in the computer room, items belonging to workers, door handles, etc.

[0112] Specifically, the corresponding image detection algorithm is run based on the target type detected by the detection results.

[0113] S604, determine whether the image detection algorithm returns a satisfactory result. If the image detection algorithm returns a satisfactory result, proceed to S606.

[0114] The criterion for judging whether the result returned by the image detection algorithm is qualified can be the status output of the image detection algorithm itself. For example, the status output of the image detection algorithm itself is "no valid inspection target identified", or the status output of the image detection algorithm itself is "valid inspection target identified".

[0115] Specifically, the image detection algorithm's own status output of "valid inspection target identified" indicates that the image detection algorithm returns a qualified result, while the image detection algorithm's own status output of "valid inspection target not identified" indicates that the image detection algorithm returns a unqualified result.

[0116] Specifically, if the image detection algorithm returns an unqualified result, it is necessary to determine whether the camera focal length has been updated. If the camera focal length has not been updated, a preset focal length is added to the current camera focal length. If the camera focal length has been updated, it is necessary to determine whether the previous image detection algorithm returned a qualified result.

[0117] S606 determines whether the lower limit of the allowable focal length is 0.

[0118] Among them, the allowable lower focal length Fsub is the minimum target focal length of the camera, and the initial value of the allowable lower focal length is set to 0mm.

[0119] S608: If the lower limit focal length is 0, then set the lower limit focal length to the current camera's focal length.

[0120] In the initial state, that is, when the allowable lower focal length Fsub is 0mm, the allowable lower focal length is set to the current camera's focal length.

[0121] S610 adds a preset focal length to the current camera's focal length to obtain the target focal length.

[0122] The preset focal length refers to the adjustment range of the camera's current focal length. The larger the preset focal length, the greater the adjustment range of the camera's current focal length. The value of the preset focal length should be flexibly selected according to the actual situation, and no limit is imposed here.

[0123] One method is to add a preset focal length to the current camera's focal length, which can make full use of the camera's depth of field and thus greatly reduce image blur.

[0124] In this embodiment, by adding a preset focal length to the camera, the depth of field of the camera is fully utilized, and the lower limit of the target focal length is obtained, providing a basis for determining the range of the target focal length in subsequent processes.

[0125] In one embodiment, such as Figure 7 The flowchart illustrating the target focal length generation method is shown. The image processing method for the inspection robot also includes:

[0126] S702, determine whether the return result of the previous image detection algorithm is qualified.

[0127] If the image detection algorithm returns an unqualified result, it determines whether the previous image detection algorithm returned a qualified result.

[0128] If the previous image detection algorithm returned a satisfactory result, S706 is executed; if the previous image detection algorithm returned an unsatisfactory result, S704 is executed.

[0129] S704, if the previous image detection algorithm returned an unqualified result, then execute the step of increasing the current camera's focal length by a preset focal length to obtain the target focal length.

[0130] S706, if the previous image detection algorithm returned a qualified result, then the upper limit of the allowable focal length is set to the current camera's focal length.

[0131] The maximum allowable focal length Fup is set to the current camera's focal length, meaning that the maximum value of the current camera's focal length is the maximum allowable focal length Fup.

[0132] In this embodiment, by determining whether the return result of the previous image detection algorithm is qualified, the current focal length range of the camera is made to fill the depth of field range of the camera as much as possible, so as to obtain a reasonable upper limit value of the current focal length, which provides a basis for subsequent calculation of the target focal length.

[0133] In one embodiment, the target focal length is obtained by calculating the upper limit of the allowable focal length and the average of the upper limit of the allowable focal length.

[0134] The target focal length is obtained by using the average of the upper limit of the allowable focal length and the upper limit of the allowable focal length (Fup+Fsub)*0.5.

[0135] It should be noted that the current focal length range of the camera is greater than or equal to the lower permissible focal length Fsub and less than or equal to the upper permissible focal length Fup. The current focal length range of the camera is included in the depth of field range.

[0136] In this embodiment, the target focal length is calculated by using the upper and lower allowable focal lengths, which provides the clearest imaging focal length for the camera and serves as a basis for subsequent shooting of the inspection target.

[0137] In one embodiment, the relative positional relationship between the actual target image and the desired target image includes a positional relationship matrix between the actual target image and the desired target image. The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image. This includes: acquiring inspection target information; determining the desired target image based on the inspection target information; and inputting the actual target image and the desired target image into a pre-trained image affine model to obtain a positional relationship matrix between the actual target image and the desired target image.

[0138] The inspection target information includes: the current pose of the camera on the inspection robot, the type of inspection target, the location of the inspection target, and the status of the inspection target.

[0139] The desired target image is a standard image predefined in the ledger. For each type of inspection target, the image of the inspection target is taken in advance to obtain the desired target image. The desired target image generally does not have distortion, blur, or other issues that image detection algorithms cannot recognize.

[0140] Specifically, the actual target image and the desired target image are input into a pre-trained image affine model TNet to obtain the positional relationship matrix (Transform matrix T) between the actual target image and the desired target image.

[0141] In this embodiment, desired target images are captured for each inspection target, and the desired target images are compared with the actual target images to obtain a positional relationship matrix between the actual target images and the desired target images, providing a basis for subsequent image conversion between the actual target images and the desired target images.

[0142] In one embodiment, performing an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image includes: performing an affine transformation on the actual target image based on the relative positional relationship to obtain the transformed actual target image; and cropping the transformed actual target image based on the region of interest of the inspection target of the inspection robot to obtain the inspection target image.

[0143] The Region of Interest (ROI) of the inspection target of the inspection robot can be obtained by reading, converting, thresholding, and filling the image of the inspection target.

[0144] Specifically, based on the region of interest (ROI) (R1) of the inspection target of the inspection robot, the transformed actual target image IMG-rect is cropped to obtain the inspection target image R2.

[0145] In this embodiment, the inspection target image is obtained by cropping the transformed actual image IMG-rect using the region of interest (ROI). This reduces the portion of the image containing objects unrelated to the inspection target, improves image recognition accuracy, and makes the image detection algorithm more robust and stable.

[0146] In one embodiment, such as Figure 8 The flowchart shown is a method for improving the visual inspection accuracy of a mobile inspection robot, including: a first part, in which the focal length is automatically set by deployment software during deployment, and a second part, in which the photo undergoes adaptive affine transformation during inspection.

[0147] The first part involves automatic focus setting via deployment software during deployment. Specifically, the deployment software automatically sets the camera's focus during the deployment process. This deployment software integrates an automatic focus setting method and includes the necessary human-computer interaction interface for running that method.

[0148] S802: If the inspection robot reaches the predetermined position, the camera posture on the inspection robot is obtained.

[0149] S804 detects inspection targets based on the camera posture on the inspection robot.

[0150] S806 runs the image detection algorithm based on the target type detected by the detection results.

[0151] S808 determines whether the image detection algorithm returns a satisfactory result.

[0152] S810 determines whether the lower limit of the allowable focal length is 0.

[0153] S812, if the lower limit focal length is 0, then set the lower limit focal length to the current camera's focal length.

[0154] S814, add a preset focal length to the current camera's focal length, then return to S804.

[0155] This involves adding a preset focal length and performing an image detection of the inspection target.

[0156] S816 determines whether the result returned by the previous image detection algorithm is qualified.

[0157] If the image detection algorithm returns an unqualified result, it determines whether the previous image detection algorithm returned a qualified result.

[0158] If the previous image detection algorithm returned a result that was unqualified, then the step of increasing the current camera's focal length by a preset focal length is performed to obtain the target focal length.

[0159] S818, if the previous image detection algorithm returned a qualified result, then set the upper limit of the allowable focal length to the current camera's focal length.

[0160] S820 calculates the upper limit of the permissible focal length and the average of the upper limit of the permissible focal length to obtain the target focal length.

[0161] The second part involves photo adaptive affine transformation during inspection. In the actual inspection process, a photo adaptive affine transformation method is used to process the actual target photos, and then visual inspection is performed on the processed photos.

[0162] S822 controls the camera to capture images of the inspection target at the target focal length, thus obtaining an image of the actual target.

[0163] S824, obtain inspection target information.

[0164] S826 determines the desired target image based on the inspection target information.

[0165] S828: The actual target image and the desired target image are input into a pre-trained affine model to obtain the positional relationship matrix between the actual target image and the desired target image.

[0166] S830 performs an affine transformation on the actual target image based on the relative positional relationship to obtain the transformed actual target image.

[0167] S832, based on the region of interest of the inspection target of the inspection robot, crops the transformed actual target image to obtain the inspection target image.

[0168] In this embodiment, the target object is detected based on the posture of the camera on the inspection robot, and the detection result is obtained. The focal length of the camera is updated according to the detection result to obtain a more suitable focal length, so that the target object is as clear as possible. Then, the camera is controlled to take pictures of the target object with a more suitable focal length to obtain the actual target image. The actual target image is transformed according to the relative position relationship to obtain the transformed target image. In this way, the actual target image and the expected target image are aligned. This method can eliminate the position error and angle error caused by the inspection robot not arriving or the camera posture being inaccurate, and improve the accuracy of the visual detection algorithm. This method (1) can make full use of the camera depth of field by using the automatic setting method of camera focal length in the deployment process to make up for potential navigation positioning and control system errors, so that the target photos taken are as clear as possible, thereby improving the accuracy of the visual detection algorithm. (2) By using the target image adaptive affine transformation method in the actual inspection process, the actual target photos can be aligned with the expected target photos in the ledger, eliminating the rotation, translation and other deflections of the target photos and ROI areas caused by navigation distance and angle errors, thereby improving the accuracy of the visual detection algorithm. (3) By combining the above-mentioned automatic camera focal length setting method and target image adaptive affine transformation method, the navigation and visual detection algorithms in the mobile inspection task are treated as a system problem. Both can be run automatically through programming, thereby effectively reducing the pressure on deployment personnel.

[0169] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0170] Based on the same inventive concept, this application also provides an image processing device for an inspection robot to implement the image processing method of the inspection robot described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the image processing device for the inspection robot provided below can be found in the limitations of the image processing method for the inspection robot described above, and will not be repeated here.

[0171] In one embodiment, such as Figure 9As shown, an image processing device for an inspection robot is provided, comprising: an acquisition module 902, a detection module 904, a preprocessing module 906, an image capture module 908, a processing module 910, and an output module 912, wherein:

[0172] The acquisition module 902 is used to acquire the camera pose on the inspection robot if the inspection robot reaches the predetermined position.

[0173] The detection module 904 is used to detect the inspection target based on the camera posture on the inspection robot;

[0174] The preprocessing module 906 is used to update the current camera's focal length based on the detection results to obtain the target focal length.

[0175] The imaging module 908 is used to control the camera to capture images of the inspection target at the target focal length to obtain actual target images;

[0176] The processing module 910 is used to input the actual target image and the desired target image into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image;

[0177] The output module 912 is used to perform an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image.

[0178] In one embodiment, the preprocessing module 906 is further configured to run an image detection algorithm based on the target type of the detection result; if the image detection algorithm returns a qualified result, determine whether the allowable lower limit focal length is 0; if the allowable lower limit focal length is 0, set the allowable lower limit focal length to the focal length of the current camera; and add a preset focal length to the focal length of the current camera to obtain the target focal length.

[0179] In one embodiment, the preprocessing module 906 is further configured to: if the image detection algorithm returns an unqualified result, determine whether the previous image detection algorithm returned a qualified result; if the previous image detection algorithm returned an unqualified result, perform the step of increasing the current camera's focal length by a preset focal length to obtain the target focal length; if the previous image detection algorithm returned a qualified result, set the allowable upper limit focal length to the current camera's focal length.

[0180] In one embodiment, the preprocessing module 906 is further configured to calculate the upper limit of the allowable focal length and the average value of the upper limit of the allowable focal length to obtain the target focal length.

[0181] In one embodiment, the relative positional relationship between the actual target image and the desired target image includes a positional relationship matrix between the actual target image and the desired target image. The processing module 910 is also used to acquire inspection target information; determine the desired target image based on the inspection target information; and input the actual target image and the desired target image into a pre-trained image affine model to obtain a positional relationship matrix between the actual target image and the desired target image.

[0182] In one embodiment, the output module 912 is further configured to perform an affine transformation on the actual target image according to the relative positional relationship to obtain a transformed actual target image; and to crop the transformed actual target image according to the region of interest of the inspection target of the inspection robot to obtain an inspection target image.

[0183] The modules in the image processing device of the aforementioned inspection robot can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0184] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores detection result data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements an image processing method for an inspection robot.

[0185] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method for an inspection robot. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0186] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above method steps.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above method steps.

[0189] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above method steps.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method for an inspection robot, characterized in that, The method includes: If the inspection robot reaches the predetermined position, the camera pose on the inspection robot is obtained; The inspection target is detected based on the camera posture on the inspection robot. Based on the target type detected, run the image detection algorithm; If the image detection algorithm returns a qualified result, it is determined whether the allowable lower limit focal length is 0; wherein, a qualified result indicates that a valid inspection target has been identified; If the lower limit of the allowable focal length is 0, then the lower limit of the allowable focal length is set to the current focal length of the camera; After increasing the focal length of the current camera by a preset focal length, the target detection is performed again. If the image detection algorithm returns a result that is unqualified, it is determined whether the previous result returned by the image detection algorithm was qualified; wherein, the return result being unqualified indicates that no valid inspection target was identified; If the image detection algorithm returned an unqualified result in the previous step, then the step of increasing the focal length of the current camera by a preset focal length and then performing the target detection again is executed. If the previous image detection algorithm returned a qualified result, the upper limit of the allowable focal length is set to the current focal length of the camera. The target focal length is obtained by calculating the average of the upper allowable focal length and the lower allowable focal length; wherein the focal length range determined by the lower allowable focal length and the upper allowable focal length is included by the depth of field range of the camera. The camera is controlled to capture an image of the inspected target at the target focal length, thereby obtaining an actual target image; The actual target image and the desired target image are input into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image. The image affine model is trained based on images of various inspection targets and the manually labeled images of various inspection targets. The inspection target image is obtained by performing an affine transformation on the actual target image based on the relative positional relationship.

2. The method according to claim 1, characterized in that, The relative positional relationship between the actual target image and the desired target image includes the positional relationship matrix between the actual target image and the desired target image. The step of inputting the actual target image and the desired target image into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image includes: obtaining inspection target information. The desired target image is determined based on the inspection target information; The actual target image and the desired target image are input into a pre-trained affine model to obtain the positional relationship matrix between the actual target image and the desired target image.

3. The method according to claim 1, characterized in that, The step of performing an affine transformation on the actual target image based on the relative positional relationship to obtain the inspection target image includes: The actual target image is transformed by performing an affine transformation on the actual target image according to the relative positional relationship; the transformed actual target image is then cropped according to the region of interest of the inspection target of the inspection robot to obtain the inspection target image.

4. The method according to claim 1, characterized in that, The inspection robot can be a tracked inspection robot or a trackless inspection robot.

5. An image processing device for an inspection robot, characterized in that, The device includes: The acquisition module is used to acquire the camera pose on the inspection robot if the inspection robot reaches the predetermined position. The detection module is used to detect the inspection target based on the camera posture on the inspection robot; The preprocessing module is used to run an image detection algorithm based on the target type detected by the detection results; if the image detection algorithm returns a qualified result, it determines whether the allowable lower limit focal length is 0; wherein, a qualified result indicates that a valid inspection target has been identified; if the allowable lower limit focal length is 0, it sets the allowable lower limit focal length to the current focal length of the camera; after increasing the current focal length of the camera by a preset focal length, it performs inspection target detection again; if the image detection algorithm returns an unqualified result, it determines whether the previous return result of the image detection algorithm was qualified; wherein, the... A return result of "unqualified" indicates that no valid inspection target was identified. If the previous image detection algorithm returned an unqualified result, the step of increasing the current camera's focal length by a preset focal length and then re-detecting the inspection target is performed. If the previous image detection algorithm returned a qualified result, the upper limit of the allowable focal length is set to the current camera's focal length. The average of the upper limit of the allowable focal length and the lower limit of the allowable focal length is calculated to obtain the target focal length. The focal length range determined by the lower limit of the allowable focal length and the upper limit of the allowable focal length is included in the camera's depth of field range. The shooting module is used to control the camera to take pictures of the inspection target at the target focal length to obtain an actual target image; The processing module is used to input the actual target image and the desired target image into a pre-trained image affine model to obtain the relative positional relationship between the actual target image and the desired target image. The image affine model is trained based on images of various inspection targets and the manually labeled results of the images of various inspection targets. The output module is used to perform an affine transformation on the actual target image according to the relative positional relationship to obtain the inspection target image.

6. The apparatus according to claim 5, characterized in that, The relative positional relationship between the actual target image and the desired target image includes a positional relationship matrix between the actual target image and the desired target image. The processing module is further configured to: Obtain inspection target information; The desired target image is determined based on the inspection target information; The actual target image and the desired target image are input into a pre-trained affine model to obtain the positional relationship matrix between the actual target image and the desired target image.

7. The apparatus according to claim 5, characterized in that, The output module is also used for: The actual target image is transformed by performing an affine transformation on the actual target image according to the relative positional relationship; the transformed actual target image is then cropped according to the region of interest of the inspection target of the inspection robot to obtain the inspection target image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images