Image enhancement method, inspection robot, remote terminal and inspection operation and maintenance system

Abstract

The embodiments of the present application provide an image enhancement method, an inspection robot, a remote terminal and an inspection and operation system, and relate to the field of robot technology. The present application obtains an initial acquisition image acquired when the inspection robot operates the target object in the target area, wherein the inspection robot operates the target object by controlling the mechanical gripper on the robotic arm; performs iterative image enhancement processing on the initial acquisition image using the enhancement sub-model in the pre-trained target model to obtain a target enhanced image; performs denoising and sharpening processing on the target enhanced image using the denoising sub-model and the sharpening sub-model in the target model to obtain a target image; and sends the target image to a remote terminal. The image enhancement method can improve the quality of images acquired by the inspection robot in an environment with poor lighting conditions. In addition, remote operation of the target object in the target image is achieved through the mechanical gripper on the robotic arm of the inspection robot.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and more specifically, to an image enhancement method, an inspection robot, a remote terminal, and an inspection and operation system. Background Art

[0002] Pipeline corridors are underground tunnel spaces that integrate various engineering pipelines such as electricity, communications, gas, heating, water supply and drainage, such as immersed tube tunnels and urban underground integrated pipeline corridors. Pipeline corridors are closed spaces with poor ventilation conditions and are prone to accidents such as water seepage, smoke and fire. Therefore, in order to maintain the safety of equipment and staff in pipeline corridors and reduce the waste of human resources, more and more inspection robots are being used in the safety monitoring of pipeline corridors.

[0003] Since there are many environments with poor lighting conditions and are not conducive to detection in the pipeline corridor, inspection robots are needed to replace manual inspections of the pipeline corridor. The inspection robot can move along the pipeline corridor while analyzing the gas or images in the pipeline corridor to promptly detect abnormalities in the operating equipment and send out alarm signals.

[0004] However, due to the structural design of the current inspection robots and the performance problems of visual collection devices such as cameras, it is difficult for them to obtain complete image information in the tunnel under low light conditions. In addition, the inspection robots in the prior art only have the function of collecting environmental information and cannot handle abnormal equipment, which reduces the working efficiency of the robots. Summary of the invention

[0005] The purpose of the present application includes, for example, providing an image enhancement method, an inspection robot, a remote terminal and an inspection operation and maintenance system, which can improve the quality of images collected by the inspection robot in an environment with poor light conditions through the image enhancement method. In addition, through the mechanical gripper on the mechanical arm of the inspection robot, a target object in the target image can be remotely operated.

[0006] The embodiments of the present application can be implemented as follows:

[0007] In a first aspect, the present application provides an image enhancement method, which is applied to an inspection robot of an inspection and maintenance system, wherein the inspection and maintenance system comprises: a remote terminal and the inspection robot, wherein the remote terminal is in communication connection with the inspection robot, and the method comprises:

[0008] Acquire an initial acquisition image acquired when the inspection robot operates on a target object in a target area, wherein the initial acquisition image is acquired by the inspection robot performing real-time image acquisition on the target area and / or the target object in the target area, wherein the inspection robot includes a mechanical arm, and the inspection robot operates on the target object by controlling a mechanical gripper on the mechanical arm;

[0009] Performing iterative image enhancement processing on the initial collected image using an enhancer model in a pre-trained target model to obtain a target enhanced image;

[0010] The target enhanced image is subjected to denoising and sharpening processing by the denoising sub-model and the sharpening sub-model in the target model to obtain a target image; the loss function of the target model is obtained by adding the first loss function, the second loss function and the third loss function, and the loss function of the target model is calculated by the following formula:

[0011] L total =λ1L spa +λ2L exp +λ3L col +λ4L s +λ5L den +λ6L reg

[0012] Among them, L spa is the spatial consistency loss function, L exp is the exposure recovery loss function, L col is the color constancy loss function, L s is the smoothness loss function, L den is the denoising loss function, L reg is the regularization term of the denoising loss function, λ1, λ2, λ3, λ4, λ5, and λ6 are the weights of the six sub-loss functions respectively; the first loss function is obtained by multiplying the spatial consistency loss function, the exposure recovery loss function, the color constancy loss function, and the smoothness loss function with the corresponding weights and then adding them together; the second loss function is obtained by multiplying the denoising loss function with the corresponding weights; and the third loss function is obtained by multiplying the regularization term of the denoising loss function with the corresponding weights;

[0013] The target image is sent to the remote terminal.

[0014] In an optional implementation, the iterative image enhancement processing is performed on the initial collected image using the enhancement sub-model in the pre-trained target model to obtain the target enhanced image, including:

[0015] Inputting the initial collected image into the convolutional neural network in the enhancement sub-model to obtain an enhancement matrix, wherein the enhancement matrix has the same size as the initial collected image;

[0016] According to the enhancement matrix, iterative image enhancement processing is performed on the initial collected image to obtain a target enhanced image.

[0017] In an optional implementation, performing iterative enhancement processing on the initial acquired image according to the enhancement matrix to obtain a target enhanced image includes:

[0018] Obtaining a first estimated enhanced image according to the enhancement matrix and the initial acquired image;

[0019] According to the enhancement matrix and the first estimated enhanced image, a second estimated enhanced image is obtained, the second estimated enhanced image is used as a new first estimated enhanced image, and this step is repeated until the number of executions reaches a preset number, and the obtained second estimated enhanced image is used as the target enhanced image.

[0020] In an optional implementation, the denoising sub-model and the sharpening sub-model in the target model perform denoising and sharpening processing on the target enhanced image to obtain the target image, including:

[0021] Inputting the target enhanced image into the denoising sub-model for denoising processing to obtain a denoised enhanced image;

[0022] The denoised enhanced image is input into the sharpening sub-model, and information is extracted from the denoised enhanced image by a Gaussian blur low-pass filter in the sharpening sub-model to obtain low-frequency information of the denoised enhanced image, and the denoised enhanced image and the low-frequency information are sharpened by a sharpening function to obtain a target image.

[0023] In an optional embodiment, the method further includes:

[0024] Input the sample image into the initial model to obtain the actual output image;

[0025] According to the actual output image, using a preset first loss function, calculating a first loss of the initial model;

[0026] Calculating the noise reduction loss of the initial model by using the sample image and the noise reduction initial sub-model in the initial model;

[0027] According to the first loss and the noise reduction loss, each sub-model in the initial model is modified to obtain the target model corresponding to the initial model.

[0028] In an optional implementation, the calculating the denoising loss of the initial model by using the sample image and the denoising initial sub-model in the initial model includes:

[0029] Input the sample image and a plurality of sample sub-images sampled from the sample image into the denoising initial sub-model in the initial model to obtain an actual denoised image and a plurality of first denoised sub-images, and sample the actual denoised image to obtain a plurality of second denoised sub-images;

[0030] Calculating a second loss of the initial model using the second loss function according to the first denoised sub-image and the second denoised sub-image;

[0031] Calculate the third loss of the initial model using the third loss function according to the sample sub-image and the first denoised sub-image;

[0032] The step of modifying each sub-model in the initial model according to the first loss and the noise reduction loss to obtain the target model corresponding to the initial model includes:

[0033] According to the first loss, the second loss and the third loss, each sub-model in the initial model is modified to obtain the target model corresponding to the initial model.

[0034] In a second aspect, the present application provides an image enhancement device, the device comprising:

[0035] an acquisition module, used to acquire an initial acquisition image acquired when the inspection robot operates on a target object in a target area, wherein the initial acquisition image is acquired by the inspection robot performing real-time image acquisition on the target area and / or the target object in the target area, wherein the inspection robot includes a mechanical arm, and the inspection robot operates on the target object by controlling a mechanical gripper on the mechanical arm;

[0036] A processing module, used for performing iterative image enhancement processing on the initial collected image using an enhancement sub-model in a pre-trained target model to obtain a target enhanced image;

[0037] The processing module is also used to perform denoising and sharpening processing on the target enhanced image by the denoising sub-model and the sharpening sub-model in the target model to obtain the target image; the loss function of the target model is obtained by adding the first loss function, the second loss function and the third loss function, and the loss function of the target model is calculated by the following formula:

[0038] L total =λ1L spa +λ2L exp+λ3L col +λ4L s +λ5L den +λ6L reg

[0039] Among them, L spa is the spatial consistency loss function, L exp is the exposure recovery loss function, L col is the color constancy loss function, L s is the smoothness loss function, L den is the denoising loss function, L reg is the regularization term of the denoising loss function, λ1, λ2, λ3, λ4, λ5, and λ6 are the weights of the six sub-loss functions respectively; the first loss function is obtained by multiplying the spatial consistency loss function, the exposure recovery loss function, the color constancy loss function, and the smoothness loss function with the corresponding weights and then adding them together; the second loss function is obtained by multiplying the denoising loss function with the corresponding weights; and the third loss function is obtained by multiplying the regularization term of the denoising loss function with the corresponding weights;

[0040] A sending module is used to send the target image to the remote terminal.

[0041] The processing module is also specifically used to input the initial acquired image into the convolutional neural network in the enhancement sub-model to obtain an enhancement matrix, and the enhancement matrix is ​​equal to the size of the initial acquired image; according to the enhancement matrix, iterative image enhancement processing is performed on the initial acquired image to obtain a target enhanced image.

[0042] The processing module is specifically further used to obtain a first estimated enhanced image based on the enhancement matrix and the initial acquired image; obtain a second estimated enhanced image based on the enhancement matrix and the first estimated enhanced image, use the second estimated enhanced image as a new first estimated enhanced image, and repeat this step until the number of executions reaches a preset number, and use the obtained second estimated enhanced image as the target enhanced image.

[0043] The processing module is also specifically used to input the target enhanced image into the denoising sub-model for denoising processing to obtain a denoised enhanced image; input the denoised enhanced image into the sharpening sub-model, extract information from the denoised enhanced image using a Gaussian blur low-pass filter in the sharpening sub-model to obtain low-frequency information of the denoised enhanced image, and use a sharpening function to sharpen the denoised enhanced image and the low-frequency information to obtain a target image.

[0044] The training module is used to input the sample image into the initial model to obtain the actual output image; according to the actual output image, use the preset first loss function to calculate the first loss of the initial model; use the sample image and the denoising initial sub-model in the initial model to calculate the denoising loss of the initial model; according to the first loss and the denoising loss, correct each sub-model in the initial model to obtain the target model corresponding to the initial model.

[0045] The training module is also used to input the sample image and multiple sample sub-images sampled from the sample image into the denoising initial sub-model in the initial model to obtain an actual denoised image and multiple first denoised sub-images, and sample multiple second denoised sub-images from the actual denoised image; calculate the second loss of the initial model using the second loss function based on the first denoised sub-image and the second denoised sub-image; calculate the third loss of the initial model using the third loss function based on the sample sub-image and the first denoised sub-image; and correct each sub-model in the initial model based on the first loss and the denoising loss to obtain the target model corresponding to the initial model, including: correcting each sub-model in the initial model based on the first loss, the second loss and the third loss to obtain the target model corresponding to the initial model.

[0046] In a third aspect, the present application provides an inspection robot, the inspection robot comprising: a sensor module, an execution module, a control module and a first communication module; the sensor module is electrically connected to the control module, the execution module is electrically connected to the control module, the control module is electrically connected to the first communication module, and the first communication module is used to establish a communication connection with a remote terminal;

[0047] The sensor module includes: a stereo camera, a pan-tilt camera and a three-dimensional laser scanner, wherein the stereo camera is used to collect image information of a target object in a target area in real time; the pan-tilt camera is used to collect image information of a target area in real time; and the three-dimensional laser scanner is used to generate three-dimensional point cloud information of the target area;

[0048] The execution module includes: a mechanical arm, a mechanical gripper and an electric push rod, wherein the mechanical gripper is installed at the end of the mechanical arm and is used to operate the target object; the mechanical arm is used to send the mechanical gripper to the target position; the electric push rod is connected to the pan-tilt camera of the sensor module and is used to adjust the height of the pan-tilt camera;

[0049] The control module includes: an industrial computer module, a single-chip computer module, an embedded computing module and a stepper motor controller module; the industrial computer module is used to control the movement of the inspection robot according to the instructions of the remote terminal; the embedded computing module is used to receive the image information collected by the stereo camera and the pan-tilt camera, and execute the image enhancement method of the first aspect to obtain the target image, and send the target image to the industrial computer module; the stepper motor controller module is used to control the movement of the electric push rod according to the level signal of the single-chip computer module;

[0050] The first communication module is used to receive the target image sent by the industrial computer module, and send the target image to the remote terminal.

[0051] In an optional embodiment, the inspection robot further includes: a power supply module and a mobile module; the power supply module is electrically connected to the sensor module, the execution module, the control module, the first communication module and the mobile module respectively, and the mobile module is electrically connected to the control module;

[0052] The power supply module includes: an inverter and a voltage converter, a power supply battery, a power supply switch and a mechanical arm voltage stabilizing module, the power supply battery is used to supply power to the mobile module, the sensor module, the execution module, the control module and the first communication module, the power supply switch is used to control the opening and closing of the power supply circuit of the inspection robot, and the mechanical arm voltage stabilizing module is used to provide a power supply voltage to the mechanical arm;

[0053] The mobile module includes: an underlying mobile platform, and the underlying mobile platform is used to control the movement of the inspection robot.

[0054] In a fourth aspect, the present application provides a remote terminal, the remote terminal comprising: a second communication module and a human-computer interaction module; the human-computer interaction module is electrically connected to the second communication module, and the second communication module is used to establish a communication connection with the inspection robot described in the third aspect;

[0055] The second communication module is used to receive the target image sent by the inspection robot, and send equipment operation instructions and state switching instructions to the inspection robot.

[0056] In a fifth aspect, the present application provides an inspection and operation system, the inspection and operation system comprising: a remote terminal and an inspection robot, the remote terminal being communicatively connected with the inspection robot;

[0057] The remote terminal is used to set the inspection route of the inspection robot, send a state switching instruction to the inspection robot, control the inspection robot to switch to a target working state, and send a device operation instruction to the inspection robot according to a target image sent by the inspection robot;

[0058] The inspection robot is used to move in the target area according to the constructed three-dimensional point cloud map, collect initial collection images, and execute the image enhancement method described in the first aspect to obtain the target image, send the target image to the remote terminal, switch the working state according to the state switching instruction sent by the inspection robot, and operate the target object by controlling the mechanical gripper on the mechanical arm according to the equipment operation instruction sent by the remote terminal.

[0059] In a sixth aspect, the present application provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image enhancement method as described in the first aspect of the aforementioned embodiment.

[0060] In a seventh aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image enhancement method described in the first aspect of the aforementioned embodiment are implemented.

[0061] The beneficial effects of the embodiments of the present application include:

[0062] By using the image enhancement method, inspection robot, remote terminal and inspection operation and maintenance system provided by the present application, firstly, by performing iterative image enhancement processing on the initial collected image collected by the inspection robot, the influence of the weak light environment on the initial collected image is eliminated as much as possible, thereby improving the quality of the target image. The visual effect of the image is further improved by noise reduction and sharpening. Secondly, according to the received target image, the staff can operate the target object to be repaired by controlling the mechanical gripper on the robotic arm, thereby realizing automatic control of the inspection robot, improving the automation level of the inspection robot, and reducing the workload of the staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 A schematic diagram of the structure of the inspection robot provided in an embodiment of the present application;

[0065] Figure 2 Another structural schematic diagram of the inspection robot provided in an embodiment of the present application;

[0066] Figure 3 An example diagram of the structure of a remote terminal provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the modules of the inspection and operation and maintenance system provided in the embodiment of the present application;

[0068] Figure 5 A schematic diagram of a target object operated by an inspection robot provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of the steps of the image enhancement method provided in the embodiment of the present application;

[0070] Figure 7 A schematic diagram of another step flow of the image enhancement method provided in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of another step flow of the image enhancement method provided in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of the target model structure of the image enhancement method provided in an embodiment of the present application;

[0073] Fig.10 A schematic diagram of another step flow of the image enhancement method provided in an embodiment of the present application;

[0074] Fig.11 A schematic diagram of another step flow of the image enhancement method provided in an embodiment of the present application;

[0075] Fig.12 A schematic diagram of the initial model training process of the image enhancement method provided in an embodiment of the present application;

[0076] Fig.13 A schematic diagram of another step flow of the image enhancement method provided in an embodiment of the present application;

[0077] Fig.14 A schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application;

[0078] Fig.15 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0079] Icons: 1-inspection robot; 11-sensor module; 111-pan-tilt camera; 112-3D laser scanner; 113-stereo camera; 12-execution module; 121-electric push rod; 122-mechanical arm; 123-mechanical gripper; 13-control module; 131-industrial computer module; 132-embedded computing module; 133-single-chip microcomputer module; 134-stepping motor controller module; 14-first communication module; 141-first wireless network card; 15-power supply module; 151-inverter and voltage converter; 152-mechanical arm voltage regulator module; 153-power switch; 154-power supply battery; 16-mobile module; 1 61-bottom mobile platform; 17-support structure; 171-aluminum profile frame; 172-first layer load-bearing plate; 173-second layer load-bearing plate; 174-top layer plate; 18-connector; 19-fixing member; 2-remote terminal; 21-display; 22-computer host; 23-second communication module; 231-second wireless network card; 24-human-computer interaction module; 241-visual interaction software; 3-gas pipeline equipment; 31-valve handwheel; 10-image enhancement device; 1001-acquisition module; 1002-processing module; 1003-sending module; 1004-training module; 2001-processor; 2002-memory; DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0081] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0082] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0083] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0084] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0085] The utility corridor is an important part of the power grid, water network and other systems in cities or densely populated areas. Safety inspections of the utility corridors in cities are an important measure to ensure the reliable and stable operation of the utility corridors. At present, most of the utility corridors are inspected by inspection robots instead of manual inspections. However, due to the relatively small interior of the utility corridor, poor lighting conditions, and performance problems of visual collection equipment such as cameras, it is difficult to obtain complete image information in the utility corridor under low light conditions. Moreover, the inspection robots in the prior art only have a single function of collecting environmental information and alarming, making it difficult to handle abnormal equipment in the utility corridor as soon as the alarm is triggered, which poses a high safety hazard.

[0086] Based on this, the applicant has proposed an image enhancement method, a patrol robot, a remote terminal, and a patrol operation and maintenance system. By iteratively enhancing the initial image collected by the patrol robot, the impact of the weak light environment on the initial image is eliminated as much as possible, thereby improving the quality of the target image. The visual effect of the image is further improved by noise reduction and sharpening. Furthermore, the staff can also operate the target object to be repaired by controlling the mechanical gripper on the robotic arm according to the state of the target object in the target image, thereby improving the automation of the patrol robot and reducing the workload of the staff.

[0087] The following is an explanation of an image enhancement method, an inspection robot, a remote terminal, and an inspection and operation and maintenance system provided in an embodiment of the present application in combination with a number of specific application examples.

[0088] Figure 1 as well as Figure 2 The figure is a schematic diagram of the structure of the inspection robot provided in an embodiment of the present application. The inspection robot includes: a sensor module 11, an execution module 12, a control module 13 and a first communication module 14.

[0089] The sensor module 11 is electrically connected to the control module 13 , the execution module 12 is electrically connected to the control module 13 , the control module 13 is electrically connected to the first communication module 14 , and the first communication module 14 is used to establish a communication connection with a remote terminal.

[0090] The sensor module 11 may include a stereo camera 113, a pan-tilt camera 111, and a three-dimensional laser scanner 112. The stereo camera 113 is used to collect image information of a target object in a target area in real time. The pan-tilt camera 111 is used to collect image information of a target area in real time. The three-dimensional laser scanner 112 is used to generate three-dimensional point cloud information of the target area.

[0091] The stereo camera 113 may be a dual-lens camera for realizing a stereo imaging function, and a set of stereo photos or videos with visual difference can be obtained through the stereo camera 113. In the embodiment of the present application, the stereo camera 113 may be mounted on the mechanical gripper 123 through the fixing member 19, and may shoot the target object when the inspection robot operates the target object, so that the staff at the remote terminal side can know the current situation of the target object.

[0092] The pan-tilt camera 111 may be a camera with a pan-tilt. Generally, the pan-tilt camera 111 has a device capable of supporting the camera to rotate in both horizontal and vertical directions, so as to drive the camera to shoot the target area from multiple angles.

[0093] The three-dimensional laser scanner 112 is mainly used to quickly acquire three-dimensional coordinate data of the surface of a target object or target area over a large area and with high resolution by means of high-speed laser scanning measurement.

[0094] The execution module 12 includes: a mechanical arm 122 , a mechanical gripper 123 and an electric push rod 121 . The mechanical gripper 123 is installed at the end of the mechanical arm 122 for operating a target object.

[0095] The mechanical arm 122 is used to send the mechanical gripper 123 to the target position; the electric push rod 121 is connected to the pan-tilt camera 111 of the sensor module 11 and is used to adjust the height of the pan-tilt camera 111.

[0096] The mechanical arm 122 is installed on the second load-bearing plate 173 of the support structure 17 through the connecting piece 18. The mechanical arm 122 and the mechanical gripper 123 together constitute a mechanical component for grasping or performing operations, and may have functions such as clamping, placing, and rotating. Optionally, the mechanical gripper 123 may be a mechanical structure similar to a human hand, such as three fingers, five fingers, etc., or a mechanical structure without fingers, such as a parallel clamp, etc., which is not limited in this application.

[0097] Optionally, the electric push rod 121 can be a mechanical component with a linear actuator, which is arranged at the lower part of the pan-tilt camera 111 to adjust the height of the pan-tilt camera 111 through up and down movement.

[0098] The control module 13 includes: an industrial computer module 131 , a single chip computer module 133 , an embedded computing module 132 and a stepping motor controller module 134 .

[0099] The industrial computer module 131 is used to control the movement of the inspection robot according to the instructions of the remote terminal.

[0100] The embedded computing module 132 is used to receive image information collected by the stereo camera 113 and the pan-tilt camera 111 , and execute the image enhancement method in the following embodiment to obtain a target image, and send the target image to the industrial computer module 131 .

[0101] The stepper motor controller module 134 is used to control the movement of the electric push rod 121 according to the level signal of the single chip computer module 133 .

[0102] The support structure 17 is installed on the mobile module 16 of the robot, and includes: an aluminum profile frame 171 , a first-layer load-bearing plate 172 , a second-layer load-bearing plate 173 , and a top-layer plate 174 installed on the top of the aluminum profile frame 171 .

[0103] Among them, the industrial computer module 131 is installed on the second load-bearing plate 173 of the support structure 17, and the embedded computing module 132, the single-chip computer module 133 and the stepper motor controller module 134 are all installed on the first load-bearing plate 172 of the support structure 17.

[0104] The industrial computer module 131 can be an industrial control computer, which generally adopts a bus structure. After receiving the device operation instruction sent by the remote terminal, the industrial computer module 131 runs the coordinate conversion algorithm, the robot arm motion planning algorithm, etc. to realize the control of the above-mentioned execution module 12. Optionally, the industrial computer module 131 can also be used to run the autonomous positioning and navigation control algorithm of the inspection robot, and then send the mobile control instruction to the mobile module 16 to control the movement of the mobile module 16. The industrial computer module 131 can also receive other modules inside the inspection robot through a wired local area network, a serial port, etc., such as the motion information of the mobile module 16 described below, the status information of the above-mentioned execution module 12, and the perception information of the sensor module 11, and send this information to the first communication module 14.

[0105] The embedded computing module 132 may be a computing device with computing processing capability and capable of performing enhanced processing on image information, for example, an embedded computing board.

[0106] The single-chip microcomputer module 133 can be a printed circuit board (PCB) with preset functions. In the embodiment of the present application, the single-chip microcomputer module 133 can be used to convert the motion control instructions sent by the industrial computer module 131 into motion level signals and send them to the stepper motor controller module 134.

[0107] The stepper motor controller module 134 may be a stepper motor controller, which can convert the received motion level signal into a pulse signal required for the electric push rod 121 of the execution module 12 to move up and down, so as to control the electric push rod 121 to move.

[0108] The first communication module 14 is used to receive the target image sent by the industrial computer module 131 and send the target image to the remote terminal.

[0109] The first communication module 14 may also include: a first wireless network card, which can be used to establish a communication connection with a remote terminal. The first communication module 14 can not only send the target image to the remote terminal, but also send the motion information of the mobile module 16 sent by the industrial computer module 131, the state information of the execution module 12, and the perception information of the sensor module 11 to the remote terminal in real time.

[0110] Optionally, continue to Figure 1 as well as Figure 2 The inspection robot also includes: a power supply module 15 and a moving module 16.

[0111] The power supply module 15 is electrically connected to the sensor module 11 , the execution module 12 , the control module 13 , the first communication module 14 and the movement module 16 , respectively, and the movement module 16 is electrically connected to the control module 13 .

[0112] The power supply module 15 includes: an inverter and voltage converter 151, a power supply battery 154, a power supply switch 153 and a robotic arm voltage stabilizing module 152. The power supply battery 154 is used to supply power to the mobile module 16, the sensor module 11, the execution module 12, the control module 13 and the first communication module 14. The power supply switch 153 is used to control the opening and closing of the power supply circuit of the inspection robot. The robotic arm voltage stabilizing module 152 is used to provide a power supply voltage to the robotic arm 122.

[0113] Optionally, in this embodiment, the power supply battery 154 may be a rechargeable storage battery, which is fixedly placed above the mobile module 16 and continuously outputs a voltage signal that can be used for the operation of other modules of the inspection robot.

[0114] The inverter and voltage converter 151 is electrically connected to the power supply battery 154 and is placed above the mobile module 16. It is a converter that can convert the voltage signal output by the power supply battery 154 into the normal working voltage of other modules of the inspection robot. The voltage signal output by the power supply battery 154 is input to the inverter and voltage converter 151, and after the voltage conversion by the inverter and voltage converter 151, the normal working voltage adapted to the mobile module 16, the sensor module 11, the execution module 12, the control module 13 and the first communication module 14 is obtained, and is output to the above modules respectively.

[0115] The robot arm voltage stabilizing module 152 is installed on the first load-bearing plate 172 and is electrically connected to the power supply module 15 . It is a device that can convert the voltage signal output by the power supply module 15 into an adaptive working voltage for the robot arm 122 .

[0116] The mobile module 16 includes: an underlying mobile platform, which can be used to control the movement of the inspection robot.

[0117] The mobile module 16 may be a bottom-layer mobile platform with a function interface for returning motion speed and acceleration information, and may be a mechanical structure composed of movable wheels and a platform frame. The specific material and appearance are not limited in this application.

[0118] The moving module 16 can control the robot to move in a specified direction according to the moving control instruction of the industrial computer module 131 in the control module 13 .

[0119] Figure 3 FIG. 1 is a diagram showing an example structure of a remote terminal provided in an embodiment of the present application. Figure 3 As shown, illustratively, the remote terminal 2 may be composed of a computer host 22 with computing and processing capabilities and a display 21 for displaying target images and working status information of the inspection robot.

[0120] Alternatively, if Figure 4 As shown, the remote terminal 2 may include: a second communication module 23 and a human-computer interaction module 24, wherein the human-computer interaction module 24 may include a visual interaction software 241 running on the computer host 22, and the visual interaction software 241 may also receive instructions from the staff through an external device and display the processing results on the display 21. The second communication module 23 may include a second wireless network card 231 installed on the computer host.

[0121] The human-machine interaction module 24 is electrically connected to the second communication module 23 , and the second communication module 23 is used to establish a communication connection with the inspection robot 1 in the above embodiment.

[0122] The second communication module 23 is used to establish a communication connection with the first communication module 14 of the above-mentioned inspection robot 1, receive the target image sent by the inspection robot 1, the motion information of the mobile module 16, the status information of the execution module 12 and the perception information of the sensor module 11, and send equipment operation instructions, status switching instructions, instructions for controlling the robot's movement, etc. to the inspection robot 1 to control the inspection robot 1 to move in a specified direction or perform specified operations.

[0123] like Figure 4 As shown, an embodiment of the present application also provides an inspection and operation and maintenance system, which includes: a remote terminal 2 and an inspection robot 1, and the remote terminal 2 is communicatively connected with the inspection robot 1.

[0124] The remote terminal 2 is used to set the inspection route of the inspection robot 1, send state switching instructions to the inspection robot 1, control the inspection robot 1 to switch to the target working state, and send equipment operation instructions to the inspection robot 1 according to the target image sent by the inspection robot 1.

[0125] The inspection robot 1 is used to move in the target area according to the constructed three-dimensional point cloud map, collect initial collection images, and execute the image enhancement method in the following embodiment to obtain the target image, send the target image to the remote terminal 2, switch the working state according to the state switching instruction sent by the remote terminal 2, and operate the target object by controlling the mechanical gripper 123 on the mechanical arm 122 according to the equipment operation instruction sent by the remote terminal 2.

[0126] Below, the workflow of the inspection and operation and maintenance system is explained in detail.

[0127] First, a communication connection is established between the first wireless network card 141 in the first communication module 14 on the inspection robot 1 side and the second wireless network card 231 in the second communication module 23 on the remote terminal 2 side.

[0128] Then, the inspection robot 1 receives the inspection route set by the staff at the remote terminal 2 through the first communication module 14 and sends it to the industrial computer module 131 for storage. The industrial computer module 131 controls the inspection robot 1 to move along the set inspection route in an autonomous inspection operation state.

[0129] It should be noted that before the first inspection of the inspection robot 1, the staff can send instructions to the inspection robot 1 on the remote terminal 2 side. After the industrial computer module 131 of the inspection robot 1 receives the instruction via the first wireless network card 141, it runs the autonomous positioning and navigation algorithm of the inspection robot 1 according to the instruction, obtains the mobile control instruction, and sends the mobile control instruction to the underlying mobile platform 161 of the mobile module 16 to control the inspection robot 1 to move in the specified direction in the target area. During the movement, the three-dimensional laser scanner 112 of the sensor module 11 of the inspection robot 1 scans the target area and constructs a three-dimensional point cloud map of the target area. After the construction is completed, the three-dimensional laser scanner 112 sends the three-dimensional point cloud map to the industrial computer module 131 for storage, and then the industrial computer module 131 sends it to the remote terminal 2 via the first wireless network card 141.

[0130] After the remote terminal 2 receives the above-mentioned three-dimensional point cloud map via the second wireless network card 231, it can store it in the memory of the computer host. Optionally, the visualization interaction software 241 of the human-computer interaction module 24 can call the three-dimensional point cloud map to display it on the display. After the staff calibrates the target object in the three-dimensional point cloud map, for example, calibrates the expected position of the inspection robot 1 when operating the target equipment, the calibrated three-dimensional point cloud map is synchronized to the industrial computer module 131 of the inspection robot 1.

[0131] Thus, in the subsequent inspection operation, after the inspection robot 1 is set to the autonomous inspection operation state according to the state switching instruction sent by the staff on the remote terminal 2 side, it can start the autonomous inspection operation according to the calibrated three-dimensional point cloud map and the inspection route set by the staff. In addition, when the inspection robot 1 is in the autonomous inspection operation state, the mechanical arm 122, the mechanical gripper 123 and the electric push rod 121 in the execution module 12 of the inspection robot 1 are in the standby state, which can be, for example, Figure 2 The posture shown in .

[0132] The target area may be a pipe gallery, a tunnel, or the like, and the target object may be equipment that needs to be operated in the pipe gallery or the tunnel, for example, Figure 5 The valve hand wheel 31 on the gas pipeline equipment 3 shown is, of course, not limited to this.

[0133] Next, during the movement of the inspection robot 1, the motion information of the mobile module 16, the status information of the execution module 12, the image information of the target object collected by the stereo camera 113 in the sensor module 11, the image information of the target area collected by the pan-tilt camera 111, and the environmental point cloud perception information collected by the three-dimensional laser scanner 112 are all sent to the industrial computer module 131 in real time. After storing the above information, the industrial computer module 131 synchronizes it to the remote terminal 2 via the first communication module 14.

[0134] Among them, the motion information of the mobile module 16 and the environmental point cloud perception information collected by the three-dimensional laser scanner 112 can be input into the autonomous positioning and navigation algorithm of the industrial computer module 131 to send a mobile control instruction to the mobile module 16 to realize the autonomous inspection of the robot in the target area. Optionally, the autonomous positioning and navigation algorithm can be a loosely coupled algorithm combining the hdl_localization three-dimensional positioning algorithm based on the unscented Kalman filter attitude estimation and the Dijkstra path planning algorithm, of course, not limited to this.

[0135] The image information of the target object collected by the stereo camera 113 and the image information of the target area collected by the pan-tilt camera 111 can be transmitted to the embedded computing module 132 of the control module 13 through the wired local area network inside the inspection robot 1. The module is used to run the image enhancement method in the following embodiments, process the above image information, obtain the target image, and send the target image to the industrial computer module 131 for storage, which is then sent by the industrial computer module 131 to the display of the remote terminal 2 via the first communication module 14 for display. In this way, the staff of the remote terminal 2 can obtain the information of the target area and the target object in real time.

[0136] Furthermore, if the staff of the remote terminal 2 determines that a certain target object in the target image sent by the inspection robot 1 needs to be repaired, the staff can send a state switching instruction to the inspection robot 1 through the remote terminal 2, so that the inspection robot 1 switches from the autonomous inspection operation state to the equipment operation and maintenance state. Optionally, when the inspection robot 1 is in the equipment operation and maintenance state, the staff can select the equipment to be operated in the target image on the remote terminal 2 side and send the equipment operation instruction to the inspection robot 1.

[0137] After receiving the equipment operation instruction, the inspection robot 1 will record the interruption position of this autonomous inspection operation in the industrial computer module 131, and run the autonomous positioning and navigation algorithm in the industrial computer module 131, move to the expected position of the target object to be operated, and start operating the target object.

[0138] For example, Figure 5 As shown, when the target object is a gas pipeline device 3, after the inspection robot moves to the desired position, the mechanical arm and the mechanical gripper are moved to the posture ready for operation, that is, the posture that enables the stereo camera on the mechanical gripper to face the valve handwheel 31 of the gas pipeline device 3. Afterwards, the stereo camera collects image information of the gas pipeline device 3 and the valve handwheel 31 thereon, and transmits it to the embedded computing module in real time, and runs the image enhancement method in the following embodiment to obtain the target image of the gas pipeline device 3 and the valve handwheel 31 thereon. Optionally, the embedded computing module can also run a device recognition algorithm to locate the valve handwheel 31 in the target image according to the target image, and transmit the recognition and positioning results to the industrial computer module.

[0139] The recognition result output by the embedded computing module is in the image coordinate system, and needs to be converted by the industrial computer module into the coordinates of the robot base coordinate system. Exemplarily, the coordinate conversion can be achieved by the following formula.

[0140]

[0141] Wherein, (u, v) is the coordinate of the valve hand wheel 31 in the image coordinate system, z c is the distance between the position of the stereo camera and the valve hand wheel 31, (x E ,y E ,z E ) is the coordinate of the valve hand wheel 31 in the robot arm base coordinate system, is the internal parameter of the stereo camera, is a transformation matrix determined according to the relative positions of the stereo camera, the robot base coordinate system and the image coordinate system of the stereo camera.

[0142] The industrial computer module can also run a robot motion planning algorithm, input the coordinates of the robot base coordinate system into the robot motion planning algorithm, solve the path of the robot through inverse kinematics, and obtain the motion planning result of the robot moving to the operating position. Optionally, the robot motion planning algorithm can be a path solving algorithm based on a random road map method or other motion planning algorithms, which is not limited in this application.

[0143] The industrial computer module transmits the motion planning results of the above-mentioned mechanical arm to the mechanical arm of the execution module in real time. The mechanical arm executes the motion rules and moves the mechanical clamp installed at the end of the mechanical arm at the same time, so that the clamping center point of the mechanical clamp coincides with the center of the valve handwheel 31. After the mechanical arm moves to the operating position, the mechanical clamp is closed so that the mechanical clamp clamps the valve handwheel 31.

[0144] The staff at the remote terminal can determine the rotation direction and angle of the valve handwheel 31 according to the target image sent by the inspection robot at this time, and send corresponding instructions to the inspection robot so that the mechanical gripper of the inspection robot drives the valve handwheel 31 to rotate to the target position.

[0145] Continue reading Figure 4 After completing the operation, the staff on the remote terminal 2 side can send a state switching instruction to the inspection robot 1 again to switch the inspection robot 1 from the equipment operation and maintenance state to the autonomous inspection operation state, run the autonomous positioning and navigation algorithm in the industrial computer module 131, return to the interruption position recorded when switching to the equipment operation and maintenance state, and continue to perform autonomous inspection operations along the inspection route.

[0146] The present application also provides an image enhancement method. Figure 6 The method can be applied to the inspection robot of the above inspection and maintenance system, such as Figure 6 As shown, the method may include the following steps.

[0147] S401, acquiring an initial captured image captured when the inspection robot operates on a target object in a target area.

[0148] The initial captured image is obtained by real-time image capture of the target area and / or the target object in the target area by the inspection robot, wherein the inspection robot includes a mechanical arm, and the inspection robot operates the target object by controlling the mechanical gripper on the mechanical arm.

[0149] Optionally, the initial captured image can be obtained by the inspection robot's stereo camera capturing real-time images of the target object, or by the inspection robot's pan-tilt camera capturing real-time images of the target area, and this application does not limit this.

[0150] In addition, the initial captured image can be acquired by a stereo camera when the inspection robot operates on the target object in the target area, or it can be acquired by capturing the target area when the inspection robot is in an autonomous inspection operation state. Of course, it is not limited to this.

[0151] S402, using the enhancement sub-model in the pre-trained target model to iteratively enhance the initial collected image to obtain a target enhanced image.

[0152] The stereo camera or gimbal camera of the sensor module sends the acquired initial acquired image to the embedded computing module of the control module in real time. The enhancement sub-model in the target model running on the embedded computing module first performs image enhancement processing on all pixels of the initial acquired image iteratively to obtain the target enhanced image.

[0153] It should be noted that the iterative image enhancement processing on the initial acquired image can be understood as performing image enhancement processing on the initial acquired image multiple times through the enhancement sub-model to obtain a target enhanced image. The specific number of iterations can be determined according to the effect to be achieved by the target enhanced image, and this application does not limit it here.

[0154] S403, the denoising sub-model and the sharpening sub-model in the target model perform denoising and sharpening processing on the target enhanced image to obtain a target image.

[0155] The loss function of the target model is obtained by adding the first loss function, the second loss function and the third loss function. The loss function of the target model is calculated by the following formula:

[0156] L total =λ1L spa +λ2L exp +λ3L col +λ4L s +λ5L den +λ6L reg

[0157] Among them, L spa is the spatial consistency loss function, Lexp is the exposure recovery loss function, L col is the color constancy loss function, L s is the smoothness loss function, L den is the denoising loss function, L reg is the regularization term of the denoising loss function, and λ1, λ2, λ3, λ4, λ5, and λ6 are the weights of the six sub-loss functions respectively.

[0158] The first loss function is obtained by multiplying the spatial consistency loss function, exposure recovery loss function, color constancy loss function and smoothness loss function with the corresponding weights and then adding them together. The second loss function is obtained by multiplying the denoising loss function with the corresponding weights. The third loss function is obtained by multiplying the regularization term of the denoising loss function with the corresponding weights.

[0159] Among them, the spatial consistency loss function L spa It can be used to keep the spatial relationship between the actual output image and the sample image unchanged, which can be expressed as follows:

[0160]

[0161] Wherein, N is the pooling kernel size for performing the averaging operation on the sample image, which is used to divide the actual output image or the sample image into regions of the pooling kernel size, and illustratively, may be 4. Ω(i) may be 4 regions adjacent to the i-th region. X i is the average pixel value of the ith region in the actual output image. i is the average pixel value of the i-th region in the sample image.

[0162] Spatial consistency loss function L spa The value of can be obtained by the average of the differences between each area of ​​the actual output image and the adjacent areas, and the average of the differences between each area of ​​the sample image and the adjacent areas.

[0163] Exposure recovery loss function L exp It is used to prevent the actual output image from being overexposed, which can be expressed by the following formula:

[0164]

[0165] Wherein, N is the pooling kernel size for performing the averaging operation on the sample image, and illustratively, it can be 4. r is a sample image, and E is the exposure level.

[0166] Color constancy loss function L col It is used to solve the problem of color distortion of the actual output image, which can be expressed by the following formula:

[0167]

[0168] Where N is the number of channels of the sample image, I i is the average pixel value of each channel in the sample image, M is the average value of the pixels in each channel, and ε is the channel of the sample image.

[0169] Smoothness loss function L s It is used to smooth the actual output image and can be expressed by the following formula:

[0170]

[0171] in and are the horizontal and vertical gradient operations of the input image, respectively. c is the transformation matrix of each channel in the image, and i is the channel of the input image.

[0172] Noise reduction loss function L den and the regularization term L of the denoising loss function reg , which is used to reduce the noise of the enhanced image, can be expressed by the following formula:

[0173] L den =(f(g1(x))-g2(x)) 2

[0174] L reg =(f(g1(x))-g1(f(x))+g2(f(x))-f(g2(x))) 2

[0175] Wherein, x is the sample image, g1(x) and g2(x) are the sample sub-images obtained after the sample image passes through the sampler, f(g1(x)) and f(g2(x)) are the first denoised sub-images obtained after the sample sub-image is input into the denoising initial sub-model, g1(f(x)) and g2(f(x)) are the second denoised sub-images obtained after the sample image passes through the sampler after the actual denoised image is obtained after the sample image passes through the initial denoising sub-model.

[0176] In the first loss function, the corresponding weight of the spatial consistency loss function is λ1, the corresponding weight of the exposure recovery loss function is λ2, the corresponding weight of the color constancy loss function is λ3, and the corresponding weight of the smoothness loss function is λ4. In the second loss function, the corresponding weight of the denoising loss function is λ5. In the third loss function, the corresponding weight of the regularization term of the denoising loss function is λ6.

[0177] The second loss function is the regularization term L of the noise loss function. reg , the third loss function, namely the denoising loss function L denThe specific calculation process is described in detail in the following examples.

[0178] The loss function L composed of the first loss function, the second loss function and the third loss function can be total The initial model in the following embodiment is trained to obtain a target model.

[0179] The denoising sub-model of the embedded computing module first performs denoising and sharpening processing on the target enhanced image in sequence to reduce the impact of noise points in the target enhanced image on the image quality and enhance the clarity of the image. After the denoising and sharpening processing is completed, the target image is obtained.

[0180] S404, sending the target image to a remote terminal.

[0181] The embedded computing module sends the target image obtained after processing the target model to the industrial computer module, and sends it to the communication terminal via the first communication module.

[0182] At this time, if the inspection robot is in the equipment operation and maintenance state, the target image is obtained by performing image enhancement, noise reduction, and sharpening processing on the initial acquired image acquired by the stereo camera. Then, based on the target image, equipment operation instructions can be sent to the inspection robot through the remote terminal, so that the inspection robot can operate the target object in the target image through the robotic arm and mechanical gripper.

[0183] In this embodiment, by performing iterative image enhancement processing on the initial collected image collected by the inspection robot, the influence of the weak light environment on the initial collected image is eliminated as much as possible, and the quality of the target image is improved. The visual effect of the image is further improved by noise reduction and sharpening. In addition, the staff can operate the target object to be repaired by controlling the mechanical gripper on the mechanical arm according to the received target image, thereby improving the automation degree of the inspection robot and reducing the workload of the staff.

[0184] Alternatively, if Figure 7 As shown, in the above step S402, the initial collected image is iteratively enhanced using the enhancement sub-model in the pre-trained target model to obtain a target enhanced image, which can be achieved by the following steps S501 to S502.

[0185] S501, input the initial collected image into the convolutional neural network in the enhancer model to obtain an enhancement matrix.

[0186] Among them, the size of the enhancement matrix is ​​equal to that of the initial acquired image.

[0187] Optionally, the enhancement matrix can be obtained by performing feature extraction on the initial acquired image by a convolutional neural network of the enhancement sub-model, and is used to represent image features of the initial acquired image.

[0188] S502: performing iterative image enhancement processing on the initial collected image according to the enhancement matrix to obtain a target enhanced image.

[0189] Optionally, the initial collected image may be enhanced by performing image enhancement processing through multiple iterations using an enhancement matrix to obtain an enhanced target enhanced image.

[0190] The target enhanced image enhances the unclear features in the initial acquired image, improves the unclear features of the target object or target area in the initial acquired image due to poor lighting conditions, enriches the information content of the initial acquired image, and facilitates the staff to interpret the image.

[0191] In this embodiment, the initial collected image is iteratively enhanced by using the enhancement matrix obtained by the convolutional neural network, thereby improving the visual effect of the initial collected image and increasing the clarity of the initial collected image.

[0192] Alternatively, if Figure 8 As shown, in the above step S502, the initial collected image is iteratively enhanced according to the enhancement matrix to obtain a target enhanced image, which can be achieved by the following steps S601 to S602.

[0193] S601, obtaining a first estimated enhanced image according to an enhancement matrix and an initial collected image.

[0194] Alternatively, if Fig. 9 As shown, the enhancement matrix and the initial acquired image can be input into the following formula to obtain a first estimated enhanced image.

[0195] LE n (x) = LE n-1 (x)+A(x)LE n-1 (x)(1-LE n-1 (x))

[0196] Among them, LE n (x) is the output estimated enhanced image. When x = 1, that is, LE n (1) is the first estimated enhanced image, LE n-1 (1) is the initial collected image. A(x) is the enhancement matrix obtained by the convolutional neural network after extracting the features of the initial collected image. n is the number of times the above formula is iterated.

[0197] According to the above formula, the first estimated enhanced image can be obtained by the product of the initial collected image, the enhanced matrix, and the sum of the initial collected image.

[0198] S602, obtaining a second estimated enhanced image according to the enhancement matrix and the first estimated enhanced image, taking the second estimated enhanced image as a new first estimated enhanced image, and repeating this step until the number of executions reaches a preset number, taking the obtained second estimated enhanced image as a target enhanced image.

[0199] Optionally, according to the above formula, the second estimated enhanced image can be calculated in the same way with the first estimated enhanced image and the enhanced matrix as input.

[0200] Then, the second estimated enhanced image and the enhanced matrix are input into the above formula to obtain the third estimated enhanced image. The above formula is repeatedly executed, and the estimated enhanced image and the enhanced matrix output by the last formula are used as input to obtain the estimated enhanced image output after the current formula is executed. When the formula is executed for a preset number of times, the execution of the formula can be stopped, and the estimated enhanced image output by the last formula is used as the target enhanced image.

[0201] The preset number n may be determined according to the required clarity of the target enhanced image, and illustratively, may be 8.

[0202] In this embodiment, the target enhanced image is obtained by iterative calculation using the above formula, thereby increasing the clarity of the initial collected image and improving the quality of the initial collected image.

[0203] Alternatively, if Fig.10 As shown, in the above step S403, the denoising sub-model and the sharpening sub-model in the target model perform denoising and sharpening processing on the target enhanced image to obtain the target image, which can be implemented by the following steps S701 to S702.

[0204] S701, input the target enhanced image into the denoising sub-model for denoising, and obtain the enhanced image after denoising.

[0205] The denoising sub-model can be a denoising algorithm based on U-Net. U-Net can denoise the image through feature extraction and upsampling to eliminate the noise generated when the initial image is collected and converted into a target enhanced image, and obtain an enhanced image after denoising.

[0206] S702, input the denoised enhanced image into the sharpening sub-model, extract information from the denoised enhanced image through a Gaussian blur low-pass filter in the sharpening sub-model to obtain low-frequency information of the denoised enhanced image, and use a sharpening function to sharpen the denoised enhanced image and the low-frequency information to obtain a target image.

[0207] Optionally, the sharpening sub-model can first extract the low-frequency information in the enhanced image after denoising obtained in the above step through a Gaussian blur low-pass filter. After that, the low-frequency information in the enhanced image is assigned a negative weight and image fused with the enhanced image after denoising to achieve sharpening of the enhanced image after denoising, and thus obtain the target image.

[0208] The execution process of the above sharpening submodule can be expressed as follows:

[0209] E img =α*D img -β·LF img

[0210] Among them, E img is the target image, D img is the enhanced image after noise reduction, LF img is the low-frequency information of the enhanced image after denoising. α and β are the weights of the sharpening submodule, which are used to adjust the sharpening effect of the sharpening module. They can be determined according to the quality of the image information collected by the stereo camera and the gimbal camera. Optionally, α can be 1.5 and β can be 0.5.

[0211] In this embodiment, the target image is obtained by performing noise reduction and sharpening processing on the target enhanced image, thereby reducing the influence of the noise reduction module on the normal pixels of the image and enhancing the edge information of the image.

[0212] Alternatively, if Fig.11 As shown, the image enhancement method provided in the embodiment of the present application may also include the following steps.

[0213] S801, input the sample image into the initial model to obtain the actual output image.

[0214] The sample image may be an image of a target area or a target object captured by a pan-tilt camera or a stereo camera, and may be the same as or different from the initial captured image. In an embodiment of the present application, an image different from the initial captured image is taken as the sample image.

[0215] The initial model is a model that is initially built and has unoptimized parameters. After further training and optimization, the target model can be obtained. The sample image can be input into the initial model to obtain the corresponding actual output image.

[0216] S802: Calculate the first loss of the initial model according to the actual output image using a preset first loss function.

[0217] like Fig.12 As shown, the first loss function can be set as the training target. When the first loss function value, that is, the first loss, is lower than the preset loss threshold, it can be considered that the initial model has been trained and the initial model at this time is used as the target model.

[0218] The first loss function can be calculated by comparing the actual output image with the sample image, and the specific content will be described in detail in the following embodiments.

[0219] S803, using the sample image and the denoising initial sub-model in the initial model, calculating the denoising loss of the initial model.

[0220] Optionally, the parameters in the U-NET-based denoising initial sub-model can be further optimized based on the sample image by using the denoising initial sub-model in the initial model and taking the denoising loss as a training target.

[0221] S804: According to the first loss and the noise reduction loss, each sub-model in the initial model is modified to obtain a target model corresponding to the initial model.

[0222] Since the initial model is trained by an unsupervised method, as described in the above embodiment, the first loss and the denoising loss can be used as the initial model and the denoising initial sub-model as the objective function of training optimization, and the parameters of the enhancement sub-model and the denoising sub-model of the initial model are respectively corrected to obtain the trained target model.

[0223] In this embodiment, the initial model is trained by the first loss and the noise reduction loss to obtain the target model, so that the parameters of the target model are accurate, thereby obtaining a target image with a good image enhancement effect.

[0224] Alternatively, if Fig.13 As shown, in the above step S803, the noise reduction loss of the initial model is calculated using the sample image and the noise reduction initial sub-model in the initial model, which can be implemented by the following steps S901 to S903.

[0225] S901, input the sample image and multiple sample sub-images sampled from the sample image into the denoising initial sub-model in the initial model to obtain an actual denoised image and multiple first denoised sub-images, and sample the actual denoised image to obtain multiple second denoised sub-images.

[0226] like Fig.12 As shown, the sample image is input into the sampler g(·) to obtain a plurality of sample sub-images. In this embodiment, two sample sub-images are taken as an example, namely, sample sub-image g1(y) and sample sub-image g2(y).

[0227] The sample image and the above sample sub-image are input into the denoising initial sub-model, and the corresponding actual denoised image f(y), the first denoised sub-image f(g1(y)), and the first denoised sub-image f(g2(y)) are obtained respectively.

[0228] The actual denoised image f(y) is input into the denoising initial sub-model to obtain a plurality of second denoised sub-images. In this embodiment, two second denoised sub-images are taken as an example, namely, the second denoised sub-image g1(f(y)) and the second denoised sub-image g2(f(y)).

[0229] S902: Calculate a second loss of the initial model using a second loss function according to the first denoised sub-image and the second denoised sub-image.

[0230] The second loss can be the regularization term L of the denoising loss function reg The values ​​are used to represent the differences between the first denoised sub-image f(g1(y)) and the first denoised sub-image f(g2(y)) and the second denoised sub-image g1(f(y)) and the second denoised sub-image g2(f(y)) respectively.

[0231] The denoising initial sub-model of the initial model can be modified through the second loss.

[0232] S903: Calculate a third loss of the initial model using a third loss function according to the sample sub-image and the first denoised sub-image.

[0233] The third loss can be the denoising loss function L den The value of is used to represent the difference between the sample sub-image g2(y) and the first denoised sub-image f(g1(y)).

[0234] The denoising initial sub-model of the initial model can be further modified by the third loss.

[0235] Optionally, in the above step S804, each sub-model in the initial model is corrected according to the first loss and the denoising loss to obtain a target model corresponding to the initial model, which may include: according to the first loss, the second loss and the third loss, each sub-model in the initial model is corrected to obtain a target model corresponding to the initial model.

[0236] The loss function L composed of the first loss function, the second loss function and the third loss function can be total The initial model is trained unsupervised to obtain the target model. In this embodiment, the initial model is trained in an unsupervised manner through a loss function to obtain the target model, so that the parameters of the target model are more accurate.

[0237] like Fig.14 As shown, the embodiment of the present application further provides an image enhancement device 10, which includes:

[0238] The acquisition module 1001 is used to acquire an initial acquisition image acquired when the inspection robot operates on the target object in the target area. The initial acquisition image is acquired by the inspection robot performing real-time image acquisition on the target area and / or the target object in the target area. The inspection robot includes a mechanical arm, and the inspection robot operates on the target object by controlling the mechanical gripper on the mechanical arm.

[0239] A processing module 1002 is used to perform iterative image enhancement processing on the initial collected image using the enhancement sub-model in the pre-trained target model to obtain a target enhanced image;

[0240] The processing module 1002 is also used to perform denoising and sharpening processing on the target enhanced image by the denoising sub-model and the sharpening sub-model in the target model to obtain the target image; the loss function of the target model is obtained by adding the first loss function, the second loss function and the third loss function, and the loss function of the target model is calculated by the following formula:

[0241] L total =λ1L spa +λ2L exp +λ3L col +λ4L s +λ5L den +λ6L reg

[0242] Among them, L spa is the spatial consistency loss function, L exp is the exposure recovery loss function, L col is the color constancy loss function, L s is the smoothness loss function, L den is the denoising loss function, L reg is the regularization term of the denoising loss function, and λ1, λ2, λ3, λ4, λ5, and λ6 are the weights of the six sub-loss functions respectively.

[0243] The first loss function is obtained by multiplying the spatial consistency loss function, exposure recovery loss function, color constancy loss function and smoothness loss function with the corresponding weights and then adding them together. The second loss function is obtained by multiplying the denoising loss function with the corresponding weights. The third loss function is obtained by multiplying the regularization term of the denoising loss function with the corresponding weights.

[0244] The sending module 1003 is used to send the target image to the remote terminal.

[0245] The processing module 1002 is also specifically used to input the initial collected image into the convolutional neural network in the enhancement sub-model to obtain an enhancement matrix, and the enhancement matrix has the same size as the initial collected image; according to the enhancement matrix, iterative image enhancement processing is performed on the initial collected image to obtain a target enhanced image.

[0246] The processing module 1002 is also specifically used to obtain a first estimated enhanced image based on the enhancement matrix and the initial acquired image; obtain a second estimated enhanced image based on the enhancement matrix and the first estimated enhanced image, use the second estimated enhanced image as a new first estimated enhanced image, and repeat this step until the number of executions reaches a preset number, and use the obtained second estimated enhanced image as the target enhanced image.

[0247] The processing module 1002 is also specifically used to input the target enhanced image into the denoising sub-model for denoising processing to obtain the enhanced image after denoising; input the enhanced image after denoising into the sharpening sub-model, extract information from the enhanced image after denoising by using a Gaussian blur low-pass filter in the sharpening sub-model to obtain the low-frequency information of the enhanced image after denoising, and use the sharpening function to sharpen the enhanced image after denoising and the low-frequency information to obtain the target image.

[0248] The training module 1004 is used to input the sample image into the initial model to obtain the actual output image; calculate the first loss of the initial model based on the actual output image using a preset first loss function; calculate the denoising loss of the initial model using the sample image and the denoising initial sub-model in the initial model; and correct each sub-model in the initial model based on the first loss and the denoising loss to obtain a target model corresponding to the initial model.

[0249] The training module 1004 is also used to input the sample image and multiple sample sub-images sampled from the sample image into the denoising initial sub-model in the initial model to obtain the actual denoised image and multiple first denoised sub-images, and sample multiple second denoised sub-images from the actual denoised image; calculate the second loss of the initial model using the second loss function based on the first denoised sub-image and the second denoised sub-image; calculate the third loss of the initial model using the third loss function based on the sample sub-image and the first denoised sub-image; correct each sub-model in the initial model according to the first loss and the denoising loss to obtain a target model corresponding to the initial model, including: correcting each sub-model in the initial model according to the first loss, the second loss and the third loss to obtain a target model corresponding to the initial model.

[0250] See also Fig.15This embodiment also provides a computer device, which includes: a processor 2001, a memory 2002 and a bus. The memory 2002 stores machine-readable instructions executable by the processor 2001. When the computer device is running, the above machine-readable instructions are executed. The processor 2001 communicates with the memory 2002 through the bus. The processor 2001 is used to execute the steps of the image enhancement method in the above embodiment.

[0251] The memory 2002, the processor 2001 and the bus components are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The data processing device of the image enhancement system includes at least one software function module that can be stored in the memory 2002 in the form of software or firmware or solidified in the operating system (OS) of the computer device. The processor 2001 is used to execute the executable modules stored in the memory 2002, such as the software function modules and computer programs included in the data processing device of the image enhancement system.

[0252] Among them, the memory 2002 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0253] Optionally, the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method embodiment are executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0254] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0255] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0256] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image enhancement method, characterized in that: An inspection robot applied to an inspection and maintenance system, the inspection and maintenance system comprising: a remote terminal and the inspection robot, the remote terminal being communicatively connected with the inspection robot, the method comprising: Acquire an initial acquisition image acquired when the inspection robot operates on a target object in a target area, wherein the initial acquisition image is acquired by the inspection robot performing real-time image acquisition on the target area and / or the target object in the target area, wherein the inspection robot includes a mechanical arm, and the inspection robot operates on the target object by controlling a mechanical gripper on the mechanical arm; Performing iterative image enhancement processing on the initial collected image using an enhancer model in a pre-trained target model to obtain a target enhanced image; The target enhanced image is subjected to denoising and sharpening processing by the denoising sub-model and the sharpening sub-model in the target model to obtain a target image; the loss function of the target model is obtained by adding the first loss function, the second loss function and the third loss function, and the loss function of the target model is calculated by the following formula: L total =λ1L spa +λ2L exp +λ3L col +λ4L s +λ5L den +λ6L reg Among them, L spa is the spatial consistency loss function, L exp is the exposure recovery loss function, L col is the color constancy loss function, L s is the smoothness loss function, L den is the denoising loss function, L reg is the regularization term of the denoising loss function, λ1, λ2, λ3, λ4, λ5, and λ6 are the weights of the six sub-loss functions respectively; the first loss function is obtained by multiplying the spatial consistency loss function, the exposure recovery loss function, the color constancy loss function, and the smoothness loss function with the corresponding weights and then adding them together; the second loss function is obtained by multiplying the denoising loss function with the corresponding weights; and the third loss function is obtained by multiplying the regularization term of the denoising loss function with the corresponding weights; sending the target image to the remote terminal; The method of performing iterative image enhancement processing on the initial collected image using the enhancement sub-model in the pre-trained target model to obtain a target enhanced image includes: Inputting the initial collected image into the convolutional neural network in the enhancement sub-model to obtain an enhancement matrix, wherein the enhancement matrix has the same size as the initial collected image; According to the enhancement matrix, performing iterative image enhancement processing on the initial acquired image to obtain a target enhanced image; The iterative enhancement processing is performed on the initial acquired image according to the enhancement matrix to obtain a target enhanced image, including: Obtaining a first estimated enhanced image according to the enhancement matrix and the initial acquired image; According to the enhancement matrix and the first estimated enhanced image, a second estimated enhanced image is obtained, the second estimated enhanced image is used as a new first estimated enhanced image, and this step is repeated until the number of executions reaches a preset number, and the obtained second estimated enhanced image is used as the target enhanced image.

2. The image enhancement method according to claim 1, characterized in that: The denoising sub-model and the sharpening sub-model in the target model perform denoising and sharpening processing on the target enhanced image to obtain the target image, including: Inputting the target enhanced image into the denoising sub-model for denoising processing to obtain a denoised enhanced image; The denoised enhanced image is input into the sharpening sub-model, and information is extracted from the denoised enhanced image by a Gaussian blur low-pass filter in the sharpening sub-model to obtain low-frequency information of the denoised enhanced image, and the denoised enhanced image and the low-frequency information are sharpened by a sharpening function to obtain a target image.

3. The image enhancement method according to claim 1, characterized in that: The method further comprises: Input the sample image into the initial model to obtain the actual output image; Calculate the first loss of the initial model using the first loss function according to the actual output image; Calculating the noise reduction loss of the initial model by using the sample image and the noise reduction initial sub-model in the initial model; According to the first loss and the noise reduction loss, each sub-model in the initial model is modified to obtain the target model corresponding to the initial model.

4. The image enhancement method according to claim 3, characterized in that: The step of calculating the noise reduction loss of the initial model by using the sample image and the noise reduction initial sub-model in the initial model comprises: Input the sample image and a plurality of sample sub-images sampled from the sample image into the denoising initial sub-model in the initial model to obtain an actual denoised image and a plurality of first denoised sub-images, and sample the actual denoised image to obtain a plurality of second denoised sub-images; Calculating a second loss of the initial model using the second loss function according to the first denoised sub-image and the second denoised sub-image; Calculate the third loss of the initial model using the third loss function according to the sample sub-image and the first denoised sub-image; The step of modifying each sub-model in the initial model according to the first loss and the noise reduction loss to obtain the target model corresponding to the initial model includes: According to the first loss, the second loss and the third loss, each sub-model in the initial model is modified to obtain the target model corresponding to the initial model.

5. A patrol robot, characterized in that: The inspection robot comprises: a sensor module, an execution module, a control module and a first communication module; the sensor module is electrically connected to the control module, the execution module is electrically connected to the control module, the control module is electrically connected to the first communication module, and the first communication module is used to establish a communication connection with a remote terminal; The sensor module includes: a stereo camera, a pan-tilt camera and a three-dimensional laser scanner, wherein the stereo camera is used to collect image information of a target object in a target area in real time; the pan-tilt camera is used to collect image information of a target area in real time; and the three-dimensional laser scanner is used to generate three-dimensional point cloud information of the target area; The execution module includes: a mechanical arm, a mechanical gripper and an electric push rod, wherein the mechanical gripper is installed at the end of the mechanical arm and is used to operate the target object; the mechanical arm is used to send the mechanical gripper to the target position; the electric push rod is connected to the pan-tilt camera of the sensor module and is used to adjust the height of the pan-tilt camera; The control module includes: an industrial computer module, a single-chip computer module, an embedded computing module and a stepper motor controller module; the industrial computer module is used to control the movement of the inspection robot according to the instructions of the remote terminal; the embedded computing module is used to receive the image information collected by the stereo camera and the pan-tilt camera, and execute the image enhancement method according to any one of claims 1 to 3 to obtain a target image, and send the target image to the industrial computer module; the stepper motor controller module is used to control the movement of the electric push rod according to the level signal of the single-chip computer module; The first communication module is used to receive the target image sent by the industrial computer module, and send the target image to the remote terminal.

6. The inspection robot according to claim 5, characterized in that: The inspection robot further includes: a power supply module and a mobile module; the power supply module is electrically connected to the sensor module, the execution module, the control module, the first communication module and the mobile module respectively, and the mobile module is electrically connected to the control module; The power supply module includes: an inverter and a voltage converter, a power supply battery, a power supply switch and a mechanical arm voltage stabilizing module, the power supply battery is used to supply power to the mobile module, the sensor module, the execution module, the control module and the first communication module, the power supply switch is used to control the opening and closing of the power supply circuit of the inspection robot, and the mechanical arm voltage stabilizing module is used to provide a power supply voltage to the mechanical arm; The mobile module includes: an underlying mobile platform, and the underlying mobile platform is used to control the movement of the inspection robot.

7. A remote terminal, characterized in that: The remote terminal comprises: a second communication module and a human-computer interaction module; the human-computer interaction module is electrically connected to the second communication module, and the second communication module is used to establish a communication connection with the inspection robot according to claim 5 or 6; The second communication module is used to receive the target image sent by the inspection robot, and send equipment operation instructions and state switching instructions to the inspection robot.

8. A patrol inspection and maintenance system, characterized in that: The inspection and operation system comprises: a remote terminal and an inspection robot, wherein the remote terminal is in communication connection with the inspection robot; The remote terminal is used to set the inspection route of the inspection robot, send a state switching instruction to the inspection robot, control the inspection robot to switch to a target working state, and send a device operation instruction to the inspection robot according to a target image sent by the inspection robot; The inspection robot is used to move in the target area according to the constructed three-dimensional point cloud map, collect initial collection images, and execute the image enhancement method described in any one of claims 1 to 4 to obtain a target image, send the target image to the remote terminal, switch the working state according to the state switching instruction sent by the inspection robot, and operate the target object by controlling the mechanical gripper on the mechanical arm according to the equipment operation instruction sent by the remote terminal.

Images