Image acquisition method and system for robots used in live working

Abstract

The invention of the present application relates to the field of image processing technology, and in particular to an image acquisition method and system for a robot for live working, which detects whether the working environment of the robot for live working includes a target object. When the target object is detected, the image acquisition device is triggered to acquire the image of the target object, and a plurality of continuously shot images are acquired by continuously shooting the target object. The image acquisition device on the robot for live working can acquire information on different attributes of the target object by controlling the exposure time, and in view of the fact that long exposure images have lower noise and higher signal-to-noise ratio than short exposure images as a whole, the principal component analysis method is used to extract images of well-exposed areas of the long exposure images, and the images are given a larger fusion weight to increase their influence on the results of image fusion, reduce the influence of the short exposure images on the results of image fusion, and ensure a smooth transition of the images acquired after the fusion.

Description

Technical Field

[0001] The invention of the present application relates to the field of image processing technology, and in particular to an image acquisition method and system for a robot for live-line working. Background Art

[0002] The multi-frame image fusion algorithm has been widely used in the industry because of its better denoising effect than single-frame and better preservation of detail texture. Considering memory and algorithm running time, the commonly used solution is: the camera takes 5-10 images continuously, selects one of them as the reference frame, performs image registration on the other images, and performs multi-frame image fusion on the registered image to obtain the denoised image.

[0003] For actual scenes, dynamic range refers to the ratio of the brightness of the brightest and darkest parts of the scene. The dynamic range of the image is expanded by processing the low dynamic range image, that is, by adjusting the overexposed and underexposed parts of the image, the details therein are enhanced to make it closer to the image captured by the human eye. In the process of image acquisition and display, due to the limitations of the CCD / CMOS image sensor and ordinary display in the camera, the generated image has poor details in the overbright or dark areas, so the brightness dynamic range of the image is very low. Therefore, an accurate multi-frame motion image fusion method is urgently needed to enhance the clarity of the image. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, an image acquisition method for a live working robot is invented, which includes detecting whether the working environment of the live working robot includes a target object, and when the target object is detected, triggering an image acquisition device to acquire an image of the target object, and acquiring multiple continuously shot images by continuously shooting the target object;

[0005] When the target object is detected to be static, a short exposure image is acquired;

[0006] When the target object is detected to be dynamic, a long exposure image is acquired;

[0007] When the acquired image includes a long exposure image and a short exposure image, obtaining a weight coefficient of the long exposure image based on the image backlight brightness;

[0008] Correction of long exposure images based on principal component analysis;

[0009] Image fusion is performed according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image.

[0010] The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ijand the maximum value M ij , and calculate the difference d between the two ij ;

[0011] d ij =M ij -A ij

[0012] d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness;

[0013] B ij =A ij +K*d ij

[0014] Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness.

[0015]

[0016] where Y min and Y max are the minimum and maximum thresholds of brightness in the long exposure image, respectively.

[0017] The correction of long exposure images based on principal component analysis includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2…g n}, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F,

[0018] F=Xμ max +a

[0019] Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

[0020] The image fusion comprises:

[0021] H ij =Z ij P i,j,L +P i,j,S (256-Z ij )

[0022] Where P i,j,S This is a short exposure image.

[0023] The beneficial effect of the present application is that the image acquisition device on the robot for live working can collect information on different attributes of the target object by controlling the exposure time, and in view of the fact that long exposure images have lower noise and higher signal-to-noise ratio than short exposure images as a whole, the principal component analysis method is used to extract images of well-exposed areas of long exposure images, give them a larger fusion weight, increase their influence on the results of image fusion, reduce the influence of short exposure images on the results of image fusion, and ensure a smooth transition of the images acquired after fusion. DETAILED DESCRIPTION

[0024] The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0025] It should be noted that although the expressions "first", "second", etc. are used herein to describe different modules, steps, data, etc. of the embodiments of the present invention, the expressions "first", "second", etc. are only used to distinguish between different modules, steps, data, etc., and do not indicate a specific order or importance. In fact, the expressions "first", "second", etc. can be used interchangeably.

[0026] The embodiment of the present application relates to an image acquisition method for a live-line working robot, which includes detecting whether the working environment of the live-line working robot includes a target object, and when the target object is detected, triggering an image acquisition device to acquire an image of the target object, and acquiring a plurality of continuously shot images by continuously shooting the target object; when the target object is detected to be static, capturing a short-exposure image;

[0027] When the target object is detected to be dynamic, a long exposure image is acquired;

[0028] When the acquired image includes a long exposure image and a short exposure image, obtaining a weight coefficient of the long exposure image based on the image backlight brightness;

[0029] Correction of long exposure images based on principal component analysis;

[0030] Image fusion is performed according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image.

[0031] The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ;

[0032] d ij =M ij -A ij

[0033] d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness;

[0034] B ij =A ij +K*d ij

[0035] Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness.

[0036]

[0037] where Y min and Y max are the minimum and maximum thresholds of brightness in the long exposure image, respectively.

[0038] The correction of long exposure images based on principal component analysis includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2...g n}, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i, i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F,

[0039] F=Xμ max +a

[0040] Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

[0041] The image fusion comprises:

[0042] H ij =Z ij P i,j,L +P i,j,S (256-Z ij )

[0043] Where P i,j,S This is a short exposure image.

[0044] The present application also relates to an image acquisition system for a robot for live working, which includes an image acquisition module, which can acquire long-exposure images and short-exposure images; an infrared sensor, which is installed on the robot and is used to collect information about the target object. The image acquisition module is a camera installed on the front end of the robot and can rotate on the robot; when the infrared sensor detects the target object, the camera can rotate to align with the target object and then take a picture, a long-exposure image weight analysis module, a long-exposure image correction module and an image fusion module; the image fusion module performs image fusion according to the weight coefficient, the corrected long-exposure image and the short-exposure image to obtain a fused image.

[0045] The functions implemented by each module in the device correspond to the steps in the method described above. For the specific technical effects achieved, please refer to the description of the method steps above, which will not be repeated here.

[0046] In the embodiment of the present invention, the processor can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0047] The memory in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD), etc.

[0048] In some embodiments, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, perform any of the methods described above.

[0049] In view of the fact that long exposure images have lower noise and higher signal-to-noise ratio than short exposure images as a whole, this application adopts the principal component analysis method to extract images of well-exposed areas of long exposure images, give them a larger fusion weight, increase their influence on the image fusion results, reduce the influence of short exposure images on the image fusion results, and ensure a smooth transition of the fused image.

[0050] In certain circumstances, multitasking and parallel processing may be advantageous.

[0051] The methods and apparatus of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0052] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

Claims

1. A method for acquiring images of a robot for live working, characterized in that: include: Detecting whether the working environment of the robot for live working includes a target object, and when the target object is detected, triggering an image acquisition device to acquire an image of the target object, and acquiring a plurality of continuously shot images by continuously shooting the target object; When the target object is detected to be static, a short exposure image is acquired; When the target object is detected to be dynamic, a long exposure image is acquired; When the acquired image includes a long exposure image and a short exposure image, obtaining a weight coefficient of the long exposure image based on the image backlight brightness; Correction of long exposure images based on principal component analysis; Perform image fusion according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image; The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ; d ij =M ij -A ij d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness; B ij =A ij +K*d ij Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness. where Y min and Y max are the minimum and maximum thresholds of brightness in long exposure images respectively; The method of correcting the long exposure image based on the principal component analysis method includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2…g n }, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F, F=Xμ max +a Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L ; The image fusion comprises: H ij =Z ij P i,j,L +P i,j,S (256-Z ij ) Where P i,j,S This is a short exposure image.

2. The image acquisition method of the robot for live working according to claim 1, characterized in that: Set the robot's motion path, collect the robot's motion posture information during the robot's motion, and determine the exposure time of the collected image based on the posture information.

3. The image acquisition method of the robot for live working according to claim 2, characterized in that: The robot’s position information is collected through inertial sensors.

4. The image acquisition method of the robot for live working according to claim 1, wherein: The long exposure image and the short exposure image are simultaneously captured by different cameras for the same scene.

5. An image acquisition system for a robot for live working, wherein: include: An image acquisition module, which acquires long-exposure images and short-exposure images; an infrared sensor, which is installed on the robot and is used to collect information about the target object. The image acquisition module is a camera installed on the front end of the robot and can rotate on the robot; a weight analysis module for long-exposure images, a long-exposure image correction module, and an image fusion module; The image fusion module performs image fusion according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image; The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ; d ij =M ij -A ij d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness; B ij =A ij +K*d ij Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness. where Y min and Y max are the minimum and maximum thresholds of brightness in long exposure images respectively; The weight analysis module of the long exposure image includes correcting the long exposure image based on the principal component analysis method; the correction of the long exposure image based on the principal component analysis method includes: dividing the long exposure image into blocks, reducing noise and Gaussian filtering the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2…g n }, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F, F=Xμ max +a Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L ; The image fusion comprises: H ij =Z ij P i,j,L +P i,j,S (256-Z ij ) Where P i,j,S This is a short exposure image.

6. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image acquisition method for a robot for live-line working according to any one of claims 1 to 4 is executed.

Images