A multi-frame motion image fusion method and system for robot vision

Abstract

The invention relates to the technical field of image processing, and provides a multi-frame motion image fusion method of robot vision, wherein a robot is continuously photographed moving in the same scene to obtain a plurality of continuously photographed images; the plurality of photographed images are long-exposure images and short-exposure images obtained according to the motion posture of the robot; a weight coefficient of the long-exposure image is obtained based on the image backlight brightness; the long-exposure image is corrected based on a principal component analysis method; and image fusion is performed according to the weight coefficient, the corrected long-exposure image and the short-exposure image to obtain a fused image. In view of the fact that the long-exposure image has lower noise and higher signal-to-noise ratio than the short-exposure image as a whole, the principal component analysis method is used to extract the image of the well-exposed area of ​​the long-exposure image, and the image is given a greater fusion weight, thereby increasing its influence on the result of image fusion, reducing the influence of the short-exposure image on the result of image fusion, and ensuring the smooth transition of the fused image.

Description

Technical Field

[0001] The invention of the application relates to the field of image processing technology, and in particular to a multi-frame motion image fusion method and system for robot vision. Background Art

[0002] The multi-frame image fusion algorithm has been widely used in the industry because of its better denoising effect than single-frame and better preservation of detail texture. Considering memory and algorithm running time, the commonly used solution is: the camera takes 5-10 images continuously, selects one of them as the reference frame, performs image registration on the other images, and performs multi-frame image fusion on the registered image to obtain the denoised image.

[0003] For actual scenes, dynamic range refers to the ratio of the brightness of the brightest and darkest parts of the scene. The dynamic range of the image is expanded by processing the low dynamic range image, that is, by adjusting the overexposed and underexposed parts of the image, the details therein are enhanced to make it closer to the image captured by the human eye. In the process of image acquisition and display, due to the limitations of the CCD / CMOS image sensor and ordinary display in the camera, the generated image has poor details in the overbright or dark areas, so the brightness dynamic range of the image is very low. Therefore, an accurate multi-frame motion image fusion method is urgently needed to enhance the clarity of the image. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a multi-frame image fusion method, which includes: obtaining multiple continuously shot images by continuously shooting a robot moving in the same scene; and the multiple shot images are long-exposure images and short-exposure images obtained according to the robot's motion posture; obtaining a weight coefficient of the long-exposure image based on the image backlight brightness;

[0005] Correction of long exposure images based on principal component analysis;

[0006] Image fusion is performed according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image.

[0007] The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ;

[0008] d ij =M ij -A ij

[0009] dij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness;

[0010] B ij =A ij +K*d ij

[0011] Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness.

[0012]

[0013] where Y min and Y max are the minimum and maximum thresholds of brightness in the long exposure image, respectively.

[0014] The correction of long exposure images based on principal component analysis includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g 1 ,g 2 …g n}, where n is the number of blocks, and the average value of the image is calculated, Then each column vector g 1 Subtract a to get the de-mean vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x 1 ,x 2 …x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F,

[0015] F=Xμ max +a

[0016] Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

[0017] The image fusion comprises:

[0018] Hij =ZijP i,j,L +P i,j,S (256-Z ij )

[0019] Where P i,j,S This is a short exposure image.

[0020] The beneficial effect of the present application lies in that, in view of the fact that long exposure images have lower noise and higher signal-to-noise ratio than short exposure images as a whole, the principal component analysis method is used to extract images of well-exposed areas of long exposure images, give them a larger fusion weight, increase their influence on the results of image fusion, reduce the influence of short exposure images on the results of image fusion, and ensure a smooth transition of the fused image. DETAILED DESCRIPTION

[0021] The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way.

[0022] It should be noted that although the expressions "first", "second", etc. are used herein to describe different modules, steps, data, etc. of the embodiments of the present invention, the expressions "first", "second", etc. are only used to distinguish between different modules, steps, data, etc., and do not indicate a specific order or importance. In fact, the expressions "first", "second", etc. can be used interchangeably.

[0023] The embodiment of the present application relates to a multi-frame motion image fusion method and system for robot vision, including: acquiring multiple continuously shot images by continuously shooting a robot moving in the same scene; and the multiple shot images are long-exposure images and short-exposure images acquired according to the robot's motion posture;

[0024] Obtaining a weight coefficient for a long exposure image based on the image backlight brightness;

[0025] Correction of long exposure images based on principal component analysis;

[0026] Image fusion is performed according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image.

[0027] The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ;

[0028] d ij =Mij -A ij

[0029] d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness;

[0030] B ij =A ij +K*d ij

[0031] Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness.

[0032]

[0033] where Y min and Y max are the minimum and maximum thresholds of brightness in the long exposure image, respectively.

[0034] The correction of long exposure images based on principal component analysis includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g 1 ,g 2 …g n}, where n is the number of blocks, and the average value of the image is calculated, Then each column vector g 1 Subtract a to get the de-mean vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x 1 ,x 2 …x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F,

[0035] F=Xμ max +a

[0036] Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

[0037] The image fusion comprises:

[0038] H ij =ZijP i,j,L +P i,j,S (256-Z ij )

[0039] Where P i,j,S This is a short exposure image.

[0040] Among them, a predetermined motion trajectory or motion path is set according to the actual work task needs of the robot, and the parameter information of the robot is collected during the movement of the robot, so as to determine the posture parameter information of the robot during the movement. The parameter information is collected according to the displacement change or angle change of the robot based on the inertial sensor or the angle sensor, and according to the characteristics of the posture change, the camera exposure time is further limited. For large changes in angle displacement or continuous movements, long exposure images are used to ensure the integrity of the motion information collected by the collected images. At the same time, the long exposure image and the short exposure image are simultaneously photographed by different cameras for the same scene. The supporting hardware facilities of this application include a memory, a controller, a camera and a processor, wherein the processor includes an image acquisition module to acquire a long exposure image and a short exposure image; a long exposure image correction module to extract an image of a well-exposed area of ​​the long exposure image using a principal component analysis method, and a long exposure image weight analysis module to give it a larger fusion weight according to the extracted long exposure image correction area to increase its influence on the result of image fusion; the image fusion module performs image fusion according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image.

[0041] The functions implemented by each module in the device correspond to the steps in the method described above. For the specific technical effects achieved, please refer to the description of the method steps above, which will not be repeated here.

[0042] In the embodiment of the present invention, the processor can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0043] The memory in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD), etc.

[0044] In some embodiments, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, perform any of the methods described above.

[0045] In view of the fact that long exposure images have lower noise and higher signal-to-noise ratio than short exposure images as a whole, this application adopts the principal component analysis method to extract images of well-exposed areas of long exposure images, give them a larger fusion weight, increase their influence on the image fusion results, reduce the influence of short exposure images on the image fusion results, and ensure a smooth transition of the fused image.

[0046] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0047] The methods and apparatus of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0048] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

Claims

1. A multi-frame motion image fusion method for robot vision, characterized in that: include: A plurality of continuously shot images are obtained by continuously shooting the robot moving in the same scene; and the plurality of shot images are long-exposure images and short-exposure images obtained according to the movement posture of the robot; Obtaining a weight coefficient for a long exposure image based on the image backlight brightness; Correction of long exposure images based on principal component analysis; Perform image fusion according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image; The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ; d ij =M ij -A ij d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness; B ij =A ij +K*d ij Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness. where Y min and Y max are the minimum and maximum thresholds of brightness in long exposure images respectively; The method of correcting the long exposure image based on the principal component analysis method includes: dividing the long exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2…g n }, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F, F=Xμ max +a Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

2. The multi-frame motion image fusion method of robot vision according to claim 1, characterized in that: The image fusion comprises: H ij =ZijP i,j,L +P i,j,S (256-Z ij ) Where P i,j,S This is a short exposure image.

3. The multi-frame motion image fusion method of robot vision according to claim 1, characterized in that: Set the robot's motion path, collect the robot's motion posture information during the robot's motion, and determine the exposure time of the collected image based on the posture information.

4. The multi-frame motion image fusion method of robot vision according to claim 1, characterized in that: The robot’s position information is collected through inertial sensors.

5. The multi-frame motion image fusion method of robot vision according to claim 1, wherein: The long exposure image and the short exposure image are simultaneously captured by different cameras for the same scene.

6. A multi-frame image fusion system, wherein: include: An image acquisition module, for acquiring a long exposure image and a short exposure image; Long exposure image weight analysis module, long exposure image correction module and image fusion module; The image fusion module performs image fusion according to the weight coefficient, the corrected long exposure image and the short exposure image to obtain a fused image; The weight coefficients include: converting the long exposure image into an RGB format image, and calculating the average value A of the R, G, and B components for the brightness value of the image pixel (i, j) ij and the maximum value M ij , and calculate the difference d between the two ij ; d ij =M ij -A ij d ij Indicates the brightness range that the current pixel can increase based on the average value, and calculates the backlight brightness; B ij =A ij +K*d ij Among them, k represents the backlight brightness increase coefficient, and the weight coefficient is calculated based on the calculated backlight brightness. where Y min and Y max are the minimum and maximum thresholds of brightness in long exposure images respectively; Correcting long-exposure images based on principal component analysis includes: dividing the long-exposure image into blocks, reducing noise and performing Gaussian filtering on the divided images, and converting each image matrix into an N 2 ×1 column vector g i , and form a set {g1,g2…g n }, where n is the number of blocks, and the average value of the image is calculated, Then subtract a from each column vector g1 to get the de-averaged vector x i , i = 1, 2, ..., n and form all vectors into a matrix X = [x1, x2 ... x n ], where X is N 2 ×n size matrix, and calculate the covariance matrix C of the long exposure image, Perform eigendecomposition on the covariance matrix C and obtain the eigenvalue λ i , i = 1, 2, ..., n, and the eigenvector μ i , i = 1, 2, ..., n, select the first principal component Xμ corresponding to the maximum eigenvalue λ max As the high-frequency component of the image, the mean vector a is the low-frequency component of the image, and the two are added to obtain the vector F, F=Xμ max +a Finally, the vector F is converted into a matrix of size N*N, which is the corrected long exposure image P. i,j,L .

7. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the multi-frame motion image fusion method of robot vision according to any one of claims 1 to 5 is executed.

Images