Image processing method and device, equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises the following steps that: binocular equipment respectively carries out image acquisition on the same scene through two monocular cameras to obtain two original monocular images; obtaining perspective transformation matrixes corresponding to the two monocular cameras respectively; for each monocular camera, performing assembly error correction on the original monocular image acquired by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain an assembly correction image corresponding to the monocular camera; and carrying out image splicing on the assembly correction images corresponding to the two monocular cameras to obtain a binocular spliced image. By adopting the method, the quality of the binocular spliced image finally obtained by the binocular equipment can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] As a fundamental task in computer vision, binocular vision is considered to enhance monocular vision with limited field of view due to optical design. By arranging or stacking two monocular cameras in a specific manner, tasks such as panoramic observation and stereoscopic vision observation can be achieved. Developed to date, binocular panoramic observation devices (i.e., binocular devices) have relatively mature technical routes and systems, and there are also many binocular panoramic observation devices based on existing technical routes in the market. Generally, it is considered that a binocular panoramic observation device needs to be equipped with two cameras, arranged or stacked in parallel at a certain angle, and ensure that the observation fields of the two cameras have a certain degree of overlap, so that the overall binocular device has a much larger field of view than that of a single monocular camera itself, achieving the purpose of panoramic observation.

[0003] In traditional technologies, usually the images collected by the two monocular cameras in the binocular device are directly stitched to obtain a binocular stitched image. Although the binocular device can greatly expand the observation range in this way, the quality of the finally obtained binocular stitched image is poor. Summary of the Invention

[0004] Based on this, it is necessary to provide an image processing method, apparatus, device, storage medium, and program product that can improve the quality of binocular stitched images for the above technical problems.

[0005] In a first aspect, this application provides an image processing method applied to a binocular device including two monocular cameras. The method includes:

[0006] Acquire two original monocular images by respectively collecting images of the same scene through the two monocular cameras;

[0007] Obtain the perspective transformation matrices respectively corresponding to the two monocular cameras;

[0008] For each monocular camera, perform assembly error correction on the original monocular image collected by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain an assembly-corrected image corresponding to the monocular camera;

[0009] Stitch the assembly-corrected images respectively corresponding to the two monocular cameras to obtain a binocular stitched image.

[0010] In a second aspect, this application provides an image processing apparatus applied to a binocular device including two monocular cameras. The apparatus includes:

[0011] An acquisition module, configured to respectively acquire images of the same scene through the two monocular cameras, and obtain two original monocular images;

[0012] An acquisition module, configured to obtain perspective transformation matrices respectively corresponding to the two monocular cameras;

[0013] A correction module, configured to, for each monocular camera, perform assembly error correction on the original monocular image acquired by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, and obtain an assembly-corrected image corresponding to the monocular camera;

[0014] A stitching module, configured to stitch the assembly-corrected images respectively corresponding to the two monocular cameras to obtain a binocular stitched image.

[0015] In a third aspect, the present application provides a binocular device, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the method embodiments of the present application are implemented.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the method embodiments of the present application are implemented.

[0017] In a fifth aspect, the present application provides a computer program product including a computer program, and when the computer program is executed by a processor, the steps in the method embodiments of the present application are implemented.

[0018] The above image processing method, apparatus, device, storage medium and program product. The binocular device acquires images of the same scene through two monocular cameras respectively, obtaining two original monocular images; obtains the perspective transformation matrices corresponding to the two monocular cameras respectively; for each monocular camera, performs assembly error correction on the original monocular image acquired by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, obtaining the assembly-corrected image corresponding to the monocular camera; stitches the assembly-corrected images corresponding to the two monocular cameras respectively to obtain a binocular stitched image. Compared with the traditional binocular device that directly stitches the images acquired by the two monocular cameras in the binocular device to obtain a binocular stitched image, in this application, by obtaining the perspective transformation matrices corresponding to the respective monocular cameras in the binocular device, and performing assembly error correction on the original monocular images acquired by the respective monocular cameras according to the perspective transformation matrices corresponding to the respective monocular cameras, and then stitching the two assembly-corrected images obtained after the assembly error correction. In this way, it can effectively avoid the large differences in the observation effects of the two monocular cameras caused by the insufficient assembly accuracy of the two monocular cameras during the production process of the binocular device, that is, the geometric features of the binocular vision homologous objects are inconsistent, there are offsets and obvious bends at the panoramic stitching, etc., which seriously affect the panoramic observation effect, thereby improving the quality of the binocular stitched image finally obtained by the binocular device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flowchart of the image processing method in one embodiment;

[0020] Figure 2 It is a schematic diagram of the binocular device acquiring images in one embodiment;

[0021] Figure 3A It is a schematic diagram of the ideal observation effect of the binocular device in one embodiment;

[0022] Figure 3B It is a schematic diagram of the observation effect of the binocular device with assembly errors in one embodiment;

[0023] Figure 3C It is a schematic diagram of the observation effect of the binocular device with assembly errors in another embodiment;

[0024] Figure 4A It is a schematic diagram of a normal checkerboard calibration image in one embodiment;

[0025] Figure 4B It is a schematic diagram of a barrel-distorted checkerboard calibration image in one embodiment;

[0026] Figure 4C It is a schematic diagram of a pillow-distorted checkerboard calibration image in one embodiment;

[0027] Figure 4D Schematic diagram of a checkerboard calibration image with tangential distortion in an embodiment;

[0028] Figure 5A Schematic diagram of a binocular stitching image obtained by using a traditional image processing method in an embodiment;

[0029] Figure 5B Schematic diagram of a binocular stitching image obtained by using a traditional image processing method in another embodiment;

[0030] Figure 6 Schematic flow diagram of an image processing method in another embodiment;

[0031] Figure 7A Schematic diagram of a calibration field image collected by the left monocular camera in a binocular device in an embodiment;

[0032] Figure 7B Schematic diagram of the original calibration field image collected by the right monocular camera in a binocular device in an embodiment;

[0033] Figure 8A Schematic diagram of the calibration field image corresponding to the left monocular camera obtained after processing by the image processing method of the present application in an embodiment;

[0034] Figure 8B Schematic diagram of the calibration field image corresponding to the right monocular camera obtained after processing by the image processing method of the present application in an embodiment;

[0035] Figure 9A Schematic diagram of a binocular stitching image obtained after processing by the image processing method of the present application in an embodiment;

[0036] Figure 9B Schematic diagram of a binocular stitching image obtained after processing by the image processing method of the present application in another embodiment;

[0037] Figure 10 Schematic block diagram of an image processing device in an embodiment;

[0038] Figure 11 Internal structure diagram of a binocular device in an embodiment. Detailed implementation manners

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] In one embodiment, as Figure 1As shown, an image processing method is provided, which is applied to a binocular device including two monocular cameras, and includes the following steps:

[0041] Step 102, respectively acquire images of the same scene through two monocular cameras to obtain two original monocular images.

[0042] In one embodiment, two monocular cameras, namely a left-eye camera and a right-eye camera, are loaded in the binocular device. As Figure 2 shown, the binocular device can respectively acquire images of the same scene through the left-eye camera and the right-eye camera to obtain two original monocular images. It can be understood that one original monocular image is acquired by the left-eye camera, and the other original monocular image is acquired by the right-eye camera.

[0043] Step 104, obtain perspective transformation matrices respectively corresponding to the two monocular cameras.

[0044] In one embodiment, the binocular device can respectively acquire images of the same calibration field through two monocular cameras to obtain two calibration field images containing calibration points. For each monocular camera, the binocular device can calculate the perspective transformation matrix corresponding to the monocular camera based on the position information of the calibration points in the real space and the calibration field images acquired by the monocular camera.

[0045] In one embodiment, the perspective transformation matrices respectively corresponding to the two monocular cameras can be directly read from the memory of the binocular device, or can be directly obtained from a third-party device. Specifically, the third-party device can store the perspective transformation matrices respectively corresponding to the two monocular cameras, and the third-party device can send the perspective transformation matrices respectively corresponding to the two monocular cameras to the binocular device, and the binocular device can receive the perspective transformation matrices respectively corresponding to the two monocular cameras sent by the third-party device.

[0046] Step 106, for each monocular camera, perform assembly error correction on the original monocular image acquired by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain an assembly correction image corresponding to the monocular camera.

[0047] It can be understood that during the production process of the binocular device, there is insufficient assembly accuracy for the two monocular cameras, resulting in a large difference in the observation effects of the two monocular cameras, that is, the geometric features of the binocular vision homonymous objects are inconsistent, which will lead to serious problems such as offset and obvious bending at the panoramic stitching, seriously affecting the panoramic observation effect. Therefore, by respectively performing assembly error correction on the two original monocular images acquired by the binocular device, the image quality problems caused by insufficient assembly accuracy of the two monocular cameras during the production process of the binocular device can be avoided.

[0048] In one embodiment, for each monocular camera, the binocular device can perform perspective transformation processing on the original monocular image collected by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, so as to correct the assembly error of the original monocular image collected by the monocular camera, and obtain the assembly correction image corresponding to the monocular camera.

[0049] Step 108: Stitch the assembly correction images corresponding to the two monocular cameras respectively to obtain a binocular stitched image.

[0050] In one embodiment, the binocular device can directly stitch the assembly correction images corresponding to the two monocular cameras respectively to obtain a binocular stitched image.

[0051] In the above image processing method, the binocular device collects images of the same scene through two monocular cameras respectively to obtain two original monocular images; obtains the perspective transformation matrices corresponding to the two monocular cameras respectively; for each monocular camera, performs assembly error correction on the original monocular image collected by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain the assembly correction image corresponding to the monocular camera; stitches the assembly correction images corresponding to the two monocular cameras respectively to obtain a binocular stitched image. Compared with the traditional method of directly stitching the images collected by the two monocular cameras in the binocular device to obtain a binocular stitched image, in this application, by obtaining the perspective transformation matrices corresponding to the respective monocular cameras in the binocular device and performing assembly error correction on the original monocular images collected by the respective monocular cameras according to the perspective transformation matrices corresponding to the respective monocular cameras, and then stitching the two assembly correction images obtained after the assembly error correction. In this way, it can effectively avoid the large difference in the observation effects of the two monocular cameras caused by the insufficient assembly accuracy of the two monocular cameras during the production process of the binocular device, that is, the geometric features of the binocular vision homologous objects are inconsistent, there are offsets and obvious bends at the panoramic stitching, etc., which seriously affect the panoramic observation effect, thereby improving the quality of the binocular stitched image finally obtained by the binocular device.

[0052] It can be understood that in an ideal state, the observation effect of the binocular device should be as Figure 3A shown. However, due to the problem of insufficient assembly accuracy of the two monocular cameras during the production process of the binocular device, the observation effect of the binocular device will appear as Figure 3B shown or as Figure 3C shown. In this application, by obtaining the perspective transformation matrices corresponding to the respective monocular cameras in the binocular device and performing assembly error correction on the original monocular images collected by the respective monocular cameras according to the perspective transformation matrices corresponding to the respective monocular cameras, and then stitching the two assembly correction images obtained after the assembly error correction, the quality of the binocular stitched image finally obtained by the binocular device can be improved.

[0053] In one embodiment, for each monocular camera, the original monocular image collected by the monocular camera is corrected for assembly error according to the perspective transformation matrix corresponding to the monocular camera to obtain an assembly-corrected image corresponding to the monocular camera, including: for each monocular camera, obtaining the distortion parameters corresponding to the monocular camera; performing optical distortion correction on the original monocular image collected by the monocular camera according to the distortion parameters corresponding to the monocular camera to obtain a distortion-corrected image corresponding to the monocular camera; and performing assembly error correction on the distortion-corrected image corresponding to the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain an assembly-corrected image corresponding to the monocular camera.

[0054] It can be understood that, as Figure 4A shown, in an ideal state, the images collected by the monocular cameras in the binocular device should not be distorted. However, due to the design structure and optical characteristics of the monocular camera (including the optical lens and sensor, etc.), the incident light will be refracted after passing through the lens group, and the projection position on the sensor will change, resulting in various forms of distortion of the image. According to the direction of the optical axis where the distortion effect is located, it can be roughly divided into radial distortion and tangential distortion. Radial distortion is the position offset on the sensor formed by the refraction of light through the lens group in the x-axis direction of the camera, including barrel distortion as Figure 4B shown and pincushion distortion as Figure 4C shown. The tangential distortion as Figure 4D shown is the position offset in the y-axis direction of the camera due to the fact that the sensor is not exactly perpendicular to the theoretical incident direction of the light, usually manifested as a passive perspective transformation, that is, the parallelism of the geometric relationship on the image is destroyed. These geometric distortion phenomena destroy the original geometric features of the image and have a serious impact on the panoramic observation result of the binocular device. Therefore, it is necessary to perform distortion correction on each monocular camera in the binocular device before image stitching to ensure that the image can display the correct geometric relationship.

[0055] In one embodiment, the binocular device can collect images of a black and white checkerboard through two monocular cameras respectively to obtain two checkerboard calibration images containing calibration points. For each monocular camera, the binocular device can calculate the distortion parameters corresponding to the monocular camera based on the position information of the calibration points in the real space and the checkerboard calibration image collected by the monocular camera.

[0056] In one embodiment, the distortion parameters corresponding to the two monocular cameras can be directly read from the memory of the binocular device, or can be directly obtained from a third-party device. Specifically, the third-party device may store the distortion parameters corresponding to the two monocular cameras respectively, and the third-party device may send the distortion parameters corresponding to the two monocular cameras respectively to the binocular device, and the binocular device may receive the distortion parameters corresponding to the two monocular cameras respectively sent by the third-party device.

[0057] In the above embodiment, by first performing optical distortion correction on the original monocular image collected by the monocular camera according to the distortion parameters corresponding to the monocular camera to obtain the distortion-corrected image corresponding to the monocular camera, and then further performing assembly error correction on the distortion-corrected image corresponding to the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain the assembly-corrected image corresponding to the monocular camera, the quality of the assembly-corrected image can be further improved, thereby further improving the quality of the binocular stitching image finally obtained by the binocular device.

[0058] In one embodiment, for each monocular camera, obtaining the distortion parameters corresponding to the monocular camera includes: for each monocular camera, collecting an image of the first black-and-white checkerboard through the monocular camera to obtain a checkerboard calibration image corresponding to the monocular camera and including the first calibration points; the first calibration points are the checkerboard inner corner points of the first black-and-white checkerboard; according to the camera parameters of the monocular camera and the first actual space coordinates of the first calibration points in the real space, determining the first ideal image coordinates of the first calibration points in the checkerboard calibration image corresponding to the monocular camera, and according to the difference between the first ideal image coordinates and the first real image coordinates of the first calibration points in the checkerboard calibration image corresponding to the monocular camera, determining the distortion parameters corresponding to the monocular camera.

[0059] In the above embodiment, by introducing the checkerboard inner corner points of the black-and-white checkerboard as calibration points, and then calculating the distortion parameters corresponding to the monocular camera based on the coordinates of the calibration points in the actual space and the checkerboard calibration image respectively, the accuracy of the finally obtained distortion parameters can be improved.

[0060] In one embodiment, obtaining perspective transformation matrices corresponding to two monocular cameras respectively includes: acquiring, by the two monocular cameras respectively, images of the same calibration field to obtain two calibration field images including second calibration points; the calibration field includes at least six second black-and-white checkerboards to form two rectangular calibration planes with a preset angle; the preset angle is consistent with the sensor angle of the binocular device; the second calibration points are the inner corner points of the checkerboards of the second black-and-white checkerboards; for each monocular camera, determining the second ideal image coordinates of the second calibration points in the calibration field image acquired by the monocular camera according to the camera parameters of the monocular camera and the second actual space coordinates of the second calibration points in the real space, and determining the perspective transformation matrix corresponding to the monocular camera according to the second ideal image coordinates and the second real image coordinates of the second calibration points in the calibration field image acquired by the monocular camera.

[0061] In the above embodiment, a calibration field is constructed by black-and-white checkerboards, and the inner corner points of the checkerboards of the black-and-white checkerboards are used as calibration points. Furthermore, based on the coordinates of the calibration points in the actual space and the calibration field image respectively, the perspective transformation matrix corresponding to the monocular camera is calculated, which can improve the accuracy of the finally obtained perspective transformation matrix.

[0062] In one embodiment, image stitching of the assembly correction images corresponding to the two monocular cameras respectively to obtain a binocular stitched image includes: projecting the assembly correction images corresponding to the two monocular cameras respectively onto the cylindrical surfaces corresponding to the two monocular cameras for image stitching to obtain a binocular stitched image.

[0063] In one embodiment, each of the two monocular cameras on the binocular device has its own sensor. The binocular device can construct a cylindrical surface with the midpoint of the line connecting the two sensors as the cylinder center to obtain the cylindrical surface jointly corresponding to the two monocular cameras. Furthermore, the binocular device can project the assembly correction images corresponding to the two monocular cameras respectively onto the cylindrical surface jointly corresponding to the two monocular cameras for image stitching to obtain a binocular stitched image.

[0064] In one embodiment, each of the two monocular cameras on the binocular device has its own sensor. The binocular device can construct a cylindrical surface with the center of one of the sensors as the cylinder center to obtain the cylindrical surface jointly corresponding to the two monocular cameras. Furthermore, the binocular device can project the assembly correction images corresponding to the two monocular cameras respectively onto the cylindrical surface jointly corresponding to the two monocular cameras for image stitching to obtain a binocular stitched image.

[0065] In the above embodiments, by projecting the assembly calibration images corresponding to the two monocular cameras onto the cylindrical surfaces corresponding to the two monocular cameras respectively for image stitching, it is possible to make the assembly calibration images corresponding to the two monocular cameras transition smoothly during stitching, avoiding the situation where the corresponding points at the seam are not stitched smoothly, thereby improving the quality of the binocular stitching image finally obtained by the binocular device.

[0066] In one embodiment, projecting the assembly calibration images corresponding to the two monocular cameras onto the cylindrical surfaces corresponding to the two monocular cameras respectively for image stitching to obtain a binocular stitching image includes: for each monocular camera, constructing a cylindrical surface with the center of the sensor in the monocular camera as the cylinder center to obtain the cylindrical surface corresponding to the monocular camera, projecting the assembly calibration image corresponding to the monocular camera onto the cylindrical surface corresponding to the monocular camera to obtain the projected image corresponding to the monocular camera; performing image stitching on the projected images corresponding to the two monocular cameras respectively to obtain a binocular stitching image.

[0067] It can be understood that the two monocular cameras in the binocular device perform cylindrical projection independently, that is, the binocular device can respectively construct two cylindrical surfaces with the sensor centers of the cameras as the cylinder centers. Therefore, the projected surfaces of the projected images in the overlapping range of the field of view angles will not actually coincide completely. However, since the observation distance of the binocular device is usually far, after projection transformation calculation, the slight differences in the overlapping part of the two cylindrical projection surfaces within the field of view angle can be ignored. Therefore, at a specific observation distance, it can be approximately considered that the projected images corresponding to the two monocular cameras are projected onto different parts of the same cylindrical surface.

[0068] In the above embodiments, by respectively constructing cylindrical surfaces with the centers of the sensors in each monocular camera as the cylinder centers, projecting the assembly calibration images corresponding to each monocular camera onto the corresponding cylindrical surfaces of the respective monocular cameras and then performing image stitching, the quality of the binocular stitching image finally obtained by the binocular device can be further improved.

[0069] It can be understood that in the traditional image processing method of directly stitching the images collected by the two monocular cameras in the binocular device, although the binocular device can greatly expand the observation range, the quality of the binocular stitching image finally obtained is poor. As Figure 5A and Figure 5B shown, it can be clearly seen that there are significant differences in the observation effects of the two monocular cameras, that is, the geometric features of the binocular vision corresponding objects are inconsistent, and there are offsets and obvious bends at the panoramic stitching, and the occurrence of these situations seriously affects the effect of panoramic observation.

[0070] In one embodiment, as Figure 6As shown, the binocular device can respectively collect images of the same scene through two monocular cameras to obtain two original monocular images. By respectively collecting images of the same calibration field through two monocular cameras, two calibration field images containing second calibration points are obtained; the calibration field includes at least six second black-and-white checkerboards to form two rectangular calibration planes with a preset angle; the preset angle is consistent with the sensor angle of the binocular device; the second calibration point is the inner corner point of the checkerboard of the second black-and-white checkerboard. For each monocular camera, according to the camera parameters of the monocular camera and the second actual space coordinates of the second calibration point in the real space, the second ideal image coordinates of the second calibration point in the calibration field image collected by the monocular camera are determined, and according to the second ideal image coordinates and the second real image coordinates of the second calibration point in the calibration field image collected by the monocular camera, the perspective transformation matrix corresponding to the monocular camera is determined. For each monocular camera, by collecting an image of the first black-and-white checkerboard through the monocular camera, a checkerboard calibration image containing the first calibration point corresponding to the monocular camera is obtained; the first calibration point is the inner corner point of the checkerboard of the first black-and-white checkerboard. According to the camera parameters of the monocular camera and the first actual space coordinates of the first calibration point in the real space, the first ideal image coordinates of the first calibration point in the checkerboard calibration image corresponding to the monocular camera are determined, and according to the difference between the first ideal image coordinates and the first real image coordinates of the first calibration point in the checkerboard calibration image corresponding to the monocular camera, the distortion parameter corresponding to the monocular camera is determined. The original monocular image collected by the monocular camera is optically distortion-corrected according to the distortion parameter corresponding to the monocular camera to obtain a distortion-corrected image corresponding to the monocular camera. The distortion-corrected image corresponding to the monocular camera is subjected to perspective transformation processing according to the perspective transformation matrix corresponding to the monocular camera to achieve assembly error correction, and an assembly-corrected image corresponding to the monocular camera is obtained. For each monocular camera, a cylindrical surface is constructed with the center of the sensor in the monocular camera as the cylinder center to obtain a cylindrical surface corresponding to the monocular camera, and the assembly-corrected image corresponding to the monocular camera is projected onto the cylindrical surface corresponding to the monocular camera to obtain a projection image corresponding to the monocular camera. The projection images respectively corresponding to the two monocular cameras are image-stitched to obtain a binocular stitched image.

[0071] In one embodiment, the binocular device can respectively collect images of the same calibration field through two monocular cameras to obtain two calibration field images containing second calibration points, as Figure 7A and Figure 7BAs shown, it can be clearly seen that there are relatively serious deformations in these two calibration field images before being processed by the image processing method of the present application. Among them, the calibration field includes six second black-and-white checkerboards to form two rectangular calibration planes with a preset included angle; the preset included angle is consistent with the sensor included angle of the binocular device; the second calibration points are the inner corner points of the checkerboards of the second black-and-white checkerboards. For each monocular camera, according to the camera parameters of the monocular camera and the second actual space coordinates of the second calibration points in the real space, the second ideal image coordinates of the second calibration points in the calibration field images collected by the monocular camera are determined. According to the second ideal image coordinates and the second real image coordinates of the second calibration points in the calibration field images collected by the monocular camera, the perspective transformation matrix corresponding to the monocular camera is determined, so as to facilitate subsequent image correction based on the perspective transformation matrix. After being corrected by the image processing method of the present application, normal images as shown in Figure 8A and Figure 8B can be obtained respectively.

[0072] In one embodiment, the binocular device projects the assembly correction images corresponding to the two monocular cameras onto the cylinders corresponding to the two monocular cameras respectively for image stitching, and a binocular stitched image with normal display as shown in Figure 9A and Figure 9B can be obtained, improving the quality of the binocular stitched image.

[0073] It should be understood that although the steps in the flowcharts of the above embodiments are shown in sequence, these steps do not necessarily have to be executed in sequence. Unless there is a clear indication in this article, the execution of these steps has no strict sequence limit, and these steps can be executed in other sequences. Moreover, at least a part of the steps in the above embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0074] In one embodiment, as shown in Figure 10 , an image processing device 1000 is provided, which is applied to a binocular device including two monocular cameras. The device specifically includes:

[0075] An acquisition module 1002, configured to acquire two original monocular images by respectively using two monocular cameras to acquire images of the same scene;

[0076] An acquisition module 1004, configured to acquire the perspective transformation matrices corresponding to the two monocular cameras respectively;

[0077] The calibration module 1006 is configured to, for each monocular camera, perform assembly error calibration on the original monocular image collected by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, so as to obtain the assembly calibration image corresponding to the monocular camera;

[0078] The stitching module 1008 is configured to stitch the assembly calibration images respectively corresponding to the two monocular cameras to obtain a binocular stitched image.

[0079] In one embodiment, the calibration module 1006 is further configured to, for each monocular camera, obtain the distortion parameters corresponding to the monocular camera; perform optical distortion calibration on the original monocular image collected by the monocular camera according to the distortion parameters corresponding to the monocular camera to obtain the distortion calibration image corresponding to the monocular camera; perform assembly error calibration on the distortion calibration image corresponding to the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain the assembly calibration image corresponding to the monocular camera.

[0080] In one embodiment, the calibration module 1006 is further configured to, for each monocular camera, collect an image of a first black and white checkerboard through the monocular camera to obtain a checkerboard calibration image corresponding to the monocular camera and including first calibration points; the first calibration points are the checkerboard inner corner points of the first black and white checkerboard; according to the camera parameters of the monocular camera and the first actual space coordinates of the first calibration points in the real space, determine the first ideal image coordinates of the first calibration points in the checkerboard calibration image corresponding to the monocular camera, and determine the distortion parameters corresponding to the monocular camera according to the difference between the first ideal image coordinates and the first real image coordinates of the first calibration points in the checkerboard calibration image corresponding to the monocular camera.

[0081] In one embodiment, the acquisition module 1004 is further configured to collect images of the same calibration field through the two monocular cameras respectively to obtain two calibration field images including second calibration points; the calibration field includes at least six second black and white checkerboards to form two rectangular calibration planes with a preset included angle; the preset included angle is consistent with the sensor included angle of the binocular device; the second calibration points are the checkerboard inner corner points of the second black and white checkerboards; for each monocular camera, according to the camera parameters of the monocular camera and the second actual space coordinates of the second calibration points in the real space, determine the second ideal image coordinates of the second calibration points in the calibration field image collected by the monocular camera, and determine the perspective transformation matrix corresponding to the monocular camera according to the second ideal image coordinates and the second real image coordinates of the second calibration points in the calibration field image collected by the monocular camera.

[0082] In one embodiment, the stitching module 1008 is further configured to project the assembly calibration images respectively corresponding to the two monocular cameras onto the cylindrical surfaces corresponding to the two monocular cameras for image stitching to obtain a binocular stitched image.

[0083] In one embodiment, the stitching module 1008 is further configured to, for each monocular camera, construct a cylindrical surface with the center of the sensor in the monocular camera as the cylinder center, obtain the cylindrical surface corresponding to the monocular camera, project the assembled and corrected image corresponding to the monocular camera onto the cylindrical surface corresponding to the monocular camera, and obtain the projected image corresponding to the monocular camera; and stitch the projected images respectively corresponding to the two monocular cameras to obtain a binocular stitched image.

[0084] For the above image processing device, the binocular device acquires two original monocular images by using two monocular cameras to respectively collect images of the same scene; obtains the perspective transformation matrices respectively corresponding to the two monocular cameras; for each monocular camera, performs assembly error correction on the original monocular image collected by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, and obtains the assembled and corrected image corresponding to the monocular camera; and stitches the assembled and corrected images respectively corresponding to the two monocular cameras to obtain a binocular stitched image. Compared with the conventional method of directly stitching the images collected by the two monocular cameras in the binocular device to obtain a binocular stitched image, in this application, the perspective transformation matrices respectively corresponding to the monocular cameras in the binocular device are obtained, and the original monocular images collected by the monocular cameras are corrected for assembly errors according to the perspective transformation matrices corresponding to the monocular cameras, and then the two assembled and corrected images obtained after the assembly error correction are stitched. In this way, it is possible to effectively avoid the large difference in the observation effects of the two monocular cameras caused by the insufficient assembly accuracy of the two monocular cameras during the production process of the binocular device, that is, the geometric features of the binocular vision homologous objects are inconsistent, there are offsets and obvious bends at the panoramic stitching, etc., which seriously affect the panoramic observation effect, thereby improving the quality of the binocular stitched image finally obtained by the binocular device.

[0085] Each module in the above image processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the binocular device in hardware form or be independent of it, or can be stored in the memory in the binocular device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0086] In one embodiment, a binocular device is provided, and its internal structure diagram can be as Figure 11As shown in the figure. The binocular device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the binocular device is used to provide computing and control capabilities. The memory of the binocular device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the binocular device is used to exchange information between the processor and external devices. The communication interface of the binocular device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the binocular device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the binocular device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the binocular device, or an external keyboard, touchpad, or mouse, etc.

[0087] Those skilled in the art can understand that Figure 11 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the binocular device to which the solution of this application is applied. The specific binocular device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0088] In one embodiment, a binocular device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0089] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0090] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0092] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0093] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0094] The above-described embodiments only represent several implementation manners of this application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of the patent of this application should be subject to the appended claims.

Claims

1. An image processing method, characterized in that: Applied to a binocular device comprising two monocular cameras, the method comprises: The two monocular cameras respectively capture images of the same scene to obtain two original monocular images; Obtain the perspective transformation matrices corresponding to the two monocular cameras respectively; For each monocular camera, performing assembly error correction on the original monocular image captured by the monocular camera according to the perspective transformation matrix corresponding to the monocular camera, so as to obtain an assembly correction image corresponding to the monocular camera; The assembled calibrated images corresponding to the two monocular cameras are stitched together to obtain a binocular stitched image.

2. The method according to claim 1, characterized in that For each monocular camera, performing assembly error correction on an original monocular image captured by the monocular camera according to a perspective transformation matrix corresponding to the monocular camera to obtain an assembly correction image corresponding to the monocular camera, including: For each monocular camera, obtaining a distortion parameter corresponding to the monocular camera; Performing optical distortion correction on the original monocular image captured by the monocular camera according to the distortion parameters corresponding to the monocular camera, so as to obtain a distortion-corrected image corresponding to the monocular camera; The assembly error correction is performed on the distortion-corrected image corresponding to the monocular camera according to the perspective transformation matrix corresponding to the monocular camera to obtain the assembly-corrected image corresponding to the monocular camera.

3. The method according to claim 2, characterized in that The step of obtaining, for each monocular camera, a distortion parameter corresponding to the monocular camera includes: For each monocular camera, an image of a first black and white chessboard is captured by the monocular camera to obtain a chessboard calibration image corresponding to the monocular camera and containing a first calibration point; the first calibration point is an inner corner point of the first black and white chessboard; According to the camera parameters of the monocular camera and the first actual space coordinates of the first calibration point in the real space, the first ideal image coordinates of the first calibration point in the checkerboard calibration image corresponding to the monocular camera are determined, and according to the difference between the first ideal image coordinates and the first real image coordinates of the first calibration point in the checkerboard calibration image corresponding to the monocular camera, the distortion parameters corresponding to the monocular camera are determined.

4. The method according to claim 1, characterized in that: The obtaining of the perspective transformation matrices corresponding to the two monocular cameras respectively includes: The two monocular cameras respectively capture images of the same calibration field to obtain two calibration field images containing the second calibration point; the calibration field includes at least six second black and white checkerboards to form two rectangular calibration planes with a preset angle; the preset angle is consistent with the sensor angle of the binocular device; the second calibration point is a checkerboard corner point of the second black and white checkerboard; For each monocular camera, the second ideal image coordinates of the second calibration point in the calibration field image captured by the monocular camera are determined according to the camera parameters of the monocular camera and the second real space coordinates of the second calibration point in the real space; the perspective transformation matrix corresponding to the monocular camera is determined according to the second ideal image coordinates and the second real image coordinates of the second calibration point in the calibration field image captured by the monocular camera.

5. The method according to claim 1, characterized in that The step of stitching the assembly correction images respectively corresponding to the two monocular cameras to obtain a binocular stitched image comprises: The assembly correction images corresponding to the two monocular cameras are projected onto the cylindrical surfaces corresponding to the two monocular cameras to perform image stitching to obtain a binocular stitching image.

6. The method according to claim 5, characterized in that The step of projecting the assembly correction images respectively corresponding to the two monocular cameras onto the cylindrical surfaces corresponding to the two monocular cameras to perform image stitching to obtain a binocular stitching image comprises: For each monocular camera, a cylindrical surface with the center of the sensor in the monocular camera as the cylinder center is constructed to obtain the cylindrical surface corresponding to the monocular camera, and the assembly correction image corresponding to the monocular camera is projected onto the cylindrical surface corresponding to the monocular camera to obtain the projection image corresponding to the monocular camera; The projection images corresponding to the two monocular cameras are stitched together to obtain a binocular stitched image.

7. An image processing device, characterized in that: Applied to a binocular device comprising two monocular cameras, the device comprises: An acquisition module, used to acquire images of the same scene through the two monocular cameras respectively, to obtain two original monocular images; An acquisition module, used to acquire the perspective transformation matrices corresponding to the two monocular cameras respectively; A correction module is used to perform assembly error correction on the original monocular image captured by each monocular camera according to the perspective transformation matrix corresponding to the monocular camera, so as to obtain an assembly correction image corresponding to the monocular camera; The stitching module is used to stitch the assembly correction images corresponding to the two monocular cameras to obtain a binocular stitching image.

8. A binocular device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images