Image registration method and device

Abstract

This application discloses an image registration method and apparatus, belonging to the field of image processing. The method comprises: obtaining a first image captured by a first camera of an electronic device, a second image captured by a second camera, and a third image captured by a third camera; performing registration processing on the first image based on a first calibration parameter to obtain a first depth image that is registered with the second image; and performing an image affine transformation on the first depth image based on a depth correction parameter to obtain a second depth image that is registered with the third image; wherein the first calibration parameter is related to the first camera and the second camera.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and specifically to an image registration method and device. Background Art

[0002] As the types of cameras and shooting functions in electronic devices continue to increase, the registration between images captured by different types of cameras is becoming increasingly important.

[0003] Taking the example of an electronic device aligning an RGB image captured by a red, green, and blue (RGB) camera with a depth image captured by a time of flight (ToF) camera, the electronic device can use the intrinsic parameters of the RGB camera, the intrinsic parameters of the ToF camera, and the extrinsic parameters between the RGB camera and the ToF camera to perform stereo correction, depth parallax conversion, pixel translation processing, post-processing and inverse correction on the depth image to obtain a target depth image that is aligned with the RGB image at the pixel level, thereby completing image registration.

[0004] However, according to the above method, the electronic device can only ensure the accuracy of image registration when the intrinsic and extrinsic parameters of each camera are fixed. However, because the autofocus function causes the structure of the camera lens assembly to change, and the optical image stabilization function causes the camera to move, the calibration parameters (i.e., intrinsic and / or extrinsic parameters) of some cameras in the electronic device will change, resulting in poor image registration accuracy of the electronic device. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image registration method and apparatus that can solve the problem of poor accuracy of image registration in electronic devices.

[0006] In a first aspect, an embodiment of the present application provides an image registration method, which includes: acquiring a first image captured by a first camera of an electronic device, a second image captured by a second camera, and a third image captured by a third camera; performing registration processing on the first image based on a first calibration parameter to obtain a first depth image registered with the second image; performing an image affine transformation on the first depth image based on a depth correction parameter to obtain a second depth image registered with the third image; wherein the first calibration parameter is related to the first camera and the second camera.

[0007] In a second aspect, an embodiment of the present application provides an image registration device, which includes an acquisition module and a processing module; the acquisition module is used to acquire a first image captured by a first camera of an electronic device, a second image captured by a second camera, and a third image captured by a third camera; the processing module is used to perform registration processing on the first image acquired by the acquisition module based on a first calibration parameter to obtain a first depth image that is registered with the second image acquired by the acquisition module; and based on the depth correction parameter, perform an image affine transformation on the processed first depth image to obtain a second depth image that is registered with the third image acquired by the acquisition module; wherein the first calibration parameter is related to the first camera and the second camera.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, a first image captured by a first camera of an electronic device, a second image captured by a second camera, and a third image captured by a third camera can be obtained; and based on a first calibration parameter, the first image is registered to obtain a first depth image registered with the second image; and based on a depth correction parameter, an image affine transformation is performed on the first depth image to obtain a second depth image registered with the third image; wherein the first calibration parameter is related to the first camera and the second camera. Through this solution, since the electronic device can perform registration processing on the first image captured by the first camera based on the calibration parameters related to the first camera and the second camera to obtain a first depth image registered with the second image captured by the second camera, and can process the first depth image based on the depth correction parameter to obtain a depth image registered with the third image captured by the third camera; that is, the electronic device can indirectly realize the registration of the first image and the third image through the second image, thereby improving the accuracy of image registration of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the image registration method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an image registration device provided in an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0016] Figure 4 This is a hardware diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0018] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0019] First, some nouns or terms involved in the description and claims of this application are explained below.

[0020] ToF: Time-of-flight ranging technology is a scene depth perception technology that analyzes the distance between the camera and the object by measuring the round-trip time of artificial light signals (such as infrared lasers).

[0021] Red Green Blue Depth (RGBD) image registration: the process of feature matching and pixel-level alignment between color images and depth images.

[0022] Camera internal parameters: including the camera's focal length, principal point, and distortion parameters.

[0023] Camera's external parameters: the camera's rotation or translation in the spatial coordinate system.

[0024] The image registration method and apparatus provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0025] In recent years, with the gradual opening of the 3D market, 3D scene perception technology has gradually entered the public eye. Currently, mainstream 3D scene perception technologies include binocular vision technology, structured light technology, and Time of Flight technology.

[0026] Among them, binocular vision technology is based on the principle of parallax and can simultaneously obtain images of the object under test from two perspectives and calculate the depth information of the object under test through triangulation. Structured light technology uses a near-infrared laser emission device to project light with structural characteristics onto the object under test, and then a dedicated infrared camera collects image information and combines it with triangulation to calculate, thereby obtaining the depth information of the object under test. ToF technology has always been a focus of research and attention in the field of computer vision. This technology measures the round-trip time of artificial light signals to analyze the distance between the camera and the object. Compared with binocular vision technology and structured light technology, ToF technology has the advantages of high ranging accuracy, low software overhead, and strong anti-interference ability. Therefore, it has broad application prospects in related fields of 3D scene perception technology, such as background blur, real-time positioning and map construction, three-dimensional reconstruction of scenes, augmented reality technology, virtual reality technology and other fields.

[0027] As electronic devices continue to expand their capabilities, ToF technology has become widely used in their photography scenarios, enabling them to capture depth images using ToF cameras. As the number of cameras in electronic devices increases, the importance of image registration between different camera types is becoming increasingly important.

[0028] The following takes the registration of heterogeneous images by an electronic device as an example to exemplify the method for registering images by an electronic device.

[0029] For example, for the registration between heterogeneous images, the homologous image registration method based on feature matching often does not work. Taking RGBD image registration as an example, the existing RGBD image registration method is usually designed and implemented based on an RGBD camera with a fixed structure (or internal parameter). The electronic device can use the offline calibration parameters of the binocular camera composed of an RGB camera and a ToF camera (i.e., the internal parameters of the RGB camera, the internal parameters of the ToF camera, and the external parameters between the RGB camera and the ToF camera) to perform stereo correction, depth parallax conversion, pixel translation processing, post-processing and inverse correction on the depth image collected by the ToF camera to obtain a depth image that is aligned at the pixel level with the image collected by the RGB camera. In this way, RGBD image registration can be achieved.

[0030] However, when the camera module's structure and installation position remain unchanged, the above offline calibration parameters can meet the requirements of RGBD image registration. However, RGB cameras in electronic devices usually have the following characteristics:

[0031] 1. The autofocus function is achieved by changing the structure of the lens group.

[0032] 2. Use a module with optical image stabilization function to achieve the anti-shake function. In order to further enhance the anti-shake effect, the RGB camera can have a micro gimbal.

[0033] Due to the structural changes in the lens group, the intrinsic parameters of the camera cannot be used directly, and the rotation and translation of the lens of the micro gimbal will make the external parameters of the binocular camera cannot be used directly. That is, the above offline calibration parameters can no longer meet the requirements of RGBD image registration. Therefore, the depth image and color image after registration cannot be aligned at the pixel level, which affects subsequent applications.

[0034] To solve the above problems, in the image registration method provided in the embodiment of the present application, the electronic device can perform registration processing on the first image captured by the ToF camera based on offline calibration parameters (e.g., the first camera in the embodiment of the present application) related to the ToF camera (e.g., the first camera in the embodiment of the present application) and the fixed-structure RGB camera (e.g., the second camera in the embodiment of the present application), thereby obtaining a first depth image that is registered with the second image captured by the fixed-structure RGB camera; and based on the depth correction parameters, perform image affine transformation on the first depth image to obtain a second depth image that is registered with the third image captured by the RGB camera with a micro-gimbal (e.g., the third camera in the embodiment of the present application). That is, the electronic device can achieve registration of the first image and the third image through the second image, thereby improving the accuracy of image registration by the electronic device.

[0035] The present application provides an image registration method. Figure 1 FIG. 1 shows a flow chart of the image registration method provided in the embodiment of the present application. Figure 1 As shown, the image registration method provided in the embodiment of the present application may include the following steps 101 to 103. The method is exemplarily described below by taking an electronic device executing the method as an example.

[0036] Step 101: An electronic device obtains a first image captured by a first camera of the electronic device, a second image captured by a second camera, and a third image captured by a third camera.

[0037] In the embodiment of the present application, the first image, the second image and the third image are images of the same scene.

[0038] Optionally, in the embodiment of the present application, the first image, the second image, and the third image may be images of the same scene captured by the first camera, the second camera, and the third camera at the same time.

[0039] Optionally, in the embodiment of the present application, the first image, the second image, and the third image may also be images of the same fixed scene (e.g., a fixed subject) captured by the first camera, the second camera, and the third camera at different times using the same intrinsic and extrinsic parameters. In the embodiment of the present application, the electronic device may include the first camera, the second camera, and the third camera.

[0040] Optionally, in an embodiment of the present application, the first camera may include a ToF camera, the second camera may include a structurally fixed RGB camera, and the third camera may include a camera with a micro gimbal.

[0041] In the embodiment of the present application, a camera with a fixed structure means that the calibration parameters (ie, intrinsic parameters and / or extrinsic parameters) of the camera will not change.

[0042] In the embodiment of the present application, the ToF camera is a camera with a fixed structure, and the camera with a micro gimbal will rotate as the micro gimbal rotates, that is, the camera with a micro gimbal is a camera with a non-fixed structure.

[0043] It can be understood that in the embodiment of the present application, the image captured by the ToF camera is a depth image, and the image captured by the RGB camera is an RGB image.

[0044] It should be noted that in the embodiments of the present application, the above embodiments are illustrated by taking a ToF camera, an RGB camera with a fixed structure, and a camera with a micro gimbal as examples; in actual implementation, the first camera may include any possible camera with a fixed structure that can capture depth images, the second camera may include any possible camera with a fixed structure that can capture color images, and the third camera may include any possible camera with a non-fixed structure (i.e., the internal parameters and / or external parameters will change), which is not limited in the embodiments of the present application.

[0045] In the embodiment of the present application, the second camera and the third camera are cameras of the same source. Wherein, multiple cameras (for example, two cameras) having the same source can be understood as: the camera type of the multiple cameras is the same.

[0046] For example, when the second camera is an RGB camera, the third camera of the same source as the second camera is also an RGB camera.

[0047] Optionally, in an embodiment of the present application, the second camera and the third camera can form a binocular camera.

[0048] In an embodiment of the present application, since the first camera may include a ToF camera, the second camera may include a structurally fixed RGB camera, and the third camera may include a camera with a micro gimbal, the electronic device can obtain the depth image captured by the first camera and the RGB images captured by the second camera and the third camera respectively, thereby improving the diversity and flexibility of the electronic device in capturing images.

[0049] Step 102: The electronic device performs registration processing on the first image based on the first calibration parameter to obtain a first depth image registered with the second image.

[0050] In the embodiment of the present application, the first calibration parameter is related to the first camera and the second camera.

[0051] Optionally, in an embodiment of the present application, the first calibration parameter may include: an internal parameter of the first camera, an internal parameter of the second camera, and an external parameter between the first camera and the second camera.

[0052] Optionally, in the embodiment of the present application, the internal parameters of the first camera may include: the focal length, principal point and distortion parameters of the first camera.

[0053] Optionally, in the embodiment of the present application, the internal parameters of the second camera may include: the focal length, principal point and distortion parameters of the second camera.

[0054] Optionally, in an embodiment of the present application, the external parameters between the first camera and the second camera may include: the rotation and translation relationship between the first camera and the second camera in the spatial coordinate system, for example, the relative angle and relative position of the first camera and the second camera in the spatial coordinate system.

[0055] In an embodiment of the present application, since the first calibration parameters may include the internal parameters of the first camera, the internal parameters of the second camera, and the external parameters between the first camera and the second camera, the electronic device may accurately determine the relationship between the parameters (including the internal parameters and the external parameters) between the first camera and the second camera based on the first calibration parameters, thereby accurately aligning the first image captured by the first camera with the second image captured by the second camera based on the first calibration parameters.

[0056] In this embodiment of the present application, the first depth image registered with the second image can be: the first depth image is aligned with the second image at the pixel level, that is, the matching degree of position information between corresponding pixels in the first depth image and the second image is greater than or equal to a matching threshold. In this way, the first image and the second image can be registered.

[0057] Optionally, in an embodiment of the present application, the above step 102 can be specifically implemented through the following steps 102a to 102d.

[0058] Step 102a: The electronic device performs depth image correction on the first image based on the first calibration parameter to obtain a first corrected image.

[0059] In an embodiment of the present application, the electronic device performs depth image correction on the first image based on the first calibration parameter, and obtaining the first corrected image can be understood as: the electronic device determines the deviation between the intrinsic parameters of the first camera and the intrinsic parameters of the second camera based on the intrinsic parameters of the first camera and the internal parameters of the second camera; then, the electronic device can perform depth image correction on the first image based on the deviation, as well as the external parameters between the first camera and the second camera to obtain the first corrected image.

[0060] Optionally, in the embodiment of the present application, performing depth image correction on the first image may be performing stereo correction on the first image.

[0061] Step 102b: The electronic device obtains disparity information of pixels in the first corrected image.

[0062] In the embodiment of the present application, the electronic device may obtain the disparity information of the pixel points in the first corrected image based on the depth information of the pixel points in the first corrected image.

[0063] Optionally, in an embodiment of the present application, the electronic device may convert depth information of pixels in the first corrected image into disparity information to obtain disparity information of pixels in the first corrected image.

[0064] Step 102c: The electronic device performs pixel shift on the first corrected image according to the parallax information of the pixels in the first corrected image to obtain a second corrected image.

[0065] Optionally, in an embodiment of the present application, the electronic device can move the pixel points in the first corrected image horizontally (x-axis direction) or vertically (y-axis direction) according to the translation amount indicated by the disparity information to obtain a second corrected image.

[0066] Step 102d: The electronic device performs inverse correction on the second corrected image based on the first calibration parameter to obtain a first depth image.

[0067] In the embodiment of the present application, the electronic device inversely correcting the second corrected image specifically includes: performing depth image inverse correction on the second corrected image. This ensures that the first depth image obtained by inversely correcting the second corrected image matches the first image, for example, where the positional deviation between the first depth image and the first image is less than a preset deviation range.

[0068] Furthermore, since the first depth image matches the first image, and the first image and the second image are images captured of the same scene, registration of the first depth image and the second image can be ensured.

[0069] For methods of performing depth image correction, obtaining parallax information, pixel translation and inverse correction on electronic devices, reference may be made to the specific descriptions in the relevant technologies. To avoid repetition, they will not be described here.

[0070] In an embodiment of the present application, since the electronic device can perform depth image correction, depth parallax conversion, pixel translation and inverse correction on the first image based on the first calibration parameters to obtain a first depth image that is aligned with the second image, the image alignment effect of the electronic device can be improved.

[0071] Step 103: The electronic device performs an image affine transformation on the first depth image based on the depth correction parameter to obtain a second depth image registered with the third image.

[0072] In the embodiment of the present application, the above-mentioned depth correction parameters are related to the second camera and the third camera.

[0073] Specifically, the depth correction parameters can be determined by the electronic device based on calibration parameters related to the second camera and the third camera, and images captured by the second camera and the third camera respectively.

[0074] The specific method for the electronic device to determine the above-mentioned depth correction parameters will be described in detail in the following embodiments, and will not be described here in order to avoid repetition.

[0075] Optionally, in an embodiment of the present application, the image affine transformation may include at least one of the following: image rotation, image translation, image scaling, image shearing or image inversion, etc., which can be determined based on actual usage requirements and is not limited in the embodiment of the present application.

[0076] Optionally, in this embodiment of the present application, the second depth image registered with the third image may be a second depth image that is pixel-aligned with the third image, i.e., the degree of match between the position information of the corresponding pixels in the second depth image and the third image is greater than or equal to the aforementioned matching threshold. In this manner, registration of the first and third images can be achieved.

[0077] In an embodiment of the present application, since the electronic device performs an image affine transformation on the first depth image that is aligned with the second image at the pixel level based on the depth correction parameters related to the second camera and the third camera, the second depth image after the image affine transformation can be aligned with the third image.

[0078] In the image registration method provided in the embodiment of the present application, since the electronic device can perform registration processing on the first image captured by the first camera based on calibration parameters related to the first camera and the second camera, and obtain a first depth image that is registered with the second image captured by the second camera, and can process the first depth image based on the depth correction parameters to obtain a depth image that is registered with the third image captured by the third camera; that is, the electronic device can indirectly realize the registration of the first image and the third image through the second image, thereby improving the accuracy of image registration of the electronic device.

[0079] It can be understood that in the embodiment of the present application, the electronic device may determine the depth correction parameters before performing an image affine transformation on the first depth image based on the depth correction parameters to obtain the second depth image.

[0080] The following describes in detail the method for the electronic device to determine the above-mentioned depth correction parameters.

[0081] Optionally, in an embodiment of the present application, before the above step 103, the heterogeneous image registration method provided in the embodiment of the present application may further include the following steps 104 and 105.

[0082] Step 104: The electronic device calibrates the second calibration parameter based on the second image, the third image, the first depth image, and the second calibration parameter to obtain a target calibration parameter.

[0083] In the embodiment of the present application, the second calibration parameter is related to the second camera and the third camera.

[0084] Optionally, in an embodiment of the present application, the second calibration parameters may include: an internal parameter of the second camera, an internal parameter of the third camera, and an external parameter between the second camera and the third camera.

[0085] Optionally, in the embodiment of the present application, the internal parameters of the second camera may include at least one of the following: a focal length, a principal point, and a distortion parameter of the second camera.

[0086] Optionally, in the embodiment of the present application, the internal parameters of the third camera may include: the focal length, principal point and distortion parameters of the third camera.

[0087] Optionally, in an embodiment of the present application, the external parameters between the second camera and the third camera may include: the rotation and translation relationship between the second camera and the third camera in the spatial coordinate system, for example, the relative angle and relative position of the second camera and the third camera in the spatial coordinate system.

[0088] In the embodiment of the present application, since the second calibration parameters may include: the internal parameters of the second camera, the internal parameters of the third camera, and the external parameters between the second camera and the third camera, when the internal parameters of the third camera (for example, a camera with a micro-pan-tilt head) and / or the external parameters of the third camera change, the second calibration parameters will also change accordingly; that is, the second calibration parameters are not fixed, making it impossible for the electronic device to directly use the second calibration parameters when performing image registration. Therefore, it is necessary to calibrate the second calibration parameters first.

[0089] In an embodiment of the present application, since the second calibration parameters may include the internal parameters of the second camera, the internal parameters of the third camera, and the external parameters between the second camera and the third camera, the electronic device may accurately determine the relationship between the parameters (including internal parameters and external parameters) between the second camera and the third camera based on the second calibration parameters, thereby being able to align the second image captured by the second camera with the third image captured by the third camera based on the second calibration parameters.

[0090] The specific method for the electronic device to obtain the target calibration parameters is described in detail below.

[0091] Optionally, in an embodiment of the present application, the above step 104 can be specifically implemented through the following steps 104a and 104b.

[0092] Step 104a: The electronic device determines N pairs of matching feature points between the second image and the third image.

[0093] Wherein, N is a positive integer.

[0094] Optionally, in an embodiment of the present application, the feature points of the image may be any points with prominent features such as vertices, corner points or center points in the image, or the feature points of the image may be points of interest in the image.

[0095] Optionally, in an embodiment of the present application, before determining the above-mentioned N pairs of matching feature points, the electronic device may use a scale-invariant feature transform (SIFT) method, a speeded up robust features (SURF) method, or an oriented fast rotation (ORB) method, etc., to extract feature points from the second image and the third image respectively; and may match the feature points of the second image with the feature points of the third image, that is, calculate the similarity of the feature point pairs in sequence to determine M pairs of feature points between the second image and the third image, where M is an integer greater than or equal to N. Then, the electronic device may use a random sample consensus algorithm (RANSAC) to optimize the M pairs of feature points, so as to determine the above-mentioned N pairs of matching feature points with high confidence.

[0096] Step 104b: The electronic device corrects the second calibration parameter by a target algorithm based on the depth information of the first depth image, the second calibration parameter, and the feature information of the N pairs of matching feature points to obtain a target calibration parameter.

[0097] In the embodiment of the present application, the second calibration parameters are: initial calibration parameters related to the second camera and the third camera (i.e., the initial internal parameters of the second camera, the initial internal parameters of the third camera, and the initial external parameters between the second camera and the third camera).

[0098] Optionally, in an embodiment of the present application, the target algorithm may be a Levenberg-Marquette LM algorithm, that is, the electronic device may perform online correction on the second calibration parameters through the LM algorithm to obtain accurate calibration parameters related to the second camera and the third camera, that is, the target calibration parameters.

[0099] It can be understood that the target calibration parameters can accurately reflect the parameter relationship between the second camera and the third camera when the electronic device captures images through the second camera and the third camera.

[0100] For a detailed description of the LM algorithm, please refer to the detailed description in the related art. To avoid repetition, it will not be repeated here.

[0101] In the embodiment of the present application, since the electronic device can correct the second calibration parameters based on the depth information of the first depth image, the second calibration parameters, and N pairs of matching feature points between the second image and the third image through the LM algorithm, it is possible to obtain an accurate parameter relationship between the second camera and the third camera. Thus, the electronic device can accurately align the second image captured by the second camera with the third image captured by the third camera based on the corrected second calibration parameters, thereby further improving the accuracy of the electronic device in aligning heterogeneous images.

[0102] Step 105: The electronic device determines a depth correction parameter based on the target calibration parameter.

[0103] Optionally, in an embodiment of the present application, since the first depth image is a depth image registered with the second image captured by the second camera, and the target calibration parameters can accurately reflect the parameter relationship between the second camera and the third camera, the electronic device can obtain pixel deviation information between the first depth image and the third image captured by the third camera based on the parameter relationship, so that the electronic device can determine the above-mentioned depth correction parameters based on the deviation information.

[0104] Furthermore, after determining the depth correction parameters, the electronic device can perform an image affine transformation on the first depth image based on the depth correction parameters, thereby obtaining a second depth image that is aligned with the third image. In this way, the electronic device can achieve accurate alignment between the first image and the third image.

[0105] In an embodiment of the present application, since the electronic device can correct the second calibration parameters and can determine the depth correction parameters based on the corrected target calibration parameters, the accuracy of the electronic device in determining the depth correction parameters can be improved, and the electronic device can ensure the accuracy of the second depth image that is aligned with the third image when processing the above-mentioned first depth image based on the depth correction parameters.

[0106] The image registration method provided in the embodiment of the present application can be executed by an image registration device. In the embodiment of the present application, the image registration device performing the image registration method is taken as an example to illustrate the image registration device provided in the embodiment of the present application.

[0107] Combine Figure 2, an embodiment of the present application provides an image registration device 20, which may include an acquisition module 21 and a processing module 22. The acquisition module 21 can be used to acquire a first image captured by a first camera of an electronic device, a second image captured by a second camera, and a third image captured by a third camera. The processing module 22 can be used to perform registration processing on the first image acquired by the acquisition module 21 based on a first calibration parameter to obtain a first depth image that is registered with the second image acquired by the acquisition module 21; and to perform image affine transformation on the processed first depth image based on a depth correction parameter to obtain a second depth image that is registered with the third image acquired by the acquisition module 21; wherein the first calibration parameter is related to the first camera and the second camera.

[0108] In one possible implementation, the image registration device 20 may further include a determination module. The processing module 22 may further be configured to, before performing an image affine transformation on the first depth image based on the depth correction parameters to obtain a second depth image to be registered with the third image, calibrate the second calibration parameters based on the second image, the third image, the first depth image, and the second calibration parameters to obtain target calibration parameters. The determination module may be configured to determine the depth correction parameters based on the target calibration parameters obtained by the processing module 22; wherein the second calibration parameters are related to the second camera and the third camera.

[0109] In one possible implementation, the processing module 22 may include a determination submodule and a first correction submodule. The determination submodule may be configured to determine N pairs of matching feature points between the second image and the third image, where N is a positive integer. The first correction submodule may be configured to calibrate the second calibration parameters using a target algorithm based on the depth information of the first depth image, the second calibration parameters, and the feature information of the N pairs of matching feature points determined by the determination submodule to obtain target calibration parameters.

[0110] In one possible implementation, the processing module 22 may further include a second correction submodule, an acquisition submodule, and a processing submodule. The second correction submodule may be configured to perform depth image correction on the first image based on the first calibration parameter to obtain a first corrected image. The acquisition submodule may be configured to obtain disparity information of pixels in the first corrected image corrected by the second correction submodule. The processing submodule may be configured to perform pixel shifting on the first corrected image based on the disparity information obtained by the acquisition submodule to obtain a second corrected image; and to perform inverse correction on the processed second corrected image based on the first calibration parameter to obtain a first depth image.

[0111] In one possible implementation, the first camera may include a ToF camera, the second camera may include a fixed-structure RGB camera, and the third camera may include a camera with a micro gimbal.

[0112] In the image registration device provided in the embodiment of the present application, since the image registration device can perform registration processing on the first image captured by the first camera based on calibration parameters related to the first camera and the second camera, and obtain a first depth image that is registered with the second image captured by the second camera, and can process the first depth image based on the depth correction parameters to obtain a depth image that is registered with the third image captured by the third camera; that is, the image registration device can indirectly realize the registration of the first image and the third image through the second image, thereby improving the accuracy of image registration.

[0113] The image registration device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0114] The image registration device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0115] The image registration device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0116] like Figure 3As shown, an embodiment of the present application further provides an electronic device 300, including a processor 301 and a memory 302, wherein the memory 302 stores a program or instruction that can be run on the processor 301, and when the program or instruction is executed by the processor 301, the various steps of the above-mentioned image registration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0117] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0118] Figure 4 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0119] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1001 , a network module 1002 , an audio output unit 1003 , an input unit 1004 , a sensor 1005 , a display unit 1006 , a user input unit 1007 , an interface unit 1008 , a memory 1009 , and a processor 1010 .

[0120] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0121] Among them, the processor 1010 can be used to obtain a first image captured by a first camera of the electronic device, a second image captured by a second camera, and a third image captured by a third camera; and based on a first calibration parameter, perform alignment processing on the acquired first image to obtain a first depth image aligned with the acquired second image; and based on a depth correction parameter, perform an image affine transformation on the processed first depth image to obtain a second depth image aligned with the acquired third image; wherein the first calibration parameter is related to the first camera and the second camera.

[0122] In one possible implementation, the processor 1010 may also be used to perform an image affine transformation on the first depth image based on the depth correction parameter to obtain a second depth image aligned with the third image, and to correct the second calibration parameter based on the second image, the third image, the first depth image, and the second calibration parameter to obtain a target calibration parameter; and determine the depth correction parameter based on the processed target calibration parameter; wherein the second calibration parameter is related to the second camera and the third camera.

[0123] In one possible implementation, the processor 1010 can be specifically used to determine N pairs of matching feature points between the second image and the third image, where N is a positive integer; and based on the depth information of the first depth image, the second calibration parameters and the determined feature information of the N pairs of matching feature points, the second calibration parameters are corrected through a target algorithm to obtain target calibration parameters.

[0124] In one possible implementation, the processor 1010 can be specifically used to perform depth image correction on the first image based on the first calibration parameter to obtain a first corrected image; and obtain disparity information of pixel points in the corrected first corrected image; and according to the obtained disparity information, perform pixel translation on the first corrected image to obtain a second corrected image; and perform de-correction on the processed second corrected image based on the first calibration parameter to obtain a first depth image.

[0125] In one possible implementation, the first camera may include a ToF camera, the second camera may include a fixed-structure RGB camera, and the third camera may include a camera with a micro gimbal.

[0126] In the electronic device provided in the embodiment of the present application, since the electronic device can perform alignment processing on the first image captured by the first camera based on calibration parameters related to the first camera and the second camera, and obtain a first depth image aligned with the second image captured by the second camera, and can process the first depth image based on the depth correction parameters to obtain a depth image aligned with the third image captured by the third camera; that is, the electronic device can indirectly realize the alignment of the first image and the third image through the second image, thereby improving the accuracy of image alignment of the electronic device.

[0127] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described here.

[0128] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0129] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0130] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0131] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image registration method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0133] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image registration method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0135] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image registration method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0136] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0138] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image registration method, characterized in that: The method comprises: Acquire a first image captured by a first camera of the electronic device, a second image captured by a second camera, and a third image captured by a third camera; Performing registration processing on the first image based on first calibration parameters between the first camera and the second camera to obtain a first depth image that is pixel-level aligned with the second image; Performing an image affine transformation on the first depth image based on the depth correction parameter to obtain a second depth image registered with the third image; Wherein, the first calibration parameter is related to the first camera and the second camera; Before performing an image affine transformation on the first depth image based on the depth correction parameter to obtain a second depth image registered with the third image, the method further includes: Correcting the second calibration parameters based on the second image, the third image, the first depth image, and the second calibration parameters to obtain target calibration parameters; determining a parameter relationship between the second camera and the third camera based on the target calibration parameters; Based on the parameter relationship, obtaining pixel deviation information between the first depth image and the third image; Determining the depth correction parameter based on the pixel deviation information; The second calibration parameter is related to the second camera and the third camera.

2. The method according to claim 1, characterized in that The correcting the second calibration parameter based on the second image, the third image, the first depth image, and the second calibration parameter to obtain the target calibration parameter includes: Determine N pairs of matching feature points between the second image and the third image, where N is a positive integer; Based on the depth information of the first depth image, the second calibration parameters, and the feature information of the N pairs of matching feature points, the second calibration parameters are corrected through a target algorithm to obtain the target calibration parameters.

3. The method according to claim 1, characterized in that The registering process of the first image based on the first calibration parameter to obtain a first depth image registered with the second image includes: Performing depth image correction on the first image based on the first calibration parameter to obtain a first corrected image; Obtaining disparity information of pixels in the first corrected image; performing pixel shifting on the first corrected image according to the disparity information to obtain a second corrected image; The second corrected image is de-corrected based on the first calibration parameters to obtain the first depth image.

4. The method according to any one of claims 1 to 3, characterized in that The first camera includes a time-of-flight ToF camera, the second camera includes a red, green, and blue (RGB) camera with a fixed structure, and the third camera includes a camera with a micro gimbal.

5. An image registration device, characterized in that: The device includes an acquisition module and a processing module; The acquisition module is configured to acquire a first image captured by a first camera, a second image captured by a second camera, and a third image captured by a third camera of the electronic device; The processing module is configured to perform registration processing on the first image acquired by the acquisition module based on a first calibration parameter between the first camera and the second camera to obtain a first depth image that is pixel-aligned with the second image acquired by the acquisition module; and perform image affine transformation on the processed first depth image based on a depth correction parameter to obtain a second depth image that is aligned with the third image acquired by the acquisition module; Wherein, the first calibration parameter is related to the first camera and the second camera; The apparatus further includes a determination module; The processing module is further configured to, before performing an image affine transformation on the first depth image based on the depth correction parameter to obtain the second depth image registered with the third image, correct the second calibration parameter based on the second image, the third image, the first depth image, and the second calibration parameter to obtain a target calibration parameter; The determination module is configured to determine a parameter relationship between the second camera and the third camera based on the target calibration parameters; obtain pixel deviation information between the first depth image and the third image based on the parameter relationship; and determine the depth correction parameter based on the pixel deviation information; The second calibration parameter is related to the second camera and the third camera.

6. The device according to claim 5, characterized in that The processing module includes a determination submodule and a first correction submodule; The determination submodule is configured to determine N pairs of matching feature points between the second image and the third image, where N is a positive integer; The first correction submodule is used to correct the second calibration parameters through a target algorithm based on the depth information of the first depth image, the second calibration parameters and the feature information of the N pairs of matching feature points determined by the determination submodule to obtain the target calibration parameters.

7. The device according to claim 5, characterized in that The processing module also includes a second correction submodule, an acquisition submodule and a processing submodule; The second correction submodule is configured to perform depth image correction on the first image based on the first calibration parameters to obtain a first corrected image; The acquisition submodule is configured to acquire disparity information of pixel points in the first corrected image corrected by the second correction submodule; The processing submodule is used to perform pixel shifting on the first corrected image according to the disparity information obtained by the acquisition submodule to obtain a second corrected image; and to perform inverse correction on the processed second corrected image based on the first calibration parameters to obtain the first depth image.

8. The device according to any one of claims 5 to 7, characterized in that The first camera includes a ToF camera, the second camera includes a fixed-structure RGB camera, and the third camera includes a camera with a micro gimbal.

Images