Image Registration Method, Device, Electronic Device and Medium

Abstract

The present application discloses an image registration method, apparatus, electronic device and medium, belonging to the field of image processing. The method includes: obtaining a first depth image and an RGB image; performing image registration on the first depth image to obtain a second depth image, where the second depth image includes at least one bright region, and the bright region includes at least one valid depth pixel; performing bright region pixel merging on the second depth image according to the image information of the RGB image to obtain a sparse depth map, and the bright region pixel merging refers to merging at least one pixel in each of the at least one bright region into one pixel.

Description

Technical Field

[0001] This application belongs to the field of image processing, and particularly relates to an image registration method, apparatus, electronic device, and medium. Background Art

[0002] The solution from depth calculation to RGBD registration is to first perform depth calculation based on the phase map of iTOF to obtain the depths of several bright regions, each bright region occupying several pixels, then average all the depths of each bright region to obtain sparse depth, each depth occupying one pixel, and finally register the sparse depth to the RGB image coordinate system, using the depth information to assist the RGB image to form 3D information. However, for the existing dot matrix iTOF depth calculation and RGB-D registration solutions, there will be deviations in the registration accuracy and the depth accuracy after registration. Summary of the Invention

[0003] The objective of the embodiments of this application is to provide an image registration method, apparatus, electronic device, and medium, which can solve the problem of low registration accuracy and low depth accuracy after registration.

[0004] In a first aspect, the embodiments of this application provide an image registration method, which includes:

[0005] Obtain a first depth image and an RGB image;

[0006] Perform image registration on the first depth image to obtain a second depth image, where the second depth image includes at least one bright region, and the bright region includes at least one valid depth pixel;

[0007] According to the image information of the RGB image, perform bright region pixel merging on the second depth image to obtain a sparse depth map, where the bright region pixel merging refers to merging at least one pixel of each bright region among the at least one bright region into one pixel.

[0008] In a second aspect, the embodiments of this application provide an image registration apparatus, including:

[0009] An image acquisition module, configured to obtain a first depth image and an RGB image;

[0010] An image registration module, configured to perform image registration on the first depth image according to the first depth image and the RGB image to obtain a second depth image;

[0011] A pixel merging module, configured to perform bright region pixel merging on the second depth image according to the image information of the RGB image to obtain a sparse depth map.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instructions are stored. When the program or instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the method described in the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the method described in the first aspect.

[0016] In the embodiment of the present application, before performing bright region pixel merging, registration of the first depth image and the RGB image is performed, and bright region pixel merging is performed based on the obtained second depth image and RGB image after registration, avoiding the deviation caused by directly performing weighted averaging on the bright region with complex conditions, and improving the accuracy of registration and the depth accuracy after registration.

[0017] In the embodiment of the present application, during registration, the first depth image in the RGB coordinate system has the same resolution as the RGB image, and the depth data in the first depth image to be registered is non-sparse depth data. Therefore, the position of the corresponding bright region in the RGB image can be accurately obtained, improving the accuracy of registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an implementation environment capable of applying an image registration method according to an embodiment and a composition structure of a device capable of implementing the method;

[0019] Figure 2 is a flowchart of an image registration method according to an embodiment;

[0020] Figure 3 is a flowchart of registering a depth image and an RGB image according to another embodiment;

[0021] Figure 4 is a flowchart of merging valid depth pixels according to still another embodiment;

[0022] Figure 5 is a schematic diagram of an electronic device in an embodiment of the present invention;

[0023] Figure 6 It is a schematic diagram of the hardware structure of the electronic device in the embodiment of the present invention. Specific embodiments

[0024] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0026] In this embodiment, the dot matrix iTOF phase diagram can provide depth information to assist the RGB camera to implement various 3D scene applications. The emitting end of the dot matrix iTOF uses a DOE. Diffractive Optical Elements is abbreviated as DOE, also known as binary optical devices, which are mainly used for laser beam shaping, such as homogenization, collimation, focusing, forming specific patterns, etc. A number of scattered points are projected by the emitting end and received by the CMOS sensor at the receiving end to form a dot matrix iTOF phase diagram. The depth calculation of the dot matrix iTOF is realized based on the indirect time-of-flight method of phase calculation. The result of the depth calculation is the first depth image, and only the depth corresponding to the scattered points at the emitting end is contained in the first depth image. Then, it is combined with the RGB camera for RGB-D registration, and finally a sparse depth map in the RGB image coordinate system is obtained.

[0027] A current solution from depth calculation to RGBD registration is to first perform depth calculation based on the iTOF phase diagram to obtain the depths of several bright regions, where each bright region occupies several pixels, then average all the depths of each bright region to obtain a sparse depth, where each depth occupies one pixel, and finally register the sparse depth to the RGB image coordinate system.

[0028] First, during the depth calculation process, the pixel merging process for each bright area depth is only a weighted average. If the bright area projects onto the object edge or there is a large amount of depth noise within the bright area, this method will cause deviations in the accuracy of the average depth and also lead to deviations in the accuracy of registration.

[0029] Secondly, during the registration process, since the depth data to be registered is sparse depth pixels and the registered result is floating-point RGB image coordinates, it is necessary to round the coordinate values to the nearest integer. Since a much smaller resolution than that of the RGB camera is used during registration, if the RGB image resolution is very small or the depth is very small, the accuracy of registration will be affected.

[0030] Therefore, for the dot matrix iTOF depth calculation and RGB-D registration scheme of the above method, there will be deviations in the accuracy of registration and the depth accuracy after registration.

[0031] Next, in conjunction with the accompanying drawings, through specific embodiments and their application scenarios, the image registration method, device, electronic device, and medium provided by the embodiments of the present application will be described in detail.

[0032] Please refer to Figure 2 , the image registration method of this embodiment may include the following steps 101 - 103:

[0033] Step 101, obtain a first depth image and an RGB image.

[0034] Step 102, perform image registration on the first depth image to obtain a second depth image, where the second depth image includes at least one bright area, and the bright area includes at least one valid depth pixel.

[0035] Step 103, according to the image information of the RGB image, perform bright area pixel merging on the second depth image to obtain a sparse depth map, where the bright area pixel merging refers to merging at least one pixel in each of the at least one bright area into one pixel.

[0036] In this embodiment, as shown in Figure 1 , use the first camera to obtain a dot matrix iTOF phase diagram, use the second camera to obtain an RGB image, collect the first internal reference information of the first camera, collect the second internal reference information of the second camera, and the external reference information between the first camera and the second camera, and perform depth calculation on the TOF diagram to obtain a first depth image, where the TOF image is an iTOF phase diagram or a dTOF phase diagram.

[0037] In this embodiment, the depth calculation principle of the dot matrix iTOF is the indirect time-of-flight method based on phase calculation. The transmitting end of the dot matrix iTOF projects a number of scattered points based on DOE, and the receiving end is a CMO sensor to obtain the dot matrix iTOF phase map. Depth calculation is performed on the dot matrix iTOF phase map, and the result of the depth calculation is the first depth image. The TOF map provides the dot matrix iTOF, and the dot matrix iTOF provides depth information. The depth calculation method of this embodiment is used to assist the RGB camera to implement various 3D scene applications.

[0038] In this embodiment, depth calculation is only performed on the bright areas of the TOF map, and the non-bright areas of the TOF map are not processed. Among them, the bright area is the area where the depth value in the depth image is greater than the depth threshold, and the non-bright area is the area where the depth value in the depth image is less than or equal to the depth threshold.

[0039] In this embodiment, the depth threshold is preset, and the depth of the depth image is divided through the depth threshold, and the area greater than the depth threshold is divided into bright areas. The depth in the depth image refers to the number of bits used to store each pixel, and is also used to measure the color resolution of the image.

[0040] In one embodiment, the method for extracting the bright area can be to extract the bright area based on the histogram information of the phase map or to extract the bright area according to the position of the projected speckle.

[0041] In this embodiment, the RGB-D device projects a dot matrix pattern onto the target area, receives the reflected light signal reflected by the object in the target area, and forms an iTOF phase map.

[0042] An RGB image refers to an image displayed in the RGB color mode. The RGB color mode is a color standard in the industrial field. It obtains various colors through the changes of the three color channels of red (R), green (G), and blue (B) and their superposition with each other. RGB represents the colors of the three channels of red, green, and blue. This standard almost includes all the colors that the human vision can perceive and is one of the most widely used color systems.

[0043] A depth image is an image that stores three-dimensional depth feature information. Depth Images, also known as Range Images, refer to an image in which the depth values of each point in the scene collected by the image acquisition module are used as pixel values. It directly reflects the geometric shape of the visible surface of the scene, and using it can easily solve the problems in 3D target description.

[0044] In this embodiment, through the internal and external parameters calibrated by the TOF map, the RGB module, and the TOF module, the first depth image containing at least one bright area is obtained, and each bright area contains at least one valid depth pixel.

[0045] In this embodiment, the RGB-D registration method is adopted to register the first depth image and the RGB image with the same resolution. The principle of RGB-D registration is to perform coordinate transformation on the depth information of the first depth image according to the internal and external parameters of the lenses of the RGB module and the TOF module. Only the depth of the bright area is registered here, and the non-bright area is not processed. Among them, RGB-D registration: transforms the image coordinate system of the depth image into the image coordinate system of the color image.

[0046] As Figure 3 shown, in this embodiment, the dot matrix iTOF phase diagram is obtained by the first camera, and the RGB image is obtained by the second camera.

[0047] In one embodiment, referring to Figure 3 , performing image registration on the first depth image to obtain a second depth image includes:

[0048] Step 1021: Collect the first internal parameter information of the first camera, the second internal parameter information of the second camera, and the external parameter information between the first camera and the second camera.

[0049] Step 1022: According to the first internal parameter information, the second internal parameter information, and the target external parameter information, perform coordinate transformation on the first depth image to obtain a second depth image aligned with the RGB image coordinate system;

[0050] Among them, the first internal parameter information is the internal parameter information of the first camera, the second internal parameter information is the internal parameter information of the second camera, and the target external parameter information is the external parameter information between the first camera and the second camera.

[0051] In this embodiment, the first internal parameter information and the second internal parameter information respectively include: fx, fy, u0, v0, where

[0052] fx = F / dx;

[0053] fy = F / dy;

[0054] Among them, F is the length of the focal length, in mm; dx and dy respectively represent: how much length unit a pixel in the x direction or y direction occupies, that is, the size of the actual physical value represented by a pixel, which is the key to realizing the conversion between the image physical coordinate system and the pixel coordinate system. u0, v0 represent the number of horizontal and vertical pixels between the center pixel coordinate of the image and the origin pixel coordinate of the image.

[0055] In this embodiment, the target extrinsic information is the extrinsic information between the first camera and the second camera. The extrinsic information between the first camera and the second camera includes 6 extrinsic parameters, which are respectively: the rotation parameters of the three axes are (ω, δ, θ). Then, the 3×3 rotation matrices of each axis are combined (that is, first multiply the matrices), and R that aggregates the rotation information of the three axes is obtained, and its size is still 3×3; the translation parameters of the three axes of T are (Tx, Ty, Tz).

[0056] Since the first depth image and the RGB image are generated based on the first intrinsic information, the second intrinsic information, and the target extrinsic information, the resolutions of the first depth image and the RGB image are consistent. By converting the first depth image to the RGB coordinate system, the first depth image and the RGB image with the same resolution are registered in the same coordinate system. In this situation, the position of the corresponding first region in the RGB image can be obtained correspondingly, that is, the second depth image is generated.

[0057] In one embodiment, referring to Figure 4 , in step 103, the following steps 1031 to 1035 may further be included:

[0058] Step 1031: Cluster the pixels in the first region of the RGB image to obtain at least one target region, where the first region is the region in the RGB image corresponding to the bright region in the second depth image.

[0059] Step 1032: When the at least one target region is one region, determine the one region as the second region.

[0060] Step 1033: When the at least one target region is at least two regions, determine the second region according to the pixel area of each region in the at least two regions. The pixel area of the second region is greater than the pixel area of at least one third region in the at least two regions, and the at least one third region does not include the second region.

[0061] Step 1034: According to the region coordinates of the second region in the RGB image, perform bright pixel merging on the second bright region in the second depth image to obtain a sparse depth map.

[0062] Wherein, the second bright region is the bright region in the second depth image corresponding to the second region in the RGB image.

[0063] In this embodiment, for the first region of the RGB image with a complex structure, that is, the first region where there is a depth difference in the image pixels, it is necessary to select the main region of the first region, replace the original first region, and determine it as the second region of the RGB image.

[0064] The selection method is not limited to: performing k-means clustering on the pixels in the first region, dividing it into several sub-regions with different colors, and selecting the largest and unique sub-region in terms of pixel area as the main region of the first region, which becomes the second region of the RGB image. That is to say, for each first region in the RGB image, select the sub-region with the largest and unique pixel area as the second region, and process the valid depth pixels in the depth image corresponding to the second region.

[0065] In one embodiment, for the first region with a simple RGB image structure, that is, when the bright area of the depth image corresponding to the first region only covers a flat region, there is no depth difference among the image pixels in this bright area. In such a situation, no clustering is performed on the pixels in the first region, and the first region is directly output as the second region of the RGB image.

[0066] In one embodiment, in step 1034, the following steps 10341 to 10342 may further be included:

[0067] Step 10341, calculate the average value of the pixel coordinates of at least one valid depth pixel in the second bright area to obtain the pixel coordinates of the target pixel in the sparse depth map.

[0068] Step 10342, calculate the depth value of the target pixel in the sparse depth map according to the depth values of at least one valid depth pixel in the second bright area.

[0069] In this embodiment, by calculating the average value of the pixel coordinates of the valid depth pixels in the second bright area, the deviation caused by directly performing weighted averaging on the bright area with complex conditions is avoided, and the registration accuracy and the depth accuracy after registration are improved.

[0070] In one embodiment, step 10342, calculating the depth value of the target pixel in the sparse depth map includes:

[0071] Calculate the weighted average depth value of the valid depth pixels in the second bright area to obtain the depth value of the target pixel in the sparse depth map.

[0072] The weight of each valid depth pixel has an associated relationship with the pixel coordinates of the valid depth pixel and the pixel coordinates of the target pixel.

[0073] In one embodiment, calculate the pixel coordinates and pixel depth values of the pixels in the sparse depth map corresponding to each bright area in the second depth image to obtain the sparse depth map.

[0074] That is, the horizontal and vertical coordinates of the valid depth pixel coordinates in the second depth image corresponding to the second region of the RGB image are averaged and rounded respectively to obtain the horizontal and vertical coordinates of the sparse depth coordinates. That is, the horizontal coordinates of several valid depth pixel coordinates in the second depth image corresponding to the second region of the RGB image are averaged and rounded to obtain the horizontal coordinate of the sparse depth coordinates, and the vertical coordinates of several valid depth pixel coordinates in the second depth image corresponding to the second region of the RGB image are averaged and rounded to obtain the vertical coordinate of the sparse depth coordinates.

[0075] In this embodiment, the steps of averaging and rounding include:

[0076]

[0077] Among them, u1, u2, u3…u n represents the horizontal or vertical coordinate of a pixel in the bright area of the depth image, represents the horizontal or vertical coordinate of the sparse depth coordinate to be output. The horizontal and vertical coordinates of the sparse depth coordinates are obtained by using the above formulas respectively.

[0078] The floor function, whose function is "round down", or "round towards zero", that is, to take the largest integer not greater than x. Different from "rounding", rounding down is to take the largest integer value not greater than the preset required value on the coordinate axis. The mean function is a function for calculating the average value of an array.

[0079] In this embodiment, the weight of each valid depth pixel in the bright area of the depth image is calculated, and the weight is negatively correlated with the distance between the current valid depth pixel coordinate and the sparse depth coordinate. The calculation formula is as follows:

[0080]

[0081] Among them, σ is a weight parameter, generally taken as 1. The subscript i corresponds to the i-th valid depth pixel in the bright area of the depth image, u i represents the horizontal or vertical coordinate of the i-th valid depth pixel in the bright area of the depth image, represents the horizontal or vertical coordinate of the sparse depth coordinate to be output, and wi represents the weight of the i-th valid depth pixel in the bright area of the depth image. The size of the weight depends on the distance between the coordinate of this valid depth pixel and the sparse depth coordinate.

[0082] Based on the weights of the valid depth pixels in the bright area, calculate the depth value of the sparse depth coordinates:

[0083]

[0084] Among them, is the depth value of the sparse depth coordinate, w i is the weight of the i-th valid depth pixel in the bright area of the depth image, d i represents the depth value of the i-th valid depth pixel in the bright area of the depth image.

[0085] Generate a sparse depth map based on the depth value of the sparse depth coordinate.

[0086] This embodiment can improve the accuracy of dot matrix iTOF depth calculation and the RGB-D registration effect. Especially for complex scenes, the improvement of data accuracy will bring effective help to the subsequent application algorithms.

[0087] In the embodiment of the present application, before merging the bright area pixels, register the first depth image and the RGB image, and merge the bright area pixels according to the registered second depth image and RGB image, avoiding the deviation caused by directly performing weighted averaging on the bright area with complex conditions, and improving the accuracy of registration and the depth accuracy after registration.

[0088] In the embodiment of the present application, during registration, the first depth image in the RGB coordinate system has the same resolution as the RGB image, and the depth data in the first depth image to be registered is non-sparse depth data. Therefore, the position of the corresponding bright area in the RGB image can be accurately obtained, improving the accuracy of registration.

[0089] For the image registration method provided by the embodiment of the present application, the execution subject can be an image registration device. In the embodiment of the present application, taking the image registration device executing the image registration method as an example, the image registration device provided by the embodiment of the present application is described.

[0090] As Figure 1 shown, the image registration device includes:

[0091] An image acquisition module 100, configured to acquire a first depth image and an RGB image.

[0092] An image registration module 200, configured to perform image registration on the first depth image according to the first depth image and the RGB image to obtain a second depth image.

[0093] An image merging module 300, configured to perform bright area pixel merging on the second depth image according to the image information of the RGB image to obtain a sparse depth map.

[0094] This device can be applied to the TOF map to provide depth information and assist the RGB camera to implement various 3D scenarios.

[0095] In an embodiment of the present invention, the image acquisition module 100 further includes: being configured to acquire a dot matrix iTOF phase map; being further configured to acquire an RGB image; and being configured to perform depth calculation on the dot matrix iTOF phase map to obtain a first depth image.

[0096] In one embodiment, the image acquisition module 100 employs an RGB-D device.

[0097] In an embodiment of the present invention, the image registration module 200 includes:

[0098] A coordinate system conversion module, configured to perform coordinate system conversion on the first depth image according to first intrinsic information, second intrinsic information, and target extrinsic information to obtain a second depth image aligned with the coordinate system of the RGB image.

[0099] In an embodiment of the present invention, the image merging module 300 includes:

[0100] A pixel clustering module, configured to cluster pixels in a first region in the RGB image to obtain at least one target region, where the first region is a region in the RGB image corresponding to the bright region in the second depth image.

[0101] A region determination module, configured to determine the one region as a second region when the at least one target region is one region;

[0102] Or, when the at least one target region is at least two regions, being configured to determine a second region according to the pixel area of each of the at least two regions, where the pixel area of the second region is greater than the pixel area of at least one third region among the at least two regions, and the at least one third region does not include the second region.

[0103] A pixel merging module, configured to perform bright region pixel merging on a second bright region in the second depth image according to the region coordinates of the second region in the RGB image to obtain a sparse depth map, where the second bright region is the bright region in the second depth image corresponding to the second region in the RGB image.

[0104] In an embodiment of the present invention, the image merging module 300 further includes:

[0105] A pixel coordinate calculation module, configured to perform mean calculation on the pixel coordinates of at least one valid depth pixel in the second bright region to obtain the pixel coordinates of a target pixel in the sparse depth map.

[0106] A sparse depth calculation module, configured to calculate the depth value of the target pixel in the sparse depth map according to the depth values of at least one valid depth pixel in the second bright region.

[0107] In an embodiment of the present invention, the sparse depth calculation module includes:

[0108] A weight allocation module, configured to perform weight allocation of valid depth pixels according to the association relationship between the weight of each valid depth pixel and the pixel coordinates of the valid depth pixel and the pixel coordinates of the target pixel.

[0109] A second depth calculation module, configured to calculate the weighted average depth value of valid depth pixels in the second bright region to obtain the depth value of the target pixel in the sparse depth map.

[0110] In an embodiment of the present invention, the pixel merging module further includes:

[0111] A sparse depth map generation module, configured to calculate the pixel coordinates and pixel depth values of pixels in the sparse depth map corresponding to each bright region in the second depth image to obtain a sparse depth map.

[0112] In an embodiment of the present application, the image registration module performs registration of the first depth image and the RGB image before merging bright region pixels, and the image merging module performs merging of bright region pixels according to the second depth image and the RGB image obtained after registration, avoiding the deviation caused by directly performing weighted averaging on bright regions with complex conditions, and improving the registration accuracy and the depth accuracy after registration.

[0113] In an embodiment of the present application, during registration, the first depth image in the RGB coordinate system has the same resolution as the RGB image, and the depth data in the first depth image to be registered is non-sparse depth data. Therefore, the position of the corresponding bright region in the RGB image can be accurately obtained, improving the registration accuracy.

[0114] The image registration device in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than terminals. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0115] The image registration device in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0116] The image registration device provided in the embodiments of the present application can implement Figures 2 to 4 each process implemented by the method embodiments. To avoid repetition, details are not described here again.

[0117] Optionally, as Figure 5 shown, the embodiments of the present application further provide an electronic device 500, including a processor 501 and a memory 502. A program or instruction that can run on the processor 501 is stored on the memory 502. When the program or instruction is executed by the processor 501, it implements each step of the above content sharing method embodiment and can achieve the same technical effect. To avoid repetition, details are not described here again.

[0118] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for implementing the embodiments of the present application.

[0119] The electronic device 6000 includes, but is not limited to: a radio frequency unit 6001, a network module 6002, an audio output unit 6003, an input unit 6004, a sensor 6005, a display unit 6006, a user input unit 6007, an interface unit 6008, a memory 6009, and a processor 6010, etc.

[0120] Those skilled in the art can understand that the electronic device 6000 may further include a power source (such as a battery) for powering each component. The power source can be logically connected to the processor 6010 through a power management system, so as to manage functions such as charging, discharging, and power consumption management through the power management system. Figure 6 The structure of the electronic device shown does not limit the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0121] Among them, the processor 6010 is used to obtain a first depth image and an RGB image; perform image registration on the first depth image to obtain a second depth image, where the second depth image includes at least one bright area, and the bright area includes at least one valid depth pixel; according to the image information of the RGB image, perform bright area pixel merging on the second depth image to obtain a sparse depth map, and the bright area pixel merging refers to merging at least one pixel in each bright area of the at least one bright area into one pixel.

[0122] Optionally, the processor 6010 is further used to perform depth calculation on the dot matrix iTOF phase map to obtain a first depth image.

[0123] Optionally, the processor 6010 is further used to perform coordinate transformation on the first depth image according to the first internal parameter information, the second internal parameter information, and the target external parameter information to obtain a second depth image aligned with the coordinate system of the RGB image; where the first internal parameter information is the internal parameter information of the first camera, the second internal parameter information is the internal parameter information of the second camera, and the target external parameter information is the external parameter information between the first camera and the second camera.

[0124] Optionally, the processor 6010 is further used to cluster the pixels in the first area of the RGB image to obtain at least one target area, where the first area is the area in the RGB image corresponding to the bright area in the second depth image; in the case where the at least one target area is one area, determining the one area as the second area; in the case where the at least one target area is at least two areas, determining the second area according to the pixel area of each area in the at least two areas, where the pixel area of the second area is greater than the pixel area of at least one third area in the at least two areas, and the at least one third area does not include the second area; performing bright area pixel merging on the second bright area in the second depth image according to the area coordinates of the second area in the RGB image to obtain a sparse depth map.

[0125] Optionally, the processor 6010 is further configured to calculate the average value of the pixel coordinates of at least one valid depth pixel in the second bright area to obtain the pixel coordinates of the target pixel in the sparse depth map; and calculate the depth value of the target pixel in the sparse depth map according to the depth values of at least one valid depth pixel in the second bright area.

[0126] Optionally, the processor 6010 is further configured to calculate the weighted average depth value of the valid depth pixels in the second bright area to obtain the depth value of the target pixel in the sparse depth map; and there is an associated relationship between the weight of each valid depth pixel and the pixel coordinates of the valid depth pixel and the pixel coordinates of the target pixel.

[0127] Optionally, the processor 6010 is further configured to calculate the pixel coordinates and pixel depth values of the pixels in the sparse depth map corresponding to each bright area in the second depth image to obtain the sparse depth map.

[0128] In the embodiment of the present application, before merging the bright area pixels, the registration of the first depth image and the RGB image is performed, and the bright area pixels are merged according to the second depth image and the RGB image obtained after registration, so as to avoid the deviation caused by directly performing weighted averaging on the bright areas with complex conditions, and improve the registration accuracy and the depth accuracy after registration.

[0129] In the embodiment of the present application, the image data to be registered is not sparse depth data, and the resolution of the first depth image in the RGB coordinate system is the same as that of the RGB image during registration. Therefore, the position of the corresponding bright area in the RGB image can be accurately obtained, and the registration accuracy and the depth accuracy after registration are improved.

[0130] It should be understood that in the embodiment of the present application, the input unit 6004 may include a Graphics Processing Unit (GPU) 60041 and a microphone 60042. The graphics processor 60041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 6006 may include a display panel 60061, and the display panel 60061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 6007 includes at least one of a touch panel 60071 and other input devices 60072. The touch panel 60071 is also called a touch screen. The touch panel 60071 may include two parts: a touch detection device and a touch controller. The other input devices 60072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0131] The memory 6009 can be used to store software programs and various data. The memory 6009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 6009 may include a volatile memory or a non-volatile memory, or the memory 6009 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 6009 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0132] The processor 6010 may include one or more processing units; optionally, the processor 6010 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 6010 either.

[0133] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above embodiment of the image registration method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0134] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0135] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the image registration method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0136] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0137] An embodiment of the present application provides a computer program product. The program product is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above embodiment of the image registration method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0138] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0139] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0140] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. An image registration method, characterized in that, Including: Obtain a first depth image and an RGB image; wherein, the first depth image is obtained by performing depth calculation on a dot matrix iTOF phase map, the dot matrix iTOF phase map is obtained by a first camera, and the RGB image is obtained by a second camera; According to first intrinsic information, second intrinsic information, and target extrinsic information, perform coordinate transformation on the first depth image to obtain a second depth image aligned with the coordinate system of the RGB image; the first intrinsic information is the intrinsic information of the first camera, the second intrinsic information is the intrinsic information of the second camera, the target extrinsic information is the extrinsic information between the first camera and the second camera, the second depth image includes at least one bright area, and the bright area includes at least one valid depth pixel; Cluster the pixels in a first area in the RGB image to obtain at least one target area, where the first area is the area in the RGB image corresponding to the bright area in the second depth image; In the case where the at least one target area is one area, determine the one area as a second area; In the case where the at least one target area is at least two areas, determine a second area according to the pixel area of each area in the at least two areas, where the pixel area of the second area is greater than the pixel area of at least one third area in the at least two areas, and the at least one third area does not include the second area; According to the area coordinates of the second area in the RGB image, perform bright area pixel merging on the second bright area in the second depth image to obtain a sparse depth map; The second bright area is the bright area in the second depth image corresponding to the second area in the RGB image.

2. The method according to claim 1, characterized in that, The performing bright area pixel merging on the second bright area in the second depth image includes: Perform mean calculation on the pixel coordinates of at least one valid depth pixel in the second bright area to obtain the pixel coordinates of the target pixel in the sparse depth map; According to the depth values of at least one valid depth pixel in the second bright area, calculate the depth value of the target pixel in the sparse depth map.

3. The method according to claim 2, characterized in that, The calculating the depth value of the target pixel in the sparse depth map according to the depth values of at least one valid depth pixel in the second bright area includes: Calculate the weighted average depth value of the valid depth pixels in the second bright area to obtain the depth value of the target pixel in the sparse depth map; The weight of each valid depth pixel has an associated relationship with the pixel coordinates of the valid depth pixel and the pixel coordinates of the target pixel.

4. The method according to any one of claims 1-3, characterized in that, The obtaining the sparse depth map includes: Calculate the pixel coordinates and pixel depth values of the pixels in the sparse depth map corresponding to each bright area in the second depth image to obtain the sparse depth map.

5. An apparatus for image registration, characterized in that, Including: An image acquisition module, configured to obtain a first depth image and an RGB image; wherein, the first depth image is obtained by performing depth calculation on a dot matrix iTOF phase map, the dot matrix iTOF phase map is obtained by a first camera, and the RGB image is obtained by a second camera; An image registration module, configured to perform coordinate transformation on the first depth image according to the first internal parameter information, the second internal parameter information, and the target external parameter information, so as to obtain a second depth image aligned with the coordinate system of the RGB image; the first internal parameter information is the internal parameter information of the first camera, the second internal parameter information is the internal parameter information of the second camera, the target external parameter information is the external parameter information between the first camera and the second camera, the second depth image includes at least one bright area, and the bright area includes at least one valid depth pixel; A pixel merging module, configured to cluster pixels in a first area of the RGB image to obtain at least one target area, where the first area is the area in the RGB image corresponding to the bright area in the second depth image; when the at least one target area is one area, determining the one area as a second area; when the at least one target area is at least two areas, determining a second area according to the pixel area of each of the at least two areas, where the pixel area of the second area is greater than the pixel area of at least one third area among the at least two areas, and the at least one third area does not include the second area; performing bright area pixel merging on a second bright area in the second depth image according to the area coordinates of the second area in the RGB image to obtain a sparse depth map; the second bright area is the bright area in the second depth image corresponding to the second area in the RGB image.

6. An electronic device, characterized in that, It includes a processor and a memory, and the memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the image registration method according to any one of claims 1-4 are implemented.

7. A readable storage medium, characterized in that, The program or instruction is stored on the readable storage medium, and when the program or instruction is executed by the processor, the steps of the image registration method according to any one of claims 1-4 are implemented.

Images