Image registration method, device, electronic device and readable storage medium

Abstract

The present application proposes an image registration method, device, electronic device and readable storage medium, which relates to the field of image processing, wherein the method includes: obtaining a depth image; determining the original coordinates of the target pixel in the depth image in the coordinate system of the depth image; querying the constant mapping relationship between the original coordinates and the coordinate constant part according to the original coordinates of the target pixel to obtain the corresponding coordinate constant part; querying the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel to obtain the corresponding coordinate variable part; determining the target coordinates of the target pixel in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part. Therefore, for each target pixel in the depth image, there is no need to repeat the matrix multiplication calculation, which can greatly improve the efficiency of image registration, and can be applied to devices with low computing power, thereby improving the applicability of the method.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, artificial intelligence products, such as robots, are becoming more and more popular. Generally, at least one color camera can be installed on a smart product to collect color images for object classification and identification, and at least one depth camera can be installed on a smart product to collect depth images to obtain the spatial position information of the object. However, due to the difference in the installation positions of the color camera and the depth camera, it is usually necessary to align the depth image with the color image before subsequent image processing.

[0003] In the related technology, during image registration, it is necessary to traverse each pixel point in the depth image, and for each pixel point, construct a vector corresponding to the pixel point in the pixel coordinate system corresponding to the depth image, and multiply the vector corresponding to the pixel point by a preset transformation matrix to obtain the vector corresponding to the pixel point in the pixel coordinate system corresponding to the color image. Therefore, based on the vector corresponding to the pixel point in the pixel coordinate system corresponding to the color image, the coordinates of the pixel point after registration in the pixel coordinate system corresponding to the color image can be determined.

[0004] However, the depth image has many pixels. For example, the resolution of a common depth image is 640*480. The above steps need to be repeated 640*480=307200 times each time the registration is performed. The amount of calculation is large and requires a lot of computing resources. It is difficult to achieve real-time calculation in devices with low computing power, such as embedded devices. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the present application proposes an image registration method, apparatus, electronic device and readable storage medium to achieve image registration for each target pixel in a depth image without the need to repeat matrix multiplication calculations. This can greatly improve the efficiency of image registration and can be used in devices with lower computing power, thereby improving the applicability of the method.

[0007] The first embodiment of the present application proposes an image registration method, comprising:

[0008] Get the depth image;

[0009] For a target pixel in the depth image, determining an original coordinate in a coordinate system of the depth image;

[0010] According to the original coordinates of the target pixel point, query the constant mapping relationship between the original coordinates and the coordinate constant part to obtain the corresponding coordinate constant part;

[0011] According to the depth carried by the target pixel point, query the variable mapping relationship between the depth and the coordinate variable part to obtain the corresponding coordinate variable part;

[0012] According to the coordinate constant part and the coordinate variable part, the target coordinates of the target pixel point registered in the coordinate system of the imaging image are determined.

[0013] As a first possible implementation manner of the embodiment of the first aspect of the present application, determining the target coordinates of the target pixel point registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part includes:

[0014] The sum of the abscissa of the coordinate constant part and the abscissa of the coordinate variable part is used as the abscissa in the target coordinate;

[0015] The sum of the ordinate of the coordinate constant part and the ordinate of the coordinate variable part is used as the ordinate in the target coordinate.

[0016] As a second possible implementation manner of the embodiment of the first aspect of the present application, before querying the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel point to obtain the corresponding coordinate variable part, it also includes:

[0017] Calibrate the depth sensor used to acquire the depth image and the image sensor used to acquire the imaging image to obtain a first transformation matrix between the depth image and the imaging image;

[0018] A variable mapping relationship between the depth and the coordinate variable part is generated according to the time-varying element whose value changes with the depth in the first transformation matrix.

[0019] As a third possible implementation manner of the embodiment of the first aspect of the present application, generating a variable mapping relationship between the depth and the coordinate variable part according to the time-varying element whose value changes with the depth in the first transformation matrix includes:

[0020] Determining a coordinate variable portion corresponding to each depth according to a ratio between the time-varying element and each depth within a depth measurement range of the depth sensor;

[0021] The variable mapping relationship is generated according to the coordinate variable parts corresponding to each depth.

[0022] As a fourth possible implementation manner of the embodiment of the first aspect of the present application, before querying the constant mapping relationship between the original coordinates and the coordinate constant part according to the original coordinates of the target pixel point to obtain the corresponding coordinate constant part, it also includes:

[0023] Performing an internal parameter calibration on a depth sensor used to acquire the depth image, and performing an external parameter calibration on the depth sensor and an image sensor used to acquire the imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor;

[0024] A constant mapping relationship between the original coordinates and the constant part of the coordinates is generated according to the constant elements with fixed values ​​in the second transformation matrix.

[0025] As a fifth possible implementation manner of the embodiment of the first aspect of the present application, generating a constant mapping relationship between the original coordinates and the constant part of the coordinates according to a constant element with a fixed value in the second transformation matrix includes:

[0026] Using the second transformation matrix, performing coordinate transformation on the original coordinates to obtain intermediate coordinates;

[0027] Performing distortion correction on the intermediate coordinates;

[0028] Using a third transformation matrix obtained by calibrating the intrinsic parameters of the image sensor, coordinate transformation is performed on the intermediate coordinates after distortion correction to obtain a coordinate constant part corresponding to the original coordinates;

[0029] The constant mapping relationship is generated according to the original coordinates and the corresponding coordinate constant part.

[0030] The image registration method of the embodiment of the present application pre-establishes a constant mapping relationship between the original coordinates and the constant part of the coordinates, and a variable mapping relationship between the depth and the variable part of the coordinates. During the image registration process, the constant part of the coordinates corresponding to the original coordinates of each target pixel in the depth image and the variable part of the coordinates corresponding to the depth carried by the target pixel are directly read out through a search method. Thus, the target coordinates of the target pixel in the coordinate system of the imaging image can be directly calculated based on the constant part of the coordinates and the variable part of the coordinates. Therefore, for each target pixel, there is no need to repeat the matrix multiplication calculation, which can greatly improve the efficiency of image registration and can be applied to devices with lower computing power, thereby improving the applicability of the method.

[0031] The second aspect of the present application provides an image registration device, including:

[0032] An acquisition module, used for acquiring a depth image;

[0033] A determination module, configured to determine, for a target pixel in the depth image, an original coordinate in a coordinate system of the depth image;

[0034] A first query module, used for querying the constant mapping relationship between the original coordinates and the coordinate constant part according to the original coordinates of the target pixel point, so as to obtain the corresponding coordinate constant part;

[0035] A second query module is used to query the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel point, so as to obtain the corresponding coordinate variable part;

[0036] The registration module is used to determine the target coordinates of the target pixel point to be registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part.

[0037] As a first possible implementation manner of the embodiment of the second aspect of the present application, the registration module is specifically used to:

[0038] The sum of the abscissa of the coordinate constant part and the abscissa of the coordinate variable part is used as the abscissa in the target coordinate;

[0039] The sum of the ordinate of the coordinate constant part and the ordinate of the coordinate variable part is used as the ordinate in the target coordinate.

[0040] As a second possible implementation manner of the embodiment of the second aspect of the present application, the device further includes:

[0041] A first calibration module, used for calibrating internal and external parameters of a depth sensor used for acquiring the depth image and an image sensor used for acquiring the imaging image, so as to obtain a first transformation matrix between the depth image and the imaging image;

[0042] The first generating module is used to generate a variable mapping relationship between the depth and the coordinate variable part according to the time-varying element whose value changes with the depth in the first transformation matrix.

[0043] As a third possible implementation manner of the embodiment of the second aspect of the present application, the first generating module is specifically configured to:

[0044] Determining a coordinate variable portion corresponding to each depth according to a ratio between the time-varying element and each depth within a depth measurement range of the depth sensor;

[0045] The variable mapping relationship is generated according to the coordinate variable parts corresponding to each depth.

[0046] As a fourth possible implementation manner of the embodiment of the second aspect of the present application, the device further includes:

[0047] A second calibration module, used to perform an internal parameter calibration on a depth sensor used to acquire the depth image, and to perform an external parameter calibration on the depth sensor and an image sensor used to acquire the imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor;

[0048] The second generating module is used to generate a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix.

[0049] As a fifth possible implementation manner of the embodiment of the second aspect of the present application, the second generating module is specifically configured to:

[0050] Using the second transformation matrix, performing coordinate transformation on the original coordinates to obtain intermediate coordinates;

[0051] Performing distortion correction on the intermediate coordinates;

[0052] Using a third transformation matrix obtained by calibrating the intrinsic parameters of the image sensor, coordinate transformation is performed on the intermediate coordinates after distortion correction to obtain a coordinate constant part corresponding to the original coordinates;

[0053] The constant mapping relationship is generated according to the original coordinates and the corresponding coordinate constant part.

[0054] The image registration device of the embodiment of the present application pre-establishes a constant mapping relationship between the original coordinates and the constant part of the coordinates, and a variable mapping relationship between the depth and the variable part of the coordinates. During the image registration process, the constant part of the coordinates corresponding to the original coordinates of each target pixel in the depth image and the variable part of the coordinates corresponding to the depth carried by the target pixel are directly read out through a search method. Therefore, the target coordinates of the target pixel in the coordinate system of the imaging image can be directly calculated based on the constant part of the coordinates and the variable part of the coordinates. Therefore, for each target pixel, there is no need to repeat the matrix multiplication calculation, which can greatly improve the efficiency of image registration and can be applied to devices with lower computing power to improve applicability.

[0055] The third aspect embodiment of the present application proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image registration method proposed in the first aspect embodiment of the present application is implemented.

[0056] The fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the image registration method proposed in the first aspect embodiment of the present application is implemented.

[0057] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0059] Figure 1 Schematic diagram of the traditional registration algorithm process Figure 1 ;

[0060] Figure 2 Schematic diagram of the traditional registration algorithm process Figure 2 ;

[0061] Figure 3 A flowchart of the image registration method provided in Example 1 of the present application;

[0062] Figure 4 A flowchart of the image registration method provided in the second embodiment of the present application;

[0063] Figure 5 This is a schematic diagram of the registration algorithm flow in the embodiment of the present application;

[0064] Figure 6 A flowchart of the image registration method provided in Embodiment 3 of the present application;

[0065] Figure 7 A flowchart of the image registration method provided in the fourth embodiment of the present application;

[0066] Figure 8 This is a schematic diagram of the structure of the image registration device provided in Embodiment 5 of the present application;

[0067] Fig. 9 A schematic diagram of the structure of an image registration device provided in Example 6 of the present application;

[0068] Fig.10 This is a schematic diagram of the structure of the electronic device provided in Example 7 of the present application. DETAILED DESCRIPTION

[0069] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0070] Generally, the color camera can be defined as camera R, and the depth camera can be defined as camera L. The process of registering the depth image to the color image is actually to map the pixel coordinate system corresponding to the depth image, that is, the point of the L pixel coordinate system, to the pixel coordinate system corresponding to the color image, that is, the R pixel coordinate system, through coordinate transformation. Specifically, the depth image can be registered to the color image in the following two ways:

[0071] The first method is the registration process without distortion correction: see Figure 1 , for each pixel in the depth image, determine its coordinates in the L pixel coordinate system (u L , v L ), and construct a vector of size 4*1 Among them, Z is the depth carried by the pixel point, and the vector is multiplied by the matrix M to obtain the vector of the R pixel coordinate system Then (u R , v R ) is the coordinate of the pixel point in the depth image after registration in the R pixel coordinate system.

[0072] Among them, the matrix M is based on the internal parameter matrix I of the camera L L , the intrinsic parameter matrix I of camera R R , the external parameter matrix E between camera L and camera R L2R The result is:

[0073]

[0074] in, Among them, the intrinsic parameter matrix I of camera L L In, f xl (unit: pixel) is related to dx (the physical size of a pixel in the x direction, unit: mm / pixel) and focal length f (unit: mm). yl (Unit: pixel) is related to dy (the physical size of a pixel in the y direction, unit: mm / pixel) and focal length f, c xl 、c yl is the vertical and horizontal offset of the image origin relative to the optical center imaging point (unit: pixel). Similarly, the intrinsic parameter matrix I of camera R R In, f xr Related to dx and focal length f, f yr Related to dy and focal length f, c xr 、c yr It is the vertical and horizontal offset of the image origin relative to the optical center imaging point.

[0075] And, in general, the external parameter matrix E L2R It is represented by [R|T], where R is the rotation matrix and T is the translation matrix. Then M is:

[0076]

[0077] In the second method, color images usually have large distortion. In order to improve the registration accuracy, a distortion correction step can be added: see Figure 2 , for each pixel in the depth image, determine its coordinates in the L pixel coordinate system (u L , v L ), and construct a vector of size 4*1 Multiply the vector on the left by the matrix M 1 , get the 4*1 vector in the R camera coordinate system And perform distortion correction in the R camera coordinate system. Specifically, (X R ,Y R ) is reorganized into a matrix of size 3*5 Then multiply the above reorganized 3*5 matrix by the 5*1 distortion coefficient vector [1k 1 k 2 p 1 p 2 ] T , and get a 3*1 matrix Then the coordinates after distortion correction (X' R ,Y' R ) is transformed to the R pixel coordinate system and multiplied by the intrinsic parameter matrix I of the R camera on the right R That is, the vector of the R pixel coordinate system for:

[0078]

[0079] Among them, k1 and k2 are the radial distortion parameters of camera R, and p1 and p2 are the tangential distortion parameters of camera R.

[0080] Then (u R , v R ) is the coordinate of the pixel point in the depth image after registration in the R pixel coordinate system.

[0081] Among them, the matrix M 1 It is based on the intrinsic parameter matrix I of camera L L , the external parameter matrix E between camera L and camera R L2R The result is:

[0082]

[0083] The definitions of the pixel coordinate system and the camera coordinate system are the same as the general definitions in computer vision, that is, the pixel coordinate system: the coordinate origin is defined at the upper left corner of the image, the X-axis is horizontal to the right, the Y-axis is horizontal downward, and the unit is pixel; the camera coordinate system: the coordinate origin is defined at the optical center of the lens, the X-axis is horizontal to the right, the Y-axis is horizontal downward, and the unit is millimeter.

[0084] However, the depth image has many pixels. For example, the resolution of a common depth image is 640*480. The above steps need to be repeated 640*480=307200 times each time the registration is performed. The amount of calculation is large and requires a lot of computing resources. It is difficult to achieve real-time calculation in devices with low computing power, such as embedded devices.

[0085] Therefore, this application mainly proposes an image registration method to address the technical problem that image registration in the prior art requires a large amount of computing resources and is difficult to achieve real-time computing in devices with low computing power, such as embedded devices.

[0086] The image registration method of the embodiment of the present application divides the image registration process into two parts, one part is a constant part and the other part is a variable part. During the initialization process of the image registration algorithm, the coordinate constant part corresponding to each original coordinate can be pre-calculated, and a constant mapping relationship between the original coordinate and the coordinate constant part can be established. In addition, according to the maximum range of the depth measured by the depth sensor, the variable mapping relationship corresponding to each depth under the maximum range can be calculated in advance, and a variable mapping relationship between the depth and the coordinate variable part can be established. During the image registration process, the coordinate constant part corresponding to the original coordinate of each target pixel point in the depth image and the coordinate variable part corresponding to the depth carried by the target pixel point are directly read out through a search method, so that the coordinate constant part and the coordinate variable part are added to obtain the target coordinates of the target pixel point registered in the coordinate system of the imaging image. Therefore, there is no need to repeat the matrix multiplication calculation for each target pixel point, which can greatly improve the efficiency of image registration, can be applied to devices with low computing power, and improve the applicability of the method.

[0087] The following describes the image registration method, device, electronic device and readable storage medium according to the embodiments of the present application with reference to the accompanying drawings.

[0088] Figure 3 A flowchart of the image registration method provided in Example 1 of the present application.

[0089] The execution subject of the embodiment of the present application is the image registration device provided by the present application, and the image registration device can be configured in an electronic device, or in a server connected to a network with the electronic device, to control the electronic device.

[0090] Among them, the electronic device can be any device, instrument or machine with computing and processing capabilities. For example, the electronic device can be a PC (Personal Computer), a mobile terminal, a robot, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a vehicle-mounted device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0091] like Figure 3 As shown, the image registration method may include the following steps:

[0092] Step 101: Acquire a depth image.

[0093] In the embodiment of the present application, the depth can be detected by a depth sensor in the electronic device to obtain a depth image, and accordingly, the image registration device can obtain the depth image collected by the depth sensor. For example, the depth can be detected by a TOF (Time of Flight) sensor to obtain a depth image, or by an RGB-D (Red Green Blue-Depth) sensor to obtain a depth image, or by a laser radar to obtain a depth image, or by a structured light sensor to project speckle structured light to the subject, and obtain the structured light reflected by the subject, and obtain an infrared speckle map according to the reflected structured light imaging, and the depth map generation chip outputs a depth image according to the infrared speckle map, etc., and the present application does not limit this.

[0094] In the embodiment of the present application, the user operation can be detected, and the depth image can be obtained in response to the user operation. Alternatively, the image acquisition can be performed continuously or intermittently to obtain the depth image. The depth image carries the depth information corresponding to each pixel.

[0095] Step 102: For a target pixel in the depth image, determine the original coordinates in the coordinate system of the depth image.

[0096] In an embodiment of the present application, the target pixel point may be each pixel point in the depth image, or, in order to improve the efficiency of image registration, the target pixel point may also be part of the pixel points in the depth image. For example, the depth image may be downsampled, such as alternate row sampling or alternate column sampling. In this case, the target pixel points may be the pixel points in odd rows, even rows, odd columns, even columns, etc. in the depth image. The present application does not impose any restrictions on this.

[0097] In the embodiment of the present application, the coordinate system of the depth image is an image coordinate system corresponding to the depth image, which is pre-established.

[0098] As an example, the coordinate origin in the coordinate system of the depth image can be the center point of the depth image, the X axis is horizontal to the right, the Y axis is horizontal downward, and the unit is pixel. At this time, the coordinate system of the depth image is a common image coordinate system for computer vision.

[0099] As another example, the coordinate origin in the coordinate system of the depth image can be the upper left corner of the depth image, the X axis is horizontal to the right, and the Y axis is horizontal downward, and the unit is pixel. At this time, the coordinate system of the depth image can be Figure 1 or Figure 2 The L pixel coordinate system in, that is, the image coordinate system coincides with the common pixel coordinate system of computer vision.

[0100] As another example, the coordinate origin in the coordinate system of the depth image may be the upper right corner of the depth image, with the X axis pointing horizontally to the left and the Y axis pointing horizontally downward, and the unit is pixel.

[0101] It should be noted that the process of establishing the coordinate system of the above-mentioned depth image is only an exemplary description. In actual application, the coordinate origin in the coordinate system of the depth image can also be the lower left corner of the depth image, with the X-axis horizontally to the right and the Y-axis horizontally upward, and the unit is pixel, or the coordinate origin in the coordinate system of the depth image can be the lower right corner of the depth image, with the X-axis horizontally to the left and the Y-axis horizontally upward, and the unit is pixel, etc. This application does not impose any restrictions on this.

[0102] For ease of explanation, in the embodiments of the present application, the coordinate system of the depth image is used as an example as a pixel coordinate system commonly used in computer vision.

[0103] In the embodiment of the present application, after determining each target pixel point, the original coordinates of each target pixel point in the coordinate system of the depth image can be determined. For example, when the coordinate system of the depth image coincides with the pixel coordinate system commonly used in computer vision, the horizontal coordinate in the original coordinate represents the column of the target pixel point in the depth image, and the vertical coordinate represents the row of the target pixel point in the depth image.

[0104] Step 103, according to the original coordinates of the target pixel point, query the constant mapping relationship between the original coordinates and the coordinate constant part to obtain the corresponding coordinate constant part.

[0105] In the embodiment of the present application, the coordinate constant part is the part related to the original coordinates of the pixel point, the internal parameters of the depth sensor for collecting the depth image, and the internal parameters of the imaging sensor for collecting the imaging image. Figure 2 , since the distortion of the image is only related to the intrinsic parameters of the imaging sensor, in order to improve the registration accuracy, the coordinate constant part can also include Figure 2 The distortion correction step in .

[0106] The imaging image is acquired synchronously with the depth image, and the imaging image may be an image based on visual perception, such as a color image or an RGB image. The imaging sensor for acquiring the imaging image may be an image sensor such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or a TFT (Thin Film Transistor).

[0107] In an embodiment of the present application, the coordinate constant part can be calculated during the initialization process of the image registration algorithm. For example, during the initialization process, the original coordinates of each pixel in the depth image can be predetermined, and the coordinate constant part corresponding to each original coordinate can be calculated based on the intrinsic parameters of the depth sensor, the intrinsic parameters of the imaging sensor, and the original coordinates. The original coordinates and the coordinate constant part are stored in an array corresponding to each other. During image registration, the coordinate constant part corresponding to the original coordinates of the target pixel can be read directly through the search method to improve the real-time performance of data calculation.

[0108] Step 104 , query the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel point, so as to obtain the corresponding coordinate variable part.

[0109] In the embodiment of the present application, the coordinate variable part is the part related to the depth measured by the depth sensor and the external parameters between the depth sensor and the imaging sensor.

[0110] In an embodiment of the present application, in order to improve the real-time performance of data calculation, the maximum range of the depth sensor measuring depth can be covered, and the coordinate variable part corresponding to each possible depth Z is calculated according to the external parameters between the depth sensor and the imaging sensor, and the depth and the coordinate variable part are stored in an array. During image registration, the coordinate variable part corresponding to the depth carried by the target pixel point is directly read out through the search method.

[0111] It should be noted that the present application only takes step 104 being executed after step 103 as an example. In actual application, in order to improve the efficiency of image registration, step 104 can also be executed in parallel with step 103, or, after each target pixel point is determined in step 102, step 104 can be executed. The present application does not impose any restrictions on this.

[0112] Step 105, determining the target coordinates of the target pixel point to be registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part.

[0113] In the embodiment of the present application, the coordinate system of the imaged image is the image coordinate system corresponding to the imaged image, which is also preset. The coordinate system of the imaged image is established in the same way as the coordinate system of the depth image. For example, when the coordinate system of the depth image is Figure 1 or Figure 2 When the L pixel coordinate system is used, the coordinate system of the image can be Figure 1 or Figure 2 The R pixel coordinate system in the L pixel coordinate system. That is, the origin of the coordinates in the L pixel coordinate system is the upper left corner of the depth image, the X axis is horizontal to the right, the Y axis is horizontally downward, and the unit is pixel. The origin of the coordinates in the R pixel coordinate system is also the upper left corner of the imaging image, the X axis is horizontal to the right, and the Y axis is horizontally downward, and the unit is pixel.

[0114] In the embodiment of the present application, after the coordinate constant part and the coordinate variable part are obtained by querying, the coordinate constant part and the coordinate variable part can be added and summed to obtain the target coordinates of the target pixel point registered in the coordinate system of the imaged image. Specifically, the sum of the abscissa of the coordinate constant part and the abscissa of the coordinate variable part can be used as the abscissa in the target coordinates, and the sum of the ordinate of the coordinate constant part and the ordinate of the coordinate variable part can be used as the ordinate in the target coordinates.

[0115] In the present application, by pre-establishing a constant mapping relationship between the original coordinates and the constant part of the coordinates, as well as a variable mapping relationship between the depth and the variable part of the coordinates, in the process of image registration, the constant part of the coordinates corresponding to the original coordinates of each target pixel point in the depth image, and the variable part of the coordinates corresponding to the depth carried by the target pixel point are directly read out through a search method, so that the constant part of the coordinates and the variable part of the coordinates are added together to obtain the target coordinates of the target pixel point registered in the coordinate system of the imaging image, which can greatly improve the efficiency of image registration, can be applied to devices with lower computing power, and improve the applicability of the method.

[0116] The image registration method of the embodiment of the present application pre-establishes a constant mapping relationship between the original coordinates and the constant part of the coordinates, and a variable mapping relationship between the depth and the variable part of the coordinates. During the image registration process, the constant part of the coordinates corresponding to the original coordinates of each target pixel in the depth image and the variable part of the coordinates corresponding to the depth carried by the target pixel are directly read out through a search method. Thus, the target coordinates of the target pixel in the coordinate system of the imaging image can be directly calculated based on the constant part of the coordinates and the variable part of the coordinates. Therefore, for each target pixel, there is no need to repeat the matrix multiplication calculation, which can greatly improve the efficiency of image registration and can be applied to devices with lower computing power, thereby improving the applicability of the method.

[0117] As a possible implementation method, a first transformation matrix can be calculated based on the intrinsic parameter matrix of the depth sensor, the intrinsic parameter matrix of the imaging sensor, and the extrinsic parameter matrix between the depth sensor and the imaging sensor, and a variable mapping relationship between the depth and the coordinate variable part can be generated based on the time-varying elements in the first transformation matrix whose values ​​vary with the depth. The above process is described in detail below in conjunction with Example 2.

[0118] Figure 4 A flowchart of the image registration method provided in Example 2 of the present application.

[0119] like Figure 4 As shown, the process of generating the variable mapping relationship may include the following steps:

[0120] Step 201 , calibrating the internal and external parameters of a depth sensor for acquiring a depth image and an image sensor for acquiring an imaging image, so as to obtain a first transformation matrix between the depth image and the imaging image.

[0121] For example, the internal parameter matrix of the depth sensor is obtained by calibrating the internal parameters of the depth sensor as I L , perform internal calibration on the imaging sensor, and obtain the internal parameter matrix of the imaging sensor as I R , perform external parameter calibration on the depth sensor and image sensor, and obtain the external parameter matrix between the depth sensor and the imaging sensor as E L2R , then the first transformation matrix can be Figure 1 The matrix M in is:

[0122]

[0123] Step 202: Generate a variable mapping relationship between the depth and the coordinate variable part according to the time-varying elements whose values ​​vary with the depth in the first transformation matrix.

[0124] For example, see Figure 5 In the first transformation matrix M, the time-varying element whose value changes with depth is m 14 and m 24 , so, according to m 14 and m 24 , directly calculate the coordinate variable part corresponding to the depth carried by the target pixel. For example, mark the depth carried by the target pixel as Z, and the horizontal coordinate of the coordinate variable part is The vertical axis is but Right now

[0125] Furthermore, in order to further improve the efficiency of image registration, during image registration, there is no need to directly calculate the corresponding coordinate variable part based on the depth carried by each target pixel point. Instead, a variable mapping relationship between the depth and the coordinate variable part is generated in advance based on the time-varying elements in the first transformation matrix whose values ​​change with the depth. Therefore, during image registration, the coordinate variable part corresponding to the depth carried by the target pixel point can be read directly through the search method.

[0126] As a possible implementation method, the maximum range of the depth sensor can be covered, and the coordinate variable part corresponding to each possible depth Z under the maximum range can be pre-calculated. The depth and the coordinate variable part are stored in the array. When registering the image, the coordinate variable part corresponding to the depth carried by the target pixel point is directly read out by the search method. Therefore, when registering the image, there is no need to calculate the corresponding coordinate variable part for each target pixel point, which can further improve the efficiency of image registration.

[0127] Specifically, the coordinate variable part corresponding to each depth can be determined according to the ratio between the time-varying element and each depth within the depth measurement range of the depth sensor. For example, if the depth is marked as Z, the horizontal coordinate of the coordinate variable part is The vertical axis is Afterwards, a variable mapping relationship is generated according to the coordinate variable parts corresponding to each depth.

[0128] It should be noted that the present application does not limit the execution process of steps 201 to 202. For example, steps 201 to 202 can be executed before step 101, or steps 201 to 202 can also be executed before steps 102, 103 or 104. The present application does not impose any restrictions on this.

[0129] In the embodiment of the present application, the coordinate variable part is related to the depth information collected by the depth sensor. During image registration, the corresponding coordinate variable part can be directly calculated in real time according to the time-varying element and the depth carried by the target pixel point. Alternatively, in order to improve the efficiency of image registration and reduce the amount of calculation, the coordinate variable part corresponding to each depth can be calculated in advance according to each depth and time-varying element within the depth measurement range of the depth sensor during the initialization process of the image registration algorithm, and a variable mapping relationship between the depth and the coordinate variable part is generated, so that during image registration, the corresponding coordinate variable part can be read out by searching. Therefore, the coordinate variable part corresponding to the target pixel point can be determined in different ways, further improving the applicability of the method.

[0130] As a possible implementation method, the second transformation matrix can be calculated based on the intrinsic parameter matrix of the depth sensor and the extrinsic parameter matrix between the depth sensor and the imaging sensor, and the constant mapping relationship between the original coordinates and the constant part of the coordinates can be generated based on the constant elements with fixed values ​​in the second transformation matrix. The above process is described in detail below in conjunction with Example 3.

[0131] Figure 6 A flowchart of the image registration method provided in Example 3 of the present application.

[0132] like Figure 6 As shown, the process of generating the constant mapping relationship may include the following steps:

[0133] Step 301 , performing an internal parameter calibration on a depth sensor used to acquire a depth image, and performing an external parameter calibration on the depth sensor and an image sensor used to acquire an imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor.

[0134] For example, the internal parameter matrix of the depth sensor is obtained by calibrating the internal parameters of the depth sensor as I L , perform external parameter calibration on the depth sensor and image sensor, and obtain the external parameter matrix between the depth sensor and the imaging sensor as E L2R , then the second transformation matrix can be Figure 2 or Figure 5 The matrix M in 1 ,Right now:

[0135]

[0136] Step 302: Generate a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix.

[0137] Depend on Figure 5 It can be seen that the second transformation matrix M 1 The constant elements with fixed values ​​are Then we can use the constant matrix Generates a constant mapping relationship between the original coordinates and the constant part of the coordinates.

[0138] It should be noted that the present application does not limit the execution process of steps 301 to 302. For example, steps 301 to 302 can be executed before step 101, or steps 301 to 302 can also be executed before step 102 or 103. The present application does not impose any restrictions on this.

[0139] Generally, the image may have a large distortion. In order to improve the accuracy of image registration, as a possible implementation method of the present application, a distortion correction step may be added when generating the coordinate constant part. The above process is described in detail below in conjunction with Example 4.

[0140] Figure 7 A flowchart of the image registration method provided in Example 4 of the present application.

[0141] like Figure 7 As shown, based on the above embodiment, step 302 may specifically include the following sub-steps:

[0142] Step 401, using the second transformation matrix, transforming the original coordinates to obtain intermediate coordinates.

[0143] For example, the original coordinates of the target pixel in the marked depth image are (u L , v L ), see Figure 5 , we can use the first transformation matrix M 1 A 3*3 constant matrix composed of constant elements with fixed values with [u L ,v L ,1] T Multiply, that is, M 1 c *[u L ,v L ,1] T , get the 3*1 vector [X R ,Y R ,1] T , that is, the intermediate coordinates can be (X R , Y R ).

[0144] Step 402: Perform distortion correction on the intermediate coordinates.

[0145] For example, see Figure 5 , we can convert (X R ,Y R ) is reorganized into a matrix of size 3*5 Then multiply the above reorganized 3*5 matrix by the 5*1 distortion coefficient vector [1k 1 k 2 p 1 p 2 ] T , and get a 3*1 matrix Then the intermediate coordinate after distortion correction is (X' R ,Y' R ).

[0146] Wherein, k1 and k2 are radial distortion parameters of the imaging sensor, and p1 and p2 are tangential distortion parameters of the imaging sensor.

[0147] Step 403 , using the third transformation matrix obtained by calibrating the intrinsic parameters of the image sensor, coordinate transformation is performed on the intermediate coordinates after the distortion correction to obtain the coordinate constant part corresponding to the original coordinates.

[0148] For example, see Figure 5 , the third transformation matrix obtained by calibrating the image sensor internal parameters is I R ,Right now Then through get That is, the coordinates of the coordinate constant part are

[0149] Step 404: Generate a constant mapping relationship according to the original coordinates and the corresponding constant part of the coordinates.

[0150] In the embodiment of the present application, after the coordinate constant part corresponding to the original coordinates is calculated, a constant mapping relationship between the original coordinates and the coordinate constant part can be generated.

[0151] In the present application, by predetermining the original coordinates corresponding to each pixel point in the depth image, calculating the coordinate constant part corresponding to the original coordinates corresponding to each pixel point, and generating a constant mapping relationship between the original coordinates and the coordinate constant part, during image registration, the coordinate constant part corresponding to each target pixel point can be read out by a search method, thereby improving the efficiency of image registration.

[0152] like Figure 5 As shown, the coordinate constant part corresponding to the target pixel point is obtained in the query And the coordinate variable part After that, the target coordinates (u R , v R )for

[0153] In order to implement the above embodiment, the present application also proposes an image registration device.

[0154] Figure 8 This is a structural schematic diagram of the image registration device provided in Example 5 of the present application.

[0155] like Figure 8 As shown, the image registration device 100 includes: an acquisition module 110 , a determination module 120 , a first query module 130 , a second query module 140 and a registration module 150 .

[0156] The acquisition module 110 is used to acquire a depth image.

[0157] The determination module 120 is used to determine the original coordinates of the target pixel point in the depth image in the coordinate system of the depth image.

[0158] The first query module 130 is used to query the constant mapping relationship between the original coordinates and the coordinate constant part according to the original coordinates of the target pixel point, so as to obtain the corresponding coordinate constant part.

[0159] The second query module 140 is used to query the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel point, so as to obtain the corresponding coordinate variable part.

[0160] The registration module 150 is used to determine the target coordinates of the target pixel point to be registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part.

[0161] As a possible implementation method, the registration module 150 is specifically used to: use the sum of the horizontal coordinate of the coordinate constant part and the horizontal coordinate of the coordinate variable part as the horizontal coordinate in the target coordinate; use the sum of the vertical coordinate of the coordinate constant part and the vertical coordinate of the coordinate variable part as the vertical coordinate in the target coordinate.

[0162] Further, in a possible implementation of the embodiment of the present application, see Fig. 9 ,exist Figure 8 Based on the illustrated embodiment, the image registration device 100 may further include:

[0163] The first calibration module 160 is used to calibrate the internal and external parameters of the depth sensor used to acquire the depth image and the image sensor used to acquire the imaging image, so as to obtain a first transformation matrix between the depth image and the imaging image.

[0164] The first generating module 170 is used to generate a variable mapping relationship between the depth and the coordinate variable part according to the time-varying element whose value changes with the depth in the first transformation matrix.

[0165] As a possible implementation method, the first generation module 170 is specifically used to: determine the coordinate variable part corresponding to each depth based on the ratio between the time-varying element and each depth within the depth measurement range of the depth sensor; and generate a variable mapping relationship based on the coordinate variable part corresponding to each depth.

[0166] The second calibration module 180 is used to perform an internal parameter calibration on a depth sensor used to acquire a depth image, and to perform an external parameter calibration on the depth sensor and an image sensor used to acquire an imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor.

[0167] The second generating module 190 is used to generate a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix.

[0168] As a possible implementation method, the second generation module 190 is specifically used to: use the second transformation matrix to perform coordinate transformation on the original coordinates to obtain intermediate coordinates; perform distortion correction on the intermediate coordinates; use the third transformation matrix obtained by calibrating the internal parameters of the image sensor to perform coordinate transformation on the intermediate coordinates after distortion correction to obtain the coordinate constant part corresponding to the original coordinates; generate a constant mapping relationship based on the original coordinates and the corresponding coordinate constant part.

[0169] It should be noted that the above explanation of the embodiment of the image registration method is also applicable to the image registration device 100 of this embodiment, and will not be repeated here.

[0170] The image registration device of the embodiment of the present application pre-establishes a constant mapping relationship between the original coordinates and the constant part of the coordinates, and a variable mapping relationship between the depth and the variable part of the coordinates. During the image registration process, the constant part of the coordinates corresponding to the original coordinates of each target pixel in the depth image and the variable part of the coordinates corresponding to the depth carried by the target pixel are directly read out through a search method. Therefore, the target coordinates of the target pixel in the coordinate system of the imaging image can be directly calculated based on the constant part of the coordinates and the variable part of the coordinates. Therefore, for each target pixel, there is no need to repeat the matrix multiplication calculation, which can greatly improve the efficiency of image registration and can be applied to devices with lower computing power to improve applicability.

[0171] In order to implement the above embodiments, the present application also proposes an electronic device.

[0172] Fig.10 This is a schematic diagram of the structure of the electronic device provided in Example 7 of the present application.

[0173] like Fig.10 As shown, the electronic device includes: a memory 201, a processor 202, and a computer program stored in the memory 201 and executable on the processor 202. When the processor 202 executes the program, the image registration method proposed in the aforementioned embodiment of the present application is implemented.

[0174] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image registration method proposed in the above embodiments of the present application is implemented.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0176] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0177] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0178] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0179] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0180] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0182] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image registration method, characterized in that: The method comprises the following steps: Get the depth image; For a target pixel in the depth image, determining an original coordinate in a coordinate system of the depth image; Performing an internal parameter calibration on a depth sensor used to acquire the depth image, and performing an external parameter calibration on the depth sensor and an image sensor used to acquire an imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor; Generate a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix According to the original coordinates of the target pixel point, query the constant mapping relationship between the original coordinates and the coordinate constant part to obtain the corresponding coordinate constant part; Calibrate the depth sensor used to acquire the depth image and the image sensor used to acquire the imaging image to obtain a first transformation matrix between the depth image and the imaging image; Generate a variable mapping relationship between the depth and the coordinate variable part according to the time-varying element whose value changes with the depth in the first transformation matrix According to the depth carried by the target pixel point, query the variable mapping relationship between the depth and the coordinate variable part to obtain the corresponding coordinate variable part; According to the coordinate constant part and the coordinate variable part, the target coordinates of the target pixel point registered in the coordinate system of the imaging image are determined.

2. The image registration method according to claim 1, characterized in that: The step of determining the target coordinates of the target pixel point to be registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part comprises: The sum of the abscissa of the coordinate constant part and the abscissa of the coordinate variable part is used as the abscissa in the target coordinate; The sum of the ordinate of the coordinate constant part and the ordinate of the coordinate variable part is used as the ordinate in the target coordinate.

3. The image registration method according to claim 1, characterized in that: The generating, according to the time-varying element whose value varies with the depth in the first transformation matrix, a variable mapping relationship between the depth and the coordinate variable part comprises: Determining a coordinate variable portion corresponding to each depth according to a ratio between the time-varying element and each depth within a depth measurement range of the depth sensor; The variable mapping relationship is generated according to the coordinate variable parts corresponding to each depth.

4. The image registration method according to claim 1, characterized in that: The step of generating a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix includes: Using the second transformation matrix, performing coordinate transformation on the original coordinates to obtain intermediate coordinates; Performing distortion correction on the intermediate coordinates; Using a third transformation matrix obtained by calibrating the intrinsic parameters of the image sensor, coordinate transformation is performed on the intermediate coordinates after distortion correction to obtain a coordinate constant part corresponding to the original coordinates; The constant mapping relationship is generated according to the original coordinates and the corresponding coordinate constant part.

5. An image registration device, characterized in that: The device comprises: An acquisition module, used for acquiring a depth image; A determination module, configured to determine, for a target pixel in the depth image, an original coordinate in a coordinate system of the depth image; A first query module, used for querying the constant mapping relationship between the original coordinates and the coordinate constant part according to the original coordinates of the target pixel point, so as to obtain the corresponding coordinate constant part; A second query module is used to query the variable mapping relationship between the depth and the coordinate variable part according to the depth carried by the target pixel point, so as to obtain the corresponding coordinate variable part; A registration module, used for determining the target coordinates of the target pixel point to be registered in the coordinate system of the imaging image according to the coordinate constant part and the coordinate variable part; A first calibration module, used for calibrating internal and external parameters of a depth sensor used for acquiring the depth image and an image sensor used for acquiring the imaging image, so as to obtain a first transformation matrix between the depth image and the imaging image; A first generating module, configured to generate a variable mapping relationship between the depth and the coordinate variable part according to a time-varying element whose value varies with the depth in the first transformation matrix; A second calibration module, used to perform an internal parameter calibration on a depth sensor used to acquire the depth image, and to perform an external parameter calibration on the depth sensor and an image sensor used to acquire the imaging image, so as to obtain a second transformation matrix between the depth image and the image sensor; The second generating module is used to generate a constant mapping relationship between the original coordinates and the constant part of the coordinates according to the constant elements with fixed values ​​in the second transformation matrix.

6. The image registration device according to claim 5, characterized in that: The registration module is specifically used for: The sum of the abscissa of the coordinate constant part and the abscissa of the coordinate variable part is used as the abscissa in the target coordinate; The sum of the ordinate of the coordinate constant part and the ordinate of the coordinate variable part is used as the ordinate in the target coordinate.

7. The image registration device according to claim 5, characterized in that: The first generating module is specifically used for: Determining a coordinate variable portion corresponding to each depth according to a ratio between the time-varying element and each depth within a depth measurement range of the depth sensor; The variable mapping relationship is generated according to the coordinate variable parts corresponding to each depth.

8. The image registration device according to claim 5, characterized in that: The second generating module is specifically used for: Using the second transformation matrix, performing coordinate transformation on the original coordinates to obtain intermediate coordinates; Performing distortion correction on the intermediate coordinates; Using a third transformation matrix obtained by calibrating the intrinsic parameters of the image sensor, coordinate transformation is performed on the intermediate coordinates after distortion correction to obtain a coordinate constant part corresponding to the original coordinates; The constant mapping relationship is generated according to the original coordinates and the corresponding coordinate constant part.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image registration method as claimed in any one of claims 1 to 4 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image registration method as described in any one of claims 1 to 4 is implemented.

Images