Fast registration method and device for RGB-D camera module and RGB-D camera module

Abstract

Disclosed are a rapid registration method and apparatus for RGB-D camera modules. This method simplifies the registration process by registering local or specific points in an image through a cropping-and-zooming approach, reducing data processing and improving registration efficiency. Furthermore, since portions of the RGB image without corresponding depth regions are not included in the subsequent registration process, registration can be completed even if at least a portion of the RGB image has no corresponding depth region.

Description

Technical Field

[0001] The present application relates to the field of RGB-D camera modules, and more specifically to a fast registration method and device for RGB-D camera modules and an RGB-D camera module. Background Art

[0002] An RGB-D camera module is a camera module that can output not only the RGB information of the object being photographed, but also its depth information. With the continuous development of machine vision, RGB-D camera modules are increasingly being used in fields such as autonomous driving, visual obstacle avoidance, and facial recognition.

[0003] In practical applications, it is usually necessary not only to collect the RGB information and depth information of the subject, but also to align the depth information and RGB information. In traditional registration methods, RGB-D registration is mainly achieved by placing pixels of the depth image on a one-to-one basis with pixels of the RGB image (i.e., point-by-point correspondence), or by interpolating pixels of the depth image and pixels of the RGB image to perform one-to-many registration.

[0004] However, traditional registration methods have some problems. For example, the computational complexity of calculating correspondences point by point is large, resulting in a very long time consumption. For another example, RGB-D registration can only be achieved under the premise that the entire RGB image has a corresponding depth area; otherwise, the registered image may be incomplete.

[0005] Therefore, an optimized registration method for RGB-D camera modules is expected to improve the registration efficiency. Summary of the Invention

[0006] One advantage of the present application is that it provides a fast registration method, device and RGB-D camera module for an RGB-D camera module, wherein the registration method for the RGB-D camera module can achieve registration only by determining the positions of specific points of the infrared image and the RGB image, thereby greatly reducing the amount of data processing to improve the registration efficiency.

[0007] Another advantage of the present application is that it provides a fast registration method, device and RGB-D camera module for an RGB-D camera module, wherein the registration method for the RGB-D camera module performs registration by cropping and zooming, and the part of the RGB image that originally has no corresponding depth area does not participate in the subsequent registration process. Therefore, the registration can be completed under the premise that at least a part of the RGB image has no corresponding depth area.

[0008] Another advantage of the present application is that it provides a fast registration method, device and RGB-D camera module for an RGB-D camera module, wherein, in the registration method for the RGB-D camera module, the RGB image is cropped around its specific reference point and with a cropping size consistent with the aspect ratio of the infrared image. In this way, registration can be achieved by achieving correspondence between a first reference point in the infrared image and a second reference point in the RGB image, as well as correspondence between at least one corner point of the infrared image and at least one corner point of the enlarged cropped area, which can greatly simplify the registration process and improve registration efficiency.

[0009] Another advantage of the present application is that a fast registration method, device and RGB-D camera module are provided for an RGB-D camera module, wherein, in the registration method for the RGB-D camera module, the first reference point of the infrared image is determined by the center point of the point cloud area, which can effectively avoid the reference point being a flying point or a pixel point without depth information.

[0010] To achieve at least one of the above advantages or other advantages and purposes, according to one aspect of the present application, a registration method for an RGB-D camera module is provided, comprising:

[0011] Get the calibration parameters of the RGB-D camera module;

[0012] Obtaining an RGB image and source data of the photographed target through the RGB camera and depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera;

[0013] Processing the source data based on the calibration parameters to obtain a depth point cloud and an infrared image, wherein the depth point cloud is aligned with the infrared image;

[0014] Calculating a focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera;

[0015] Determining the size of the to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio;

[0016] Calculating an offset between a first reference point in the infrared image and a second reference point in the RGB image;

[0017] Determining a position of the area to be cropped in the RGB image based on the offset; and

[0018] The area to be cropped is magnified using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image.

[0019] In the registration method for an RGB-D camera module according to the present application, the first reference point is the optical center of the infrared image, and the second reference point is the optical center of the RGB image.

[0020] In the alignment method for an RGB-D camera module according to the present application, calculating the offset between a first reference point in the infrared image and a second reference point in the RGB image includes calculating the offset between the first reference point and the second reference point using the following formula, wherein the formula is: x=x2–x1 / a, y=y2–y1 / a, wherein (x1, y1) represents the optical center coordinates of the infrared image, (x2, y2) represents the optical center coordinates of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

[0021] In the registration method for an RGB-D camera module according to the present application, determining the position of the area to be cropped in the RGB image based on the offset includes: using the offset as the coordinates of a corner point of the area to be cropped in the RGB image; and determining the position of the area to be cropped in the RGB image based on the coordinates of the corner point and the size of the area to be cropped.

[0022] In the registration method for an RGB-D camera module according to the present application, the offset between the first reference point in the infrared image and the second reference point in the RGB image is calculated, including: extracting a point cloud area with a preset size in the central area of the depth point cloud; counting the depth values of each pixel point in the point cloud area to obtain the center point of the point cloud area, and then obtaining the center point of the effective area in the central area of the infrared image as the first reference point, wherein the center point of the effective area in the central area of the infrared image is aligned with the center point of the point cloud area; projecting the point cloud area to the coordinate system of the RGB image based on the calibration parameters of the RGB-D camera module to obtain a projection area; and counting the values of each pixel point in the projection area to obtain the center point of the projection area as the second reference point.

[0023] In the alignment method for an RGB-D camera module according to the present application, the offset between the first reference point and the second reference point is calculated using the following formula, wherein the formula is: x = x2'–x1' / a, y = y2'–y1' / a, wherein (x1', y1') represent the coordinates of the center point of the effective area in the central area of the infrared image, (x2', y2') represent the coordinates of the center point of the projection area of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

[0024] In the registration method for an RGB-D camera module according to the present application, determining the position of the area to be cropped in the RGB image based on the offset includes: using the offset as the coordinates of a corner point of the area to be cropped in the RGB image; and determining the position of the area to be cropped in the RGB image based on the coordinates of the corner point and the size of the area to be cropped.

[0025] In the registration method for an RGB-D camera module according to the present application, the shape of the point cloud area is a square, and its size is 10*10 to 50*50.

[0026] In the registration method for an RGB-D camera module according to the present application, the area to be cropped is magnified using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is aligned with the infrared image, including: rotating the area to be cropped based on the calibration parameters; and magnifying the area to be cropped using the focal length ratio as a magnification factor to obtain the registration area.

[0027] In the registration method for an RGB-D camera module according to the present application, the difference between the field of view angles of the RGB camera and the depth camera in the horizontal direction is greater than 2°, and the distance between the center points of the RGB camera and the depth camera is 10 mm.

[0028] According to another aspect of the present application, there is provided an RGB-D camera module, comprising:

[0029] RGB camera;

[0030] a depth camera, the field of view of the RGB camera including the field of view of the depth camera; and

[0031] A data processing device is used to execute the fast registration method for RGB-D camera modules as described above.

[0032] In the RGB-D camera module according to the present application, the difference between the field of view angles of the RGB camera and the depth camera in the horizontal direction is greater than 2°.

[0033] In the RGB-D camera module according to the present application, the distance between the optical centers of the RGB camera and the depth camera in the horizontal direction is less than 15 mm.

[0034] According to another aspect of the present application, a registration device for an RGB-D camera module is provided, comprising:

[0035] A calibration parameter acquisition unit, used to obtain calibration parameters of the RGB-D camera module;

[0036] A source image acquisition unit, configured to obtain an RGB image and an infrared image of the object through the RGB camera and the depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera;

[0037] a depth calculation unit, configured to process the infrared image based on the calibration parameters to obtain a depth point cloud;

[0038] A focal length ratio calculation unit, configured to calculate a focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera;

[0039] a size determining unit, configured to determine a size of a to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio;

[0040] an offset calculation unit, configured to calculate an offset between a first reference point in the infrared image and a second reference point in the RGB image;

[0041] a to-be-cropped area determining unit, configured to determine a position of the to-be-cropped area in the RGB image based on the offset; and

[0042] The registration area generating unit is configured to magnify the area to be cropped using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image.

[0043] According to another aspect of the present application, there is provided an electronic device, comprising:

[0044] processor; and

[0045] A memory, in which computer program instructions are stored, which, when executed by the processor, enable the processor to execute the fast registration method for the RGB-D camera module as described above.

[0046] Further objectives and advantages of the present application will be fully reflected through understanding of the following description and drawings.

[0047] These and other objects, features and advantages of the present application are fully reflected in the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] These and / or other aspects and advantages of the present application will become more clear and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:

[0049] Figure 1The figure shows a flow chart of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0050] Figure 2 The figure shows a schematic diagram of the field of view range of the RGB camera and the depth camera in the horizontal direction in the registration method for the RGB-D camera module according to an embodiment of the present application.

[0051] Figure 3 The figure shows another schematic diagram of the field of view range of the RGB camera and the depth camera in the horizontal direction in the registration method for the RGB-D camera module according to an embodiment of the present application.

[0052] Figure 4 The figure illustrates a flow chart of a specific example of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0053] Figure 5A The figure illustrates one of the process schematic diagrams of a specific example of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0054] Figure 5B The figure illustrates a second process diagram of a specific example of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0055] Figure 5C The figure illustrates a third process diagram of a specific example of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0056] Figure 5D The figure illustrates a fourth process diagram of a specific example of a registration method for an RGB-D camera module according to an embodiment of the present application.

[0057] Figure 6 The figure illustrates a flow chart of calculating the offset between the first reference point in the infrared image and the second reference point in the RGB image in another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0058] Figure 7 The figure illustrates a flow chart of another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0059] Figure 8A The figure illustrates one of the process schematic diagrams of another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0060] Figure 8BThe figure illustrates a second process diagram of another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0061] Figure 8C The figure illustrates a third process diagram of another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0062] Figure 8D The figure illustrates a fourth process diagram of another specific example of the registration method for an RGB-D camera module according to an embodiment of the present application.

[0063] Figure 9 The figure shows a schematic block diagram of a registration device for an RGB-D camera module according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following description is intended to disclose the present application so that those skilled in the art can implement the present application. The embodiments described below are for illustrative purposes only, and those skilled in the art may conceive of other obvious variations. The basic principles of the present application defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present application.

[0065] Application Overview

[0066] As mentioned above, in the traditional registration method, RGB-D registration is mainly achieved by making a one-to-one correspondence between the pixels of the depth image and the pixels of the RGB image (i.e., a point-by-point correspondence method), or by interpolating the pixels of the depth image and the pixels of the RGB image to perform registration in a one-to-many manner.

[0067] However, traditional registration methods have some problems. For example, the computational complexity of calculating correspondences point by point is large, resulting in a very long time consumption. For another example, RGB-D registration can only be achieved under the premise that the entire RGB image has a corresponding depth area; otherwise, the registered image may be incomplete.

[0068] In response to the time-consuming problem of registration, the applicant proposes that if the registration can be performed on a local or specific point of the image, the registration process can be greatly simplified, the amount of data processing can be reduced, and the registration efficiency can be improved. Specifically, the second reference point in the RGB image corresponding to the first reference point of the infrared image is cropped with a cropping size consistent with the aspect ratio of the infrared image. By achieving a correspondence between the first reference point in the infrared image and the second reference point in the RGB image, as well as a correspondence between at least one corner point of the infrared image and at least one corner point of the magnified cropped area, registration can be achieved to simplify the registration process and improve registration efficiency. In particular, in this registration scheme, since the portion of the RGB image that originally has no corresponding depth area does not participate in the subsequent registration process, registration can be completed under the premise that at least a portion of the RGB image has no corresponding depth area.

[0069] Based on this, the present application proposes a registration method for an RGB-D camera module, which includes: obtaining calibration parameters of the RGB-D camera module; obtaining an RGB image and source data of the photographed target through the RGB camera and depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera; processing the source data based on the calibration parameters to obtain a depth point cloud and an infrared image, wherein the depth point cloud is aligned with the infrared image; calculating the focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera; determining the size of the area to be cropped in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio; calculating the offset between the first reference point in the infrared image and the second reference point in the RGB image; determining the position of the area to be cropped in the RGB image based on the offset; and enlarging the area to be cropped with the focal length ratio as the magnification to obtain a registration area in the RGB image that is registered with the infrared image.

[0070] In addition, the present application also provides a registration device for an RGB-D camera module, which includes: a calibration parameter acquisition unit, a source image acquisition unit, a depth calculation unit, a focal length ratio calculation unit, a size determination unit, an offset calculation unit, a to-be-cropped area determination unit, and a registration area generation unit. The calibration parameter acquisition unit is used to acquire calibration parameters of the RGB-D camera module. The source image acquisition unit is used to obtain an RGB image and an infrared image of the photographed target through the RGB camera and the depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera. The depth calculation unit is used to process the infrared image based on the calibration parameters to obtain a depth point cloud. The focal length ratio calculation unit is used to calculate the focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera. The size determination unit is used to determine the size of the to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio. The offset calculation unit is used to calculate the offset between a first reference point in the infrared image and a second reference point in the RGB image. The to-be-cropped region determining unit is configured to determine a position of the to-be-cropped region in the RGB image based on the offset. The registration region generating unit is configured to magnify the to-be-cropped region using the focal length ratio as a magnification factor to obtain a registration region in the RGB image that is registered with the infrared image.

[0071] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0072] Exemplary Registration Methods

[0073] like Figure 1As shown, according to an embodiment of the present application, a registration method for an RGB-D camera module is illustrated, and the registration method for an RGB-D camera module includes: S110, obtaining calibration parameters of the RGB-D camera module; S120, obtaining an RGB image and source data of the photographed target through the RGB camera and the depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera; S130, processing the source data based on the calibration parameters to obtain a depth point cloud and an infrared image, wherein the depth point cloud is aligned with the infrared image; S140, calculating the depth point cloud. The method further comprises the steps of: determining a focal length ratio between an effective focal length of a high-resolution camera and an effective focal length of the RGB camera; S150, determining a size of a to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio; S160, calculating an offset between a first reference point in the infrared image and a second reference point in the RGB image; S170, determining a position of the to-be-cropped area in the RGB image based on the offset; and S180, magnifying the to-be-cropped area using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image.

[0074] In step S110, calibration parameters of the RGB-D camera module 60 are obtained. Specifically, the RGB-D camera module 60 includes an RGB camera 61 for collecting RGB information and a depth camera 62 for collecting depth information. The specific type of the depth camera 62 is not limited by this application, and the depth camera 62 can be a structured light camera, a binocular camera, or a TOF (Time of Flight) camera.

[0075] The calibration parameters of the RGB-D camera module 60 include calibration internal parameters and calibration external parameters. The calibration internal parameters include the principal point (c) and focal length (f) of the RGB camera 61, and the calibration external parameters include the pose transformation parameters between the RGB camera 61 and the depth camera 62, and the pose transformation parameters include a first parameter (R) containing axis rotation information and a second parameter (T) containing axis translation information. In some embodiments of the present application, the calibration parameters include distortion parameters, and the distortion parameters include a tangential distortion parameter (p), a radial distortion parameter (k), and a fisheye model distortion parameter (θ).

[0076] In step S120, the RGB image and source data of the subject are obtained by the RGB camera 61 and the depth camera 62 of the RGB-D camera module 60, respectively. Specifically, the RGB camera 61 can obtain an RGB image containing the color information of the subject. The source data and the depth image containing the depth information are obtained. The source data varies depending on the type of the depth camera 62. For example, the photoelectric conversion value of different phases of each pixel can be obtained by an i-TOF (Indirect Time of Flight) camera.

[0077] like Figure 2 As shown, in this embodiment of the present application, the field of view of the RGB camera 61 includes the field of view of the depth camera 62, so that the field of view of the RGB image covers the field of view of the depth image or infrared image subsequently generated by the depth camera 62, which facilitates the subsequent registration process. When the optical axis of the RGB camera 61 and the optical axis of the depth camera 62 deviate in a certain direction (for example, the horizontal direction), the difference in the field of view angles of the RGB camera 61 and the depth camera 62 in that direction, as well as the distance between the RGB camera 61 and the depth camera 62 in that direction will affect the registration effect.

[0078] The greater the difference in the field of view angles of the RGB camera 61 and the depth camera 62 in this direction, the closer the field of view intersection point of the RGB camera 61 and the depth camera 62 in this direction is to the RGB camera 61 and the depth camera 62, such as Figure 3 When the difference in the field of view angles of the RGB camera 61 and the depth camera 62 in this direction is small, the distance between the field of view intersection points of the RGB camera 61 and the depth camera 62 in this direction is far.

[0079] The farther the distance between the RGB camera 61 and the depth camera 62 in this direction, the farther the field of view intersection point of the RGB camera 61 and the depth camera 62 in this direction is from the RGB camera 61 and the depth camera 62. When the distance between the RGB camera 61 and the depth camera 62 in this direction is far, the distance between the field of view intersection point of the RGB camera 61 and the depth camera 62 in this direction is far.

[0080] The long distance between the intersection points of the fields of view of the RGB camera 61 and the depth camera 62 in this direction may cause the field of view of the RGB image of the subject at a close distance (for example, 300 mm) to not completely cover the field of view of the depth image or the infrared image, resulting in a decrease in the image registration accuracy.

[0081] Accordingly, it is necessary to limit the field of view of the RGB camera 61 and the depth camera 62, as well as the distance between the RGB camera 61 and the depth camera 62. In some embodiments of the present application, the difference in the field of view of the RGB camera 61 and the depth camera 62 in the horizontal direction is greater than 2°, and the distance between the optical centers of the RGB camera 61 and the depth camera 62 in the horizontal direction is less than 15 mm.

[0082] In a specific example of the present application, the RGB camera 61 has a vertical field of view of 80° and a horizontal field of view of 53°. The depth camera 62 has a vertical field of view of 75° and a horizontal field of view of 50°. The difference in the horizontal field of view between the RGB camera 61 and the depth camera 62 is 3°. In this specific example, the horizontal distance between the optical centers of the RGB camera 61 and the depth camera 62 is 10 mm.

[0083] In step S130, the source data is processed based on the calibration parameters to obtain a depth point cloud and an infrared image. Specifically, a depth image and an infrared image can be calculated from the source data, and the depth image can be converted into a depth point cloud based on the calibration parameters. The size of the depth image is consistent with the size of the infrared image, and the pixels of the depth image and the infrared image have a one-to-one correspondence. Accordingly, the depth point cloud and the infrared image are aligned, that is, the pixels of the depth point cloud correspond one-to-one with the pixels of the infrared image.

[0084] In an embodiment of the present application, a coordinate system can be established based on the calibration parameters so that the pixels of the depth point cloud correspond to the pixels of the depth image. The pixel coordinate system can be established using the following formula:

[0085]

[0086] In the process of establishing the coordinate system, distortion correction can be performed. Specifically, radial distortion correction can be performed using the following formula:

[0087] x0=x(1+k1r 2 +k2r 4 +k3r 6 ), y0=y(1+k1r 2 +k2r 4 +k3r 6 ), where k1, k2, and K3 are radial distortion parameters.

[0088] Tangential distortion correction can be performed using the following formula:

[0089] x0=x+[2p1y+p2(r 2+2x 2 )],y0=y+[2p2x+p1(r 2 +2y 2 )], where p1 and p2 are tangential distortion parameters.

[0090] The fisheye model distortion correction can be performed using the following formula:

[0091] x0=(θ d / r)*a0,y0=(θ d / r)*b0, where θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ),θ=a tan(r),r 2 =a0 2 +b0 2 .

[0092] In an embodiment of the present application, since the infrared image is aligned with the depth image and the depth point cloud, the infrared image and the RGB image are registered to achieve the registration between the depth image or the depth point cloud and the RGB image. As mentioned above, in the traditional registration method, RGB-D registration is mainly achieved by making a one-to-one correspondence between the pixels of the depth image and the pixels of the RGB image (i.e., a point-by-point correspondence method). The point-by-point calculation of the correspondence relationship is computationally intensive, resulting in a long time consumption. Moreover, RGB-D registration can only be achieved under the premise that the entire RGB image has a corresponding depth area.

[0093] In particular, the present application simplifies the registration process by registering local or specific points of the image, reducing the amount of data processing to improve registration efficiency. Specifically, first, the RGB image is cropped based on the size of the infrared image and the focal length ratio; then, the area to be cropped is enlarged and enlarged to the same size as the infrared image. In an embodiment of the present application, by registering the area to be cropped of the RGB image, the part of the RGB image that originally had no corresponding depth area does not participate in the subsequent registration process. Therefore, registration can be completed under the premise that at least a part of the RGB image has no corresponding depth area.

[0094] More specifically, in this embodiment of the present application, the RGB image has a large field of view and needs to be cropped. Since the focal lengths of the depth camera 62 and the RGB camera 61 are different, before cropping the RGB image, the focal length ratio between the effective focal length of the depth camera 62 and the effective focal length of the RGB camera 61 needs to be determined. Accordingly, in step S140, the focal length ratio between the effective focal length of the depth camera 62 and the effective focal length of the RGB camera 61 is calculated. Here, the effective focal length refers to the distance from the center of the lens to the focal point.

[0095] In step S150, the size of the area to be cropped in the RGB image corresponding to the infrared image is determined based on the size of the infrared image and the focal length ratio. In an embodiment of the present application, the size of the area to be cropped is the ratio of the size of the infrared image to the focal length ratio. That is, if the width of the infrared image is w and the height is h, the width of the area to be cropped is w / a and the height is h / a, where a represents the focal length ratio. The aspect ratio of the area to be cropped is consistent with the aspect ratio of the infrared image, so that the area to be cropped can be consistent with the size of the infrared image after being magnified using the focal length ratio as the magnification factor.

[0096] In the embodiment of the present application, the RGB image is cropped around a specific reference point, and an offset of the to-be-cropped region is determined to determine a positional relationship between the to-be-cropped region and the infrared image. Accordingly, in step S160, an offset between a first reference point in the infrared image and a second reference point in the RGB image is calculated.

[0097] like Figures 4 to 5D As shown, in a specific example of the present application, the optical center of the infrared image is used as the first reference point, and the optical center of the RGB image is used as the second reference point. The offset between the first reference point and the second reference point is calculated using the following formula, wherein the formula is: x = x2–x1 / a, y = y2–y1 / a, where (x1, y1) represents the coordinates of the optical center of the infrared image, (x2, y2) represents the coordinates of the optical center of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

[0098] like Figures 6 to 8D As shown, in another specific example of the present application, the center point of the effective area of the central area of the infrared image is used as the first reference point, and the center point of the projection area corresponding to the effective area of the central area of the infrared image in the RGB image is used as the second reference point, so as to avoid the first reference point corresponding to the second reference point being a flying point, or a pixel point without depth information.

[0099] In this specific example, the offset between the first reference point and the second reference point is calculated using the following formula, wherein the formula is: x = x2'–x1' / a, y = y2'–y1' / a, wherein (x1', y1') represent the coordinates of the center point of the effective area in the central area of the infrared image, (x2', y2') represent the coordinates of the center point of the projection area of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

[0100] In this specific example, first, a point cloud area with a preset size in the central area of the depth point cloud is extracted; then, the depth values of each pixel point in the point cloud area are counted to obtain the center point of the point cloud area, and then the center point of the effective area in the central area of the infrared image is obtained as the first reference point, wherein the center point of the effective area in the central area of the infrared image is aligned with the center point of the point cloud area; then, based on the calibration parameters of the RGB-D camera module 60, the point cloud area is projected to the coordinate system of the RGB image to obtain a projection area; subsequently, the values of each pixel point in the projection area are counted to obtain the center point of the projection area as the second reference point.

[0101] Correspondingly, step S160 includes: S161, extracting a point cloud area with a preset size in the central area of the depth point cloud; S162, counting the depth values of each pixel point in the point cloud area to obtain the center point of the point cloud area, and then obtaining the center point of the effective area in the central area of the infrared image as the first reference point, wherein the center point of the effective area in the central area of the infrared image is aligned with the center point of the point cloud area; S163, projecting the point cloud area to the coordinate system of the RGB image based on the calibration parameters of the RGB-D camera module 60 to obtain a projection area; and, S164, counting the values of each pixel point in the projection area to obtain the center point of the projection area as the second reference point.

[0102] In step S161, a point cloud region of appropriate size should be extracted to avoid a decrease in projection stability due to a small size and a large computational complexity due to a large size. In some embodiments of the present application, the point cloud region is square in shape and has a size of 10*10 to 50*50. In a specific example, the point cloud region has a size of 20*20.

[0103] In step S162, the valid area of the RGB image refers to the area in the central area corresponding to the depth point cloud where the depth value is not zero, that is, the depth value of each pixel in the point cloud area is not zero. Accordingly, it is necessary to count the maximum depth and minimum depth of the pixels in the horizontal direction (excluding zero value) and the maximum depth and minimum depth of the pixels in the vertical direction (excluding zero value) in the central area of the depth point cloud to obtain the center point of the point cloud area. Since the depth point cloud is aligned with the infrared image, the center point of the valid area of the central area of the infrared image can be determined by the center point of the point cloud area.

[0104] After determining the offset between the first reference point and the second reference point, the position of the area to be cropped in the RGB image can be determined. In an embodiment of the present application, the offset is used as the coordinates of a corner point of the area to be cropped in the RGB image, and the position of the area to be cropped in the RGB image is determined based on the coordinates of the corner point and the size of the area to be cropped.

[0105] Accordingly, step S170 includes: using the offset as the coordinates of a corner point of the area to be cropped in the RGB image; and determining the position of the area to be cropped in the RGB image based on the coordinates of the corner point and the size of the area to be cropped.

[0106] In step S180, the region to be cropped is magnified using the focal length ratio as a magnification factor to obtain a registered region in the RGB image that is registered with the infrared image. Because the aspect ratio of the region to be cropped is consistent with that of the infrared image, the region to be cropped, after being magnified using the focal length ratio as a magnification factor, can be consistent with the size of the infrared image. Furthermore, since the offset (x = x2 – x1 / a, y = y2 – y1 / a) is used as the coordinates of a corner point of the region to be cropped in the RGB image, after the region to be cropped is magnified using the focal length ratio as a magnification factor, the second reference point corresponds to the first reference point, and the corner point of the region to be cropped in the RGB image corresponds to the corresponding corner point of the infrared image, thereby obtaining a registered region in the RGB image that is registered with the infrared image.

[0107] Specifically, first, the region to be cropped is rotated based on the calibration parameters to perform rotation correction, ensuring no rotation in the z-direction. The rotation angle is obtained from the calibration parameters. Next, the region to be cropped is magnified using the focal length ratio as a magnification factor to obtain the registration region. Specifically, the magnification can be performed using bilinear interpolation or other methods, and this is not limited to the present application.

[0108] In summary, the rapid registration method for RGB-D camera modules has been described. This method significantly reduces data processing through specific point registration or local registration, thereby improving registration efficiency. Furthermore, this method can complete registration even when at least a portion of the RGB image has no corresponding depth region.

[0109] Example RGB-D camera module

[0110] According to another aspect of the present application, an RGB-D camera module 60 is further provided. The RGB-D camera module 60 includes an RGB camera 61, a depth camera 62, and a data processing device 63. The positional relationship between the RGB camera 61 and the depth camera 62 is such that the field of view of the RGB camera 61 includes the field of view of the depth camera 62.

[0111] Specifically, in some embodiments of the present application, the difference in the field of view angles of the RGB camera 61 and the depth camera 62 in the horizontal direction is greater than 2°, and the distance between the optical centers of the RGB camera 61 and the depth camera 62 in the horizontal direction is less than 15 mm.

[0112] In a specific example of the present application, the RGB camera 61 has a vertical field of view of 80° and a horizontal field of view of 53°. The depth camera 62 has a vertical field of view of 75° and a horizontal field of view of 50°. The difference in the horizontal field of view between the RGB camera 61 and the depth camera 62 is 3°. In this specific example, the horizontal distance between the optical centers of the RGB camera 61 and the depth camera 62 is 10 mm.

[0113] The data processing device 63 is used to perform Figures 1 to 8D The fast registration method for RGB-D camera module is shown in FIG. Here, the fast registration method for RGB-D camera module has been described in the reference to FIG. Figures 1 to 8D The illustrated fast registration method for the RGB-D camera module is described in detail, and therefore, its repeated description will be omitted.

[0114] Exemplary Registration Device

[0115] According to another aspect of the present application, a fast registration device 10 for an RGB-D camera module is also provided. Figure 9As shown, the rapid registration device 10 for the RGB-D camera module includes: a calibration parameter acquisition unit 11, a source image acquisition unit 12, a depth calculation unit 13, a focal length ratio calculation unit 14, a size determination unit 15, an offset calculation unit 16, a to-be-cropped area determination unit 17, and a registration area generation unit 18.

[0116] Specifically, the calibration parameter acquisition unit 11 is used to obtain the calibration parameters of the RGB-D camera module. The source image acquisition unit 12 is used to obtain an RGB image and an infrared image of the subject through the RGB camera 61 and the depth camera 62 of the RGB-D camera module 60, respectively, wherein the field of view of the RGB camera 61 includes the field of view of the depth camera 62. The depth calculation unit 13 is used to process the infrared image based on the calibration parameters to obtain a depth point cloud. The focal length ratio calculation unit 14 is used to calculate the focal length ratio between the effective focal length of the depth camera 62 and the effective focal length of the RGB camera 61. The size determination unit 15 is used to determine the size of the area to be cropped corresponding to the infrared image in the RGB image based on the size of the infrared image and the focal length ratio; the offset calculation unit 16 is used to calculate the offset between the first reference point in the infrared image and the second reference point in the RGB image. The area to be cropped determination unit 17 is used to determine the position of the area to be cropped in the RGB image based on the offset. The registration region generating unit 18 is configured to magnify the to-be-cropped region using the focal length ratio as a magnification factor to obtain a registration region in the RGB image that is registered with the infrared image.

[0117] The specific functions of each unit have been referenced above. Figures 1 to 8D The illustrated fast registration method for the RGB-D camera module is described in detail, and therefore, its repeated description will be omitted.

[0118] In summary, the rapid registration device 10 for an RGB-D camera module has been described. The rapid registration device 10 for an RGB-D camera module significantly reduces data processing through specific point registration or local registration, thereby improving registration efficiency. Furthermore, registration can be performed even when at least a portion of the RGB image has no corresponding depth region.

[0119] Exemplary electronic devices

[0120] According to another aspect of the present application, an electronic device 80 is provided, the electronic device 80 comprising: a memory 81 and a processor 82, wherein the memory 81 stores computer program instructions, and when the computer program instructions are executed by the processor 82, the processor 82 executes a reference Figures 1 to 8D The fast registration method for RGB-D camera module is shown in FIG. Here, the fast registration method for RGB-D camera module has been described in the reference to FIG. Figures 1 to 8D The illustrated fast registration method for the RGB-D camera module is described in detail, and therefore, its repeated description will be omitted.

[0121] In summary, the electronic device 80 is explained, and the electronic device 80 can execute a fast registration method for an RGB-D camera module to improve registration efficiency.

[0122] Those skilled in the art will understand that the embodiments of the present application described above and shown in the accompanying drawings are intended only as examples and do not limit the present application. The objectives of the present application have been fully and effectively achieved. The functional and structural principles of the present application have been demonstrated and explained in the embodiments. The embodiments of the present application may be modified or altered in any manner without departing from the principles described.

Claims

1. A fast registration method for RGB-D camera modules, characterized in that: include: Get the calibration parameters of the RGB-D camera module; Obtaining an RGB image and source data of the photographed target through the RGB camera and depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera; Processing the source data based on the calibration parameters to obtain a depth point cloud and an infrared image, wherein the depth point cloud is aligned with the infrared image; Calculating a focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera; Determining the size of the to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio; Calculating an offset between a first reference point in the infrared image and a second reference point in the RGB image; Determining a position of the area to be cropped in the RGB image based on the offset; and The area to be cropped is magnified using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image.

2. The fast registration method for RGB-D camera module according to claim 1, wherein: The first reference point is the optical center of the infrared image, and the second reference point is the optical center of the RGB image.

3. The fast registration method for RGB-D camera module according to claim 2, wherein: Calculating an offset between a first reference point in the infrared image and a second reference point in the RGB image includes: The offset between the first reference point and the second reference point is calculated using the following formula, where the formula is: x = x2–x1 / a, y = y2–y1 / a, where (x1, y1) represents the optical center coordinates of the infrared image, (x2, y2) represents the optical center coordinates of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

4. The rapid registration method for an RGB-D camera module according to claim 3, wherein: Determining a position of the to-be-cropped area in the RGB image based on the offset includes: Using the offset as the coordinates of a corner point of the area to be cropped in the RGB image; and Based on the coordinates of the corner points and the size of the area to be cropped, a position of the area to be cropped in the RGB image is determined.

5. The fast registration method for RGB-D camera module according to claim 1, wherein: Calculating an offset between a first reference point in the infrared image and a second reference point in the RGB image includes: Extracting a point cloud region having a preset size in a central region of the depth point cloud; Counting the depth values of each pixel point in the point cloud area to obtain a center point of the point cloud area, and then obtaining a center point of a valid area in the central area of the infrared image as the first reference point, wherein the center point of the valid area in the central area of the infrared image is aligned with the center point of the point cloud area; Projecting the point cloud area onto the coordinate system of the RGB image based on the calibration parameters of the RGB-D camera module to obtain a projection area; and The values of each pixel in the projection area are counted to obtain the center point of the projection area as the second reference point.

6. The fast registration method for RGB-D camera module according to claim 5, wherein: The offset between the first reference point and the second reference point is calculated using the following formula: x = x2'-x1' / a, y = y2'-y1' / a, wherein (x1', y1') represent the coordinates of the center point of the effective area in the central area of the infrared image, (x2', y2') represent the coordinates of the center point of the projection area of the RGB image, a represents the focal length ratio, x represents the offset in the u direction, and y represents the offset in the v direction.

7. The fast registration method for RGB-D camera module according to claim 6, wherein: Determining a position of the to-be-cropped area in the RGB image based on the offset includes: Using the offset as the coordinates of a corner point of the area to be cropped in the RGB image; and Based on the coordinates of the corner points and the size of the area to be cropped, a position of the area to be cropped in the RGB image is determined.

8. The fast registration method for RGB-D camera module according to claim 5, wherein: The point cloud area is in the shape of a square, and its size is 10*10 to 50*50.

9. The fast registration method for RGB-D camera module according to claim 1, wherein: The method further comprises: enlarging the area to be cropped by using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image, comprising: Rotating the area to be cropped based on the calibration parameters; and The area to be cropped is magnified using the focal length ratio as a magnification factor to obtain the registration area.

10. The fast registration method for RGB-D camera module according to claim 1, wherein: The difference between the field of view angles of the RGB camera and the depth camera in the horizontal direction is greater than 2°, and the distance between the center points of the RGB camera and the depth camera is 10 mm.

11. An RGB-D camera module, characterized in that: include: RGB camera; A depth camera, wherein the field of view of the RGB camera includes the field of view of the depth camera; as well as A data processing device for executing the fast registration method for an RGB-D camera module as described in any one of claims 1 to 10.

12. The RGB-D camera module according to claim 11, wherein: The difference between the field of view angles of the RGB camera and the depth camera in the horizontal direction is greater than 2°.

13. The RGB-D camera module according to claim 12, wherein: The distance between the optical centers of the RGB camera and the depth camera in the horizontal direction is less than 15 mm.

14. A rapid registration device for an RGB-D camera module, characterized in that: include: A calibration parameter acquisition unit, used to obtain calibration parameters of the RGB-D camera module; A source image acquisition unit, configured to obtain an RGB image and an infrared image of the object through the RGB camera and the depth camera of the RGB-D camera module, respectively, wherein the field of view of the RGB camera includes the field of view of the depth camera; a depth calculation unit, configured to process the infrared image based on the calibration parameters to obtain a depth point cloud; A focal length ratio calculation unit, configured to calculate a focal length ratio between the effective focal length of the depth camera and the effective focal length of the RGB camera; a size determining unit, configured to determine a size of a to-be-cropped area in the RGB image corresponding to the infrared image based on the size of the infrared image and the focal length ratio; an offset calculation unit, configured to calculate an offset between a first reference point in the infrared image and a second reference point in the RGB image; a to-be-cropped area determining unit, configured to determine a position of the to-be-cropped area in the RGB image based on the offset; and The registration area generating unit is configured to magnify the area to be cropped using the focal length ratio as a magnification factor to obtain a registration area in the RGB image that is registered with the infrared image.

15. An electronic device, characterized in that: include: processor; as well as A memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the fast registration method for an RGB-D camera module as described in any one of claims 1 to 10.

Images