Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device and a storage medium. The method comprises: acquiring two sets of combined images of an object; wherein one set of combined images comprises a first depth image captured by a first camera and a first plane image captured by a second camera, and the other set of combined images comprises a second depth image captured by the first camera and a second plane image captured by the second camera; the depth image and the plane image in any set of combined images have a mapping relationship; determining a homography matrix between the first plane image and the second plane image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information represents the distance between the object and the first camera. Through this method, the accuracy of homography estimation can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to an image processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the field of image processing, a homography matrix is defined as a projection mapping matrix from one plane to another. Assuming that the homography between two images has been obtained, the homography matrix H can be used to associate the two images, for example, by using the formula A = H * B to associate image A and image B, that is, using the homography matrix H to transform image B to image A.

[0003] The homography matrix has many practical applications, such as image correction, registration, and estimation of the motion between two cameras in SLAM. However, how to accurately estimate the homography matrix is crucial. SUMMARY

[0004] The present disclosure provides an image processing method and device, an electronic device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising:

[0006] obtaining two sets of combined images of a collection object; wherein one set of combined images comprises a first depth image collected by a first camera and a first plane image collected by a second camera, and the other set of combined images comprises a second depth image collected by the first camera and a second plane image collected by the second camera; the depth image and the plane image in any set of combined images have a mapping relationship;

[0007] determining a homography matrix between the first plane image and the second plane image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information represents the distance between the collection object and the first camera.

[0008] In some embodiments, the determination of the homography matrix between the first plane image and the second plane image based on the mapping relationship and the depth information included in the first depth image and the second depth image comprises:

[0009] performing depth region division on the first depth image and the second depth image to obtain first depth sub-images of different depth regions and second depth sub-images of different depth regions; wherein the different depth regions represent local regions on the collection object that have different distances to the first camera;

[0010] According to the mapping relationship, a first planar sub-image corresponding to each depth region first depth sub-image on the first planar image is obtained, and a second planar sub-image corresponding to each depth region second depth sub-image on the second planar image is obtained.

[0011] According to a group of the first planar sub-images and the second planar sub-images of the same depth region, a homography sub-matrix is established.

[0012] According to the homography sub-matrix corresponding to the combination of multiple groups of the first planar sub-images and the second planar sub-images, a homography matrix between the first planar image and the second planar image is determined.

[0013] In some embodiments, the method comprises:

[0014] Feature point matching is performed on the first planar image and the second planar image to obtain a matching combination between pixel points in the first planar image and pixel points in the second planar image.

[0015] The homography sub-matrix is established according to a group of the first planar sub-images and the second planar sub-images of the same depth region, comprising:

[0016] From the matching combination of the pixel points, a target matching combination belonging to the first planar sub-image and the second planar sub-image of the same depth region is selected.

[0017] Based on the target matching combination, the homography sub-matrix is calculated.

[0018] In some embodiments, the weights of different homography sub-matrices in the homography matrix are different; wherein the weight of the homography sub-matrix corresponding to a target depth region is greater than the weight of the homography sub-matrix corresponding to a depth region other than the target depth region.

[0019] In some embodiments, the method comprises:

[0020] The first camera is calibrated offline to obtain a first calibration result; the first calibration result is used to correct the image collected by the first camera to obtain the first depth image and the second depth image.

[0021] The second camera is calibrated offline to obtain a second calibration result; the second calibration result is used to correct the image collected by the second camera to obtain the first planar image and the second planar image.

[0022] Based on the first calibration result and the second calibration result, the mapping relationship is determined.

[0023] In some embodiments, the method comprises:

[0024] According to the homography matrix, the first planar image and the second planar image are registered.

[0025] According to a second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, comprising:

[0026] The acquisition module is configured to acquire two groups of combined images of a collection object; wherein one group of combined images comprises a first depth image collected by a first camera and a first planar image collected by a second camera, and the other group of combined images comprises a second depth image collected by the first camera and a second planar image collected by the second camera; the depth image and the planar image in any group of combined images have a mapping relationship therebetween;

[0027] The determination module is configured to determine a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information represents the distance between the collection object and the first camera.

[0028] In some embodiments, the determination module is further configured to perform depth region division on the first depth image and the second depth image to obtain first depth sub-images of different depth regions and second depth sub-images of different depth regions; wherein different depth regions represent local regions with different distances to the first camera on the collection object; according to the mapping relationship, the first depth sub-image of each depth region is mapped to a corresponding first planar sub-image on the first planar image, and the second depth sub-image of each depth region is mapped to a corresponding second planar sub-image on the second planar image; a homography sub-matrix is established according to a group of first planar sub-images and second planar sub-images of the same depth region; and the homography matrix between the first planar image and the second planar image is determined according to the homography sub-matrices corresponding to combinations of multiple groups of first planar sub-images and second planar sub-images.

[0029] In some embodiments, the apparatus comprises:

[0030] The feature matching module is configured to perform feature point matching on the first planar image and the second planar image to obtain a matching combination between pixel points in the first planar image and pixel points in the second planar image.

[0031] The determination module is further configured to select target matching combinations belonging to the first planar sub-image and the second planar sub-image of the same depth region from the matching combination of the pixel points; and calculate the homography sub-matrix based on the target matching combinations.

[0032] In some embodiments, different homographic sub-matrices in the homographic matrix have different weights; and a homographic sub-matrix corresponding to a target depth region has a greater weight than a homographic sub-matrix corresponding to a depth region other than the target depth region.

[0033] In some embodiments, the apparatus comprises:

[0034] a first calibration module configured to calibrate the first camera offline to obtain a first calibration result; the first calibration result being used to correct an image captured by the first camera to obtain the first depth image and the second depth image;

[0035] a second calibration module configured to calibrate the second camera offline to obtain a second calibration result; the second calibration result being used to correct an image captured by the second camera to obtain the first plane image and the second plane image;

[0036] a mapping module configured to determine the mapping relationship based on the first calibration result and the second calibration result.

[0037] In some embodiments, the apparatus comprises:

[0038] a registration module configured to register the first plane image and the second plane image according to the homographic matrix.

[0039] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, comprising:

[0040] a processor;

[0041] a memory for storing processor-executable instructions;

[0042] The processor is configured to perform the image processing method as described in the first aspect above.

[0043] According to a fourth aspect of embodiments of the present disclosure, a storage medium is provided, comprising:

[0044] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in the first aspect above.

[0045] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0046] In the embodiments of the present disclosure, the images captured by the combination of the depth camera (first camera) and the plane camera (second camera) can fuse the depth information in the depth image when performing the homographic estimation between the plane images, thereby improving the accuracy of the homographic estimation.

[0047] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, in which:

[0049] Figure 1 is a flow chart of an image processing method according to an example embodiment of the present disclosure.

[0050] Figure 2 is an example diagram of an application based on a homography matrix according to an example embodiment of the present disclosure.

[0051] Figure 3 is an example diagram of a depth region division according to an example embodiment of the present disclosure.

[0052] Figure 4 is an example diagram of obtaining a calibration result according to an example embodiment of the present disclosure.

[0053] Figure 5 is an example diagram of image correction based on a calibration result according to an example embodiment of the present disclosure.

[0054] Figure 6 is a schematic diagram of an image processing method according to an example embodiment of the present disclosure.

[0055] Figure 7 is an example diagram of obtaining a planar sub-image according to an example embodiment of the present disclosure.

[0056] Figure 8 is a diagram of an image processing device according to an example embodiment.

[0057] Figure 9 is a block diagram of an electronic device apparatus according to an example embodiment. DETAILED DESCRIPTION

[0058] The example embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following example embodiments do not represent all the implementations in accordance with the present disclosure. Instead, they only represent examples of apparatus and methods in accordance with some aspects of the present disclosure, as detailed in the appended claims.

[0059] Figure 1 is a flow chart of an image processing method according to an example embodiment of the present disclosure, as shown in Figure 1 the image processing method applied in an electronic device comprises the following steps:

[0060] S11, acquire two sets of combined images of a collection object; wherein one set of combined images comprises a first depth image collected by a first camera and a first plane image collected by a second camera, and another set of combined images comprises a second depth image collected by the first camera and a second plane image collected by the second camera; the depth image and the plane image in any set of combined images have a mapping relationship therebetween;

[0061] S12, determine a homography matrix between the first plane image and the second plane image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information represents a distance between the collection object and the first camera.

[0062] In an embodiment of the present disclosure, the electronic device can be any server or terminal having data processing capability, including but not limited to a mobile phone, a tablet computer, or a personal computer (PC).

[0063] In step S11, the electronic device acquires two sets of combined images of a collection object. The collection object can be a person, an animal, a plant, or a natural scene such as a blue sky, white clouds, or a sunset. The two sets of combined images of the collection object can be images corresponding to short-focus shooting or images corresponding to long-focus shooting, which is not limited in the embodiment of the present disclosure.

[0064] In an embodiment of the present disclosure, any set of combined images comprises a depth image collected by a depth camera (first camera) and a plane image collected by a normal color camera (second camera). The depth image includes depth information, which reflects the distance between each part of the collection object and the first camera.

[0065] In an embodiment of the present disclosure, the internal parameters of the first camera and the second camera for collecting any set of combined images can be the same, and the external parameters of the first camera and the second camera can also be the same. The internal parameters of the camera are related to the characteristics of the camera itself, such as the focal length and pixel size of the camera; the external parameters are in the world coordinate system, such as the position and rotation direction of the camera. Of course, the internal parameters of the first camera and the second camera for collecting any set of combined images can be different, and the external parameters can also be different. The present disclosure can determine the mapping relationship between the depth image and the plane image in any set of combined images based on the external parameters of the first camera and the second camera.

[0066] In step S12, the electronic device can determine the homography matrix between the first plane image and the second plane image based on the mapping relationship between the depth image and the plane image, and the depth information included in the first depth image and the second depth image.

[0067] As described above, the internal parameters and the external parameters of the first camera and the second camera for collecting any set of combined images can be the same, so that the information of the two-dimensional plane images included in the images collected by the first camera and the second camera in the same set of combinations is consistent, and only the additional distance information is included in the depth image. At this time, the mapping relationship between the depth image and the plane image in the same combination is a one-to-one mapping row by row and column by column. In this embodiment, when determining the homography matrix between the first plane image and the second plane image based on the mapping relationship and the depth information, for example, the average collection distances of the first camera in each set of combined images can be calculated based on the depth information of each pixel point included in the first depth image and the second depth image, and the difference between the average collection distances can be used for subsequent estimation of the homography matrix. Specifically, when subsequently estimating the homography matrix, feature point detection and matching can be performed based on the values of each pixel point in the first plane image and the second plane image and the ratio between the average collection distances, and then the mapping relationship between the feature points is calculated, that is, the homography matrix of the present disclosure is obtained. The ratio between the average collection distances can be used as a position constraint condition for matching pixel points when performing feature point detection and matching. For example, when the collection directions of the first plane image and the second plane image are consistent, the ratio between the average collection distances determined based on the first depth image and the second depth image is 2, and then for a certain pixel point A on the first plane image, a matching pixel point can be found in the range of 2 times the pixel in the up, down, left and right of the position of the pixel point A in the second plane image, so as to further calculate the homography matrix based on the matching pixel point.

[0068] In this embodiment, the accuracy of feature point detection and matching can be improved based on the constraint of the depth information in the first depth image and the second depth image, so that the accuracy of the homography matrix estimation can be improved.

[0069] In another embodiment, considering that the distances of different regions on the collected object to the first camera are not the same, the first plane image and the second plane image can also be subjected to block homography matrix estimation after region division based on different depth information on the depth image and in combination with the mapping relationship, so as to improve the accuracy of the homography matrix by more detailed division, and the block homography matrix estimation method will be described in detail later in the present disclosure.

[0070] It can be understood that in the embodiments of the present disclosure, the images collected by the combination of the depth camera (the first camera) and the plane camera (the second camera) enable the depth information in the depth image to be fused when performing homography estimation between the plane images, so as to improve the accuracy of the homography estimation.

[0071] In an embodiment, the method comprises:

[0072] According to the homography matrix, the first planar image and the second planar image are registered.

[0073] After the homography matrix is determined, the first planar image and the second planar image can be matched and superimposed by using the homography matrix, for example, multi-pose face registration. Of course, the embodiments of the present disclosure are not limited to registration by using the homography matrix, but also can use the homography matrix for image stitching, etc.

[0074] Figure 2 An example diagram for the application based on the homography matrix in the embodiments of the present disclosure is shown in FIG. 1. Figure 2 As shown in FIG. 1, after the electronic device obtains two groups of images with different acquisition poses by using a depth camera (i.e., a first camera) and an RGB camera (i.e., a second camera), the depth image (i.e., a first depth image) and the depth image (i.e., a second depth image) are combined to perform multi-pose registration on the color image (i.e., a first planar image) and the color image (i.e., a second planar image). Figure 1 Figure 2 Figure 1 Figure 2

[0075] As described above, after the region division based on different depth information on the depth image, the first planar image and the second planar image can be subjected to block homography matrix estimation in combination with the mapping relationship. In this embodiment, step S12 can specifically include the following steps.

[0076] The first depth image and the second depth image are subjected to depth region division to obtain a first depth sub-image of different depth regions and a second depth sub-image of different depth regions; wherein the different depth regions represent local regions with different distances from the first camera on the acquisition object.

[0077] According to the mapping relationship, a first planar sub-image corresponding to the first depth sub-image of each depth region on the first planar image and a second planar sub-image corresponding to the second depth sub-image of each depth region on the second planar image are obtained.

[0078] A homography sub-matrix is established according to a group of the first planar sub-images and the second planar sub-images with the same depth region.

[0079] According to the homography sub-matrices corresponding to the combinations of the groups of first planar sub-images and second planar sub-images, a homography matrix between the first planar image and the second planar image is determined.

[0080] Figure 3 An example diagram for depth region division shown in the embodiments of the present disclosure is shown in FIG. 2. Figure 3 ​​​​As shown, when depth region division is performed on the first depth image and the second depth image, pixels with depth information less than 50 centimeters (cm) can be divided into one depth region, pixels with depth information between 50 cm and 100 cm can be divided into one depth region, pixels with depth information between 100 cm and 200 cm can be divided into one depth region, pixels with depth information between 200 cm and 350 cm can be divided into one depth region, and pixels with depth information greater than 350 cm can be divided into another depth region. Of course, the depth region division of the present disclosure is not limited to the above manner, and the depth region division manner can be determined in combination with the accuracy requirement of the determination of the homogeneity matrix and the calculation amount. In general, the more the depth region division, the higher the accuracy, but the calculation amount can also be greater. It should be noted that in the embodiments of the present disclosure, the depth region division manners of the first depth image and the second depth image are the same.

[0081] After obtaining the first depth sub-image of different depth regions based on the first depth image and the second depth sub-image of different depth regions based on the second depth image, the first plane sub-image corresponding to the first depth sub-image on the first plane image and the second plane sub-image corresponding to the second depth sub-image on the second plane can be obtained according to the mapping relationship. It should be noted that in the embodiments of the present disclosure, the mapping relationship between the first depth image and the first plane image can be different from the mapping relationship between the second depth image and the second plane image. When the mapping relationships of the two groups of combined images are different, for example, the mapping relationship can include a first mapping relationship and a second mapping relationship, the first plane sub-image corresponding to the first depth sub-image on the first plane image can be obtained according to the first mapping relationship, and the second plane sub-image corresponding to the second depth sub-image on the second plane can be obtained according to the second mapping relationship.

[0082] It can be understood that since different depth sub-images include different depth information, the plane sub-images corresponding to different depth sub-images also correspond to different depth information. Subsequently, a homography sub-matrix can be established according to a group of first plane sub-images and second plane sub-images with the same depth region, and the homography sub-matrix represents the mapping matrix of the local region on the different first plane image and second plane image captured by the second camera, and the local region corresponds to the same depth region on the first depth image and the second depth image.

[0083] After determining the homography sub-matrix of each local region on the first plane image and the second plane image, the homography sub-matrix can be combined according to the region position to obtain the homography matrix between the first plane image and the second plane image.

[0084] For example, it is assumed that H iis a homography sub-matrix of the i-th block region, then the homography matrix between the first planar image and the second planar image can be expressed as formula (1) as follows:

[0085]

[0086] wherein n represents the number of the block regions, and n is a positive integer greater than 1.

[0087] It should be noted that in this embodiment, the acquisition poses of the two groups of combined images can be different, but the distances of the first camera to the acquisition object in different combinations can be consistent.

[0088] It can be understood that the present disclosure performs depth region division based on the depth information included in the first depth image and the second depth image, then obtains the first planar sub-image and the second planar sub-image of the same depth region according to the mapping relationship, and then solves the homography sub-matrix corresponding to the depth region, and finally combines each homography sub-matrix to obtain the homography matrix corresponding to the first planar image and the second planar image. The limitation brought by estimating the homography matrix only relying on the planar image information is reduced, and the accuracy of homography matrix estimation can be improved based on the region division mapping of depth information.

[0089] It should be noted that in one embodiment of the present disclosure, when the homography sub-matrix is established according to a group of first planar sub-images and second planar sub-images of the same depth region, feature point detection and matching can be directly performed on the first planar sub-image and the second planar sub-image, so as to calculate the homography sub-matrix between the first planar sub-image and the second planar sub-image based on the matched feature point pairs.

[0090] In another embodiment of the present disclosure, other methods can also be used to calculate the homography matrix between the first planar sub-image and the second planar sub-image, and the method comprises:

[0091] Feature point matching is performed on the first planar image and the second planar image to obtain a matching combination between pixel points in the first planar image and pixel points in the second planar image;

[0092] The homography sub-matrix is established according to a group of first planar sub-images and second planar sub-images of the same depth region, and the method comprises:

[0093] From the matching combination of the pixel points, a target matching combination belonging to the first planar sub-image and the second planar sub-image of the same depth region is selected;

[0094] The homography sub-matrix is calculated based on the target matching combination.

[0095] In an embodiment of the present disclosure, feature point matching can be performed based on a Scale Invariant Feature Transform (SIFT) algorithm, and feature point detection and matching can be performed based on a Speeded Up Robust Features (SURF) algorithm or the like. The present disclosure does not make specific limitations on the manner of feature point detection and matching.

[0096] In this embodiment, feature point matching can be performed on the first planar image and the second planar image to obtain matching combinations between pixel points in the first planar image and pixel points in the second planar image. Then, according to the region information of the first planar sub-image and the second planar sub-image, target matching combinations belonging to the first planar sub-image and the second planar sub-image are selected from the matching combinations of the pixel points. For example, the feature points in the matching combinations are blocked, and a mask is used. The regions to which the first planar sub-image and the second planar sub-image belong are set to 1, and the remaining regions are set to 0, so that the target matching combinations are filtered out from the matching combinations of the pixel points. Assuming that the matching combinations of the first planar image and the second planar image are Feature1 and Feature2, the filtered target matching combinations are Feature1_Roi and Feature2_Roi, where Featuren_Roi is a region-limited Featuren, and Featuren_Roi is a subset of Featuren.

[0097] In an embodiment, the weights of different homography sub-matrices in the homography matrix are different; and the weight of the homography sub-matrix corresponding to the target depth region is greater than the weight of the homography sub-matrix corresponding to a depth region other than the target depth region.

[0098] In an embodiment of the present disclosure, the weight of the homography sub-matrix corresponding to the target depth region is set to be relatively large, so as to improve the importance of the homography sub-matrix of the target depth region in the entire homography matrix, thereby obtaining a better matching effect or splicing effect when the homography matrix is used for subsequent image matching or splicing operations.

[0099] For example, when a portrait is collected, the preferred collection distance is 100-300 cm, so the depth region of 100-300 cm can be determined as the focus, i.e., the target depth region of the present disclosure, and other background regions are non-focus regions. When a close-up is collected, the focus distance is usually less than 50 cm, so the depth region less than 50 cm can be determined as the target depth region. The present disclosure can set the target depth region according to the collection method of the collection object, and set the weight corresponding to the homography sub-matrix of the target depth region to be relatively large.

[0100] In the embodiments of the present disclosure, assuming that the weight corresponding to the homography sub-matrix is W=(w1 … w n ), the weighted homography matrix can be expressed as shown in the following formula (2):

[0101] H result =W·H (2)

[0102] wherein H is the matrix shown in the aforementioned formula (1).

[0103] In an embodiment, the method comprises:

[0104] calibrating the first camera to obtain a first calibration result; the first calibration result is used to correct the image collected by the first camera to obtain the first depth image and the second depth image;

[0105] calibrating the second camera to obtain a second calibration result; the second calibration result is used to correct the image collected by the second camera to obtain the first plane image and the second plane image;

[0106] determining the mapping relationship based on the first calibration result and the second calibration result.

[0107] Since the pose difference between the first camera and the second camera in the same group of combined images when collecting images by the camera will cause the position of the pixel points included in the depth image to be different from the position of the pixel points in the plane image, and the distortion of the lens of the camera may exist, which causes the image to be deformed, and cannot truly reflect the original appearance of the collected object, etc., it is necessary to establish a camera imaging geometric model through camera calibration to reduce the influence of the above phenomena on the estimation of the homography matrix. The process of calibrating the camera is also the process of obtaining the intrinsic parameters, extrinsic parameters and distortion parameters of the camera.

[0108] In the embodiments of the present disclosure, the first calibration result of the first camera and the second calibration result of the second camera can be obtained by using an offline calibration method. The offline calibration method, for example, uses a three-dimensional calibration object with non-coplanar special calibration marks to provide the correspondence between the image points and their corresponding three-dimensional space points and calculate the calibration parameters. Of course, in the absence of a calibration object, the present disclosure can also use an online calibration method, such as a scene constraint-based self-calibration and a geometric constraint-based self-calibration method.

[0109] Figure 4 An example diagram for obtaining a calibration result in the embodiments of the present disclosure is shown in FIG. 1. Figure 4 As shown in FIG. 1, offline calibration is performed on the depth image and the RGB image to obtain a calibration result, which includes the intrinsic parameters, extrinsic parameters and distortion parameters of the camera.

[0110] This disclosure allows for offline calibration of a first camera and a second camera using a checkerboard pattern, thereby obtaining calibration results (including a first calibration result and a second calibration result) for the cameras (including the first camera and the second camera). Based on the intrinsic parameters and distortion parameters in the calibration results, the depth image acquired by the first camera and the planar image acquired by the second camera can be corrected. Furthermore, based on the aforementioned extrinsic parameters, a mapping relationship can be established between the depth image acquired by the first camera and the planar image acquired by the second camera. If the relative positions of the first camera and the second camera are different in pose during the acquisition of the two sets of combined images, the aforementioned first mapping relationship and second mapping relationship need to be established separately.

[0111] Figure 5 This is an example diagram illustrating image correction based on calibration results in an embodiment of this disclosure, such as... Figure 5 As shown, the depth camera is the first camera of this disclosure, and the RGB camera is the second camera of this disclosure. The depth image obtained by the depth camera can be stereo corrected using the calibration results to obtain a corrected depth image (including a first depth image and a second depth image). The RGB image obtained by the RGB camera can be stereo corrected using the calibration results to obtain a corrected RGB image (including a first planar image and a second planar image). In addition, it is necessary to combine the extrinsic parameters in the calibration results to establish a mapping relationship between the corrected depth image and the planar image.

[0112] This disclosure uses camera calibration to correct the acquired images, and then estimates the homography matrix based on the two sets of corrected combined images. This reduces erroneous mappings caused by distortions in the uncorrected images, thus improving the accuracy of homography matrix estimation.

[0113] Figure 6 This is a schematic diagram of an image processing method according to an embodiment of the present disclosure, such as... Figure 6 As shown, in depth Figure 1 Using the first depth image and planar image 1 (first planar image) as examples, the initial processing procedure is explained, including depth... Figure 1 The first image is an image obtained by image correction of the image acquired by the depth camera (first camera). The second image is an image obtained by image correction of the image acquired by the RGB camera (second camera). The image correction method can be as described above. Figure 4 and Figure 5 As shown. After obtaining the corrected first depth image, depth region segmentation can be performed. Simultaneously, based on the mapping relationship between the depth image and the planar image, the first planar sub-image (planar sub-image 1) corresponding to the first depth sub-image after depth region segmentation is obtained on the first planar image. Similarly, for depth... Figure 2The same operation is also performed on the second depth image (second depth image) and the planar image 2 (second planar image), to obtain the second planar sub-image (planar sub-image 2) corresponding to the second depth sub-image after depth region division of the second depth image on the second planar image. Figure 7 An example diagram for obtaining a planar sub-image in an embodiment of the present disclosure is shown as follows. Figure 7 After depth region division of the depth image, different depth regions in the depth image are obtained, and further remapping is performed based on the correspondence of the stereo rectification to obtain different regions in the planar image, i.e., to obtain the planar sub-image. The correspondence of the stereo rectification is the mapping relationship mentioned above. As shown in Figure 6 For the planar image 1 and the planar image 2, feature point detection and matching are also performed to obtain the feature point set 1 and the feature point set 2. Each feature point in the feature point set 1 and each feature point in the feature point set 2 have a one-to-one correspondence relationship, and the feature point set 1 and the feature point set 2 are the matching combination between the pixel points in the first planar image and the pixel points in the second planar image. Subsequently, based on the planar sub-image and the feature point set, feature point blocking is performed in the form of a mask to obtain the region feature point subset 1 belonging to the planar sub-image 1 and the region feature point subset 2 belonging to the planar sub-image 2. The region feature point subset 1 and the region feature point subset 2 are the target matching combination mentioned in the present disclosure. Subsequently, the homography sub-matrix H i The homography matrix between the planar image 1 and the planar image 2 can be obtained by combining the homography sub-matrices according to the region positions.

[0114] Figure 8 An image processing apparatus diagram according to an example embodiment is shown. Referring to Figure 8 , the apparatus includes:

[0115] The acquisition module 101 is configured to acquire two sets of combined images of an acquisition object. One set of combined images includes a first depth image acquired by a first camera and a first planar image acquired by a second camera, and the other set of combined images includes a second depth image acquired by the first camera and a second planar image acquired by the second camera. The depth image and the planar image in any set of combined images have a mapping relationship.

[0116] The determination module 102 is configured to determine a homography matrix between the first planar image and the second planar image based on the mapping relationship and depth information included in the first depth image and the second depth image. The depth information represents the distance between the acquisition object and the first camera.

[0117] In some embodiments, the determining module 102 is further configured to perform depth region division on the first depth image and the second depth image to obtain first depth sub-images of different depth regions and second depth sub-images of the different depth regions; wherein the different depth regions represent local regions on the collection object that are different in distance from the first camera; according to the mapping relationship, obtain corresponding first plane sub-images of the first depth sub-images of each depth region on the first plane image, and corresponding second plane sub-images of the second depth sub-images of each depth region on the second plane image; according to a group of the first plane sub-images and the second plane sub-images of the same depth region, establish a homography sub-matrix; and according to the homography sub-matrices corresponding to combinations of multiple groups of the first plane sub-images and the second plane sub-images, determine the homography matrix between the first plane image and the second plane image.

[0118] In some embodiments, the apparatus comprises:

[0119] The feature matching module 103 is configured to perform feature point matching on the first plane image and the second plane image to obtain matching combinations between pixel points in the first plane image and pixel points in the second plane image.

[0120] The determining module 102 is further configured to select target matching combinations belonging to the first plane sub-images and the second plane sub-images of the same depth region from the matching combinations of the pixel points; and calculate the homography sub-matrix based on the target matching combinations.

[0121] In some embodiments, the weights of different homography sub-matrices in the homography matrix are different; wherein the weight of the homography sub-matrix corresponding to a target depth region is greater than the weight of the homography sub-matrix corresponding to a depth region other than the target depth region.

[0122] In some embodiments, the apparatus comprises:

[0123] The first calibration module 104 is configured to perform offline calibration on the first camera to obtain a first calibration result; and the first calibration result is used to correct images collected by the first camera to obtain the first depth image and the second depth image.

[0124] The second calibration module 105 is configured to perform offline calibration on the second camera to obtain a second calibration result; and the second calibration result is used to correct images collected by the second camera to obtain the first plane image and the second plane image.

[0125] The mapping module 106 is configured to determine the mapping relationship based on the first calibration result and the second calibration result.

[0126] In some embodiments, the apparatus includes:

[0127] a registration module 107, configured to register the first planar image and the second planar image according to the homography matrix.

[0128] With reference to the apparatus in the above-described embodiments, specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0129] Figure 9 is a block diagram of an electronic device apparatus 800 according to an exemplary embodiment. The apparatus 800 can be a terminal such as a server or a mobile phone, for example.

[0130] With reference to Figure 9 , the apparatus 800 can include one or more of the following components: a processing component 802, a memory component 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0131] The processing component 802 usually controls overall operations of the apparatus 800, such as operations associated with display, phone call, data communication, camera operation and recording operation. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. Further, the processing component 802 can include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0132] The memory component 804 is configured to store various types of data to support operations of the apparatus 800. Examples of these data include instructions for any application or methods operating on the apparatus 800, contact data, phonebook data, messages, pictures, videos and so on. The memory component 804 can be realized by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0133] The power supply component 806 provides power for various components of the apparatus 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the apparatus 800.

[0134] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors for sensing a touch, a slide and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0135] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting an audio signal.

[0136] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0137] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the device 800, a change in position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and a temperature change of the device 800, among a plethora of other examples. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor (e.g., a CMOS or CCD image sensor) configured to work in conjunction with the camera component 808. In some embodiments, the sensor component 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0138] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0139] In an example embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic elements, for performing the above-described methods.

[0140] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0141] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform an image processing method, the method comprising:

[0142] obtaining two groups of combined images of an acquisition object; wherein one group of combined images comprises a first depth image acquired by a first camera and a first plane image acquired by a second camera, and the other group of combined images comprises a second depth image acquired by the first camera and a second plane image acquired by the second camera; the depth image and the plane image in any one group of combined images have a mapping relationship therebetween;

[0143] determining a homography matrix between the first plane image and the second plane image based on the mapping relationship and depth information included in the first depth image and the second depth image; wherein the depth information represents a distance between the acquisition object and the first camera.

[0144] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns of this disclosure that can be derived from the description and illustrations presented herein without departing from the scope and spirit of the disclosure. The specification and examples given are considered exemplary only, and the true scope and spirit of the disclosure are indicated by the following claims.

[0145] It is to be understood that the disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the disclosure is limited only by the claims that follow.

Claims

1. An image processing method, characterized by, The method comprises: acquiring two sets of combined images of a collection object; wherein one set of combined images comprises a first depth image collected by a first camera and a first plane image collected by a second camera, and the other set of combined images comprises a second depth image collected by the first camera and a second plane image collected by the second camera; the depth image and the plane image in any set of combined images have a mapping relationship; based on the depth information included in the first depth image and the second depth image, calculating the average collection distance of the first camera when the first depth image is acquired and the average collection distance of the first camera when the second depth image is acquired; based on the mapping relationship and the ratio between the average collection distance corresponding to the first depth image and the average collection distance corresponding to the second depth image, determining the homography matrix between the first plane image and the second plane image; wherein the depth information represents the distance between the collection object and the first camera.

2. The method of claim 1, wherein, The method comprises: performing depth region division on the first depth image and the second depth image to obtain first depth sub-images of different depth regions and second depth sub-images of different depth regions; wherein different depth regions represent local regions with different distances to the first camera on the collection object; according to the mapping relationship, acquiring the first plane sub-image corresponding to each depth region first depth sub-image on the first plane image, and the second plane sub-image corresponding to each depth region second depth sub-image on the second plane image; establishing a homography sub-matrix according to a group of first plane sub-images and second plane sub-images of the same depth region; determining the homography matrix between the first plane image and the second plane image according to the homography sub-matrices corresponding to the combinations of multiple groups of first plane sub-images and second plane sub-images.

3. The method of claim 2, wherein, The method comprises: performing feature point matching on the first plane image and the second plane image to obtain a matching combination between pixel points in the first plane image and pixel points in the second plane image; establishing a homography sub-matrix according to a group of first plane sub-images and second plane sub-images of the same depth region, comprising: selecting target matching combinations belonging to the first plane sub-image and the second plane sub-image of the same depth region from the matching combination of the pixel points; calculating the homography sub-matrix based on the target matching combination.

4. The method of claim 2, wherein, The weights of different homography sub-matrices in the homography matrix are different; wherein the weight of the homography sub-matrix corresponding to the target depth region is greater than the weight of the homography sub-matrix corresponding to the depth region outside the target depth region.

5. The method of claim 1, wherein, The method comprises: calibrating the first camera to obtain a first calibration result; the first calibration result is used to correct the images collected by the first camera to obtain the first depth image and the second depth image; calibrate the second camera to obtain a second calibration result; the second calibration result is used to correct images collected by the second camera to obtain the first planar image and the second planar image; determine the mapping relationship based on the first calibration result and the second calibration result.

6. The method of claim 1, wherein, The method comprises: According to the homography matrix, the first planar image and the second planar image are registered.

7. An image processing apparatus characterized by comprising: The device comprises: The acquisition module is configured to acquire two sets of combined images of a collection object; wherein one set of combined images comprises a first depth image collected by a first camera and a first planar image collected by a second camera, and the other set of combined images comprises a second depth image collected by the first camera and a second planar image collected by the second camera; the depth image and the planar image in any set of combined images have a mapping relationship; The determination module is configured to calculate the average collection distance of the first camera when the first depth image is acquired and the average collection distance of the first camera when the second depth image is acquired based on the depth information included in the first depth image and the second depth image; determine the homography matrix between the first planar image and the second planar image based on the mapping relationship and the ratio between the average collection distance corresponding to the first depth image and the average collection distance corresponding to the second depth image; wherein the depth information represents the distance between the collection object and the first camera.

8. The device of claim 7, wherein The determination module is further configured to divide the first depth image and the second depth image into depth regions to obtain first depth sub-images of different depth regions and second depth sub-images of different depth regions; wherein different depth regions represent local regions with different distances to the first camera on the collection object; according to the mapping relationship, obtain first planar sub-images corresponding to the first depth sub-images of each depth region on the first planar image, and second planar sub-images corresponding to the second depth sub-images of each depth region on the second planar image; establish homography sub-matrices according to a group of the first planar sub-images and the second planar sub-images of the same depth region; and determine the homography matrix between the first planar image and the second planar image according to the homography sub-matrices corresponding to the combination of multiple groups of first planar sub-images and second planar sub-images.

9. The apparatus of claim 8, wherein, The device comprises: The feature matching module is configured to perform feature point matching on the first planar image and the second planar image to obtain matching combinations between pixel points in the first planar image and pixel points in the second planar image; The determination module is further configured to select target matching combinations belonging to the first planar sub-images and the second planar sub-images of the same depth region from the matching combinations of the pixel points; and calculate the homography sub-matrices based on the target matching combinations.

10. The apparatus of claim 8, wherein, Different homographic sub-matrices in the homographic matrix have different weights; wherein a homographic sub-matrix corresponding to a target depth region has a greater weight than a homographic sub-matrix corresponding to a depth region other than the target depth region.

11. The apparatus of claim 7, wherein, The apparatus comprises: A first calibration module configured to calibrate the first camera offline to obtain a first calibration result; the first calibration result is used to correct an image captured by the first camera to obtain the first depth image and the second depth image; A second calibration module configured to calibrate the second camera offline to obtain a second calibration result; the second calibration result is used to correct an image captured by the second camera to obtain the first plane image and the second plane image; A mapping module configured to determine the mapping relationship based on the first calibration result and the second calibration result.

12. The apparatus of claim 7, wherein, The apparatus comprises: A registration module configured to register the first plane image and the second plane image according to the homographic matrix.

13. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; Wherein the processor is configured to execute the image processing method as claimed in any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the image processing method as claimed in any one of claims 1 to 6.

Images