Image consistency processing method, system, computer system and storage medium

By extracting and matching the rigid body transformation constant features in the second frame depth map from the first frame depth map, the rigid body transformation parameters are calculated, which solves the efficiency and accuracy of image relationship acquisition in the camera motion scene, and achieves high efficiency image consistency processing.

CN111754555BActive Publication Date: 2024-12-03YAOKE INTELLIGENT TECH SHANGHAI CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201910238362.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-03-27
Publication Date
2024-12-03
Estimated Expiration
2039-03-27

AI Technical Summary

Technical Problem

In a scene where the camera is movable, how to efficiently obtain the accurate rigid body transformation relationship between image frames in the same image sequence, for applications such as target tracking.

Method used

By extracting the first rigid body transformation invariant features from the first frame depth map, selecting and matching the corresponding features in the second frame depth map, the rigid body transformation parameters are calculated, including translation matrix and rotation matrix, and using covariance matrix and curvature information , combined with SVD algorithm and search area matching, satisfies geometric condition constraints.

Benefits of technology

It realizes accurate and fast acquisition of rigid body transformation relationships between continuous frame depth maps, improves the efficiency and accuracy of image consistency processing, and is suitable for applications such as target recognition and tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111754555B_ABST
    Figure CN111754555B_ABST
Patent Text Reader

Abstract

The image consistency processing method, system, computer system and storage medium of the present application respectively extract rigid body transformation invariant features from the first frame depth map and the second frame depth map; select part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtain the second rigid body transformation invariant features that match the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the rigid body transformation parameters are calculated according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set; the present application provides accurate and fast acquisition of the rigid body transformation relationship between consecutive frame depth maps to solve the problems of the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image consistency processing method, system, computer system and storage medium. Background Art

[0002] In scenes where the camera can move, multiple frames in the same image sequence may be taken when the camera is in different positions, and it is often necessary to obtain the rigid transform relationship between the coordinate systems of different frames to achieve applications, such as target tracking.

[0003] How to find an efficient computational solution for obtaining the accurate rigid body transformation relationship between image frames in the same image sequence has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide an image consistency processing method, system, computer system and storage medium to solve the problems in the prior art.

[0005] To achieve the above objectives and other related objectives, the present application provides an image consistency processing method, which is applied to a computer system connected to a camera device for communication, and is used to obtain rigid body transformation parameters of rigid body transformation between a first frame depth map and a second frame depth map of continuous image frames in an image sequence provided by the camera device; the method comprises: extracting a first rigid body transformation invariant feature from the first frame depth map; wherein the first rigid body transformation invariant feature is calculated based on each voxel mapped by at least part of the pixels in the first frame depth map, and the voxel is mapped by a rectangular area determined by at least two pixels as diagonal corner points; selecting part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtaining second rigid body transformation invariant features matching each first rigid body transformation invariant feature in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; and the rigid body transformation parameters are calculated based on the rigid body transformation relationship of each matching feature pair in the matching feature pair set.

[0006] In one embodiment of the present application, the second frame depth map is acquired by the camera device at a second position that moves relative to the first position at which the first frame depth map is acquired.

[0007] In one embodiment of the present application, the rigid body transformation parameters include a translation matrix and a rotation matrix.

[0008] In one embodiment of the present application, the method for obtaining the first rigid body transformation invariant feature includes: according to rectangular areas of the same size uniformly distributed in the local or entire area on the first frame depth map, obtaining the voxels mapped by each rectangular area to form a first voxel set; for the first voxel set, calculating the covariance matrix corresponding to each voxel therein, and calculating the corresponding eigenvalue according to the covariance matrix as the first rigid body transformation invariant feature.

[0009] In one embodiment of the present application, the method for calculating the corresponding eigenvalue according to the covariance matrix includes: SVD algorithm.

[0010] In one embodiment of the present application, the method comprises: calculating the curvature of a voxel according to its eigenvalue, and selecting the first rigid body transformation invariant features of a plurality of voxels with the largest curvature to form the first invariant feature set.

[0011] In one embodiment of the present application, the method for obtaining the second rigid body transformation invariant feature includes: obtaining each search area that encloses each first rectangular area on the second frame depth map according to each corresponding first rectangular area of ​​each voxel on the first frame depth map that serves as the basis for obtaining the first invariant feature set; and calculating the second rigid body transformation invariant feature that is closest to the first rigid body transformation invariant feature corresponding to the first rectangular area as a matching feature based on each second rectangular area selected from the search area that has the same size as the first rectangular area it encloses.

[0012] In one embodiment of the present application, the search area is a rectangular shape having the same center as the first rectangular area it encloses; and / or, when the search area and the first rectangular area are rectangular, assuming that the side length of the search area is h and the side length of the first rectangular area is r, then the number of selected second rectangular areas is (hr)×(hr).

[0013] In one embodiment of the present application, the covariance matrix and / or rectangular area is calculated based on a summed area table generated according to the corresponding depth map; wherein the summed area table is a set of three-dimensional coordinate information of each pixel point in the corresponding depth map.

[0014] In one embodiment of the present application, the rigid body transformation parameters are calculated based on the rigid body transformation relationship of each matching feature pair in the matching feature pair set, including: obtaining the first center of mass of each first voxel corresponding to each first rigid body transformation invariant feature and the second center of mass of each second voxel corresponding to each second rigid body transformation invariant feature according to the matching feature pair set; obtaining the part of each first voxel after eliminating the translation difference with the first center of mass, forming a first adjustment set; obtaining the part of each second voxel after eliminating the translation difference with the second center of mass, forming a second adjustment set; calculating the rotation matrix according to the relationship between the first adjustment set and the second adjustment set; projecting the first center of mass to the coordinate system where the second center of mass is located according to the rotation matrix, and calculating the translation difference between the projection of the first center of mass and the second center of mass to obtain the translation matrix.

[0015] In one embodiment of the present application, the rigid transformation parameters are subject to geometric conditions; wherein the geometric conditions include: the rigid transformation parameters calculated by each of the matching feature pair sets are applied to the first frame depth map to project it to the coordinate system of the second frame depth map, and the sum of the projection errors between each first pixel point on the projection of the first frame depth map and each second pixel point with the same pixel coordinates on the second frame depth map is less than a projection error threshold.

[0016] In one embodiment of the present application, the rigid body transformation parameters are calculated based on the rigid body transformation relationship of each matching feature pair in the matching feature pair set, including: a rigid body transformation parameter evaluation step, which includes: randomly selecting a number of the matching feature pairs to form a matching feature pair subset, and calculating the alternative rigid body transformation parameters based on them; judging whether each matching feature pair in the matching feature pair subset is subject to the geometric condition constraints; if the judgment result is yes, the matching feature pairs are considered valid to obtain the number of valid matching feature pairs in the matching feature pair subset; generating a score associated with the alternative rigid body transformation parameter based on the number of valid matching feature pairs; repeating the rigid body transformation parameter evaluation step a predetermined number of times to select the alternative rigid body transformation parameter with the highest score as the calculated rigid body transformation parameter.

[0017] To achieve the above objectives and other related objectives, the present application provides an image consistency processing system, including: a communication module, used to obtain a first frame depth map and a second frame depth map of consecutive image frames in an image sequence provided by a camera device; a processing module, used to obtain rigid body transformation parameters of a rigid body transformation between the first frame depth map and the second frame depth map, including: extracting rigid body transformation invariant features from the first frame depth map and the second frame depth map respectively; wherein the rigid body transformation invariant features are at least calculated based on each voxel in a point cloud mapped by each pixel point in a frame depth map, and the voxel is a combination of at least two predetermined images in the depth map. The element coordinates are obtained by mapping the rectangular area framed by the diagonal corner points; some or all of the first rigid body transformation invariant features obtained from the first frame depth map are selected to form a first invariant feature set, and the second rigid body transformation invariant features matching the first rigid body transformation invariant features in the first invariant feature set are obtained in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the rigid body transformation parameters are calculated according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set.

[0018] To achieve the above objectives and other related objectives, the present application provides a computer system, including: a transceiver, which is communicatively connected to a camera device; a memory, which is used to store computer instructions; and a processor, which is used to run the computer instructions to execute the described method.

[0019] To achieve the above objectives and other related objectives, the present application provides a non-transitory computer-readable storage medium storing computer instructions, which are executed to perform the method described above.

[0020] As described above, the image consistency processing method, system, computer system and storage medium of the present application respectively extract rigid body transformation invariant features from the first frame depth map and the second frame depth map; select part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtain the second rigid body transformation invariant features that match the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the rigid body transformation parameters are calculated according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set; the present application provides accurate and fast acquisition of the rigid body transformation relationship between consecutive frame depth maps to solve the problems of the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1Shown is a structural schematic diagram of an application scenario in an embodiment of the present application.

[0022] Figure 2 Shown is a schematic diagram of the structure of a processing device in one embodiment of the present application.

[0023] Figure 3 Shown is a flowchart of an image consistency processing method in one embodiment of the present application.

[0024] Figure 4 Shown is a flowchart of a method for acquiring the first rigid body transformation invariant feature in one embodiment of the present application.

[0025] Figure 5 Shown is a flow chart of a method for acquiring the second rigid body transformation invariant feature in one embodiment of the present application.

[0026] Figure 6 Shown is a schematic diagram of a process of obtaining rigid body transformation parameters according to a first invariant feature set and a second invariant feature set in one embodiment of the present application.

[0027] Figure 7 Shown is a schematic diagram of functional modules of an image consistency processing system in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0029] The technical solution of the present application relates to the technical field of image processing.

[0030] The technical solution of the present application is applied to a light field camera system. The light field camera system includes one or more camera arrays, and the multiple cameras included in each camera array are arranged at a certain angle and spacing; wherein the number, angle and spacing of the cameras can be adjusted according to the size and complexity of the scene. The depth information of the objects in the scene is calculated by the light field depth estimation algorithm, and a depth map (or corresponding point cloud) is generated as input data.

[0031] It should be noted that there are many methods for obtaining depth information, such as structured light solutions or TOF solutions, and any technology that can obtain depth information can be included in the scope of this application.

[0032] like Figure 1As shown, a structural schematic diagram of an application scenario in an embodiment of the present application is shown.

[0033] In this application scenario, a camera device 101 and a processing device 102 are provided.

[0034] Among them, the camera device 101 may include one or more camera arrays, each camera array includes multiple cameras; the camera array in the camera device 101 collects image data (which may include images taken by each camera, such as photos, videos, etc.); the processing device 102 is communicatively connected with the camera device 101 so that the camera device 101 can transmit image data to the processing device 102, and / or the processing device 102 sends control instructions to the camera device 101.

[0035] In some examples, the communication connection may be a wired connection through an electrical line, such as a connection to a corresponding standard line through a USB interface, an HDMI interface, etc. at the other end.

[0036] In some examples, the communication connection may also be a wireless connection, for example, connected through a wireless transceiver at the other end, such as WiFi, Bluetooth, a mobile communication module (2G / 3G / 4G / 5G), etc.

[0037] In some examples, the communication connection may also be a network connection, that is, a long-distance communication connection via a local area network and / or the Internet.

[0038] The processing device 102 has processing and computing capabilities, and can extract depth information from the obtained image data, generate a depth map (or a corresponding point cloud) as input data, and perform data processing to obtain the desired results.

[0039] The specific implementation of the processing device 102 may vary according to different application scenarios of the above various embodiments. For example, in some examples, the processing device 102 may be integrated with the camera device 101 as a component in the same device; or, in some examples, the processing device 102 may be located in a different device from the camera device 101, for example, the processing device 102 is implemented in a server / server group, etc., and is communicated with the camera device 101 via the Internet.

[0040] like Figure 2 As shown, a schematic diagram of the structure of the processing device 200 described in an embodiment of the present application is shown.

[0041] The processing device 200 may be implemented through the architecture of a computer system, and includes: one or more transceivers 201 , a memory 202 , and a processor 203 .

[0042] It should be noted that, in this embodiment, the number of components shown in the figure is only for example and is not intended to be limiting.

[0043] The one or more transceivers 201 include: a first transceiver 201 that communicates with the camera device, which can be implemented by an interface circuit for communicating by, for example, a wired connection (such as USB, HDMI) or a wireless connection (such as WiFi, 2G / 3G / 4G / 5G).

[0044] The memory 202 is used to store computer instructions;

[0045] The processor 203 is coupled to the transceiver 201 and the memory 202, and is used to execute the computer instructions to implement the required image processing function.

[0046] The memory 202 may include, but is not limited to, a high-speed random access memory 202 and a non-volatile memory 202, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices.

[0047] The processor 203 can be a general-purpose processor 203, including a central processing unit 203 (CPU), a network processor 203 (NP), etc.; it can also be a digital signal processor 203 (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0048] In one embodiment, the camera device is movable. For example, the camera in the camera device is disposed on a movable carrier, so that it can move along with the movement of the carrier.

[0049] The movement of the carrier may be translation or rotation. In some embodiments, the carrier may be a pan-tilt platform, or a slide platform mounted on a slide rail.

[0050] When the camera is moving, it is still collecting images. The image frames collected by the same camera at different times constitute an image sequence. Since the camera may be movable, each frame in the image sequence may be collected in different motion postures, and the shooting angles are different.

[0051] In such a case, the processing device 200 in the above embodiment can implement the image consistency processing method by running computer instructions, thereby obtaining the most consistent rigid body transformation parameters for rigid body transformation between depth images of consecutive frames in the same image sequence of the camera device, thereby achieving more accurate output results, such as target recognition, tracking and other applications.

[0052] like Figure 3 As shown, a flow chart of the image consistency processing method in an embodiment of the present application is shown.

[0053] In this embodiment, the image consistency processing method includes:

[0054] Step S301: extracting a first rigid body transformation invariant feature from the first frame depth map.

[0055] In this embodiment, first, the first frame depth map is assumed to be I R , the depth map of the second frame is I M , the rigid body transformation parameters are the rotation matrix and translation matrix, namely {R, t}. We introduce the extraction from I R ,I M The rigid transformation invariant features (RTIF) are used as an effective implementation to extract reliable key points.

[0056] The principle of RTIF is explained below:

[0057] For example, convert the depth map I of size m×n into a point cloud Given the intrinsic parameter matrix K of the depth camera, each pixel of I is mapped to the corresponding three-dimensional coordinates to generate a point cloud.

[0058] Since each pixel in the depth map (pixel coordinates are set to (u, v)) has a depth value I(u, v) = z, the 2D image coordinates are mapped to the three-dimensional space through the mapping function f:I→Ω. The mapping function is defined as:

[0059]

[0060] Then the point cloud Ω is defined as:

[0061]

[0062] Voxel Φ is a subset of the corresponding point cloud Ω of I. Φ is a rectangular region R from I. I The mapping pixels of (u1, v1, u2, v2) are generated, where (u1, v1) and (u2, v2) are the pixel coordinates of a pair of corner points of the rectangular area.

[0063]

[0064] According to this definition, Ω can also be considered to be the R given the depth map size mxn I The generated voxel is (0,0,m,n).

[0065] RTIF is calculated for each depth map voxel Φ, defining the rigid body transformation invariant feature Λ Φ =F(Φ); the rigid body transformation invariant feature can be used to describe the voxel covariance matrix C Φ and curvature information ξ Φ , curvature information ξ Φ Is to use C Φ The eigenvalue of (ascending) derived.

[0066] thus,

[0067]

[0068] Covariance matrix C Φ Defined as:

[0069]

[0070] Conversely, the covariance matrix C Φ Can be used to calculate each eigenvalue and then get Λ Φ .

[0071] Curvature ξ Φ Calculated as:

[0072]

[0073] For a depth map sequence, our goal is to estimate the optimal rigid body transformation parameters {R, t} of two consecutive depth maps caused by the camera motion. Therefore, the problem can be formulated as a constrained optimization problem:

[0074] max|η|

[0075] That is, maximize the number of valid matching RTIF pairs between the two depth maps to achieve the maximum geometric consistency correspondence:

[0076]

[0077] Under such a correspondence of maximum geometric consistency, the rigid body transformation parameters finally calculated should also meet the geometric constraint conditions; that is, the rigid body transformation parameters calculated by each matching feature pair set act on the first frame depth map to project it to the coordinate system where the second frame depth map is located, and the sum of the projection errors between each first pixel point on the projection of the first frame depth map and each second pixel point with the same pixel coordinates on the second frame depth map is less than a projection error threshold e, which is expressed as:

[0078]

[0079] Our goal is to find the optimal solution for {R, t}, and we solve this problem by R and I M The mutually matching RTIFs are obtained to form a RTIF corresponding set, and then the optimal solution of {R, t} is obtained based on it.

[0080] like Figure 4 As shown, a method for obtaining the first rigid body transformation invariant feature is shown, including:

[0081] Step S401: according to rectangular areas of the same size uniformly distributed partially or entirely on the first frame depth map, voxels mapped by the rectangular areas are acquired to form a first voxel set.

[0082] Step S402: for the first voxel set, calculate the covariance matrix corresponding to each voxel therein, and calculate the corresponding eigenvalue according to the covariance matrix as the first rigid body transformation invariant feature.

[0083] For example: From I R Extract RTIF from:

[0084] From I R Divide N evenly R Rectangular areas of the same size form a rectangular area set The rectangular area can be a rectangle with unequal length and width, or a square with equal length and width. In this example, it is set to be a square with a size of r×r. Then, a first voxel set can be obtained by the aforementioned method of mapping voxels through a rectangular area.

[0085] For each voxel Calculate its covariance matrix As shown in formula (5); then, through, for example, the singular value decomposition method (SVD), its

[0086] Then, corresponding to the first voxel set We can obtain a first invariant feature set

[0087] Step S302: select part or all of the first rigid body transformation invariant features obtained from the first frame depth map to constitute a first invariant feature set, and obtain the second rigid body transformation invariant features that match the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to constitute a second invariant feature set; and, the matching feature pairs formed by the matched first rigid body transformation invariant features and second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set.

[0088] like Figure 5 As shown, a flow chart showing the method for obtaining the second rigid body transformation invariant feature in an embodiment of the present application is shown.

[0089] Specifically include:

[0090] Step S501: According to each corresponding first rectangular area of ​​each voxel on the first frame depth map serving as a basis for obtaining the first invariant feature set, each search area encompassing each first rectangular area on the second frame depth map is obtained respectively.

[0091] Step S502: Calculate, based on each second rectangular area selected from the search area and having the same size as the first rectangular area enclosed by the second rectangular area, a second rigid body transformation invariant feature closest to the first rigid body transformation invariant feature corresponding to the first rectangular area as a matching feature.

[0092] Optionally, for efficient calculation, only a part of To produce the first invariant feature set, the selection criterion is the curvature calculated according to formula (6). The first rigid body transformation invariant features of several voxels with the largest curvature can be selected to form the first invariant feature set.

[0093] Optional, if based on I R and I M Assuming that the baseline between is small, the position of the same physical point in the depth map of the two frames should not differ too much, so it is very likely that R with I M A second rigid body transformation invariant feature matching the first rigid body transformation invariant feature is obtained at a similar position on the image processing unit.

[0094] Therefore, each of the search areas encloses a corresponding first rectangular area, and assuming that the size of the first rectangular area is r×r as shown in the above example, the size of the search area can be set to h×h, where h is greater than r, preferably slightly greater, for example, h is greater than r by 1, 2, 3, 4, 5 or more pixels.

[0095] Assumptions is calculated from pixels (u i ,v i) is the center of the region, the center of which is at I R on I M On the same position pixel (u i ,v i ) is the center of the search area. In this search area (assuming the size is h×h), we can try to find a matching second rigid body transformation invariant feature generated by an area of ​​size r×r by minimizing the difference between the features. Therefore, for each We can find multiple candidate second rectangular areas in the search area and then select the one with the smallest difference.

[0096] For example, the number of the second rectangular areas may be (hr)×(hr); and The smallest difference It is expressed as:

[0097]

[0098] Among them, 0≤j<(hr)×(hr).

[0099] Of course, the calculation of the number of the second rectangular areas is only an example and is not limited to this; in other calculation methods, the number of the second rectangular areas can also be, for example, (h-r+1)×(h-r+1), etc.

[0100] By efficiently calculating the matching second rigid body transformation invariant features by narrowing the search area at similar positions through the above embodiment, the amount of calculation and time complexity are effectively reduced compared with methods such as iterative closest point (ICP) searching on the entire frame image.

[0101] Step S303: Calculate the rigid body transformation parameters according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set.

[0102] Optionally, we try to recover the optimal rigid body transformation parameters {R, t} under geometric constraints:

[0103]

[0104] Wherein, e comes from formula 8.

[0105] like Figure 6 As shown, a schematic diagram of the process of obtaining rigid body transformation parameters according to the first invariant feature set and the second invariant feature set in an embodiment of the present application is shown.

[0106] Specifically include:

[0107] Step S601: obtaining, according to the matching feature pair set, a first centroid of each first voxel corresponding to each first rigid body transformation invariant feature, and a second centroid of each second voxel corresponding to each second rigid body transformation invariant feature;

[0108] Step S602: obtaining the parts of each first voxel after eliminating the translation difference with the first centroid, forming a first adjustment set; obtaining the parts of each second voxel after eliminating the translation difference with the second centroid, forming a second adjustment set;

[0109] Step S603: Calculating the rotation matrix according to the relationship between the first adjustment set and the second adjustment set;

[0110] Step S604: Acting on the first center of mass according to the rotation matrix to project to the coordinate system where the second center of mass is located, and calculating the translation difference between the projection of the first center of mass and the second center of mass to obtain the translation matrix.

[0111] For example, given I R and I M The k pairs of RTIFs matched in , the centroids of their corresponding voxel sets are and

[0112] First, the centroid of each voxel is calculated using the formula:

[0113]

[0114] Among them, x, y, z are the spatial coordinates of each element in the voxel.

[0115] Then, the average centroid of the first invariant feature set is determined (i.e., the first centroid), and the average centroid of the second invariant feature set (i.e. the second centroid):

[0116]

[0117]

[0118] Then, the portion of each voxel that deviates from the centroid in the two voxel sets is calculated, as shown in the following sequence p R and p M , which removes the translation component and retains the rotation component:

[0119]

[0120]

[0121] Represent p by C R and p MThe rotation component between the two is then decomposed using singular value decomposition (SVD) to obtain an optimal rotation matrix:

[0122] C=(p R ) T ·p M (17)

[0123] [U,S,V]=SVD(C) (18)

[0124] R=V·U T (19)

[0125] After that, the translation matrix is ​​calculated as:

[0126]

[0127] Alternatively, in some embodiments, a summed area table (SAT) may be used to further speed up the search efficiency for obtaining a matched second rigid body transformation invariant feature in the search area.

[0128] Specifically, the covariance matrix and / or rectangular area is calculated based on a summed area table generated from a corresponding frame depth map; wherein the summed area table is a collection of three-dimensional coordinate information of each point in the corresponding frame depth map.

[0129] For example, the SAT of the depth map allows the set of 3D coordinate information of all elements mapped from a rectangular area to voxels to be calculated by accessing only 4 data in memory. It not only makes the calculation very efficient, but also makes the calculation cost independent of the size of the rectangular area.

[0130] In one example, the depth map I m×n The summed area table S is defined as:

[0131]

[0132] Among them, H is the three-dimensional coordinate information of the pixel (i, j), such as xx, xy, xz, yx, yy, yz, zx, zy, and zz.

[0133] In addition, the above formula (21) can be effectively iterated and calculated as:

[0134] S I (0, 0, m, n) = H (m,n) +S I (m-1, n)+S I (m, n-1)-S I (m-1, n-1) (22)

[0135] Therefore, a single pass over the input depth map is sufficient to compute its corresponding summed area table. ∑R is computed based on the rectangular area R from pixel (u1, v1) to pixel (u2, v2).

[0136] ∑R=S I (u2, v2)-S I (u1, v2)-S I (u2,v1)+S l (u1,v1) (23)

[0137] Let's take another example to illustrate how to use SAT to calculate the covariance matrix:

[0138] According to the definition of formula (5), the covariance matrix C can be derived as

[0139]

[0140] in,

[0141] In order to efficiently calculate the covariance, 9 SATs can be pre-computed:

[0142] The summation used in Equation 24 can be calculated as:

[0143]

[0144] It can be seen that by using SAT, we can efficiently complete the calculation of formula (5), thereby speeding up I M The process of searching for matching features in .

[0145] In the above embodiment, it is assumed that k pairs of matching RTIFs are obtained, but there may be invalid matches among them. For example, in I M There is actually no match in , but since the smallest difference is selected, it is still possible to assign it a match during the calculation.

[0146] In this case, the pair of matches can be considered as outliers and need to be eliminated.

[0147] The above elimination can be accomplished using the rigid body transformation parameter evaluation step:

[0148] The rigid body transformation parameter evaluation step includes: randomly selecting a number of the matching feature pairs to form a matching feature pair subset, and calculating the alternative rigid body transformation parameters based on them; judging whether each matching feature pair in the matching feature pair subset is subject to the geometric condition constraints; if the judgment result is yes, the matching feature pair is considered valid to obtain the number of valid matching feature pairs in the matching feature pair subset; and generating a score associated with the alternative rigid body transformation parameters according to the number of valid matching feature pairs.

[0149] Then, the rigid body transformation parameter evaluation step is repeated a predetermined number of times to select the candidate rigid body transformation parameter with the highest score as the calculated rigid body transformation parameter.

[0150] Let me give you an example to illustrate this.

[0151] First, k pairs of matching RTIFs are randomly selected from the obtained set of matching feature pairs as a subset, and {R, t} is calculated based on the subset. Next, all pairs in the subset are calculated by the projection error e, as shown in the above formula (8). If the calculated projection error is less than the preset threshold δ, the pair of RTIFs is considered to be valid; thus, the number of all valid matching feature pairs in the subset (necessarily ≤ k) can be counted and recorded as the score of this {R, t}.

[0152] Then, the subset extraction and the corresponding {R, t} score acquisition are repeated N times, and the {R, t} with the highest score is selected as the maximum value corresponding to the geometric consistency.

[0153] For the number of iterations, N is chosen to be high enough to guarantee a success probability p (usually set to 99%), which means that after N iterations, we can find the optimal solution {R, t} with probability p as the final rigid body transformation parameters.

[0154] N can be calculated. Specifically, for the outlier ratio β, the appropriate number of iterations N is calculated by the following formula:

[0155]

[0156] Where s is the subset size. Through the optimization of this embodiment, not only the time complexity of maximizing consistency is greatly reduced, but also the robustness of the system to outliers is improved.

[0157] In addition, the various computer instructions involved in the aforementioned method embodiments may be loaded into a non-transitory computer-readable storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (compact disk-read only memory), a magneto-optical disk, a ROM (read only memory), a RAM (random access memory), an EPROM (erasable programmable read-only memory), an EEPROM (electrically erasable programmable read-only memory), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The non-transitory computer-readable storage medium may be a product that is not connected to a computer device, or may be a component that is connected to a computer device for use.

[0158] In a specific implementation, the computer instructions are routines, instructions, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.

[0159] like Figure 7 , which shows a schematic diagram of the architecture of the image consistency processing system 700 in an embodiment of the present application.

[0160] The image consistency processing system 700 includes: a communication module 701 and a processing module 702 .

[0161] The communication module 701 is configured to obtain a first depth map and a second depth map of consecutive image frames in an image sequence provided by a camera device.

[0162] The processing module 702 is used to obtain rigid body transformation parameters of rigid body transformation between the first frame depth map and the second frame depth map, including: extracting rigid body transformation invariant features from the first frame depth map and the second frame depth map respectively; wherein the rigid body transformation invariant features are calculated based on at least each voxel in the point cloud mapped by each pixel point in a frame depth map, and the voxel is mapped by a rectangular area framed by at least two predetermined pixel coordinates in the depth map as diagonal corner points; selecting part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtaining second rigid body transformation invariant features matching each first rigid body transformation invariant feature in the first invariant feature set in the second frame depth map to form a second invariant feature set; and matching feature pairs formed by the matched first rigid body transformation invariant features and second rigid body transformation invariant features, each of the matching feature pairs forming a matching feature pair set; and calculating the rigid body transformation parameters according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set.

[0163] It should be noted that it should be understood that the division of the various modules of the above image consistency processing system 700 is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the processing module 702 can be a separately established processing element, or it can be integrated in a certain chip of the above image consistency processing system 700. In addition, it can also be stored in the memory of the above image consistency processing system 700 in the form of program code, and called and executed by a certain processing element of the above image consistency processing system 700. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0164] For example, the above modules may be configured to implement the above methods (eg Figure 2 to Figure 6 For example, one or more integrated circuits (such as those shown in the embodiments), such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when one of the above modules is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0165] Since the implementation principle in the present system embodiment is similar to that in the aforementioned method embodiment, the technical features described in the embodiments can be mutually called upon, and therefore, the same technical details will not be described repeatedly here.

[0166] In summary, the image consistency processing method, system, computer system and storage medium of the present application respectively extract rigid body transformation invariant features from the first frame depth map and the second frame depth map; select part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtain the second rigid body transformation invariant features that match the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the rigid body transformation parameters are calculated according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set; the present application provides accurate and fast acquisition of the rigid body transformation relationship between consecutive frame depth maps to solve the problems of the prior art.

[0167] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for image consistency processing, characterized in that: A computer system is applied to a computer system connected to a camera device for obtaining continuous depth maps of at least a first frame and a second frame according to continuous image frames provided by the camera device, and obtaining rigid body transformation parameters of rigid body transformation between the first frame and the second frame; the method comprises: Extracting a first rigid body transformation invariant feature from the first frame depth map; wherein the first rigid body transformation invariant feature is calculated based on each voxel mapped by at least part of the pixels in the first frame depth map, and the voxel is mapped by a rectangular area determined by at least two pixels as diagonal corner points; the process of calculating the first rigid body transformation invariant feature comprises: according to each rectangular area of ​​the same size uniformly distributed in a local or entire area on the first frame depth map, obtaining each voxel mapped by each rectangular area to form a first voxel set; for the first voxel set, calculating the covariance matrix corresponding to each voxel therein, and calculating the corresponding eigenvalue according to the covariance matrix as the first rigid body transformation invariant feature; Selecting part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtaining second rigid body transformation invariant features matching the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the process of obtaining the second rigid body transformation invariant features matching the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map includes: obtaining each search area containing each first rectangular area on the second frame depth map according to each corresponding first rectangular area corresponding to each voxel as the basis for obtaining the first invariant feature set; calculating the second rigid body transformation invariant feature closest to the first rigid body transformation invariant feature corresponding to the first rectangular area according to each second rectangular area selected from the search area and having the same size as the first rectangular area contained therein, as the matching feature; The rigid body transformation parameters are obtained by calculating the rigid body transformation relationship of each matching feature pair in the matching feature pair set. The process of calculating the rigid body transformation parameters by calculating the rigid body transformation parameters includes: obtaining the first center of mass of each first voxel corresponding to each first rigid body transformation invariant feature and the second center of mass of each second voxel corresponding to each second rigid body transformation invariant feature according to the matching feature pair set; obtaining the part of each first voxel after eliminating the translation difference with the first center of mass to form a first adjustment set; obtaining the part of each second voxel after eliminating the translation difference with the second center of mass to form a second adjustment set; calculating the rotation matrix according to the relationship between the first adjustment set and the second adjustment set; projecting the first center of mass to the coordinate system where the second center of mass is located according to the rotation matrix, and calculating the translation difference between the projection of the first center of mass and the second center of mass to obtain the translation moment.

2. The method according to claim 1, characterized in that The second frame depth map is acquired by the camera device at a second position that moves relative to the first position at which the first frame depth map is acquired.

3. The method according to claim 1, characterized in that The rigid body transformation parameters include a translation matrix and a rotation matrix.

4. The method according to claim 1, characterized in that: The method for acquiring the first rigid body transformation invariant feature includes: According to rectangular areas of the same size uniformly distributed locally or in the entire area on the first frame depth map, voxels mapped to the rectangular areas are obtained to form a first voxel set; For the first voxel set, a covariance matrix corresponding to each voxel therein is calculated, and a corresponding eigenvalue is calculated according to the covariance matrix as the first rigid body transformation invariant feature.

5. The method according to claim 4, characterized in that The method for calculating the corresponding eigenvalue according to the covariance matrix includes: SVD algorithm.

6. The method according to claim 1 or 4, characterized in that: include: The curvature of the voxel is calculated according to its eigenvalue, and the first rigid body transformation invariant features of several voxels with the largest curvature are selected to form the first invariant feature set.

7. The method according to claim 1, characterized in that The method for acquiring the second rigid body transformation invariant feature includes: According to each corresponding first rectangular area on the first frame depth map corresponding to each voxel serving as a basis for obtaining the first invariant feature set, each search area encompassing each first rectangular area on the second frame depth map is obtained respectively; A second rigid body transformation invariant feature closest to the first rigid body transformation invariant feature corresponding to the first rectangular area is calculated based on each second rectangular area selected from the search area and having the same size as the first rectangular area enclosed by the second rectangular area as the matching feature.

8. The method according to claim 7, characterized in that The search area is a rectangular shape with the same center as the first rectangular area it contains; and / or, when the search area and the first rectangular area are rectangular, the number of selected second rectangular areas is, where h is the side length of the search area and r is the side length of the first rectangular area.

9. The method according to claim 4, characterized in that The covariance matrix and / or rectangular area are calculated based on a summed area table generated according to the corresponding depth map; wherein the summed area table is a collection of three-dimensional coordinate information of each pixel point in the corresponding depth map.

10. The method according to claim 3, characterized in that The step of calculating the rigid body transformation parameters according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set includes: Obtaining, according to the matching feature pair set, a first center of mass of each first voxel corresponding to each first rigid body transformation invariant feature, and a second center of mass of each second voxel corresponding to each second rigid body transformation invariant feature; Obtaining the parts of each first voxel after eliminating the translation difference with the first center of mass, forming a first adjustment set; obtaining the parts of each second voxel after eliminating the translation difference with the second center of mass, forming a second adjustment set; Calculate the rotation matrix according to the relationship between the first adjustment set and the second adjustment set; The rotation matrix is ​​applied to the first center of mass to project the first center of mass to the coordinate system where the second center of mass is located, and the translation difference between the projection of the first center of mass and the second center of mass is calculated to obtain the translation matrix.

11. The method according to claim 1, characterized in that: The rigid body transformation parameters are subject to geometric conditions; wherein, the geometric conditions include: the rigid body transformation parameters calculated by each of the matching feature pair sets are applied to the first frame depth map to project it to the coordinate system of the second frame depth map, and the sum of the projection errors between each first pixel point on the projection of the first frame depth map and each second pixel point with the same pixel coordinates on the second frame depth map is less than a projection error threshold.

12. The method according to claim 11, characterized in that The step of calculating the rigid body transformation parameters according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set includes: The rigid body transformation parameter evaluation step includes: randomly selecting a number of the matching feature pairs to form a matching feature pair subset, and calculating the candidate rigid body transformation parameters based on the matching feature pair subset; judging whether each matching feature pair in the matching feature pair subset is subject to the geometric condition constraint; if the judgment result is yes, the matching feature pair is considered valid to obtain the number of valid matching feature pairs in the matching feature pair subset; and generating a score associated with the candidate rigid body transformation parameter according to the number of valid matching feature pairs; The rigid body transformation parameter evaluation step is repeated a predetermined number of times to select the candidate rigid body transformation parameter with the highest score as the calculated rigid body transformation parameter.

13. An image consistency processing system, characterized in that: include: A communication module, configured to obtain a first depth map and a second depth map of consecutive image frames in an image sequence provided by a camera device; A processing module, used to obtain rigid body transformation parameters of rigid body transformation between the first frame depth map and the second frame depth map, including: Extracting rigid body transformation invariant features from the first frame depth map and the second frame depth map respectively; wherein the rigid body transformation invariant features are calculated based on at least each voxel in the point cloud mapped by each pixel point in a frame depth map, and the voxel is mapped by a rectangular area framed by at least two predetermined pixel coordinates in the depth map as diagonal corner points; the process of calculating the first rigid body transformation invariant features comprises: according to each rectangular area of ​​the same size uniformly distributed in a local or entire area on the first frame depth map, obtaining each voxel mapped by each rectangular area to form a first voxel set; for the first voxel set, calculating the covariance matrix corresponding to each voxel therein, and calculating the corresponding eigenvalue according to the covariance matrix as the first rigid body transformation invariant feature; Selecting part or all of the first rigid body transformation invariant features obtained from the first frame depth map to form a first invariant feature set, and obtaining second rigid body transformation invariant features matching the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map to form a second invariant feature set; and the matching feature pairs formed by the matched first rigid body transformation invariant features and the second rigid body transformation invariant features, each of the matching feature pairs constitutes a matching feature pair set; the process of obtaining the second rigid body transformation invariant features matching the first rigid body transformation invariant features in the first invariant feature set in the second frame depth map includes: obtaining each search area containing each first rectangular area on the second frame depth map according to each corresponding first rectangular area corresponding to each voxel as the basis for obtaining the first invariant feature set; calculating the second rigid body transformation invariant feature closest to the first rigid body transformation invariant feature corresponding to the first rectangular area according to each second rectangular area selected from the search area and having the same size as the first rectangular area contained therein, as the matching feature; The rigid body transformation parameters are obtained by calculation according to the rigid body transformation relationship of each matching feature pair in the matching feature pair set; the process of calculating the rigid body transformation parameters includes: obtaining the first center of mass of each first voxel corresponding to each first rigid body transformation invariant feature and the second center of mass of each second voxel corresponding to each second rigid body transformation invariant feature according to the matching feature pair set; obtaining the part of each first voxel after eliminating the translation difference with the first center of mass to form a first adjustment set; obtaining the part of each second voxel after eliminating the translation difference with the second center of mass to form a second adjustment set; calculating the rotation matrix according to the relationship between the first adjustment set and the second adjustment set; projecting the first center of mass to the coordinate system where the second center of mass is located according to the rotation matrix, and calculating the translation difference between the projection of the first center of mass and the second center of mass to obtain the translation moment.

14. A computer system, characterized in that: include: A transceiver, communicatively connected to the camera device; Memory, for storing computer instructions; A processor, configured to execute the computer instructions to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, characterized in that Computer instructions are stored, and when the computer instructions are executed, the method according to any one of claims 1 to 12 is performed.

Citation Information

Patent Citations

  • Method for complementing object shape in RGBD image

    CN105913492A

  • Keypoint-based point-pair-feature for scalable automatic global registration of large RGB-d scans

    WO2017096299A1