A method for repairing depth information of an RGB-D space of a special-shaped projection surface

Abstract

The application discloses a depth information repairing method for an RGB-D space of a special-shaped projection surface, a geometric shape change trend of the special-shaped projection surface is presented through a Gray code sequence image, a smooth transition region is divided according to the geometric shape change trend of the special-shaped projection surface, three-dimensional depth information of the smooth transition region is accurately fitted by using a three-dimensional curved surface mathematical model, and basic guarantee is provided for a geometric correction processing process of a subsequent special-shaped projection surface projection display system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction technology and projection display, in particular, a depth information repairing method for RGB-D space of a special-shaped projection surface. BACKGROUND

[0002] Unlike conventional projection surface display systems, special-shaped projection surface display systems will bring people a better sense of on-site immersion and augmented reality, and the spatial display characteristics of special-shaped projection surfaces will greatly stimulate people's visual perception, and they are widely used in virtual reality, artistic performance, cultural display, science and technology popularization, innovation and creativity and many other fields. With the improvement of the visual performance of projection display systems, the accuracy requirements of special-shaped projection surface display systems for depth information are more stringent, and the quality of the acquired depth information will directly affect the presentation effect of the projection display picture.

[0003] With the help of an RGB-D depth camera, the depth information of the projection surface can be collected in real time, but due to the influence or limitation of factors such as the defects of the RGB-D depth camera itself, the collection environment, the collection distance, and the inherent properties of the measured surface, the depth information collected will have important problems such as surface area loss, depth value jitter, and insufficient spatial resolution. If the original depth information collected is directly applied to the special-shaped projection surface display system, it will inevitably affect the reliability of the subsequent geometric correction stage. Many researchers have carried out a lot of work on hole filling and depth optimization, and various research results have further enhanced the usability of depth information.

[0004] In the depth information repairing algorithm based on filtering theory, the depth information of the neighborhood around the to-be-repaired region or the color image is used to estimate the relevant depth value. When such a depth information repairing algorithm is used, if the depth information around the to-be-repaired region is high in similarity, over-repairing of the depth information will occur; if the to-be-repaired region is at the edge part collected by the RGB-D depth camera, the accuracy of the repaired depth information cannot be guaranteed. At present, the filtering repairing algorithm for depth information mainly focuses on: bilateral filtering, joint bilateral filtering, joint trilateral filtering, and other extended filtering based on the three.

[0005] Tomasi et al. proposed a depth information inpainting algorithm based on non-linear bilateral filtering, "Bilateral Filtering for Gray and Color Images" combined with spatial information and gray similarity, realized the smoothing inpainting and edge-preserving denoising of depth information. Petschnigg et al. proposed a depth information inpainting algorithm based on joint bilateral filtering, "Digital Photography with Flash and no-Flash Image Pairs" guided the inpainting of depth information with color image, to a certain extent, solved the problem of edge blur in the process of depth information inpainting. SeungWonJung et al. proposed a depth information inpainting algorithm based on joint trilateral filtering, taking the regional similarity of depth information and color image as the breakthrough point, realized the inpainting of depth information and the enhancement of color image.

[0006] "A New Method for Inpainting of Depth Maps from Time-of-Flight Sensors based on a Modified Closing by Reconstruction Algorithm", using closing reconstruction and iterative filtering principle to inpaint depth information. "Kinect Depth Recovery based on Local Filters and Plane Primitives" fully considers the relationship between the depth information of the to-be-inpainted region and the pixel information, and effectively realizes the inpainting processing of small area depth information by combining bilateral filtering. "Texture Edge-Guided Depth Recovery for Structured Light-based Depth Sensor" combines pixel filtering and median filtering to inpaint depth information, achieving the purpose of smoothing depth information and edge-preserving denoising.

[0007] The three-dimensional shape of the depth information of the special-shaped projection surface plays a crucial role in the special-shaped projection surface display system, and the lack and jitter of the depth information will seriously affect the correction result of the geometric deformation, and the projection display picture cannot meet the normal visual perception of people. The most core function of the projection display system is to present the viewing picture that people can normally perceive visually, and once the projection display picture occurs geometric deformation, it will seriously affect the normal visual experience of people. Therefore, the present application provides a depth information inpainting method for special-shaped projection surface RGB-D space, to achieve the purpose of filling and smoothing the depth information of the special-shaped projection surface. SUMMARY

[0008] The present application aims to provide a depth information repairing method for a special-shaped projection surface RGB-D space, so as to solve the technical problems of unexpected jitter, surface sampling loss and influence on geometric deformation processing of the depth information of the special-shaped projection surface RGB-D space caused by the limitation of the hardware device itself or the influence of the depth information collection environment.

[0009] In order to achieve this purpose, the technical scheme of the present application is as follows: a depth information repairing method for a special-shaped projection surface RGB-D space, characterized in that it comprises the following steps,

[0010] Step S1, define the Gray code sequence image control parameter params using the Params structure body of the GrayCodePattern class in the OpenCV library, and set the width attribute of the Gray code sequence image control parameter params to PWs pixels and the height attribute of the Gray code sequence image control parameter params to PHs pixels;

[0011] Step S2, perform instantiation operation through the create function of the GrayCodePattern class in the OpenCV library, and when instantiating the GrayCodePattern class, pass the Gray code sequence image control parameter params to the create function in the form of a function argument, so as to obtain the instantiation object GCPO of the GrayCodePattern class;

[0012] Step S3, generate the Gray code sequence image GCPs through the instantiation object GCPO of the GrayCodePattern class in the OpenCV library by calling the generate function of the instantiation object GCPO, so as to obtain the Gray code sequence image GCPs={GCPs 1 ,GCPs 2 , ……, GCPs t-1 , GCPs t}, wherein: GCPs t represents the t-th Gray code image, and the value range of t is determined by the Gray code sequence image control parameter params;

[0013] Step S4, set the horizontal resolution of the projector display picture to PWs pixels and the vertical resolution to PHs pixels, and set the color image collection resolution of the RGB-D depth camera to W RGB ×H RGB , and the depth image collection resolution to W D ×H D , wherein the horizontal resolution of the color image of the RGB-D depth camera is WRGB , the color image longitudinal resolution is H RGB , the depth image transverse resolution is W D , the depth image longitudinal resolution is H D ;

[0014] Step S5, each coded image of the Gray code sequence image GCPs is projected on the special-shaped projection surface by the projector in sequence, and meanwhile, each coded image projected on the special-shaped projection surface is collected by the RGB-D depth camera in sequence, so as to obtain a color image sequence CGCPs collected by the RGB-D depth camera, a depth image sequence DGCPs collected by the RGB-D depth camera, and a color image sequence CGCPs and a depth image sequence DGCPs having an alignment relationship, and the depth image sequence DGCPs being aligned with the color image sequence CGCPs, wherein CGCPs 1 , CGCPs 2 , ……, CGCPs t-1 , CGCPs t , DGCPs 1 , DGCPs 2 , ……, DGCPs t-1 , DGCPs t .

[0015] Step S6, according to each depth image data of the depth image sequence DGCPs, the image coordinate value and the depth value of all pixel points in each depth image are respectively transmitted to the space coordinate mapping processing process by using the space coordinate mapping processing process built in the RGB-D depth camera;

[0016] Step S7, through the three-dimensional space coordinate value solution of the space coordinate mapping processing process, the three-dimensional space coordinate value sequence DSPVs corresponding to all pixel points in the color image sequence CGCPs can be obtained, and DSPVs={{DSPVs 1 (1,1) ,DSPVs 1 (1,2) , ……, DSPVs 1 (m,n-1) , DSPVs 1 (m,n)}, {DSPVs 2 (1,1) , DSPVs 2 (1,2) , ……,DSPVs 2 (m,n-1) , DSPVs 2 (m,n)}, ……, {DSPVs t-1 (1,1) , DSPVst-1 (1,2) , ..., DSPVs t-1 (m,n-1) ,DSPVs t-1 (m,n)}, {DSPVs t (1,1) , DSPVs t (1,2) , ..., DSPVs t (m,n-1) , DSPVs t (m,n)}}, where: DSPVs t (m,n) Represents the three-dimensional coordinate value of the pixel point in the mth row and nth column in the tth color image of the color image sequence CGCPs, 1 ≤ m ≤ H RGB , 1 ≤ n ≤ W RGB ;

[0017] Step S8, set the minimum effective detection distance mEDv to ε, and create a cumulative sequence of effective three-dimensional space coordinate values ​​ESTs={ESTs (1,1) , ESTs (1,2) , ..., ESTs (m,n-1) , ESTs (m,n)}, create a counting sequence of valid three-dimensional space coordinate values ​​ESCs={ESCs (1,1) , ESCs (1,2) , …, ESCs (m,n-1) , ESCs (m,n)}, and assign the initial value of the cumulative sequence ESTs and the counting sequence ESCs to 0, so that the three-dimensional space coordinate value ESTs (p,q) X-axis coordinate value ESTs (p,q) .x=0, ​​three-dimensional space coordinate value ESTs (p,q) Y-axis coordinate value ESTs (p,q) .y=0, three-dimensional space coordinate value ESTs (p,q) Z-axis coordinate value ESTs (p,q) .z=0、ESCs (p,q) =0, where: 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0018] Step S9, traverse each three-dimensional space coordinate value DSPVs in the three-dimensional space coordinate value sequence DSPVs c (p,q) Z-axis coordinate value DSPVs c (p,q) .z, if the three-dimensional space coordinate value DSPVs c(p,q) Z-axis coordinate value DSPVs c (p,q) If z is greater than mEDv, then let TXv=ESTs (p,q) .x、TYv=ESTs (p,q) .y、TZv=ESTs (p,q) .z、TCv=ESCs (p,q) Then, ESTs are queried through expression 1, expression 2, and expression 3. (p,q) .x, ESTs (p,q) .y, ESTs (p,q) .z performs the accumulation operation of the three-dimensional space coordinate values ​​and uses expression 4 to count the valid three-dimensional space coordinate values; if the three-dimensional space coordinate value DSPVs c (p,q) Z-axis coordinate value DSPVs c (p,q) If z is less than or equal to mEDv, the 3D space coordinate value DSPVs is ignored. c (p,q) Accumulation and counting operations, where: 1 ≤ c ≤ t, 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0019] Expression 1: ESTs (p,q) .x = TXv + DSPVs c (p,q) .x

[0020] Expression 2: ESTs (p,q) .y = TYv + DSPVs c (p,q) .y

[0021] Expression 3: ESTs (p,q) .z = TZv + DSPVs c (p,q) .z

[0022] Expression 4: ESCs (p,q) = TCv + 1

[0023] Step S10, create a trusted sequence of three-dimensional space coordinate values ​​ECPs={ECPs (1,1) , ECPs (1,2) , ...,ECPs (m,n-1) , ECPs (m,n)}, according to the traversed processed accumulated sequence ESTs and the counted sequence ESCs, the X-axis coordinate value, the Y-axis coordinate value and the Z-axis coordinate value of each three-dimensional space coordinate value in the trusted sequence ECPs are calculated respectively by expression 5, expression 6 and expression 7, so as to obtain the trusted value of all three-dimensional space coordinate values in the trusted sequence ECPs, wherein: 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0024] Expression 5: ECPs (p,q) .x = ESTs (p,q) .x / ESCs (p,q)

[0025] Expression 6: ECPs (p,q) .y = ESTs (p,q) .y / ESCs (p,q)

[0026] Expression 7: ECPs (p,q) .z = ESTs (p,q) .z / ESCs (p,q)

[0027] Step S11, using the color image sequence CGCPs collected by the RGB-D depth camera and the Gray code decoding principle, the decoding operation is performed on all pixel points corresponding to the trusted sequence ECPs, and the code value sequence EGDs={EGDs (1,1) , EGDs (1,2) ,……, EGDs (m,n-1) , EGDs (m,n)} is obtained, and all pixel point positions of the trusted sequence ECPs and the code value sequence EGDs have one-to-one correspondence;

[0028] Step S12, according to the same Gray code value in the code value sequence EGDs, the region classification processing is performed on all three-dimensional space coordinate values in the trusted sequence ECPs, and the initial block region sequence CDBZs of three-dimensional space coordinate values is formed, that is, CDBZs={DGC 1 {CDBZs 1 1, CDBZs 1 2, ……, CDBZs 1 pc_1}, DGC 2 {CDBZs 2 1, CDBZs 2 2, ……, CDBZs 2 pc_2},……, DGC bc-1 {CDBZs bc-1 1, CDBZs bc-12, ..., CDBZs bc-1 pc_bc-1}, DGC bc {CDBZs bc 1. CDBZs bc 2,……, CDBZs bc pc_bc}}, where: DGC bc Indicates the Gray code value of the bcth block region in the initial block region sequence CDBZs, CDBZs bc pc_bc Indicates the pc_bc-th three-dimensional space coordinate value of the bc-th block region in the initial block region sequence CDBZs, bc indicates the number of block regions contained in the initial block region sequence CDBZs, pc_1, pc_2, ..., pc_bc indicate the number of three-dimensional space coordinate values ​​contained in the 1st, 2nd, ..., bc-th block regions respectively;

[0029] Step S13, using the mathematical solution method of plane equations, according to all the three-dimensional space coordinate values ​​of each block region in the initial block region sequence CDBZs, solve the regional plane equation sequence BPSEs={BPSEs1, BPSEs2, ..., BPSEs bc-1 , BPSEs bc}, where: BPSEs bc Represents the plane equation corresponding to the bcth block region in the initial block region sequence CDBZs;

[0030] Step S14, set the minimum smooth obtuse angle mSGv between two planes that meets the smooth transition principle as θ, and traverse each regional plane equation BPSEs of the regional plane equation sequence BPSEs k , if the domain plane equations BPSEs k and other BPSEs in the BPSEs of area plane equations e If the angle between the two is greater than mSGv, the regional plane equation BPSEs e All corresponding three-dimensional space coordinate values ​​{CDBZs e 1. CDBZs e 2, ..., CDBZs e pc_e Add to the area plane equations BPSEs k In the corresponding block area, the regional plane equation BPSEs k The corresponding k-th block area becomes {CDBZs k 1. CDBZs k 2,……, CDBZs kpc_k , CDBZs e 1, CDBZs e 2, ……, CDBZs e pc_e}, where: 1 ≤ k ≤ bc, 1 ≤ e ≤ bc and e ≠ k, pc_k, pc_e represent the number of three-dimensional coordinate values contained in the kth and eth block region respectively;

[0031] Step S15, through the traversal processing of the region plane equation sequence BPSEs, and according to the recombination of all three-dimensional coordinate values in the initial block region sequence CDBZs in the traversal process, the maximum smooth transition region sequence SDBGs = {DGC 1 {SDBGs 1 1, SDBGs 1 2, ……, SDBGs 1 sc_1}, DGC 2 {SDBGs 2 1, SDBGs 2 2, ……, SDBGs 2 sc_2}, ……, DGC bc-1 {SDBGs bc-1 1, SDBGs bc-1 2, ……, SDBGs bc-1 sc_bc-1}, DGC bc {SDBGs bc 1, SDBGs bc 2, ……, SDBGs bc sc_bc}}, where: SDBGs bc sc_bc represents the sc_bc three-dimensional coordinate value of the bc smooth transition region in the smooth transition region sequence SDBGs, bc represents the number of smooth transition regions contained in the smooth transition region sequence SDBGs, sc_1, sc_2, …, sc_bc represent the number of three-dimensional coordinate values contained in the first, second, …, bc smooth transition regions respectively;

[0032] Step S16, traverse all smooth transition regions of the smooth transition region sequence SDBGs, and calculate the surface fitting parameter sequence SFPVs corresponding to the smooth transition region sequence SDBGs according to the surface fitting expression 8 and using the least square fitting method, and SFPVs = { {V a 1 , V b 1 , Vc 1 , V d 1 , V e 1 , V f 1}, {V a 2 , V b 2 , V c 2 , V d 2 , V e 2 , V f 2},……, {V a bc-1 , V b bc-1 , V c bc-1 , V d bc-1 , V e bc-1 , V f bc-1}, {V a bc , V b bc , V c bc , V d bc , V e bc ,V f bc}}, where: {V a bc , V b bc , V c bc , V d bc , V e bc , V f bc} represents the bcth group of surface fitting parameter values ​​of the surface fitting parameter sequence SFPVs, and X, Y, and Z in Expression 8 represent the X-axis coordinate value, Y-axis coordinate value, and Z-axis coordinate value of the three-dimensional space coordinate value, respectively;

[0033] Expression 8: Z = V a × X 2 + V b × Y 2 + V cX X Y + V d X X + V e X Y + V f

[0034] Step S17, create the optimized value sequence DOVs = {DOVs (1,1) , DOVs (1,2) , ……, DOVs (m,n-1) , DOVs (m,n)} of the depth information, traverse the three-dimensional space coordinate values ECPs (w,r) of the trusted sequence ECPs one by one, and the three-dimensional space coordinate values ECPs (w,r) correspond to the Gray code values EGDs (w,r) , if the Gray code value EGDs (w,r) is equal to the Gray code value DGC h of the hth smooth transition region in the smooth transition region sequence SDBGs, then the X-axis coordinate value DOVs (w,r) .x, the Y-axis coordinate value DOVs (w,r) .y and the Z-axis coordinate value DOVs (w,r) .z of DOVs (w,r) in the optimized value sequence DOVs can be calculated by using the expression 9, the expression 10, the expression 11 and the hth set of surface fitting parameter values in the surface fitting parameter sequence SFPVs, where: 1 ≤ w ≤ m, 1 ≤ r ≤ n, 1 ≤ h ≤ bc.

[0035] Expression 9: DOVs (w,r) .x = ECPs (w,r) .x

[0036] Expression 10: DOVs (w,r) .y = ECPs (w,r) .y

[0037] Expression 11: DOVs (w,r) .z = V a h × (DOVs (w,r) .x) 2 + V b h × (DOVs (w,r) .y) 2 + V c h ×DOVs (w,r) .x × DOVs (w,r) .y + V d h × DOVs (w,r) .x + Ve h × DOVs (w,r) .y + V f h

[0038] Step S18, through the fitting optimization calculation of each three-dimensional space coordinate value in the trusted sequence ECPs in step S17, the optimized value sequence DOVs after the depth information repair is obtained, that is, DOVs = {{DOVs (1,1) .x, DOVs (1,1) .y, DOVs (1,1) .z}, {DOVs (1,2) .x, DOVs (1,2) .y, DOVs (1,2) .z}, ……, {DOVs (m,n-1) .x, DOVs (m,n-1) .y, DOVs (m,n-1) .z}, {DOVs (m,n) .x, DOVs (m,n) .y, DOVs (m,n) .z}}.

[0039] Compared with the prior art, the beneficial effects of the present application at least lie in that the method presents the geometric morphological change trend of the special-shaped projection surface through the Gray code sequence image, divides the smooth transition area according to the geometric morphological change trend of the special-shaped projection surface, and accurately fits the three-dimensional depth information of the smooth transition area by using the three-dimensional curved surface mathematical model, thereby providing basic guarantee for the subsequent geometric correction process applied to the special-shaped projection surface projection display system. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a structural schematic diagram of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described in detail by connecting the drawings through specific implementation manners. It needs to be explained here that the description of these implementation manners is used to help understand the present application, but does not constitute the limitation to the present application. In addition, the technical features involved in each implementation manner of the present application described below can be combined with each other as long as there is no conflict.

[0042] In order to realize the depth information repair of the special-shaped projection surface RGB-D space, the original depth information and the texture features collected must be reasonably used, and the texture features of the special-shaped projection surface are needed to provide reference for the repair process of the original depth information. Since the texture features of the special-shaped projection surface itself often cannot clearly show the geometric shape change trend, especially when the special-shaped projection surface does not have available texture features, the repair result of the original depth information of the special-shaped projection surface becomes less reliable. Therefore, in the depth information repair method of the special-shaped projection surface RGB-D space, the special-shaped projection surface is provided with high available texture features by projecting the Gray code sequence image using the projector, which effectively overcomes the problem of lack of available texture features of the special-shaped projection surface.

[0043] Required equipment: as shown in Figure 1 In the depth information repair method of the special-shaped projection surface RGB-D space, the special-shaped projection surface display system is composed of a computer, an RGB-D depth camera, a projector and a special-shaped projection surface, wherein the RGB-D depth camera is connected to the computer through a data transmission line, the projector is connected to the computer through a video transmission line, and the computer acquires the color image and the depth information of the special-shaped projection surface by using the RGB-D depth camera.

[0044] This method separately processes the original depth information of the special-shaped projection surface in each smooth transition area according to the geometric shape change trend of the special-shaped projection surface, so that the depth information of each smooth transition area can be smooth and continuous, greatly eliminating the over-repair, under-repair and other situations of the depth information of this method, and effectively enhancing the availability and reliability of the depth information of the special-shaped projection surface RGB-D space. The required equipment and implementation steps of this method are as follows:

[0045] Step S1, define the Gray code sequence image control parameter params using the Params structure body of the GrayCodePattern class in the OpenCV library, and set the width attribute of the Gray code sequence image control parameter params to PWs pixels and the height attribute of the Gray code sequence image control parameter params to PHs pixels;

[0046] Step S2, perform instantiation operation through the create function of the GrayCodePattern class in the OpenCV library, and when instantiating the GrayCodePattern class, pass the Gray code sequence image control parameter params to the create function as a function argument, so as to obtain the instantiation object GCPO of the GrayCodePattern class;

[0047] Step S3, using the instantiated object GCPO of the GrayCodePattern class in the OpenCV class library, generating the Gray code sequence image GCPs by calling the generate function of the instantiated object GCPO, obtaining the Gray code sequence image GCPs={GCPs 1 , GCPs 2 , ……, GCPs t-1 , GCPs t}, wherein: GCPs t represents the t-th Gray code image, and the value range of t is determined by the Gray code sequence image control parameter params;

[0048] Step S4, setting the horizontal resolution of the projection display picture as PWs pixels, the vertical resolution as PHs pixels, setting the color image acquisition resolution of the RGB-D depth camera as W RGB ×H RGB , and the depth image acquisition resolution as W D ×H D , wherein the color image horizontal resolution of the RGB-D depth camera is W RGB , the color image vertical resolution is H RGB , the depth image horizontal resolution is W D , and the depth image vertical resolution is H D ;

[0049] Step S5, sequentially projecting each coded image of the Gray code sequence image GCPs on the special-shaped projection surface by the projector, and simultaneously using the RGB-D depth camera to sequentially acquire each coded image projected on the special-shaped projection surface, obtaining the color image sequence CGCPs={CGCPs 1 , CGCPs 2 , ……, CGCPs t-1 , CGCPs t} acquired by the RGB-D depth camera and the depth image sequence DGCPs={DGCPs 1 , DGCPs 2 , ……, DGCPs t-1 , DGCPs t} acquired by the RGB-D depth camera, the color image sequence CGCPs and the depth image sequence DGCPs have an alignment relationship, and the depth image sequence DGCPs is aligned with the color image sequence CGCPs;

[0050] Step S6, using the space coordinate mapping processing process built-in the RGB-D depth camera, respectively transferring the image coordinate values and depth values of all pixel points in each depth image of the depth image sequence DGCPs to the space coordinate mapping processing process.

[0051] Step S7, through the three-dimensional space coordinate value solution of the space coordinate mapping process, the three-dimensional space coordinate value sequence DSPVs corresponding to all pixel points in the color image sequence CGCPs can be obtained, and DSPVs={{DSPVs 1 (1,1) ,DSPVs 1 (1,2) , ……, DSPVs 1 (m,n-1) , DSPVs 1 (m,n)}, {DSPVs 2 (1,1) , DSPVs 2 (1,2) , ……,DSPVs 2 (m,n-1) , DSPVs 2 (m,n)}, ……, {DSPVs t-1 (1,1) , DSPVs t-1 (1,2) , ……, DSPVs t-1 (m,n-1) ,DSPVs t-1 (m,n)}, {DSPVs t (1,1) , DSPVs t (1,2) , ……, DSPVs t (m,n-1) , DSPVs t (m,n)}}. Wherein: DSPVs t (m,n) represents the three-dimensional space coordinate value corresponding to the pixel point in the mth row and the nth column of the tth color image in the color image sequence CGCPs, 1 ≤ m ≤ H RGB , 1 ≤ n ≤ W RGB ;

[0052] Step S8, set the minimum effective detection distance mEDv as ε, create the accumulation sequence ESTs of effective three-dimensional space coordinate values ESTs={ESTs (1,1) , ESTs (1,2) , ……, ESTs (m,n-1) , ESTs (m,n)}, create the count sequence ESCs of effective three-dimensional space coordinate values ESCs={ESCs (1,1) , ESCs (1,2) , ……, ESCs(m,n-1) , ESCs (m,n)}, and assign the initial value of the cumulative sequence ESTs and the counting sequence ESCs to 0, so that the three-dimensional space coordinate value ESTs (p,q) X-axis coordinate value ESTs (p,q) .x=0, ​​three-dimensional space coordinate value ESTs (p,q) Y-axis coordinate value ESTs (p,q) .y=0, three-dimensional space coordinate value ESTs (p,q) Z-axis coordinate value ESTs (p,q) .z=0、ESCs (p,q) =0, where: 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0053] Step S9, traverse each three-dimensional space coordinate value DSPVs in the three-dimensional space coordinate value sequence DSPVs c (p,q) Z-axis coordinate value DSPVs c (p,q) .z, if the three-dimensional space coordinate value DSPVs c (p,q) Z-axis coordinate value DSPVs c (p,q) If z is greater than mEDv, then let TXv=ESTs (p,q) .x、TYv=ESTs (p,q) .y、TZv=ESTs (p,q) .z、TCv=ESCs (p,q) Then, ESTs are queried through expression 1, expression 2, and expression 3. (p,q) .x, ESTs (p,q) .y, ESTs (p,q) .z performs the accumulation operation of the three-dimensional space coordinate values ​​and uses expression 4 to count the valid three-dimensional space coordinate values; if the three-dimensional space coordinate value DSPVs c (p,q) Z-axis coordinate value DSPVs c (p,q) If z is less than or equal to mEDv, the 3D space coordinate value DSPVs is ignored. c (p,q) Accumulation and counting operations, where: 1 ≤ c ≤ t, 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0054] Expression 1: ESTs (p,q) .x = TXv + DSPVs c (p,q) .x

[0055] Expression 2: ESTs(p,q) .y = TYv + DSPVs c (p,q) .y

[0056] Expression 3: ESTs (p,q) .z = TZv + DSPVs c (p,q) .z

[0057] Expression 4: ESCs (p,q) = TCv + 1

[0058] Step S10, create a trusted sequence of three-dimensional space coordinate values ECPs = {ECPs (1,1) , ECPs (1,2) , ……, ECPs (m,n-1) , ECPs (m,n)} according to the accumulated sequence ESTs and the count sequence ESCs that have been traversed and processed, calculate the X-axis coordinate value, Y-axis coordinate value and Z-axis coordinate value of each three-dimensional space coordinate value in the trusted sequence ECPs respectively through Expression 5, Expression 6 and Expression 7, so as to obtain the trusted value of all three-dimensional space coordinate values in the trusted sequence ECPs, wherein: 1 ≤ p ≤ m, 1 ≤ q ≤ n;

[0059] Expression 5: ECPs (p,q) .x = ESTs (p,q) .x / ESCs (p,q)

[0060] Expression 6: ECPs (p,q) .y = ESTs (p,q) .y / ESCs (p,q)

[0061] Expression 7: ECPs (p,q) .z = ESTs (p,q) .z / ESCs (p,q)

[0062] Step S11, using the color image sequence CGCPs collected by the RGB-D depth camera and the Gray code decoding principle, perform decoding operation on all pixel points corresponding to the trusted sequence ECPs to obtain a code value sequence EGDs = {EGDs (1,1) , EGDs (1,2) ,……, EGDs (m,n-1) , EGDs (m,n)}, and all pixel point positions of the trusted sequence ECPs and the code value sequence EGDs have one-to-one correspondence;

[0063] Step S12, based on the same Gray code value in the code value sequence EGDs, all three-dimensional space coordinate values in the trusted sequence ECPs are regionally classified to form an initial block region sequence CDBZs={DGC 1 {CDBZs 1 1, CDBZs 1 2, ……, CDBZs 1 pc_1}, DGC 2 {CDBZs 2 1, CDBZs 2 2, ……, CDBZs 2 pc_2},……, DGC bc-1 {CDBZs bc-1 1, CDBZs bc-1 2, ……, CDBZs bc-1 pc_bc-1}, DGC bc {CDBZs bc 1, CDBZs bc 2, ……, CDBZs bc pc_bc}}. bc represents the Gray code value of the bc-th block region in the initial block region sequence CDBZs, CDBZs bc pc_bc represents the pc_bc-th three-dimensional space coordinate value of the bc-th block region in the initial block region sequence CDBZs, bc represents the number of block regions contained in the initial block region sequence CDBZs, pc_1, pc_2, ……, pc_bc respectively represent the number of three-dimensional space coordinate values contained in the 1st, 2nd, ……, bc-th block region;

[0064] Step S13, using the mathematical solving method of plane equation, according to all three-dimensional space coordinate values of each block region in the initial block region sequence CDBZs, the region plane equation sequence BPSEs corresponding to the initial block region sequence CDBZs is calculated as {BPSEs1, BPSEs2, ……, BPSEs bc-1 , BPSEs bc}, wherein: BPSEs bc represents the plane equation corresponding to the bc-th block region in the initial block region sequence CDBZs.

[0065] Step S14, set the minimum smooth obtuse angle mSGv of two planes meeting the smooth transition principle as θ, traverse each region plane equation BPSEs in the region plane equation sequence BPSEs kIf the angle between the region plane equation BPSEs k and other region plane equations BPSEs in the region plane equation sequence BPSEs e is greater than mSGv, the region plane equation BPSEs e corresponding to all three-dimensional space coordinate values {CDBZs e 1, CDBZs e 2, ……, CDBZs e pc_e} is added to the region plane equation BPSEs k corresponding to the sub-block region, so that the region plane equation BPSEs k corresponding to the kth sub-block region becomes {CDBZs k 1, CDBZs k 2,……, CDBZs k pc_k , CDBZs e 1, CDBZs e 2, ……, CDBZs e pc_e}, where 1 ≤ k ≤ bc, 1 ≤ e ≤ bc and e ≠ k, pc_k and pc_e respectively represent the number of three-dimensional space coordinate values contained in the kth and e th sub-block regions;

[0066] Step S15, through the traversal processing of the region plane equation sequence BPSEs, and according to the recombination of all three-dimensional space coordinate values in the initial sub-block region sequence CDBZs in the traversal process, the maximized smooth transition region sequence SDBGs={DGC 1 {SDBGs 1 1, SDBGs 1 2, ……, SDBGs 1 sc_1}, DGC 2 {SDBGs 2 1, SDBGs 2 2, ……, SDBGs 2 sc_2}, ……, DGC bc-1 {SDBGs bc-1 1, SDBGs bc-1 2, ……, SDBGs bc-1 sc_bc-1}, DGC bc {SDBGs bc 1, SDBGs bc 2, ……, SDBGs bc sc_bc}, wherein: SDBGs bc sc_bc represents the sc bc-th three-dimensional space coordinate value of the bc-th smooth transition region in the smooth transition region sequence SDBGs, and bc represents the number of smooth transition regions included in the smooth transition region sequence SDBGs, and sc 1, sc 2, …, sc bc respectively represent the number of three-dimensional space coordinate values included in the 1st, 2nd, …, bc-th smooth transition region;

[0067] In step S16, all smooth transition regions in the smooth transition region sequence SDBGs are traversed, and the surface fitting parameter sequence SFPVs corresponding to the smooth transition region sequence SDBGs is calculated according to the surface fitting expression 8 and by using the least square fitting method, and SFPVs={{V a 1 , V b 1 , V c 1 , V d 1 , V e 1 , V f 1}, {V a 2 , V b 2 , V c 2 , V d 2 , V e 2 , V f 2},……, {V a bc-1 , V b bc-1 , V c bc-1 , V d bc-1 , V e bc-1 , V f bc-1}, {V a bc , V b bc , V c bc , V d bc , V e bc ,V f bc}}, wherein: {Va bc , V b bc , V c bc , V d bc , V e bc , V f bc} represents the bcthgroup of surface fitting parameter values of the surface fitting parameter sequence SFPVs, and X, Y, Z in expression 8 represent the X-axis coordinate value, the Y-axis coordinate value, and the Z-axis coordinate value of the three-dimensional space coordinate value, respectively;

[0068] Expression 8: Z = V a × X 2 + V b × Y 2 + V c × X × Y + V d × X + V e × Y + V f

[0069] Step S17, create the optimized value sequence DOVs = {DOVs (1,1) , DOVs (1,2) , ……, DOVs (m,n-1) , DOVs (m,n)} of the depth information, traverse the three-dimensional space coordinate values ECPs (w,r) of the trusted sequence ECPs one by one, and the three-dimensional space coordinate value ECPs (w,r) corresponds to the Gray code value EGDs (w,r) , if the Gray code value EGDs (w,r) is equal to the Gray code value DGC h of the hths mooth transition region in the smooth transition region sequence SDBGs, then the X-axis coordinate value DOVs (w,r) .x, the Y-axis coordinate value DOVs (w,r) .y, and the Z-axis coordinate value DOVs (w,r) .z of DOVs (w,r) in the optimized value sequence DOVs can be calculated by using expression 9, expression 10, expression 11, and the hths m of surface fitting parameter values in the surface fitting parameter sequence SFPVs, wherein: 1 ≤ w ≤ m, 1 ≤ r ≤ n, and 1 ≤ h ≤ bc;

[0070] Expression 9: DOVs (w,r) .x = ECPs (w,r) .x

[0071] Expression 10: DOVs (w,r) .y = ECPs (w,r) .y

[0072] Expression 11: DOVs (w,r) .z = V a h × (DOVs (w,r) .x) 2 + V b h × (DOVs (w,r) .y) 2 + V c h ×DOVs (w,r) .x × DOVs (w,r) .y + V d h × DOVs (w,r) .x + V e h × DOVs (w,r) .y + V f h

[0073] Step S18, the fitting optimization calculation of each three-dimensional space coordinate value in the trusted sequence ECPs through step S17, that is, the depth information repaired optimized value sequence DOVs={{DOVs (1,1) .x, DOVs (1,1) .y, DOVs (1,1) .z}, {DOVs (1,2) .x, DOVs (1,2) .y, DOVs (1,2) .z}, ……, {DOVs (m,n-1) .x, DOVs (m,n-1) .y,DOVs (m,n-1) .z}, {DOVs (m,n) .x, DOVs (m,n) .y, DOVs (m,n) .z}}.

[0074] The above only expresses the embodiments of the present application, which are described in more detail and in more detail, but cannot be understood as limiting the scope of the patent application. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application patent should be subject to the appended claims.

Claims

1. A method for repairing depth information of an anamorphic projection surface RGB-D space, characterized in that, Comprising the following steps, Step S1, using the OpenCV library GrayCodePattern class of the structure body definition of the Gray code sequence image control parameters params, and set the Gray code sequence image control parameters params width attribute for PWs pixels, set the Gray code sequence image control parameters params height attribute for PHs pixels; Step S2, through the OpenCV library GrayCodePattern class of create function instantiation object operation, in GrayCodePattern class instantiation object, the Gray code sequence image control parameters params is passed to the create function in the form of function argument, thus obtaining the instantiation object GCPO of GrayCodePattern class; Step S3, using the instantiated object GCPO of the GrayCodePattern class in the OpenCV class library, generating the Gray code sequence image GCPs by calling the generate function of the instantiated object GCPO, to obtain the Gray code sequence image GCPs={GCPs 1 ,GCPs 2 ,……,GCPs t-1 ,GCPs t}, wherein: GCPs t represents the t-th Gray code image, and the value range of t is determined by the Gray code sequence image control parameter params; Step S4, set the horizontal resolution of the projector display picture as PWs pixels, the vertical resolution as PHs pixels, set the color image acquisition resolution of the RGB-D depth camera as W RGB ×H RGB , the depth image acquisition resolution as W D ×H D , wherein the horizontal resolution of the color image of the RGB-D depth camera is W RGB , the vertical resolution is H RGB , the horizontal resolution of the depth image is W D , and the vertical resolution is H D . Step S5, project each coded image of the Gray code sequence image GCPs on the irregular projection surface by the projector in sequence, and collect each coded image projected on the irregular projection surface by using the RGB-D depth camera in sequence to obtain a color image sequence CGCPs={CGCPs 1 , CGCPs 2 , ……, CGCPs t-1 , CGCPs t} collected by the RGB-D depth camera, a depth image sequence DGCPs={DGCPs 1 , DGCPs 2 , ……, DGCPs t-1 , DGCPs t} collected by the RGB-D depth camera, the color image sequence CGCPs and the depth image sequence DGCPs have an alignment relationship, and the depth image sequence DGCPs is aligned with the color image sequence CGCPs; Step S6, using the space coordinate mapping process built-in RGB-D depth camera, according to the depth image sequence DGCPs of each depth image data, respectively, the image coordinates and depth values of all pixel points in each depth image are transmitted to the space coordinate mapping process; Step S7, the three-dimensional space coordinate value of the color image sequence CGCPs is obtained by the three-dimensional space coordinate value of the space coordinate mapping process, that is, the three-dimensional space coordinate value sequence DSPVs corresponding to all pixel points in the color image sequence CGCPs, and DSPVs = { { DSPVs 1 (1,1) , DSPVs 1 (1,2) , ……, DSPVs 1 (m,n-1) , DSPVs 1 (m,n)}, { DSPVs 2 (1,1) , DSPVs 2 (1,2) , ……, DSPVs 2 (m,n-1) , DSPVs 2 (m,n)}, ……, { DSPVs t-1 (1,1) , DSPVs t-1 (1,2) , ……, DSPVs t-1 (m,n-1) , DSPVs t-1 (m,n)}, { DSPVs t (1,1) , DSPVs t (1,2) , ……, DSPVs t (m,n-1) , DSPVs t (m,n)}}, wherein: DSPVs t (m,n) represents the three-dimensional space coordinate value corresponding to the pixel point of the mth row and the nth column in the tth color image of the color image sequence CGCPs, 1≤m≤H RGB , 1≤n≤W RGB ; Step S8, set the minimum effective detection distance mEDv as ε, create an accumulated sequence of effective three-dimensional space coordinate values ESTs = {ESTs (1,1) , ESTs (1,2) , …, ESTs (m,n-1) , ESTs (m,n)}, create a counting sequence of effective three-dimensional space coordinate values ESCs = {ESCs (1,1) , ESCs (1,2) , …, ESCs (m,n-1) , ESCs (m,n)}, and perform an assignment operation on the accumulated sequence ESTs and the counting sequence ESCs with an initial value of 0, so that the X-axis coordinate value ESTs (p,q) .x = 0 of the three-dimensional space coordinate value ESTs (p,q) , the Y-axis coordinate value ESTs (p,q) .y = 0 of the three-dimensional space coordinate value ESTs (p,q) , the Z-axis coordinate value ESTs (p,q) .z = 0 of the three-dimensional space coordinate value ESTs (p,q) , and ESCs (p,q) = 0, where: 1≤p≤m, 1≤q≤n; Step S9, traversing each three-dimensional space coordinate value DSPVs in the three-dimensional space coordinate value sequence DSPVs c (p,q) Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) If the Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) If the Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs (p,q) .x, TYv = ESTs (p,q) .y, TZv = ESTs (p,q) .z, TCv = ESCs (p,q) Then, the three-dimensional space coordinate value accumulation operation is performed on ESTs (p,q) .x, ESTs (p,q) .y, ESTs (p,q) .z by using the expression 1, the expression 2, and the expression 3, and the counting operation on the effective three-dimensional space coordinate value is performed by using the expression 4; if the Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) If the Z-axis coordinate value DSPVs of the three-dimensional space coordinate value DSPVs c (p,q) The accumulation operation and the counting operation are ignored, wherein 1≤c≤t, 1≤p≤m, and 1≤q≤n. Expression 1 : ESTs (p,q) .x = TXv + DSPvs c (p,q) .x Expression 2: ESTs (p,q) .y=TYv+DSPVs c (p,q) .y Expression 3: ESTs (p,q) z = TZv + DSPvs c (p,q) z Expression 4: ESCs (p,q) = TCv+1 Step S10, creating a trusted sequence ECPs={ECPs (1,1) ,ECPs (1,2) ,……,ECPs (m,n-1) ,ECPs (m,n)} of three-dimensional space coordinate values, calculating the X-axis coordinate value, the Y-axis coordinate value and the Z-axis coordinate value of each three-dimensional space coordinate value in the trusted sequence ECPs respectively according to expression 5, expression 6 and expression 7 based on the accumulated sequence ESTs and the count sequence ESCs that have been traversed and processed, so as to obtain the trusted values of all three-dimensional space coordinate values in the trusted sequence ECPs, wherein 1≤p≤m, 1≤q≤n. Expression 5: ECPs (p,q) .x = ESTs (p,q) .x / ESCs (p,q) Expression 6: ECPs (p,q) .y = ESTs (p,q) .y / ESCs (p,q) Expression 7: ECPs (p,q) .z = ESTs (p,q) .z / ESCs (p,q) Step S11, using the color image sequence CGCPs collected by the RGB-D depth camera and the Gray code decoding principle, decoding all pixel points corresponding to the trusted sequence ECPs to obtain the code value sequence EGDs={EGDs (1,1) ,EGDs (1,2) ,……,EGDs (m,n-1) ,EGDs (m,n)}, and all pixel point positions of the trusted sequence ECPs and the code value sequence EGDs have a one-to-one correspondence; Step S12: Based on the code value sequence EGDs having the same Gray code value, all three-dimensional space coordinate values ​​in the credible sequence ECPs are subjected to regional classification processing to form an initial block region sequence CDBZs of three-dimensional space coordinate values ​​= {DGC 1 {CDBZs 1 1,CDBZs 1 2,……,CDBZs 1 pc_1 },DGC 2 {CDBZs 2 1,CDBZs 2 2,……,CDBZs 2 pc_2 },……,DGC bc-1 {CDBZs bc-1 1,CDBZs bc-1 2,……,CDBZs bc-1 pc_bc-1 },DGC bc {CDBZs bc 1,CDBZs bc 2,……,CDBZs bc pc_bc }}, where: DGC bc Indicates the Gray code value of the bcth block region in the initial block region sequence CDBZs, CDBZs bc pc_bc Indicates the pc_bc-th three-dimensional space coordinate value of the bc-th block region in the initial block region sequence CDBZs, bc indicates the number of block regions contained in the initial block region sequence CDBZs, pc_1, pc_2, ..., pc_bc indicate the number of three-dimensional space coordinate values ​​contained in the 1st, 2nd, ..., bc-th block regions respectively; Step S13, using the mathematical solution method of plane equation, according to all three-dimensional coordinate values of each block region in the initial block region sequence CDBZs, the region plane equation sequence BPSEs corresponding to the initial block region sequence CDBZs is calculated as BPSEs={BPSEs1, BPSEs2, …, BPSEs bc-1 , BPSEs bc}, wherein: BPSEs bc represents the plane equation corresponding to the bc-th block region in the initial block region sequence CDBZs. Step S14, set the minimum smooth obtuse angle mSGv of two planes meeting the smooth transition principle as θ, traverse each region plane equation BPSEs of the region plane equation sequence BPSEs k If the included angle between the region plane equation BPSEs k and other region plane equations BPSEs in the region plane equation sequence BPSEs e is greater than mSGv, add all three-dimensional space coordinate values {CDBZs e 1,CDBZs e 2,……,CDBZs e corresponding to the region plane equation BPSEs e to the corresponding sub-block region, so that the region plane equation BPSEs pc_e corresponding to the kth sub-block region becomes {CDBZs k 1,CDBZs k 2,……,CDBZs k , where: 1≤k≤bc, 1≤e≤bc and e≠k, pc_k and pc_e respectively represent the number of three-dimensional space coordinate values contained by the kth and e th sub-block regions; k k pc_k e e e pc_e , where: 1≤k≤bc, 1≤e≤bc and e≠k, pc_k and pc_e respectively represent the number of three-dimensional space coordinate values contained by the kth and e th sub-block regions;​​​ Step S15, by traversing the regional plane equation sequence BPSEs and recombining all three-dimensional space coordinate values ​​in the initial block region sequence CDBZs during the traversal process, a maximized smooth transition region sequence SDBGs={DGC 1 {SDBGs 1 1,SDBGs 1 2,……,SDBGs 1 sc_1 },DGC 2 {SDBGs 2 1,SDBGs 2 2,……,SDBGs 2 sc_2 },……,DGC bc-1 {SDBGs bc-1 1,SDBGs bc-1 2,……,SDBGs bc-1 sc_bc-1 },DGC bc {SDBGs bc 1,SDBGs bc 2,……,SDBGs bc sc_bc }}, where: SDBGs bc sc_bc represents the sc_bc-th three-dimensional space coordinate value of the bc-th smooth transition region in the smooth transition region sequence SDBGs, bc represents the number of smooth transition regions contained in the smooth transition region sequence SDBGs, sc_1, sc_2, ..., sc_bc represent the number of three-dimensional space coordinate values ​​contained in the 1st, 2nd, ..., bcth smooth transition regions respectively; Step S16, traversing all the smooth transition regions of the smooth transition region sequence SDBGs, the surface fitting parameter sequence SFPVs corresponding to the smooth transition region sequence SDBGs is calculated according to the surface fitting expression 8 and by using the least square fitting method, and SFPVs = { {V a 1 ,V b 1 ,V c 1 ,V d 1 ,V e 1 ,V f 1},{V a 2 ,V b 2 ,V c 2 ,V d 2 ,V e 2 ,V f 2},……,{V a bc-1 ,V b bc-1 ,V c bc-1 ,V d bc-1 ,V e bc-1 ,V f bc-1},{V a bc ,V b bc ,V c bc ,V d bc ,V e bc ,V f bc}}. a 1 ,V b 1 ,V c 1 ,V d 1 ,V e 1 ,V f 1} represents the first group of surface fitting parameter values of the surface fitting parameter sequence SFPVs, {V a 2 ,V b 2 ,V c 2 ,V d 2 ,V e 2 ,V f 2} represents the 2nd group of surface fitting parameter values of the surface fitting parameter sequence SFPVs, {V a bc-1 ,V b bc-1 ,V c bc-1 ,V d bc-1 ,V e bc-1 ,V f bc-1} represents the bc-1th group of surface fitting parameter values of the surface fitting parameter sequence SFPVs, {V a bc ,V b bc ,V c bc ,V d bc ,V e bc ,V f bc} represents the bcth group of surface fitting parameter values of the surface fitting parameter sequence SFPVs, X, Y, Z in expression 8 respectively represent X-axis coordinate value, Y-axis coordinate value, Z-axis coordinate value of three-dimensional space coordinate value. Expression 8: Z = V a + X 2 + V b + Y 2 + V c + X*Y + V d + X + V e + Y + V f Step S17, create the optimized value sequence DOVs of the depth information DOVs = {DOVs (1,1) , DOVs (1,2) , …, DOVs (m,n-1) , DOVs (m,n)}, traverse the three-dimensional space coordinate values ECPs (w,r) of the trusted sequence ECPs one by one, and the corresponding Gray code value of the three-dimensional space coordinate value ECPs (w,r) is EGDs (w,r) , if the Gray code value EGDs (w,r) is equal to the Gray code value DGC h of the hth smooth transition region in the smooth transition region sequence SDBGs, then the X-axis coordinate value DOVs (w,r) .x, the Y-axis coordinate value DOVs (w,r) .y, and the Z-axis coordinate value DOVs (w,r) .z of DOVs (w,r) in the optimized value sequence DOVs can be calculated by using the expression 9, the expression 10, the expression 11, and the hth set of surface fitting parameter values in the surface fitting parameter sequence SFPVs, wherein: 1≤w≤m, 1≤r≤n, 1≤h≤bc. Expression 9: DOVs (w,r) .x = ECPs (w,r) .x Expression 10: DOVs (w,r) .y = ECPs (w,r) .y Expression 11: DOVs (w,r) .z = V a h x (DOVs (w,r) .x) 2 + V b h x (DOVs (w,r) .y) 2 + V c h x DOVs (w,r) .x x DOVs (w,r) .y + V d h x DOVs (w,r) .x + V e h x DOVs (w,r) .y + V f h Step S18, the fitting optimization calculation of each three-dimensional space coordinate value in the trusted sequence ECPs through step S17, that is, the optimized value sequence DOVs of the depth information repair is obtained, DOVs = {{DOVs (1,1) .x, DOVs (1,1) .y, DOVs (1,1) .z}, {DOVs (1,2) .x, DOVs (1,2) .y, DOVs (1,2) .z}, …, {DOVs (m,n-1) .x, DOVs (m,n-1) .y, DOVs (m,n-1) .z}, {DOVs (m,n) .x, DOVs (m,n) .y, DOVs (m,n) .z}}.

2. The method of claim 1, wherein, In step S6, the space coordinate mapping process built-in RGB-D depth camera includes the MapDepthFrameToColorSpace function of Kinect v2 or the deproject function of RealSense R200.

Images