Repairing image depth value of object using light absorbing surface

By using three-dimensional model and neural network technology, the problem of inaccurate detection of depth information on absorbed materials in time-of-flight depth sensors is solved, and the depth image is repaired, and the complete color depth image is generated, which improves the reconstruction accuracy of the three-dimensional environment. .

CN119013689APending Publication Date: 2024-11-22MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202380027164.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-04
Filing Date
2023-02-21
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When the time-of-flight depth sensor detects objects containing absorbing emitted light materials, it cannot accurately determine the depth information, resulting in dark holes in the depth image, affecting the accurate reconstruction of the three-dimensional environment.

Method used

Use the object's accessible three-dimensional model to detect and track the three-dimensional pose of the object in the color image, predict the alignment of two-dimensional points and the three-dimensional model through neural networks, apply prediction algorithms to calculate the three-dimensional pose of the object in the depth image, and repair pixels in the depth image value.

Benefits of technology

Generating color depth images without dark holes improves the reconstruction accuracy and integrity of the three-dimensional environment and accurately represents the head and eyes of the head and eyes of the head and eyes of the head.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013689A_ABST
    Figure CN119013689A_ABST
Patent Text Reader

Abstract

The technology described herein discloses a system configured to detect and track a three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values associated with a region (e.g., a surface) of the object that is composed of a material that absorbs light emitted by a time-of-flight depth sensor to determine the depth. Thus, a color depth image (e.g., a red-green-blue depth image or an RGB-D image) may be generated that does not include dark holes on and around a region of an object composed of a material that absorbs light emitted by a time-of-flight depth sensor.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Applications can use depth images to display or reconstruct a three-dimensional environment. Some image capture devices use infrared (IR) technology or other light-based technologies to determine the depth in a scene and create a depth image (e.g., a depth map). For example, a camera can use a time-of-flight depth sensor (e.g., an array of time-of-flight pixels) to illuminate a scene with light (e.g., an IR pattern) emitted from an artificial light source and detect the reflected light. The phase shift between the emitted light and the reflected light is measured, and the depth information of various pixels in the depth image can be determined based on the phase shift.

[0002] Unfortunately, time-of-flight depth sensors may experience problems with accurately determining depth information for a scene. For example, objects in the scene may include surfaces made of materials that absorb emitted light (e.g., IR patterns), and thus the time-of-flight depth sensor cannot clearly detect or see the reflected light. This lack of detection and / or visibility translates into lost or corrupted depth values ​​in the depth image. In one example, a head-mounted device (e.g., an augmented reality device, a mixed reality device, etc.) includes a transparent visor that includes a material that absorbs emitted light. As a result, the time-of-flight depth sensor cannot accurately determine depth values ​​for pixels associated with the transparent eye visor, and therefore, the resulting depth image includes dark holes on and around the user's eyes.

[0003] It is with respect to these and other considerations that the disclosure herein is presented. Summary of the invention

[0004] The technology disclosed herein enables a system to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values ​​associated with an area (e.g., a surface) of the object composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Thus, a color depth image can be generated that does not include dark holes on and around the area of ​​the object composed of material that absorbs light emitted by the time-of-flight depth sensor.

[0005] The system is configured to obtain image data of a scene captured by an image capture device (e.g., a camera). The image data may include a sequence of frames, the sequence of frames comprising a video (e.g., a video of a user wearing a head-mounted display device). The image capture device includes a color (e.g., red, green, blue, or RGB) sensor and a time-of-flight depth sensor, and therefore, each frame includes a color image in a color coordinate space and a corresponding depth image in a depth coordinate space. In various examples, both coordinate spaces are right-handed coordinate systems (e.g., X, Y, Z), where Z points outward (e.g., toward the camera lens) and Y points upward, but due to differences in cameras / sensors, the coordinate spaces do not have the same origin and the axes are not colinear.

[0006] As described above, if the scene includes an object having an area composed of a material that absorbs the emitted light (e.g., an IR pattern), the time-of-flight depth sensor cannot clearly detect or see the reflected light. This lack of detection and / or visibility translates into missing or corrupted depth values ​​in the depth image. That is, the depth image may include dark holes over and around the areas that absorb the light emitted by the time-of-flight depth sensor.

[0007] To address the depth problem, the system is configured to detect (e.g., identify) an object in a color image that is known to include an area (e.g., a surface) that absorbs light emitted by a time-of-flight depth sensor. Once detected, the system predicts a set of two-dimensional points on the object in the color image, the two-dimensional points corresponding to predefined three-dimensional points in an accessible three-dimensional model of the object. For example, the three-dimensional points on the three-dimensional model of the object (which may alternatively be referred to as key points or landmarks) are manually defined in advance and can be any point on the three-dimensional model. In some instances, they may be points associated with important / unique corners and edges of the object's area that absorbs light emitted by the time-of-flight depth sensor.

[0008] A first neural network or other form of artificial intelligence can be used to detect objects. For example, a deep neural network (DNN) model can be trained using thousands or even millions of color image frames, each of which is individually annotated to indicate the shape, position, and / or orientation of an object known to cause problems with respect to depth value determination. A second neural network or other form of artificial intelligence can be used to predict two-dimensional points. An example three-dimensional model can be easily generated using a computer-aided design (CAD) software program, and therefore, the three-dimensional model can be a three-dimensional CAD "mesh" model.

[0009] Next, the system is configured to apply a prediction algorithm to the color image to calculate the three-dimensional pose of the object in the color space. The prediction algorithm calculates the three-dimensional pose of the object in the color space by placing and / or rotating the three-dimensional model of the object until a two-dimensional point on the color image is aligned with a corresponding three-dimensional point predefined in the three-dimensional model of the object. In various examples, the prediction algorithm uses a six-degree-of-freedom (6DoF) method to predict the alignment. For example, a perspective n-point (PnP) algorithm is configured to estimate the pose of the image capture device, and the estimation can be extended to align a three-dimensional landmark defined via an accessible three-dimensional mesh model with a two-dimensional landmark on the color image.

[0010] Now that the system has predicted the 3D pose of the object in the color image, the system applies a transformation between the color space of the color image and the depth space of the depth image to compute the 3D pose of the object in the depth space of the depth image. This transformation may be necessary in scenarios where the color and depth coordinate spaces do not have the same origin and the axes are not colinear due to camera / sensor differences. The system can then use the 3D pose of the object in the depth space of the depth image to inpaint the depth values ​​of pixels in the depth image associated with areas of the object that absorbed light emitted by the time-of-flight depth sensor.

[0011] In various examples, the color image and the inpainted depth image enable the generation of an RGB depth (RGB-D) image. In addition, once an object has been detected in a first color image frame of a video, the system is configured to track the object in subsequent color image frames of the video. In various examples, different neural networks may be used to first detect an object, and then to track the object. Testing has shown that a first neural network for detection requires approximately nine milliseconds per frame to inpaint the depth values, and a second neural network for tracking (once an object is detected) requires only approximately three milliseconds per frame to inpaint the depth values.

[0012] This summary is provided to introduce a series of concepts in a simplified form, which are further described in the detailed description below. This summary is neither intended to identify key or essential features of the claimed subject matter nor to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technique" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations as described above and as allowed by the context throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items in a plurality of items may use reference numbers with letters or sequences of letters to refer to each individual item. General references to items may use specific reference numbers without using sequences of letters.

[0014] Figure 1 An example environment is shown in which a depth image inpainting system can inpaint a depth image using a corresponding color image and an accessible three-dimensional model of an object that caused corrupted or missing depth values ​​in the depth image.

[0015] Figure 2 Further components and / or modules that may be used in the deep image inpainting system are shown.

[0016] Figure 3 Shows how a bounding box can be used to focus a recognition and tracking module on an area of ​​a color image frame where an object is located and / or may be moving in that area from one color image frame to the next in a sequence of color image frames (e.g., a video).

[0017] Figure 4 A transformation for converting between color coordinate space and depth coordinate space is shown so that the depth image can be inpainted.

[0018] Figure 5 An example environment is shown in which a deep image inpainting system may be accessed by any of a number of different applications via an application programming interface.

[0019] Figure 6 An example process for inpainting a depth image using a corresponding color image and an accessible three-dimensional model of an object that caused corrupted or missing depth values ​​in the depth image is shown.

[0020] Figure 7 Additional details of an example computer architecture are shown for a computer (eg, a server and / or server cluster) capable of executing the program components described herein. DETAILED DESCRIPTION

[0021] The following specific embodiments disclose an image restoration system that is configured to detect and track the three-dimensional pose of an object (e.g., a head-mounted display device) in a color image using an accessible three-dimensional model of the object. The system uses the three-dimensional pose of the object to repair pixel depth values ​​associated with areas of the object (e.g., a surface) composed of material that absorbs light emitted by a time-of-flight depth sensor to determine depth. Thus, a color depth image (e.g., a red, green, and blue depth image, or an RGB-D image) can be generated that does not include dark holes on and around areas of the object composed of material that absorbs light emitted by a time-of-flight depth sensor. Refer to the following Figure 1-Figure 7 Various examples, scenarios, and aspects of the disclosed technology are described.

[0022] Figure 1 An example environment 100 is shown in which a depth image restoration system 102 can restore a depth image using a corresponding color image and an accessible three-dimensional model of an object that causes corrupted or missing depth values ​​in the depth image. The depth image restoration system 102 includes an image capture device 104, or is connected to the image capture device 104 in some manner (e.g., via a network connection).

[0023] The image capture device 104 includes a time-of-flight (ToF) depth sensor 106 (e.g., a ToF depth sensor that emits infrared signals) and a color sensor 108 (e.g., an RGB sensor). The image capture device 104 is configured to capture a sequence of frames (e.g., images or video frames) representing a real-world scene 110, the scene 110 including a physical object that is known to absorb light emitted by the ToF depth sensor 106. In one example, the object is a head-mounted display device 112, where a transparent visor prevents light from being reflected. Therefore, the ToF depth sensor 106 cannot accurately determine the depth value of pixels associated with areas of the object that absorb light emitted by the ToF depth sensor 106.

[0024] To this end, the color sensor 108 is configured to generate a color image in the color space 114, and the ToF depth sensor 106 is configured to generate a depth image in the depth space 116. Figure 1 As shown, the color image 114 of the person wearing the head mounted display device 112 does not have any visible problems. However, the depth image 116 of the person wearing the head mounted display device 112 includes dark holes around the user's eyes where the transparent visor is located. Ultimately, this results in reproducing an incomplete color depth (e.g., RGB-D) image 118 that has obvious problems with representing and / or reconstructing the head mounted display device and / or the user's head and eyes.

[0025] To address this issue, the image capture device 104, or an application using the images 114, 116 captured by the image capture device 104, is configured to provide captured image data 120 to the repair module 122. As described above, the captured image data 120 may include a sequence of frames that include a video (e.g., a video of a person wearing the head mounted display device 112). In one example, the image capture device 104 is stationary. However, in another example, the image capture device 104 is movable so that image frames can be captured from multiple different viewpoints within the physical environment.

[0026] The inpainting module 122 is configured to inpaint the depth image 126 using accessible three-dimensional model data 124 (e.g., a CAD mesh model) associated with an object (e.g., a head mounted display device) such that the depth image 126 no longer includes the dark holes displayed in the originally captured depth image 116. Thus, a complete color depth image (e.g., RGB-D) image 128 can be reproduced that is no longer problematic with respect to representing and / or reconstructing the head mounted display device and / or the user's head and eyes.

[0027] Figure 2 Further components and / or modules that may be used in the depth image inpainting system 102 are shown. As shown, the depth image inpainting system 102 obtains a color image in a color space 114 of the image capture device 104 and a depth image in a depth space 116 of the image capture device 104.

[0028] The recognition and tracking module 202 is configured to detect (e.g., recognize) an object 112 in the color image 114 that is known to include an area (e.g., a surface) that absorbs light emitted by the time-of-flight depth sensor 106. In one example, the recognition and tracking module 202 is configured with a neural network 204 or another form of artificial intelligence that can detect any of a plurality of objects known to cause the aforementioned problems in the depth image. For example, a deep neural network (DNN) model may be trained using thousands or even millions of color image frames, each of which is individually annotated to indicate the shape, position, and / or orientation of an object known to cause problems with respect to depth value determination.

[0029] Once detected, the recognition and tracking module 202 is configured to predict a two-dimensional point 206 on the object in the color image. The prediction may be accomplished via another neural network 207. The two-dimensional point 206 corresponds to a three-dimensional point predefined in an accessible three-dimensional model 124 of the object. For example, the three-dimensional points in the three-dimensional model 124 of the object are manually defined in advance and may be any point on the three-dimensional model 124 of the object. In some instances, they may be points associated with important / unique corners and edges of the object 206 that absorb light emitted by the time-of-flight depth sensor. An example three-dimensional model may be easily generated using a computer-aided design (CAD) software program, and therefore, the three-dimensional model data 124 defining the three-dimensional points may be a three-dimensional CAD "mesh" model.

[0030] The color image with the two-dimensional points 206 is then passed to an alignment module 208, which is configured to apply a prediction algorithm 210 to the color image to calculate the three-dimensional pose of the object in the color space of the color image 212. The prediction algorithm 210 calculates the three-dimensional pose of the object in the color space of the color image 212 by placing and / or rotating the three-dimensional model 124 of the object until the two-dimensional points on the object in the color image are aligned with the corresponding three-dimensional points predefined in the three-dimensional model 124 of the object.

[0031] In various examples, the prediction algorithm 210 uses a six degrees of freedom (6DoF) approach to predict alignment. For example, a perspective n-point (PnP) algorithm is configured to estimate the pose of the image capture device 104 relative to the captured scene, and the estimation can be extended to align three-dimensional landmarks defined via an accessible three-dimensional mesh model with two-dimensional landmarks on objects in the color image.

[0032] Now that the alignment module 208 has calculated the predicted 3D pose of the object in the color image 212, the transformation module 214 applies a transformation 216 between the color space of the color image and the depth space of the depth image to calculate the 3D pose of the object in the depth space of the depth image 218. This transformation 216 may be necessary in scenarios where the color and depth coordinate spaces do not have the same origin and the axes are not co-linear due to camera / sensor differences.

[0033] Next, the depth determination module 220 may use the three-dimensional pose of the object in the depth space of the depth image 218 to inpaint depth values ​​of pixels in the depth image 118 associated with areas of the object that absorb light emitted by the ToF depth sensor 106. For example, the depth determination module 220 may apply a rasterization algorithm 222 configured to determine distances between points (e.g., pixels) on the image capture device 104 (e.g., the ToF depth sensor 106) and the three-dimensional pose of the object in the depth space of the depth image 218. In one example, the rasterization algorithm 222 projects the vertices of the constituent triangles onto a depth plane and uses a technique to fill in the pixels covered by the triangles with new depth values ​​224.

[0034] The depth determination module 220 may determine whether a new depth value 224 for a pixel associated with a three-dimensional pose of an object in the depth space of the depth image should replace a previous depth value initially captured and calculated for the depth image 118. The depth determination module 220 may be configured to replace a previous depth value for the pixel with the new depth value 224 if the previous depth value is corrupted (e.g., completely lost or greater than the new depth value 224).

[0035] Thus, the original depth image is inpainted to have more accurate depth values. This allows for the generation of an improved RGB depth (RGB-D) image. In addition, once an object has been detected in a first color image frame of a video, the recognition and tracking module 202 is configured to track the object in subsequent color image frames of the video. In various examples, different neural networks may be used to first detect an object, and then track the object. Testing has shown that a first neural network for detection requires approximately nine milliseconds per frame to inpaint the depth values, and a second neural network for tracking (once an object is detected) requires approximately three milliseconds per frame to inpaint the depth values.

[0036] Figure 3 302 is shown to focus the recognition and tracking module 202 on an area of ​​a color image frame where an object is located and / or is likely moving from one color image frame to the next color image frame in a sequence of color image frames (e.g., a video). This enables the neural network 204 to operate more efficiently with respect to tracking the movement of the object within the color image frame, since the entire image frame does not need to be analyzed for tracking purposes.

[0037] Figure 4 A transformation function 402 is shown for converting between a color coordinate space 404 and a depth coordinate space 406 so that a depth image can be inpainted. In one example, the transformation function 402 includes the following 4x4 matrix multiplication:

[0038] pose_in_color_space=pose_indepth_space*depth_to_clor_transform

[0039] Here, depth_to_clor_transform is a 4x4 rigid transform that may be derived or retrieved based on calibration information (eg, a calibration function) associated with the image capture device 104 (eg, the calibration information may be built into the image capture device 104 at the factory).

[0040] Figure 5 An example environment 500 is shown in which the depth image inpainting system 102 may be accessed by any of a number of different applications 502 via an application programming interface 504. For example, the application 502 may be configured to obtain color and depth image frames 506 of a real-world scene 508 from an image capture device 104. As described above, these image frames 506 may include corrupted depth data 510 (e.g., missing or incorrect depth values ​​for pixels) due to objects in the scene that absorb light emitted by the ToF depth sensor 106.

[0041] Therefore, the application 502 calls the deep image restoration system 102 and submits an image frame 506 with corrupted depth data 510 via an application programming interface 504. The deep image restoration system 102 may store or access a number of neural networks 512 and three-dimensional models 514 of objects known to absorb light emitted by the ToF depth sensor 106. The deep image restoration system 102 is configured to restore the depth data, as described above with respect to Figure 1-Figure 4 The invention further provides a method for reconstructing a user 518 wearing a head mounted display device in a different real world scene 520 than the scene 508 in which the user is actually located, and returns the repaired depth data 516 (e.g., a complete RGB-D image) to the application 502. In one example, the application 502 is a teleportation application that teleports and reconstructs a user 518 wearing a head mounted display device in a different real world scene 520 than the scene 508 in which the user is actually located.

[0042] Now go to Figure 6 , describing process 600, which facilitates repairing a depth image using a corresponding color image and an accessible three-dimensional model of an object that causes corrupted or missing depth values ​​in the depth image. It should be understood that the operations of the methods disclosed herein are not presented in any particular order, and it is possible and contemplated to perform some or all of the operations in an alternate order. For ease of description and illustration, the operations have been presented in the order in which they are presented. Operations may be added, omitted, and / or performed simultaneously without departing from the scope of the appended claims.

[0043] It should also be understood that the term "computer-readable instructions" and variations thereof as used in the specification and claims are used broadly herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0044] At operation 602, image data including a color image and a depth image is obtained. As described above, the color image and the depth image are captured by an image capture device configured with a color sensor and a time-of-flight depth sensor.

[0045] At operation 604 , an object is detected in the color image that is known to include a region that absorbs light emitted by the time-of-flight depth sensor.

[0046] Next, at operation 606, a three-dimensional model of the object is accessed.

[0047] Moving to operation 608, two-dimensional points on the color image corresponding to predefined three-dimensional points in the three-dimensional object model are predicted.

[0048] At operation 610, a prediction algorithm is applied to calculate a three-dimensional pose of the object in the color space of the color image. In various examples, the prediction algorithm calculates the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until a two-dimensional point on the color image is aligned with a corresponding three-dimensional point predefined in the three-dimensional model of the object.

[0049] At operation 612 , a transformation between the color space and the depth space is applied to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image.

[0050] At operation 614 , depth values ​​of pixels in the depth image associated with regions of the object that absorb light emitted by the time-of-flight depth sensor are inpainted using the three-dimensional pose of the object in the depth space of the depth image.

[0051] Figure 7 700 illustrates additional details of an example computer architecture 700 of a computer (eg, a server and / or server cluster) capable of executing the program components described herein. Figure 7 The computer architecture 700 shown in shows the architecture of a server computer, a mobile phone, a PDA, a smart phone, a desktop computer, a netbook computer, a tablet computer, and / or a laptop computer. The computer architecture 700 can be used to execute any aspect of the software components presented herein.

[0052] Figure 7 The illustrated computer architecture 700 includes a central processing unit 702 ("CPU"), a system memory 704 (including a random access memory 706 ("RAM") and a read-only memory ("ROM") 708), and a system bus 710 that couples the memory 704 to the CPU 702. The ROM 708 stores a basic input / output system containing basic routines that help transfer information between elements within the computer architecture 700, such as during startup. The computer architecture 700 also includes a mass storage device 712 for storing an operating system 707, other data, and one or more applications. The mass storage device 712 may also store computer executable instructions for implementing the image depth restoration system 102.

[0053] The mass storage device 712 is connected to the CPU 702 through a mass storage controller connected to the bus 710. The mass storage device 712 and its associated computer readable media provide non-volatile storage for the computer architecture 700. Although the description of computer readable media contained herein refers to mass storage devices, such as solid state drives, hard disks, or CD-ROM drives, those skilled in the art will appreciate that computer readable media can be any available computer storage media or communication media that can be accessed by the computer architecture 700.

[0054] Communication media includes computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any delivery medium. The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in a certain way so that information is encoded in the signal. By way of example and not limitation, communication media include wired media, such as a wired network or a direct wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Combinations of any of the above items should also be included within the scope of computer readable media.

[0055] By way of example and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computer architecture 700. For purposes of the claims, the phrases "computer storage medium", "computer readable storage medium" and variations thereof do not by themselves include waves, signals and / or other transient and / or intangible communication media.

[0056] According to various configurations, the computer architecture 700 can operate in a networked environment using logical connections to remote computers through the network 756 and / or another network. The computer architecture 700 can be connected to the network 756 via a network interface unit 714 connected to the bus 710. It should be understood that the network interface unit 714 can also be used to connect to other types of networks and remote computer systems. The computer architecture 700 can also include an input / output controller 716 for receiving and processing input from a plurality of other devices (including a keyboard, a mouse, or an electronic stylus). Similarly, the input / output controller 716 can provide output to a display screen, a printer, or other types of output devices.

[0057] It should be understood that the software components described herein, when loaded into the CPU 702 and executed, can transform the CPU 702 and the entire computer architecture 700 from a general-purpose computing system into a special-purpose computing system customized to facilitate the functions presented herein. The CPU 702 can be composed of any number of transistors or other discrete circuit elements, which can present any number of states individually or together. More specifically, the CPU 702 can operate as a finite state machine in response to the executable instructions contained in the software modules disclosed herein. These computer executable instructions can transform the CPU 702 by specifying how the CPU 702 transforms between states, thereby transforming the transistors or other discrete hardware elements that constitute the CPU 702.

[0058] Encoding the software modules presented herein can also transform the physical structure of the computer-readable medium presented herein. In different implementations of this specification, the specific transformation of the physical structure may depend on various factors. Examples of these factors may include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as a primary storage device or an auxiliary storage device, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of the transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also transform the physical state of such components to store data thereon.

[0059] As another example, the computer readable medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, the software presented herein can transform the physical state of magnetic or optical media when the software is encoded therein. These transformations can include changing the magnetic properties of a specific location in a given magnetic medium. These transformations can also include changing the physical characteristics or properties of a specific location in a given optical medium to change the optical properties of these locations. Without departing from the scope and spirit of this specification, other transformations of physical media are possible, and the aforementioned examples are provided only to facilitate this discussion.

[0060] In view of the above, it should be appreciated that many types of physical transformations occur in computer architecture 700 in order to store and execute the software components presented herein. It should also be appreciated that computer architecture 700 may include other types of computing devices, including handheld computers, embedded computer systems, personal digital assistants, and other types of computer devices known to those skilled in the art. It is also contemplated that computer architecture 700 may not include Figure 7 All components shown in the Figure 7 Other components not explicitly shown in the drawings or may be used in conjunction with Figure 7 Completely different architecture shown.

[0061] The disclosure presented herein also includes the subject matter set forth in the following clauses.

[0062] Example clause A, a method includes: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object being known to include an area that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; in response to the detected object, predicting a two-dimensional point on the color image corresponding to a three-dimensional point predefined in the three-dimensional object model using a second neural network; applying a prediction algorithm to calculate a three-dimensional pose of the object in a color space of the color image, wherein the prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until the two-dimensional point on the color image is aligned with the corresponding three-dimensional point predefined in the three-dimensional model of the object; applying a transformation between the color space and the depth space to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and using the three-dimensional pose of the object in the depth space of the depth image, repairing the depth values ​​of pixels in the depth image associated with the area of ​​the object that absorbs light emitted by the time-of-flight depth sensor.

[0063] Example clause B, a method according to example clause A, wherein repairing the depth value associated with an area of ​​the object that absorbs light emitted by the time-of-flight depth sensor includes: applying a rasterization algorithm to determine a new depth value for a pixel associated with a three-dimensional pose of the object in the depth space of a depth image; and replacing the previous depth value with the new depth value if the previous depth value of the pixel is missing or greater than the new depth value.

[0064] Example clause C, a method according to example clause A or example clause B, wherein the color image and the depth image are configured to generate an RGB-D image.

[0065] Example clause D, a method according to any one of example clauses A to C, wherein the prediction algorithm includes a perspective n-point algorithm.

[0066] Example clause E, according to any one of example clauses A to D, further comprising: using the first neural network to configure a bounding box to track the movement of the object in the scene.

[0067] Example clause F, a method according to any of example clauses A to E, wherein the transformation between the color space and the depth space comprises a four-by-four matrix multiplication rigid transformation.

[0068] Example clause G. The method of example clause F, wherein the four-by-four matrix multiplication rigid transform is defined via calibration functions defined for the color sensor and the time-of-flight depth sensor.

[0069] Example clause H, a method according to any one of example clauses A to G, wherein: the image frame is obtained from an application via an application programming interface as part of a sequence of image frames; the depth values ​​associated with the areas of the object that absorb infrared signals emitted by the time-of-flight depth sensor are repaired so that a corrected RGB-D image can be generated; the method also includes: providing the corrected RGB-D image to the application.

[0070] Example clause I, a system includes: one or more processing units; and a computer storage medium storing instructions, which, when executed by the one or more processing units, cause the system to perform operations including: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object being known to include an area that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; in response to the detected object, predicting a two-dimensional point on the color image corresponding to a predefined three-dimensional point in the three-dimensional object model using a second neural network; and applying a prediction algorithm to calculate a prediction of the object. A method of calculating the three-dimensional pose of an object in a color space of a color image, wherein a prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until a two-dimensional point on the color image is aligned with a corresponding three-dimensional point predefined in the three-dimensional model of the object; a transformation between the color space and the depth space is applied to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and using the three-dimensional pose of the object in the depth space of the depth image, depth values ​​of pixels in the depth image associated with areas of the object that absorb light emitted by a time-of-flight depth sensor are repaired.

[0071] Example clause J, a system according to example clause I, wherein repairing a depth value associated with an area of ​​an object that absorbs light emitted by a time-of-flight depth sensor includes: applying a rasterization algorithm to determine a new depth value for a pixel associated with a three-dimensional pose of the object in a depth space of a depth image; and replacing the previous depth value with the new depth value if the previous depth value of the pixel is missing or greater than the new depth value.

[0072] Example clause K, a system according to example clause I or example clause J, wherein the color image and the depth image are configured to generate an RGB-D image.

[0073] Example clause L, a system according to any of example clauses I to K, wherein the prediction algorithm includes a perspective n-point algorithm.

[0074] Example clause M, a system according to any of example clauses I to L, wherein the operations further include: configuring a bounding box using a first neural network to track the movement of an object in a scene.

[0075] Example clause N, a system according to any of example clauses I to M, wherein the transformation between the color space and the depth space comprises a four-by-four matrix multiplication rigid transformation.

[0076] Example clause O, a system according to example clause N, wherein the four by four matrix multiplication rigid transformation is defined via calibration functions defined for the color sensor and the time-of-flight depth sensor.

[0077] Example clause P, a system according to any of example clauses I to O, wherein: the image frame is obtained from an application via an application programming interface as part of a sequence of image frames; depth values ​​associated with areas of the object that absorb infrared signals emitted by a time-of-flight depth sensor are repaired so that a corrected RGB-D image can be generated; and the operation also includes: providing the corrected RGB-D image to the application.

[0078] Example clause Q, a computer storage medium storing instructions that, when executed by one or more processing units, cause a system to perform operations including: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object being known to include an area that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; in response to the detected object, predicting two-dimensional points on the color image corresponding to three-dimensional points predefined in the three-dimensional object model using a second neural network; applying a prediction algorithm to calculate a three-dimensional pose of the object in a color space of the color image, wherein the prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until the two-dimensional points on the color image are aligned with corresponding three-dimensional points predefined in the three-dimensional model of the object; applying a transformation between color space and depth space to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and using the three-dimensional pose of the object in the depth space of the depth image, repairing depth values ​​of pixels in the depth image associated with the area of ​​the object that absorbs light emitted by the time-of-flight depth sensor.

[0079] Example clause R, a computer storage medium according to example clause Q, wherein repairing a depth value associated with an area of ​​an object that absorbs light emitted by a time-of-flight depth sensor comprises: applying a rasterization algorithm to determine a new depth value for a pixel associated with a three-dimensional pose of the object in a depth space of a depth image; and replacing a previous depth value of the pixel with the new depth value if the previous depth value is missing or greater than the new depth value.

[0080] Example clause S, a computer storage medium according to example clause Q or example clause R, wherein the prediction algorithm comprises a perspective n-point algorithm.

[0081] Example clause T, a computer storage medium according to any one of example clauses Q to S, wherein the operations further comprise: configuring a bounding box using the first neural network to track movement of the object in the scene.

[0082] Although specific example embodiments have been described, these embodiments have been presented by way of example only and are not intended to limit the scope of the invention disclosed herein. Therefore, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or indispensable. In fact, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention disclosed herein. The attached claims and their equivalents are intended to cover forms or modifications that fall within the scope and spirit of the specific invention disclosed herein.

[0083] It should be understood that any reference to "first", "second", etc. elements in the summary and / or detailed description are not intended to and should not be interpreted as necessarily corresponding to the "first" and "second", etc. elements in the claims. Rather, any use of "first" and "second" in the summary, detailed description, and / or claims can be used to distinguish two different instances of the same element (e.g., two different images).

[0084] Finally, although various configurations have been described using language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A method comprising: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object known to include a region that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; In response to the detected object, predicting, using a second neural network, two-dimensional points on the color image corresponding to three-dimensional points predefined in the three-dimensional object model; applying a prediction algorithm to calculate a three-dimensional pose of the object in a color space of the color image, wherein the prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until the two-dimensional points on the color image are aligned with corresponding three-dimensional points predefined in the three-dimensional model of the object; applying the transformation between the color space and the depth space to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and Using the three-dimensional pose of the object in the depth space of the depth image, depth values ​​for pixels in the depth image associated with the region of the object that absorbs light emitted by the time-of-flight depth sensor are inpainted.

2. The method according to claim 1, wherein: Repairing the depth value associated with the area of ​​the object that absorbs light emitted by the time-of-flight depth sensor includes: applying a rasterization algorithm to determine new depth values ​​for pixels associated with the three-dimensional pose of the object in the depth space of the depth image; and If a previous depth value for the pixel is missing or greater than the new depth value, the previous depth value is replaced with the new depth value.

3. The method according to claim 1, wherein: The color image and the depth image are configured to generate an RGB-D image.

4. The method according to claim 1, wherein: The prediction algorithm includes a perspective n-point algorithm.

5. The method according to claim 1, further comprising: A bounding box is configured using the first neural network to track movement of the object in the scene.

6. The method according to claim 1, wherein: The transformation between the color space and the depth space comprises a four by four matrix multiplication rigid transformation.

7. The method according to claim 6, wherein: The four-by-four matrix multiplication rigid transformation is defined via calibration functions defined for the color sensor and the time-of-flight depth sensor.

8. The method according to claim 1, wherein: The image frame is obtained from the application via an application programming interface as part of a sequence of image frames; inpainting depth values ​​associated with the regions of the object that absorb infrared signals emitted by the time-of-flight depth sensor, such that a corrected RGB-D image can be produced; The method further includes providing the corrected RGB-D image to the application.

9. A system comprising: one or more processing units; as well as A computer storage medium storing instructions that, when executed by the one or more processing units, cause the system to perform operations including: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object known to include a region that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; In response to the detected object, predicting, using a second neural network, two-dimensional points on the color image corresponding to three-dimensional points predefined in the three-dimensional object model; applying a prediction algorithm to calculate a three-dimensional pose of the object in a color space of the color image, wherein the prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until the two-dimensional points on the color image are aligned with corresponding three-dimensional points predefined in the three-dimensional model of the object; applying the transformation between the color space and the depth space to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and Using the three-dimensional pose of the object in the depth space of the depth image, depth values ​​for pixels in the depth image associated with the region of the object that absorbs light emitted by the time-of-flight depth sensor are inpainted.

10. The system according to claim 9, wherein: Repairing the depth value associated with the area of ​​the object that absorbs light emitted by the time-of-flight depth sensor includes: applying a rasterization algorithm to determine new depth values ​​for pixels associated with the three-dimensional pose of the object in the depth space of the depth image; and If a previous depth value for the pixel is missing or greater than the new depth value, the previous depth value is replaced with the new depth value.

11. The system according to claim 9, wherein: The color image and the depth image are configured to generate an RGB-D image.

12. The system according to claim 9, wherein: The prediction algorithm includes a perspective n-point algorithm.

13. The system according to claim 9, wherein: The operations also include configuring, using the first neural network, a bounding box to track movement of the object in the scene.

14. The system according to claim 9, wherein: The transform between the color space and the depth space comprises a four by four matrix multiplication rigid transform defined via a calibration function defined for the color sensor and the time-of-flight depth sensor.

15. A computer storage medium storing instructions that, when executed by one or more processing units, cause a system to perform operations comprising: obtaining image data including a color image and a depth image captured by an image capture device configured with a color sensor and a time-of-flight depth sensor; detecting an object in the color image using a first neural network, the object known to include a region that absorbs light emitted by the time-of-flight depth sensor; accessing a three-dimensional model of the object; In response to the detected object, predicting, using a second neural network, two-dimensional points on the color image corresponding to three-dimensional points predefined in the three-dimensional object model; applying a prediction algorithm to calculate a three-dimensional pose of the object in a color space of the color image, wherein the prediction algorithm is applied to calculate the three-dimensional pose of the object in the color space of the color image by at least one of positioning or rotating the three-dimensional model of the object until the two-dimensional points on the color image are aligned with corresponding three-dimensional points predefined in the three-dimensional model of the object; applying the transformation between the color space and the depth space to the three-dimensional pose of the object in the color space of the color image to calculate the three-dimensional pose of the object in the depth space of the depth image; and Using the three-dimensional pose of the object in the depth space of the depth image, depth values ​​for pixels in the depth image associated with the region of the object that absorbs light emitted by the time-of-flight depth sensor are inpainted.

Citation Information

Patent Citations

  • Depth from time of flight camera

    CN106796656A

  • Pose estimation and model retrieval for objects in images

    CN111328396A

  • Mapping object instances using video data

    CN112602116A

  • Indoor high-precision real-time modeling and spatial positioning device and method based on multi-TOF laser radar and RGB camera

    CN113888702A

  • 3-dimensional scene analysis for augmented reality operations

    US20170243352A1