A method and system for augmenting depth data from a depth sensor by using data from a multi-view camera system.

Abstract

Methods and related systems for determining depth of a scene are disclosed herein. In some embodiments, the method can include augmenting depth data of a scene captured using a depth sensor with depth data from one or more images of the scene. For example, the method can include capturing images of the scene using multiple cameras. The method can further include generating a point cloud representing the scene based on the depth data from the depth sensor and identifying missing regions in the point cloud, such as regions occluded from the perspective of the depth sensor. The method can then include generating depth data for the missing regions based on the image data. Finally, the depth data for the missing regions can be fused with the depth data from the depth sensor to generate a fused point cloud representing the scene.

Description

Technical Field

[0005] , ,

[0001]

Cross - Reference to Related Applications

[0002] The technology of the present invention generally relates to imaging systems and, more specifically, to an imaging system for generating a virtual view of a scene for an indirect reality observer.

Background Art

[0003] In an indirect reality system, an image - processing system adds, removes, and / or modifies visual information representing the environment. For surgical applications, the indirect reality system can enable a surgeon to view the surgical site from a desired perspective, along with situational information that assists the surgeon in performing the surgical task more efficiently and accurately. Such an indirect reality system relies on image data from multiple camera angles and depth information about the environment to reconstruct an image of the environment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] Depth information about the environment is typically obtained through dedicated depth sensors, such as structured optical depth sensors. However, acquiring complex 3D shapes using structured optical depth sensors requires that all surfaces of the environment be exposed to the depth sensor in order to obtain complete, error-free data. In practice, this is not feasible due to shapes and / or occluding surfaces hidden from the depth sensor. [Brief explanation of the drawing]

[0006] Many aspects of the disclosure of this invention can be better understood by referring to the following drawings. The components in the drawings are not necessarily to scale. Instead, the emphasis is on clearly illustrating the principles of the disclosure of this invention.

[0007] [Figure 1] This is a schematic diagram of an imaging system configured according to an embodiment of the present invention. [Figure 2] Figure 1 is a perspective view of a surgical environment in which the imaging system shown in Figure 1 is used for surgical applications according to an embodiment of the technology of the present invention. [Figure 3] This is an isometric projection view of a portion of an imaging system comprising a depth sensor and a plurality of cameras, configured according to an embodiment of the technology of the present invention. [Figure 4] This is a schematic diagram of a point cloud generated by an imaging system according to an embodiment of the present invention. [Figure 5] This is a flowchart illustrating a process or method for augmenting depth data acquired using a depth sensor using image data acquired by a camera, according to an embodiment of the technology of the present invention. [Figure 6] This is a schematic diagram of a fused point cloud in which depth data has been filled in the missing regions of the point cloud in Figure 4 according to an embodiment of the technology of the present invention. [Figure 7] This is an isometric projection view of a portion of an imaging system configured according to an embodiment of the present invention. [Figure 8]This is a flowchart illustrating a process or method for augmenting depth data acquired using a depth sensor with depth data from a patient's medical scan, according to an embodiment of the present invention. [Figure 9A] This is a schematic diagram of a point cloud generated by an imaging system according to an embodiment of the present invention. [Figure 9B] This is a schematic diagram of CT scan data corresponding to the point cloud in Figure 9A, according to an embodiment of the technology of the present invention. [Figure 9C] This is a schematic diagram of a fused point cloud obtained by fusing the CT scanning data shown in Figure 9B with the point cloud shown in Figure 9A, according to an embodiment of the technology of the present invention. [Modes for carrying out the invention]

[0008] Aspects of the disclosure of the present invention generally relate to a method for determining the depth of a scene, such as a surgical scene, and for reconstructing a virtual camera view of the scene using depth information. In some embodiments described below, for example, the method includes the step of augmenting depth data of a scene captured using a depth sensor with depth data from one or more images of the scene. For example, the method may include the step of (i) capturing depth data of a scene from a depth sensor and (ii) capturing images of the scene from multiple cameras. The method may further include the step of generating a point cloud representing the scene based on the depth data from the depth sensor and the step of identifying missing areas in the point cloud, such as areas hidden from the viewpoint of the depth sensor. The method may then include the step of generating depth data for the missing areas based on images from cameras. The image may be a light field image containing information about the intensity of light rays emanating from the scene and information about the direction in which the light rays are traveling through space. The method may further include the step of (i) fusing the depth data for the missing areas derived from the image with (ii) the depth data from the depth sensor to generate a fused point cloud representing the scene.

[0009] In one aspect of the present invention, a fused point cloud can have higher accuracy and / or resolution than a point cloud generated solely from depth data from a depth sensor. In another aspect of the present invention, depth information is rapidly determined for as much of the scene as possible using a depth sensor, and light field processing is used only for relatively small areas of the scene (e.g., missing areas) where depth information cannot be determined or accurately determined using a depth sensor. Thus, the present invention can provide real-time or near real-time depth and image processing, while also providing improved accuracy. That is, the combined depth determination method of the present invention can provide (i) improved latency compared to light field processing alone, and (ii) improved accuracy compared to depth sensor processing alone.

[0010] Specific details of several embodiments of the technology of the present invention are described herein with reference to Figures 1-9C. However, the technology of the present invention can be carried out without some of these specific details. In some cases, known structures and techniques, often associated with camera arrays, light field cameras, image reconstruction, and depth sensors, are not shown in detail so as not to obscure the technology of the present invention. The terms used in the following descriptions are intended to be interpreted in their broadest and most reasonable form, even if they are used in connection with a detailed description of certain particular embodiments of the disclosure of the present invention. Certain terms may even be emphasized below, but any term intended to be interpreted in any limited form will be explicitly and specifically defined in this “Modes for Carrying Out the Invention” section.

[0011] The accompanying figures illustrate embodiments of the technology of the present invention and are not intended to limit its scope. The sizes of the various elements depicted are not necessarily to scale, and these various elements can be enlarged as needed to improve readability. Details of components, such as the location of components and certain precise connections between such components, may be omitted in the figures when such details are unnecessary for a complete understanding of how the technology of the present invention is manufactured and used. Many of the details, dimensions, angles, and other features shown in the figures are merely illustrative of particular embodiments of the disclosure of the present invention. Accordingly, other embodiments may have other details, dimensions, angles, and features without departing from the spirit or scope of the technology of the present invention.

[0012] The headings provided herein are for convenience only and should not be construed as limiting the subject matter disclosed.

[0013] I. Selected Embodiments of the Imaging System Figure 1 is a schematic diagram of an imaging system 100 ("System 100") configured according to an embodiment of the technology of the present invention. In the illustrated embodiment, System 100 includes one or more display devices 104, one or more input controllers 106, and an image processing device 102 operably / communicatively coupled to a camera array 110. In other embodiments, System 100 may comprise additional, fewer, or different components. In some embodiments, System 100 may include features substantially similar to or identical to those of an indirect reality imaging system disclosed in U.S. Patent Application No. 16 / 586, 375, entitled "Camera Array for Indirect Reality Systems," which is incorporated herein by reference in its entirety.

[0014] In the illustrated embodiment, the camera array 110 includes a plurality of cameras 112 (individually identified as cameras 112a-112n) each configured to capture an image of the scene 108 from a different perspective. In some embodiments, the cameras 112 are positioned in fixed locations and orientations relative to each other. For example, the cameras 112 can be structurally fixed by a mounting structure (e.g., a frame) at predetermined fixed locations and orientations. In some embodiments, the cameras 112 can be positioned such that adjacent cameras share overlapping perspectives of the scene 108. In some embodiments, the cameras 112 within the camera array 110 are synchronized to capture images of the scene 108 substantially simultaneously (e.g., within a threshold time error). In some embodiments, all or a subset of the cameras 112 can be light field / prenoptic / RGB cameras configured to capture information regarding the light field emitted from the scene 108 (e.g., information regarding the intensity of light rays within the scene 108 and also information regarding the direction in which the light rays are traveling through space).

[0015] In the illustrated embodiment, the camera array 110 further includes (i) one or more projectors 114 configured to project a structured light pattern onto / into the scene 108 and (ii) one or more depth sensors 116 configured to estimate the depth of surfaces within the scene 108. In some embodiments, the depth sensors 116 can estimate depth based on the structured light pattern emitted from the projectors 114.

[0016] The image processing device 102 is configured to (i) receive an image (e.g., a light field image, light field image data) captured by the camera array 110 and depth information from the depth sensor 116, and (ii) process the image and the depth information to synthesize an output image corresponding to a virtual camera view. In the illustrated embodiment, the output image corresponds to an approximation of an image of the scene 108 that is considered to be captured using a camera placed at any position and orientation corresponding to the virtual camera view. More specifically, the depth information can be combined with the image from the camera 112 to synthesize the output image as a three-dimensional rendering of the scene 108 when viewed from the virtual camera view. In some embodiments, the image processing device 102 can synthesize the output image using any of the methods disclosed in U.S. Patent Application No. 16 / 457,780, entitled "Synthesis of Images from a Virtual View Using Pixels from a Physically Imaged Array Weighted Based on Depth Error Sensitivity," which is hereby incorporated by reference in its entirety.

[0017] The image processing device 102 can synthesize output images from a subset (e.g., two or more) of the cameras 112 in the camera array 110, but does not necessarily utilize images from all of the cameras 112. For example, given a virtual camera field of view, the image processing device 102 can select stereoscopic images from two of the cameras 112 that are positioned and directed to best fit the virtual camera field of view. In some embodiments, the image processing device 102 (and / or depth sensor 116) is configured to estimate the depth for each surface point of the scene 108 and generate a point cloud and / or a three-dimensional (3D) mesh representing the surfaces of the scene 108. For example, in some embodiments, the depth sensor 116 can detect structured light projected onto the scene 108 by the projector 114 and estimate the depth information of the scene 108. Alternatively or in addition to this, the image processing device 102 can perform depth estimation based on depth information received from the depth sensor 116. As will be described in detail below, in some embodiments, the image processing device 102 can estimate depth from multi-view image data from the camera 112, with or without using information collected by the projector 114 or the depth sensor 116.

[0018] In some embodiments, functions attributed to the image processing device 102 can actually be performed by two or more physical devices. For example, in some embodiments, a synchronization controller (not shown) controls the image displayed by the projector 114 and transmits synchronization signals to the camera 112, ensuring synchronization between the camera 112 and the projector 114 to enable high-speed, multi-frame, multi-camera structured optical scanning. In addition, such a synchronization controller can act as a parameter server storing hardware-specific configurations such as structured optical scanning parameters, camera setpoints, and camera calibration data specific to the camera configuration of the camera array 110. The synchronization controller can be implemented in a separate physical device from the display controller that controls the display device 104, or the devices can be integrated with each other.

[0019] The image processing device 102 may comprise a processor and a non-temporary computer-readable storage medium for storing instructions that, when executed by the processor, perform functions attributed to the image processing device 102 as described herein. While not essential, aspects and embodiments of the present invention may be described in the general context of computer-executable instructions, such as routines executed by general-purpose computers, e.g., servers or personal computers. Those skilled in the art will recognize that the present invention can be implemented using other computer system configurations, including internet appliances, portable devices, wearable computers, cellular or mobile phones, multiprocessor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, minicomputers, and mainframe computers. The present invention can be embodied in a dedicated computer or data processor specifically programmed, set up, or configured to execute one or more of the computer-executable instructions described in detail below. In practice, the terms “computer” (and similar terms) as used herein refer to any of the devices described above, as well as any data processor or network-communicating device, including consumer electronic products such as gaming devices, cameras, or other electronic devices having a processor and other components, such as network communication circuits.

[0020] The present invention can also be implemented in a distributed computing environment in which tasks or modules are executed by remote processing devices linked through a communication network such as a local area network ("LAN"), a wide area network ("WAN"), or the Internet. In a distributed computing environment, program modules or subroutines can reside in both local and remote memory storage devices. The embodiments of the present invention described below can be stored or distributed on computer-readable media, including magnetically and optically readable and removable computer disks stored as chips (e.g., EEPROM or flash memory chips). Alternatively, embodiments of the present invention can be electronically distributed over the Internet or other networks (including wireless networks). Those skilled in the art will recognize that each part of the technology of the present invention can reside on a server computer, while the corresponding parts reside on client computers. Specific data structures and data transmissions in embodiments of the technology of the present invention are also included within the scope of the invention.

[0021] The virtual camera field of view can be controlled by an input controller 106 that provides control inputs corresponding to the location and orientation of the virtual camera field of view. The output image corresponding to the virtual camera field of view is output to a display device 104. The display device 104 is configured to receive the output image (e.g., a composite 3D rendering of scene 108) and display the output image for viewing by one or more observers. The image processing device 102 can efficiently process the input received from the input controller 106 and process the captured images from the camera array 110 to generate an output image corresponding to the virtual field of view in substantially real time (e.g., at least as fast as the frame rate of the camera array 110) when perceived by the observer on the display device 104.

[0022] The display device 104 may include, for example, a head-mounted display device, a monitor, a computer display, and / or another display device. In some embodiments, the input controller 106 and the display device 104 are integrated with the head-mounted display device, and the input controller 106 includes motion sensors for detecting the position and orientation of the head-mounted display device. The virtual camera field of view can then be derived to correspond to the position and orientation of the head-mounted display device 104, such that the virtual field of view corresponds to the field of view that is thought to be seen by an observer wearing the head-mounted display device 104. That is, in such embodiments, the head-mounted display device 104 can provide real-time rendering of the scene 108, as it is thought to be seen by an observer without the head-mounted display device 104. Alternatively, the input controller 106 may include a user-controlled control device (e.g., a mouse, a pointing device, a portable controller, a motion recognition controller) that allows the observer to manually control the virtual field of view displayed by the display device 104.

[0023] Figure 2 is a perspective view of a surgical environment in which System 100 is used for surgical applications according to an embodiment of the technology of the present invention. In the illustrated embodiment, the camera array 110 is positioned over a scene 108 (e.g., a surgical site) and supported / positioned via a swing arm 222 operably coupled to a workstation 224. In some embodiments, the swing arm 222 can be manually moved to position the camera array 110, while in other embodiments, the swing arm 222 can be robotically controlled in response to an input controller 106 (Figure 1) and / or another controller. In the illustrated embodiment, the display device 104 is embodied as a head-mounted display device (e.g., a virtual reality headset, an augmented reality headset). The workstation 224 may include a computer for controlling various functions of the image processing device 102, the display device 104, the input controller 106, the camera array 110, and / or other components of System 100 shown in Figure 1. Thus, in some embodiments, the image processing device 102 and the input controller 106 are integrated into the workstation 224, respectively. In some embodiments, the workstation 224 includes a secondary display 226 that can display a user interface for performing various configuration functions, a mirror image of the display on the display device 104, and / or other useful visual images / displays.

[0024] II. Selected Embodiments of Depth Data Augmentation from Depth Sensors Figure 3 is an isometric projection of a portion of a system 100 showing four of the cameras 112 and a depth sensor 116 according to an embodiment of the technology of the present invention. Other components of the system 100 (e.g., other parts of the camera array 110, image processing device 102, etc.) are not shown in Figure 3 for clarity. In the illustrated embodiment, each of the cameras 112 has a field of view 330 and is oriented so that the field of view 330 aligns with a portion of the scene 108. Similarly, the depth sensor 116 may have a field of view 332 aligning with a portion of the scene 108. In some embodiments, portions of some or all of the fields of view 330, 332 may overlap.

[0025] In the illustrated embodiment, a portion of the spine 309 of a patient (e.g., a human patient) is positioned within / in Scene 108. In many cases, the spine 309 (or other surfaces positioned within Scene 108) will have complex 3D shapes that make it difficult to accurately determine its surface depth and therefore difficult to accurately model in point clouds, 3D meshes, and / or other mathematical representations. For example, if a portion of the surface of Scene 108 (e.g., a portion of the spine 309) is obscured from the field of view 332 of the depth sensor 116, the depth sensor 116 will be unable to determine the depth of the obscured area. Similarly, accurately determining the depth along steep surfaces of Scene 108 can be difficult. More specifically, for a structured light system to recover depth at a given location, a structured light projector (e.g., projector 114) needs to illuminate that location with pixels or blocks of pixels of structured illumination. Similarly, an imager measuring a projection (e.g., depth sensor 116) should have pixels / blocks that see its illumination. Both conditions must be satisfied in order to perform a location measurement. In practice, it is generally impossible to achieve 100% satisfaction, where every pixel has a valid depth value, because real-world scenes have complex shapes that cause occlusion of the projector, the imager / sensor, or both. Therefore, if system 100 determines depth using only the depth sensor 116, the depth model generated by system 100 may have missing areas (e.g., holes) corresponding to each part (e.g., each surface) of scene 108 where depth information is unavailable.

[0026] In some embodiments, the system 100 may not be able to adequately or accurately generate an output image of scene 108 for such portions of scene 108 that have insufficient and / or inaccurate depth information. For example, Figure 4 is a schematic diagram of a point cloud 440 generated by the depth sensor 116 of the system 100 for the plane of the spine 309 shown in Figure 3 according to an embodiment of the technology of the present invention. Referring together to Figures 1-4, the point cloud 440 generally comprises a plurality (e.g., hundreds, thousands, millions, or more) of data points corresponding to the plane distance (e.g., depth of the spine plane) of the spine 309 and / or other features in scene 108 relative to the sensor 116. The point cloud 440 can therefore be used by the image processing device 102 to map / represent the 3D plane of the spine 309 and to synthesize the image from the camera 112 into an output image of scene 108 rendered from any desired virtual field of view, as described in detail above. In the illustrated embodiment, the point cloud 440 includes one or more missing regions 442 corresponding to each part (e.g., each surface) of the scene 108 where depth information is insufficient and / or unreliable (such as areas where the depth sensor 116 is obscured). Consequently, the system 100 may not be able to render an accurate output image for those parts of the scene 108.

[0027] In some embodiments, the image processing device 102 can process image data from one or more of the cameras 112 to determine the depth of the vertebral surface at one or more locations where the depth information from the depth sensor 116 is insufficient, unreliable, and / or inaccurate. That is, the image data from the cameras 112 can be used to "fill in" missing regions 442 of the point cloud 440. More specifically, Figure 5 is a flowchart of a process or method 550 for augmenting depth data acquired using a depth sensor with image data acquired by a camera according to an embodiment of the technology of the present invention. Some features of method 550 will be described in relation to the embodiments shown in Figures 1-4 for illustrative purposes, but those skilled in the art will readily understand that method 550 can be carried out using other suitable systems and / or devices described herein.

[0028] In block 551, method 550 includes the steps of acquiring (a) depth data of a scene 108 (e.g., spine 309) from a depth sensor 116 and (b) image data of the scene from one or more of the cameras 112. The depth data may include, for example, data relating to a structured light pattern projected onto / into the scene 108 (e.g., from a projector 114). The image data may be light field data including data relating to the intensity of light rays emanating from the scene 108 and information relating to the direction in which the light rays are traveling. In some embodiments, the depth sensor 116 and the camera 112 may acquire the depth data and image data simultaneously or substantially simultaneously and / or in real time or near real time. In other embodiments, the depth data may be acquired before the image data.

[0029] In block 552, method 550 includes the step of generating a point cloud of scene 108, such as point cloud 440, based on depth data from depth sensor 116. In some embodiments, an image processing device 102 can receive depth data from depth sensor 116 and generate a point cloud based on the depth data. In some embodiments, method 550 may further include the step of generating a 3D mesh instead of or in addition to the point cloud. In other embodiments, in block 552, method 550 may include the step of generating another mathematical representation of the physical shape of scene 108.

[0030] In block 553, method 550 includes the step of projecting the point cloud and / or depth data associated with the point cloud back onto individual cameras 112 and / or image processing devices 102. By projecting the point cloud back onto the cameras 112, an image of scene 108 can be reconstructed. More specifically, the back projection correlates the 2D pixel locations in the image from the cameras 112 with their 3D locations from the point cloud. By projecting the point cloud back onto each of the images from the cameras 112, each pixel in the image can be associated with a 3D point, or not, if the 3D location cannot be determined for the reasons described in detail above. A simpler classifier is to label each pixel in the 2D image as to whether or not it has a valid 3D correspondence. In some embodiments, this classification can be used to generate a binary mask for each of the cameras 112 indicating which pixels have a valid 3D point.

[0031] In block 554, method 550 includes the step of identifying regions of missing data in the point cloud. For example, method 550 may include the step of identifying missing regions 442 in point cloud 440 where depth data is missing or incomplete. In some embodiments, the step of identifying missing data may include filtering the point cloud data to find holes that exceed a predetermined threshold (e.g., a user-specified threshold), for example, using an inverse Euler method. In some embodiments, the step of identifying missing data may include scanning the point cloud to determine regions with sparse or absent points. In some embodiments, a mesh may be generated over the point cloud (e.g., in block 552), and holes may be identified in the mesh using, for example, a method for identifying triangles in the mesh that have at least one edge not shared by another triangle. In yet another embodiment, missing regions 442 may be identified by finding image regions from camera 112 where no valid 3D correspondence exists (e.g., by examining the binary mask of each image). In some embodiments, blocks 553 and 554 may be performed using the same algorithm and / or as part of the same computer processing.

[0032] In some embodiments, block 554, method 550 may, in addition to or instead of, include a step of identifying areas of invalid depth data and unreliable depth data and / or other potentially problematic areas of the point cloud. For example, the depth sensor 116 may be configured to tag the depth data it captures with a validation or confidence level, and method 550 may include a step of identifying areas of the point cloud and / or mesh that have a validation or confidence level below a predetermined threshold (e.g., a user-specified threshold). Such invalid or unreliable areas may be areas of the point cloud or mesh that have discontinuities, sparse depth data, and poorly behaving normal values, etc. In some embodiments, method 550 may not identify a single missing pixel as a missing or invalid area, and / or conversely, it may identify a missing pixel with some "valid" pixels scattered among it as a missing / invalid area.

[0033] In some embodiments, in block 554, method 550 may further include the step of determining depth data for the region surrounding the missing region 442 of the point cloud. This surrounding depth data can help inform / predict the depth of the missing region 442, assuming that there is no large discontinuity between the missing region 442 and the surrounding region, and therefore the missing depth value can be expected to be close to the surrounding depth.

[0034] In block 555, method 550 includes the step of extracting / identifying image data corresponding to missing or invalid areas of a point cloud or mesh. For example, the image processing device 102 can determine which camera 112 has its field of view 330 aligned with the area of ​​scene 108 corresponding to the missing area. In some embodiments, the image processing device 102 can make this determination based on prior information relating to (i) the position and orientation of the cameras (and thus the extent of their field of view 330), (ii) back projection of depth data onto the cameras 112 (block 553), (iii) processing of the point cloud or mesh, and / or (iv) other data. Furthermore, in some embodiments, the system 100 can identify and extract image data only from the cameras 112 that are determined to have sufficient optical coverage of the missing area. In some embodiments, at least a portion of the cameras 112 may have at least partially overlapping fields of view 330, so that at least one of the cameras 112 is very likely to have a field of view 330 aligned with the area of ​​scene 108 corresponding to the missing area, even if the others of the cameras 112 are obscured. Accordingly, in one aspect of the art of the present invention, the system 100 is configured to robustly capture image data relating to the missing area even if substantial occlusion exists in scene 108. In some embodiments, blocks 553-554 can be executed using the same algorithm and / or as part of the same computer processing.

[0035] In block 556, method 550 includes the step of processing the extracted image data to generate depth data for missing or invalid regions. For example, the image processing device 102 can generate depth data for the missing region using parallax from a camera 112 that has a missing region within its field of view 330 (e.g., pointed towards the missing region). In other embodiments, other suitable image processing techniques (e.g., computer algorithms) can be used to determine depth from light field data. In some embodiments, determining depth by processing image data from camera 112 may be more computationally expensive (e.g., slower) than determining depth using depth sensor 116 due to the complexity of computer algorithms that process depth information from light field data. As a result, depth data for missing or invalid regions can be generated using image data from fewer cameras than all of camera 112. In some embodiments, the processing of the extracted image data can be accelerated by using depth information about the region surrounding the missing or invalid region (e.g., captured using block 554). Specifically, many depth processing algorithms are iterated over with respect to depth to obtain true values. Therefore, by limiting the depth range based on the depth of the surrounding region, it becomes necessary to determine smaller ranges of depth, parallax, plane, etc. In other words, the search can be accelerated by avoiding local minimums that may exist outside this expected region / range.

[0036] In block 557, method 550 includes the step of fusing / combining depth data of missing or invalid regions with the original depth data (e.g., acquired using block 551) to generate a fused point cloud. For example, Figure 6 is a schematic diagram of a fused point cloud 640 in which the missing regions 442 of the point cloud 440 shown in Figure 4 are filled with image-based depth data 644 according to an embodiment of the technology of the present invention. Thus, the fused point cloud 640 can provide a more accurate and robust depth map of scene 108 that facilitates better reconstruction and synthesis of output images of scene 108 rendered from any desired virtual field of view, as described in detail above.

[0037] In block 558, method 550 may optionally include a step of generating a 3D mesh based on the fused point cloud. The 3D mesh can be used to reconstruct / combine the output image of scene 108. In some embodiments, method 550 can return to block 551 to update the depth information of scene 108. In some embodiments, method 550 can proceed to project the fused point cloud back onto camera 112 (block 553).

[0038] As described above, determining depth by processing light field image data can be computationally more expensive than determining depth using a depth sensor. In fact, if depth information for the entire scene were to be completely determined by light field image processing, even very fast systems would not be able to measure and process a considerable amount of data fast enough, making it difficult / impossible to render output images in real time or near real time. However, in one aspect of the present invention, depth information for as much of the scene as possible is rapidly determined using a depth sensor, and light field processing is used only for relatively small areas of the scene where insufficient and / or unreliable depth information from the depth sensor exists. Thus, the present invention can provide real-time or near real-time depth and image processing while simultaneously improving accuracy. That is, the combined depth determination method of the present invention can provide (i) improved latency compared to light field processing alone, and (ii) improved accuracy and resolution compared to depth sensor processing alone.

[0039] In some embodiments, the latency of system 100 can be further improved by updating depth information only for missing or invalid regions of the point cloud where accuracy improvement is desired. For example, Figure 7 is an isometric projection of a portion of system 100 showing two of the cameras 112, a display device 104, and the generated point cloud 440 (Figure 4) according to an embodiment of the technology of the present invention. Other components of system 100 (e.g., other parts of the camera array 110, image processing device 102, etc.) are not shown in Figure 7 for clarity.

[0040] In the illustrated embodiment, the display device 104 is a head-mounted display device (e.g., a headset) configured to be worn by a user (e.g., a surgeon) and having a field of view 736 aligned with only a portion of the scene 108 (e.g., a portion of the spine 309 shown in Figure 3). The head-mounted display device 104 may include a display 705 configured to display a rendered output image of the scene 108 to the user. The display 705 may be opaque or partially transparent. In some embodiments, the field of view 736 of the head-mounted display device 104 corresponds to a foveal region representing a relatively narrow field of view that the user's eye can perceive.

[0041] System 100 (e.g., image processing device 102) can track the position and orientation of the field of view 736 relative to the scene 108 and update only the missing regions 442 of the point cloud 440 within the field of view 736 using method 550 (Figure 5), without updating regions outside the field of view 736. In some embodiments, system 100 can identify a camera 112 that has the best optical coverage for that portion of the scene 108 within the field of view 736. As the user changes the position and / or orientation of the head-mounted display device 104 and thus the field of view 736, system 100 can seamlessly update (e.g., fill in) the missing regions 442 within the field of view 736 in real time or near real time. In one aspect of the technology of the present invention, the latency of the image presented to the user through the head-mounted display device 104 is reduced because the missing regions 442 outside the user's foveal region are not updated.

[0042] Referring again to Figure 1, in some embodiments, the camera 112 may have a higher resolution than the depth sensor 116 so that more depth detail about the scene 108 can be extracted from the camera 112 than from the depth sensor 116. Therefore, even if depth information from the depth sensor 116 is present and at least sufficient to determine the general depth of the scene 108, it may be advantageous to include image data from the camera 112 to increase the depth resolution and, accordingly, increase the resolution of the image output to the user through the display device 104. Accordingly, in some embodiments, the system 100 may process image data for specific local areas of the scene 108 and complement or replace the depth data captured by the depth sensor 116 for those local areas. In some embodiments, background processing performed on the image processing device 102 may automatically update local areas of the scene 108, for example, if the depth data from the depth sensor 116 is of low quality in those areas. In other embodiments, the user may select specific areas for which resolution can be improved. In yet another embodiment, the system 100 can improve resolution by processing light field data corresponding to all or part of the user's foveal region 736, as shown in Figure 7.

[0043] In other embodiments, depth data acquired by the depth sensor 116 can be supplemented or replaced with depth information obtained from means other than processing image data from the camera 112. For example, Figure 8 is a flowchart of a process or method 800 for augmenting depth data acquired using the depth sensor 116 with depth data from one or more medical scans of a patient according to an embodiment of the art of the present invention. Some features of method 860 are described in relation to system 100 shown in Figure 1 for illustrative purposes, but those skilled in the art will readily understand that method 860 can be carried out using other suitable systems and / or devices described herein. In addition, method 860 is described in relation to augmenting depth data of a patient's anatomical structures in a medical scan of a patient, but method 860 can be carried out to update / augment depth data based on other scenes and / or other data from other imaging / scanning techniques.

[0044] In block 861, method 860 includes the step of acquiring depth data of scene 108 from depth sensor 116 (e.g., live data), such as data relating to a structured light pattern projected onto / into scene 108. Scene 108 may include, for example, a portion of a patient undergoing surgery. As an example, the portion of the patient may be a portion of the patient's spine exposed during spinal surgery. Block 862 of method 860 can proceed substantially or identically to block 552 of method 550 in Figure 5 to generate, for example, a point cloud representation of the depth of scene 108.

[0045] In block 863, method 860 includes the step of aligning a point cloud with medical scanning data (e.g., patient data). In some embodiments, the medical scanning may be a computed tomography (CT) scan of a patient's spine providing a complete 3D dataset for at least a portion of scene 108. The alignment process matches points in the point cloud to corresponding 3D points in the medical scanning. System 100 can align the point cloud with the medical scanning data by detecting the locations of reference markers and / or feature points that are visible in both datasets. For example, if volumetric data includes CT data, rigid bodies of bone surfaces calculated from the CT data can be aligned to corresponding points / faces in the point cloud. In other embodiments, System 100 may use other alignment processes based on other methods of shape correspondence, and / or alignment processes that do not depend on reference markers (e.g., markerless alignment processes). In some embodiments, the alignment / alignment process may include features substantially similar to or identical to the alignment / alignment process disclosed in U.S. Provisional Patent Application No. 62 / 796,065, filed January 23, 2019, entitled “Method for Aligning Preoperative Images with Real-Time Surgical Images to an Indirect Reality Viewpoint of a Surgical Site,” which is incorporated herein by reference in its entirety as Appendix A.

[0046] In block 864, method 860 includes the step of identifying missing / invalid regions in the point cloud. In some embodiments, block 864 can proceed substantially the same as or identically to block 554 of method 550 in Figure 5. In one aspect of the art of the present invention, medical scanning data includes 3D depth data corresponding to missing or invalid regions in the point cloud. For example, Figure 9A is a schematic diagram of a point cloud 970 corresponding to a portion of a patient's spine and including a missing region 972. Figure 9B is a schematic diagram of the corresponding CT scanning data 974 of the patient's spine according to an embodiment of the art of the present invention. Referring together to Figures 9A and 9B, the CT scanning data 974 may include 3D volumetric depth data 976 corresponding to at least a portion of the missing region 972 in the point cloud 970.

[0047] In block 865, method 860 includes the step of fusing / combining 3D data from a medical scan with original depth data (e.g., acquired using block 861) to generate a fused point cloud containing data points for missing or invalid regions. Generally, medical scan data can replace and / or complement data within the point cloud. For example, medical scan data can replace data in point cloud regions where acquired data is insufficient and complement (e.g., fill in) missing regions of the point cloud. Thus, the fused point cloud can provide a more accurate and robust depth map of scene 108, facilitating better reconstruction and synthesis of output images of scene 108 rendered from any desired virtual field of view, as described in detail above.

[0048] More specifically, in some embodiments, data from a medical scan is used to fill in missing areas in the point cloud. For example, Figure 9C is a schematic diagram of a fused point cloud 980 in which CT scan data 974 shown in Figure 9B fills in missing areas 972 of point cloud 970 shown in Figure 9A. In some embodiments, suitable areas of CT scan data 974 corresponding to missing areas 972 of point cloud 970 can be found by comparing nearest neighbors between the aligned CT scan data 974 and point cloud 970. That is, for example, points in the medical scan that do not have neighbors in the aligned point cloud (e.g., below a threshold) can be identified and fused into / with the point cloud data. In other embodiments, as much of the original depth data (e.g., point cloud 970) as possible can be replaced with aligned medical scan data (e.g., CT scan data 974). In some embodiments, a nearest neighbor algorithm can be used to determine which areas of the original depth data should be removed and replaced. In yet another embodiment, medical scanning data and point clouds can be directly fused using a volumetric (e.g., voxel) representation such as a truncated signed distance function (TSDF).

[0049] In block 866, method 860 may optionally include a step of generating a 3D mesh based on the fused point cloud. The 3D mesh can be used to reconstruct / combine the output image of scene 108. In some embodiments, method 860 can return to block 851 to update the depth information of scene 108. In some embodiments, when the medical scan data and the original depth data are fused directly using TSDF, the 3D mesh can be generated using marching cubes or other suitable algorithms.

[0050] In some embodiments, the medical scanning data is prior to known and therefore does not require significant processing. Accordingly, in one aspect of the art of the present invention, method 860 can rapidly update (e.g., interpolate and / or replace) the original depth based on the medical scanning, enabling real-time or near real-time processing and generation of the output image of scene 108.

[0051] In other embodiments, medical scanning data can act as an initial state for depth optimization processing, which can be further refined. For example, medical scanning data can be aligned with live data to fill in gaps, as described in detail with reference to Figures 8-9C. However, in some embodiments, camera 112 may have higher resolution / accuracy than medical scanning data. Therefore, depth information fused from depth sensor 116 and medical scanning data can be used to initialize a 3D reconstruction process using images from camera 112. Depth information from images can then be fused with or replace depth information from medical scanning. In some embodiments, the depth range of missing / invalid regions is known based on medical scanning data, i.e., the range of iterations required to determine true depth values ​​from image data is minimized, thus accelerating depth processing of image data.

[0052] III. Here's yet another example The following examples illustrate some embodiments of the technology of the present invention: 1. A method for determining the depth of a scene, comprising the steps of: acquiring depth data of the scene with a depth sensor; acquiring image data of the scene with multiple cameras; generating a point cloud representing the scene based on the depth data; identifying a region of the point cloud; generating depth data for the region based on the image data; and fusing the depth data for the region with depth data from the depth sensor to generate a fused point cloud representing the scene. 2. The method of Example 1, where the region of the point cloud is a missing region of the point cloud that contains no data or sparse data. 3. The method of Example 2, wherein the step of identifying missing regions in the point cloud includes the step of determining that the missing regions in the point cloud have fewer data points than a predetermined threshold number. 4. The method of Example 2 or Example 3, wherein the step of identifying missing regions in the point cloud includes a step of identifying holes in the point cloud that are larger than a user-defined threshold. 5. The step of generating depth data for the missing region is one of the methods in Examples 2-4, which further utilizes a portion of the depth data captured by the depth sensor surrounding the missing region. 6. One of the methods from Examples 1 to 5, wherein the depth data for the region has a higher resolution than the depth data acquired using a depth sensor. 7. One of the methods from Examples 1 to 6, further comprising the step of generating a 3D mesh representing the scene based on the fused point cloud. 8. One of the following methods, where the setting is a surgical scene. 9. One of the methods in Examples 1-8, where each of the multiple cameras has a different position and orientation relative to the scene, and the image data is light field image data. 10. Any one of Examples 1 to 9 further comprising the steps of processing image data and fused point cloud to synthesize an output image of a scene corresponding to the virtual camera field of view, and transmitting the output image to a display for display to the user. 11. The method of Example 10, wherein the display is a head-mounted display worn by the user, and the step of identifying a region of a scene is based on at least one of the position and orientation of the head-mounted display. 12. A system for imaging a scene, comprising: a plurality of cameras arranged at different positions and orientations relative to the scene and configured to capture image data of the scene; a depth sensor configured to capture depth data relating to the depth of the scene; and a computer device communicatively coupled to the cameras and the depth sensor, wherein the computer device has a memory containing computer executable instructions and a processor for executing the computer executable instructions contained in the memory, and the computer executable instructions include instructions for the steps of: receiving image data from the cameras; receiving depth data from the depth sensor; generating a point cloud representing the scene based on the depth data; identifying a region of the point cloud; generating depth data for the region based on the image data; and fusing the depth data for the region with depth data from the depth sensor to generate a fused point cloud representing the scene. 13. The system in Example 12 where the region of the point cloud is a missing region of the point cloud where the point cloud contains no data or contains sparse data. 14. A system of Example 12 or Example 13 in which the point cloud region is user-selectable. 15. Any one of Examples 12 to 14, further including a display, wherein the computer device is communicatively coupled to the display, and the computer executable instructions further include instructions for the steps of processing image data and fused point clouds to synthesize an output image of a scene corresponding to a virtual camera field of view, and transmitting the output image to the display for display to a user. 16. A system of Example 15 in which the step of identifying a scene area is based on at least one of the position and orientation of the display. 17. A method for determining the depth of a scene, comprising the steps of: acquiring scene depth data with a depth sensor; generating a point cloud representing the scene based on the depth data; identifying the region of the point cloud; aligning the point cloud with three-dimensional (3D) medical scanning data; and fusing at least a portion of the 3D medical scanning data with depth data from the depth sensor to generate a fused point cloud representing the scene. 18. The method of Example 17, where the region of the point cloud is a missing region of the point cloud that contains no data or sparse data. 19. The method of Example 18, where the scene is a medical scene including a part of a patient, the missing area in the point cloud corresponds to a part of the patient, and that part of the 3D medical scanning data corresponds to the same part of the patient. 20.3D medical scanning data is computed tomography (CT) data, one of the methods shown in Examples 17-19.

[0053] IV. conclusion The above detailed description relating to embodiments of the technology of the present invention is not intended to be comprehensive or to limit the technology of the present invention to the forms disclosed above. While specific embodiments of the technology of the present invention and examples thereof have been described for illustrative purposes, various equivalent modifications are possible within the scope of the technology of the present invention, as will be recognized by those skilled in the art. For example, while the steps are presented in a given order, in alternative embodiments the steps may be performed in a different order. Similarly, various embodiments described herein can be combined to provide yet another embodiment.

[0054] From the above, while specific embodiments of the technology of the present invention have been described herein for illustrative purposes, it will be acknowledged that known structures and functions have not been shown or described in detail so as not to unnecessarily obscure the description of embodiments of the technology of the present invention. To the extent that relevance permits, singular or plural terms may encompass plural or singular terms, respectively.

[0055] Furthermore, with respect to a list of two or more items, unless the word “or” is expressly limited to mean only a single item limited to other items, the use of “or” in such a list should be interpreted as including (a) any single item in the list, (b) all items in the list, or (c) any combination of items in the list. In addition, the term “equipped with” is used throughout to mean including at least the enumerated features so as not to exclude more of the same features and / or other features of additional kinds. Similarly, while certain embodiments have been described herein for illustrative purposes, it will be acknowledged that various modifications can be made without departing from the art of the present invention. Furthermore, while advantages associated with some embodiments of the art of the present invention have been described in relation to those embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily have to exhibit such advantages to fall within the scope of the art of the present invention. Accordingly, the disclosure of the present invention and related art may encompass other embodiments not expressly shown or described herein. [Explanation of symbols]

[0056] 100 Imaging Systems 102 Image Processing Devices 108 Scenes 114 Projector 116 Depth Sensor

Claims

1. A method for operating a device that determines the depth within a surgical field, including the bone and / or soft tissue of a patient, A step of acquiring depth data of the bone and / or soft tissue using a depth sensor, A step of acquiring image data of the bone and / or soft tissue using multiple cameras, A step of generating a point cloud representing the bone and / or soft tissue based on the depth data, The steps include identifying a missing region of the point cloud that includes a number of point cloud data points less than a predetermined threshold for the number of point clouds, and / or a hole larger than a predetermined threshold for size, and that corresponds to a missing region of bone and / or soft tissue that is obscured from the depth sensor, A step of determining at least one depth value from the point cloud for the region adjacent to the missing region, A step of processing the image data by a depth processing algorithm in order to generate depth data for the missing region, wherein processing the image data by the depth processing algorithm includes restricting the depth processing algorithm to determine the depth data for the missing region to a depth range encompassing at least one depth value and to a depth value smaller than the maximum depth range of the image data, The steps include: fusing the depth data for the missing region with the depth data from the depth sensor to generate a fused point cloud representing the bone and / or soft tissue; A method characterized by including the following.

2. The method according to claim 1, characterized in that the depth data for the missing region has a higher resolution than the depth data acquired using the depth sensor.

3. The method according to claim 1, further comprising the step of generating a three-dimensional mesh representing the surgical scene based on the fused point cloud.

4. The method according to claim 1, characterized in that the camera and the depth sensor are mounted and fixed to a common frame.

5. The method according to claim 4, characterized in that the camera is an RGB camera.

6. The method according to claim 1, characterized in that the image data is light field image data.

7. A system for imaging surgical scenes including the bones and / or soft tissues of a patient, Multiple cameras are positioned at different locations and orientations relative to the surgical scene and configured to capture image data of the bone and / or soft tissue, A depth sensor configured to acquire depth data of the bone and / or soft tissue, A computer device that is communicatively coupled to the camera and the depth sensor, The computer device comprises a memory containing computer executable instructions and a processor for executing the computer executable instructions contained in the memory, and when the computer executable instructions are executed by the processor, Receiving depth data of the bone and / or soft tissue from the depth sensor, Receiving the image data of the bone and / or soft tissue from the camera, Based on the depth data, a point cloud representing the bone and / or soft tissue is generated, The method for identifying a missing region in the point cloud that includes a number of point cloud data points less than a predetermined threshold for the number of point clouds, and / or holes larger than a predetermined threshold for size, and that corresponds to a portion of the bone and / or soft tissue that is obscured from the depth sensor, Determining at least one depth value from the point cloud for the region adjacent to the missing region, Processing the image data by a depth processing algorithm to generate depth data for the missing region, wherein the computer executable instruction, when executed by the processor, causes the processor to further restrict the depth processing algorithm to determine the depth data for the missing region within a depth range encompassing at least one depth value and smaller than the maximum depth range of the image data. The depth data for the missing region is fused with the depth data from the depth sensor to generate a fused point cloud representing the bone and / or soft tissue. A system characterized by having the processor perform the aforementioned task.

8. The system according to claim 7, characterized in that the depth data for the missing region has a higher resolution than the depth data acquired using the depth sensor.

9. The system according to claim 7, characterized in that when the computer executable instruction is executed by the processor, it causes the processor to further generate a three-dimensional mesh representing the surgical scene based on the fused point cloud.

10. The system according to claim 7, characterized in that the camera and the depth sensor are mounted and fixed to a common frame.

11. The system according to claim 10, characterized in that the camera is an RGB camera.

12. A method for operating a device that generates output images of a surgical scene including the bones and / or soft tissues of a patient, A step of acquiring depth data of the patient's bone and / or soft tissue using a depth sensor, A step of acquiring light field image data of the bone and / or soft tissue using multiple cameras, A step of generating a point cloud representing the bone and / or soft tissue based on the depth data, The steps include identifying within the field of view of a virtual camera a missing region of the point cloud that includes a number of point cloud data points less than a predetermined threshold for the number of point clouds, and / or holes larger than a predetermined threshold for the size of the point cloud, and which corresponds to a part of the bone and / or soft tissue that is hidden from the depth sensor; A step of determining at least one depth value from the point cloud for the region adjacent to the missing region, A step of processing the light field image data by a depth processing algorithm to generate depth data for the missing region, wherein processing the light field image data by the depth processing algorithm includes restricting the depth processing algorithm to determine the depth data for the missing region to a depth range encompassing at least one depth value and smaller than the maximum depth range of the light field image data, The steps include: fusing the depth data for the missing region with the depth data from the depth sensor to generate a fused point cloud representing the surgical scene; The steps include processing the light field image data and the fused point cloud to synthesize the output image of the surgical scene, The step of transmitting the output image to a display for display to the user, A method characterized by including the following.

13. The method according to 12, characterized in that the output image is from the field of view of a virtual camera having a field of view corresponding to a part of the surgical scene.

14. The method according to 13, characterized in that each of the plurality of cameras has a different field of view of the surgical scene, and the field of view of the virtual camera is different from any of the fields of view of the cameras.

15. The method according to 12, characterized in that the camera and the depth sensor are mounted and fixed to a common frame.

16. The method according to 12, characterized in that the camera is an RGB camera.

17. A method for operating a device that generates an output image of a portion of a scene, wherein the output image is from the field of view of a virtual camera having a field of view corresponding to the portion of the scene, and the method is The steps include: acquiring depth data of the scene using a depth sensor; A step of capturing light field image data of the scene using multiple cameras, each having a different position and orientation relative to the scene, A step of generating a point cloud representing the scene based on the depth data, The steps include identifying missing regions of the point cloud within the field of view of the virtual camera, where the point cloud has fewer data points than a predetermined threshold number, A step of determining at least one depth value from the point cloud for the region adjacent to the missing region, A step of processing the light field image data by a depth processing algorithm to generate depth data for the missing region, wherein processing the light field image data by the depth processing algorithm includes restricting the depth processing algorithm to determine the depth data for the missing region to a depth range encompassing at least one depth value and smaller than the maximum depth range of the light field image data, The steps include: merging the depth data for the missing region with the depth data from the depth sensor to generate a fused point cloud representing the scene; The steps include processing the light field image data and the fused point cloud to synthesize the output image of a portion of the scene from the field of view of the virtual camera, The step of transmitting the output image to a display for display to the user, A method characterized by including the following.

18. The method according to 17, characterized in that the step of identifying the missing region of the point cloud includes the step of identifying holes in the point cloud that are larger than a threshold defined by the user.

19. The method according to 17, characterized in that the step of generating depth data for the missing region is further based on a portion of the depth data captured by the depth sensor surrounding the missing region.

20. The method according to 17, characterized in that the depth data for the missing region has a higher resolution than the depth data acquired using the depth sensor.

21. The method according to 17, further comprising the step of generating a three-dimensional mesh representing the portion of the scene based on the fused point cloud.

22. The method according to 17, characterized in that the aforementioned scene is a surgical scene.

23. The method according to 17, wherein the display is a head-mounted display worn by the user, and the method further comprises the step of determining the field of view of the virtual camera based on at least one of the position and orientation of the head-mounted display.

24. A system for capturing images of a scene, Multiple cameras are positioned at different locations and orientations relative to the aforementioned scene and configured to capture light field image data of the scene, A depth sensor configured to acquire depth data relating to the depth of the aforementioned scene, A computer device that is communicatively coupled to the camera and the depth sensor, The computer device comprises a memory containing computer executable instructions and a processor for executing the computer executable instructions contained in the memory, the computer executable instructions include instructions for generating an output image of a portion of the scene, the output image is from the field of view of a virtual camera having a field of view corresponding to the portion of the scene, and generating the output image is Receiving the light field image data from the camera, Receiving the depth data from the depth sensor, Based on the depth data, a point cloud representing the scene is generated, Identifying missing regions of the point cloud within the field of view of the virtual camera where the point cloud has fewer data points than a predetermined threshold number, Determining at least one depth value from the point cloud for the region adjacent to the missing region, Processing the light field image data by a depth processing algorithm to generate depth data for the missing region, wherein the processing of the light field image data by the depth processing algorithm includes restricting the depth processing algorithm to determine the depth data for the missing region to a depth range encompassing at least one depth value and smaller than the maximum depth range of the light field image data. The depth data for the missing region is fused with the depth data from the depth sensor to generate a fused point cloud representing the scene. The process of processing the light field image data and the fused point cloud to synthesize the output image of a portion of the scene from the field of view of the virtual camera, The output image is transmitted to a display for display to the user, A system characterized by including

Images