Generation of 3D images by digital enlargement

Abstract

To provide a system that generates a 3D image from an image of a captured target. [Solution] The controller performs digital magnification on a first image of the target captured by the first image sensor and a second image of the target captured by the second image sensor. The controller crops the first and second images so that the first portion of the target captured by the first image sensor overlaps with the second portion of the target captured by the second image sensor. The controller adjusts the cropping of the first and second images to provide a binocular overlap between the first and second portions of the target. The displayed cropped first and second images show the user a digitally magnified 3D image.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 029,831, filed May 26, 2020, which is hereby incorporated by reference in its entirety.

[0002] This disclosure relates to the generation of three-dimensional (3D) images, and more particularly, to the generation of 3D images from digital magnification of a captured image of a target.

Background Art

[0003] Conventionally, surgical loupes have been widely used in various types of surgeries. A surgical loupe is a pair of optical magnifiers that magnify the surgical field and provide magnified stereoscopic vision. However, conventional surgical loupes have significant limitations. For example, a conventional single set of surgical loupes only provides a fixed magnification level, such as 2x, and has no ability to change such magnification. Thus, surgeons typically require several pairs of surgical loupes with different magnification levels for each pair to accommodate different magnification levels. Exchanging surgical loupes in the operating room is costly and inconvenient for a single surgeon to have several customized sets of surgical loupes with different magnifications.

[0004] However, equipping conventional surgical loupes with magnifying lenses usually results in increased length, which in turn leads to an increase in form factor and weight, thereby limiting the magnification level. The increase in form factor and weight also limits the duration of surgical procedures that a surgeon can perform. Furthermore, conventional surgical loupes implement a non-imaging configuration, whereby the magnifying lenses magnify and form a pair of virtual images, thereby reducing the surgeon's working distance and depth of focus. Thus, a surgeon must restrict the position of their head and neck to a specific position when using conventional surgical loupes. This leads to neck pain and cervical diseases for surgeons who use conventional surgical loupes for long periods of time.

[0005] Rather than simply using a non-imaging configuration with surgical loupes, conventional imaging configurations in the non-surgical space include stereoscopic imaging systems and imaging systems equipped with zoom lenses, and such conventional imaging configurations generate 3D images while allowing adjustment of magnification. However, incorporating such conventional imaging configurations into the surgical space requires the implementation of two displays and / or zoom lenses for the surgeon. The two stereoscopic displays included in such conventional stereoscopic imaging systems must be mechanically adjusted and calibrated for each magnification level. Such mechanical adjustment and calibration is not feasible in the surgical space. To change the magnification of the two conventional zoom lenses, each image at each magnification level must always capture the center of the initial image, and each magnification level must continue to capture the center of the initial image. The resulting 3D image displayed to the surgeon is significantly distorted, making it impossible to incorporate conventional zoom lenses into the surgical space. [Overview of the Initiative] [Means for solving the problem]

[0006] Embodiments of the present disclosure will be described with reference to the accompanying drawings. In the drawings, similar reference numerals indicate identical or functionally similar elements. Furthermore, the leftmost digit of a reference numeral usually identifies the drawing in which the reference numeral first appears. [Brief explanation of the drawing]

[0007] [Figure 1A] This is a schematic diagram of the binocular overlap in the structure of the human eye. The area visible to both eyes is the overlapping area included in the scene visible to both eyes. [Figure 1B] This is a block diagram of a two-image sensor configuration in which two image sensors, each having two lenses, are used in a side-by-side arrangement. [Figure 1C] This is a block diagram of the overlapping area of ​​the two imaging sensors, where the area visible to both imaging sensors is the overlapping area. [Figure 2]This is a schematic diagram of a conventional digital zoom configuration, where the original image has been cropped and resized (from left to right). [Figure 3] This is a block diagram of a digital magnification 3D imaging system that can generate a 3D image when performing digital magnification on a target capture image. [Figure 4] This is a schematic diagram of a conventional digital zoom configuration, where the zoomed left image and the zoomed right image are not aligned, leading to a decrease in 3D vision and depth perception. [Figure 5] This is a schematic diagram of digital magnification using a configuration that maintains vertical alignment of both eyes. The magnified left image and the magnified right image are aligned vertically, thereby improving 3D visualization. [Figure 6] This is a schematic diagram of a digitally enlarged stereoscopic image while maintaining its vertical alignment configuration. When digital enlargement is applied, the binocular overlap between the cropped left image and the cropped right image gradually decreases. [Figure 7] This is a schematic diagram of maintaining the binocular overlap and vertical alignment configuration of both eyes. The cropped image on the left and the cropped image on the right have 75% binocular overlap and vertical alignment at 2.3x, 5.3C, and 12x magnification, respectively, which can provide the user with an improved 3D visualization experience and depth perception. [Figure 8] This is a schematic diagram of a physical embodiment of a digital magnifying surgical loupe configuration. [Modes for carrying out the invention]

[0008] The following embodiments for carrying out the invention are illustrated with reference to the accompanying drawings and illustrate exemplary embodiments consistent with the present disclosure. References to “exemplary embodiment,” “exemplary embodiment,” and “exemplary embodiment of examples” in the embodiments for carrying out the invention indicate that the described exemplary embodiments may include certain features, structures, or properties, but not all exemplary embodiments may necessarily include certain features, structures, or properties. Furthermore, such phrases do not necessarily refer to the same exemplary embodiment. Moreover, where certain features, structures, or properties can be described in relation to an exemplary embodiment, it is within the knowledge of those skilled in the art that such features, structures, or properties may be brought about in relation to other exemplary embodiments, whether or not they are explicitly stated.

[0009] The exemplary embodiments described herein are provided for illustrative purposes only and are not limiting. Other exemplary embodiments are possible, and modifications may be made to the exemplary embodiments within the spirit and scope of this disclosure. Accordingly, the modes for carrying out the invention are not intended to limit this disclosure. Rather, the scope of this disclosure is defined solely by the appended claims and their equivalents.

[0010] The embodiments of this disclosure may be implemented in hardware, firmware, software, or any combination thereof. The embodiments of this disclosure may also be implemented as instructions applied by a machine-readable medium, which may be read and executed by one or more processors. The machine-readable medium may include any mechanism for storing or transmitting information in a format readable by a machine (e.g., a computing device). For example, the machine-readable medium may include read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, electro-optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.). Further firmware, software routines, and instructions may be described herein as performing specific actions. However, such descriptions are for convenience only, and it should be understood that such actions actually originate from computing devices, processors, controllers, or other devices that execute firmware, software, routines, instructions, etc.

[0011] For the purposes of this explanation, the various components described may each be considered a module, and the term “module” shall be understood to include at least one piece of software, firmware, and hardware (such as one or more circuits, microchips, or devices, or any combination thereof), and any combination thereof. In addition, it will be understood that each module may contain one or more components within an actual device, and that each component forming part of a described module may function in conjunction with or independently of any other components forming part of the module. Conversely, multiple modules described herein may represent a single component within an actual device. Furthermore, components within a module may reside within a single device or be distributed across multiple devices via wired or wireless connections.

[0012] The exemplary embodiments for carrying out the following inventions fully illustrate the general nature of this disclosure, so that others can readily modify and / or adapt such exemplary embodiments to various uses without excessive experimentation and without departing from the spirit and scope of this disclosure, by applying the knowledge of those skilled in the art. Such adaptations and modifications are therefore intended to be within the meaning of the exemplary embodiments and their equivalents based on the teachings and guidance presented herein. It should be understood that the terminology and language used herein are for illustrative purposes only, not limiting, so that they may be interpreted by those skilled in the art in light of the teachings herein.

[0013] System Overview Figure 1A shows a schematic diagram of the binocular overlapping area of ​​the human eye configuration 100, where the area visible to both eyes is the overlapping area included in the scene visible to both eyes. The binocular overlapping area of ​​the human eye configuration 100 includes the right eye 110a, the left eye 110b, the image visible to the right eye 120a, the image visible to the left eye 120b, and the binocular overlapping area 120c visible to both eyes.

[0014] This invention describes apparatus, systems, and methods for constructing augmented reality devices for medical and dental magnification. One of the key concepts in 3D imaging and visualization is the binocular overlap 120c. The binocular overlap 120c represents the overlap between the image 120b seen by the left eye and the image 120a seen by the right eye. In humans, the binocular overlap 120c is approximately 70%.

[0015] Figure 1B shows a block diagram of a two-image sensor configuration 150 in which two image sensors, each having two lenses, are used in a side-by-side configuration. The two-image sensor configuration 150 includes a right image sensor 130a, a left image sensor 130b, a right lens 140a, and a left lens 140b. Figure 1C shows a block diagram of the binocular overlap of a two-image sensor configuration 175, where the overlap is the area visible to both image sensors. The binocular overlap of the two-image sensor configuration 175 includes the area captured by the right image sensor 150a, the area captured by the left image sensor 150b, and the binocular overlap area 150c. Figure 1C shows the binocular overlap area 150c generated when the right image sensor 130a and the left image sensor 130b are used in a side-by-side configuration as shown in Figure 1B.

[0016] Figure 2 shows a schematic diagram of a conventional digital zoom configuration 200, where the original image has been cropped and resized (from left to right). The cropped and resized image is displayed to the user after conventional digital zoom. Traditionally, digital zoom has been commonly used to zoom in on images. The principle of conventional digital zoom is shown in Figure 2. Conventional digital zoom can enlarge an image without requiring a zoom lens, but it is not suitable for 3D enlargement.

[0017] Figure 3 shows a block diagram of the digital magnification of a 3D imaging system 300 that can generate a 3D image when digital magnification is performed on a target capture image. The digital magnification of the 3D imaging system 300 includes a right lens 340a, a left lens 340b, a right image sensor 330a, a left image sensor 330b, a controller 310, a near-eye 3D display 320, and an eyeglass frame 350. In one embodiment, the eyeglass frame 350 is head-mounted. In another embodiment, the eyeglass frame 350 is a conventional eyeglass frame that sits on the user's nose and ears.

[0018] The digital zoom of the 3D image system 300 can generate a 3D image from the captured image of the target such that the generated three-dimensional (3D) image of the target is maintained after digital zoom when performing digital zoom on the captured image. A first image sensor (such as the right image sensor 330a) can capture a first image at the original size of the target. A second image sensor (such as the left image sensor 330b) can be arranged on the same x-axis as the first image sensor 330a to capture a second image at the original size of the target. It should be understood that the first image sensor 330a and the second image sensor 330b can be arranged at either a converging angle or a diverging angle.

[0019] The controller 310 can perform digital zoom on the first image captured by the first image sensor 330a at the original size of the target and the second image captured by the second image sensor 330b at the original size of the target. The controller 310 can crop the first image captured by the first image sensor 330a and the second image captured by the second image sensor 330b so that the first portion of the target captured by the first image sensor 330a overlaps with the second portion of the target captured by the second image sensor 330b. The first portion of the target captured by the first image sensor 330a overlaps with the second portion of the target captured by the second image sensor 330b. In one aspect, the first image sensor 330a is further coupled to a first autofocus lens, and the second image sensor 330b is further coupled to a second autofocus lens. The autofocus lens can enable autofocus.

[0020] The controller 310 can adjust the cropping of the first image and the second image to provide a binocular overlap of the first part of the target and the second part of the target. The binocular overlap between the first image and the second image is a overlap threshold. When the overlap threshold is satisfied, a 3D image of the target to be displayed to the user after digital magnification can be obtained. The controller can instruct a display (such as the near-eye 3D display 320) to display the cropped first image and the cropped second image including the binocular overlap to the user. The displayed cropped first image and cropped second image display the 3D image to the user by digital magnification.

[0021] The controller 310 can resize the cropped first image to the original size of the first image captured by the first image sensor 330a and resize the cropped second image to the original size of the second image captured by the second image sensor 330b. The resized cropped first image and the resized cropped second image include the binocular overlap between the first image and the second image. The controller 310 can instruct the near-eye 3D display 320 to display the resized cropped first image and the resized cropped second image including the binocular overlap to the user. The displayed resized cropped first image and resized cropped second image display the 3D image to the user by digital magnification. In one embodiment, it should be understood that the controller 310 can crop the first image captured by the first image sensor 330a to generate both a left cropped image and a right cropped image. In this embodiment, the second image captured by the second image sensor 330b is not used.

[0022] In one embodiment, the display 320 is a near-eye display. In one embodiment, the display 320 is a 2D display. In another embodiment, the display 320 is a 3D display. It should be further understood that the near-eye display 320 may include LCD (liquid crystal) microdisplays, LED (light-emitting diode) microdisplays, organic LED (OLED) microdisplays, liquid crystal on silicon (LCOS) microdisplays, retinal scanning displays, virtual retinal displays, optical see-through displays, video see-through displays, convertible video optical see-through displays, wearable projection displays, projection displays, etc. It should be understood that the display 320 may be a stereoscopic display to enable the display of 3D content. In another embodiment, the display 320 is a projection display. It should be understood that the display 320 may be a monitor placed close to the user.

[0023] It should be further understood that the display 320 may be a 3D monitor positioned close to the user, and that the user may wear polarizing glasses or active shutter glasses. It should be further understood that the display 320 may be a semi-transparent mirror positioned close to the user to reflect the image projected by the projector. It should be further understood that the projector may be 2D or 3D. It should be further understood that the projector may be used with the user wearing polarizing glasses or active shutter glasses. In one embodiment, the display 320 is a flat-panel 2D monitor or TV. In another embodiment, the display 320 is a flat-panel 3D monitor or 3D TV. The 3D monitor / TV may need to work in conjunction with a passive polarizer or active shutter glasses. In one embodiment, the 3D monitor / TV is glassless. It should be understood that the display 320 may be a touchscreen or a projector. In one example, the display 320 includes a semi-transparent mirror that can reflect the projected image to the user's eyes. The projected image can be in 3D, and the user may wear 3D glasses (e.g., polarizer, active shutter 3D glasses) to visualize the 3D image data reflected by the semi-transparent microscope. The semi-transparent microscope may be positioned above the surgical field so that the user can visualize the surgical field through it.

[0024] It should be understood that, depending on the specific application, the system's binoculars can be set as high as 100% or as low as 0%. In one embodiment, the binocular overlap can be set to a range of 60% to 100%. In another embodiment, the binocular overlap is dynamic and not static.

[0025] In one embodiment, the digital augmentation of the 3D imaging system 300 may further include additional sensors or components. In one embodiment, the system 300 further includes a microphone capable of enabling audio recording and / or communication. In one embodiment, the system 300 further includes a proximity sensor capable of sensing whether a user is wearing the system. In another embodiment, the system 300 further includes an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetometer, or a combination thereof. In one embodiment, the system 300 further includes a loudspeaker or earphone capable of enabling audio playback or communication.

[0026] It should be further understood that this system can be applied to a wide range of uses, including but not limited to the fields of surgery, medicine, veterinary medicine, military, tactics, education, industry, consumer goods, and jewelry.

[0027] Digital magnification using vertical alignment of both eyes Figure 4 shows a schematic diagram of a conventional digital zoom configuration 400, in which the zoomed left image and the zoomed right image are not aligned, leading to a decrease in 3D vision and depth perception. The conventional digital zoom configuration 400 includes a zoomed left image 410b and a zoomed right image 410a that is not aligned. Conventional digital zoom does not work well for enlarging stereoscopic images for 3D display. Figure 3 shows an example of applying conventional digital zoom directly to a stereoscopic image. Conventional digital zoom is not suitable for enlarging 3D stereoscopic images because it causes vertical misalignment between the two eyes.

[0028] The controller 310 may crop the first image captured by the first image sensor 330a and the second image captured by the second image sensor 330b in order to vertically align the overlapping portion between the first and second portions of the target. When each vertical coordinate of the cropped first image is aligned with the corresponding vertical coordinate of the cropped second image, the cropped first image is vertically aligned with the cropped second image. The controller 310 may adjust the cropping of the first and second images to provide a binocular overlap between the first and second portions of the target. To generate a 3D image of the target that will be displayed to the user after digital magnification is performed, the binocular overlap between the first and second images is vertically aligned to satisfy an overlap threshold.

[0029] The present invention discloses a digital magnification method that also ensures vertical alignment of both eyes. In one embodiment, the left image is captured by the left image sensor 330b and cropped by the controller 310, and the right image is captured by the right image sensor 330a and cropped by the controller 310, while the cropping of the left and right images maintains vertical alignment. The left and right images are cropped so that the vertical coordinates of the cropped left image and the cropped right image are aligned.

[0030] In one embodiment, a left image sensor 330b having a left lens 340b worn by the user can capture a left image. A right image sensor 330a having a right lens 340a worn by the user can capture a right image. The left and right images can be provided to a controller 310. The controller 310 can crop the left image to generate a cropped left image. The controller 310 can crop the right image to generate a cropped right image and maintain the vertical alignment of the cropped right image with respect to the cropped left image. The controller 310 can resize the cropped left image to generate a cropped and resized left image. The controller 310 can resize the cropped right image to generate a cropped and resized right image. A near-eye 3D display 320 worn by the user can display the cropped and resized left image to the user's left eye. The user-worn near-eye 3D display 320 can display a cropped and resized right image to the user's right eye. It should be understood that the controller may be a microcontroller, a computer, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof.

[0031] In one embodiment, the left and right image sensors are identical. The image sensors may use the same type of image lens. The left and right image sensors may be positioned and calibrated before the digital magnification process so that the captured left and captured right images are vertically aligned. The digital magnification process preserves the vertical alignment. For example, assume that the left and right images each have 800 (horizontal, columns) × 600 (vertical, rows) pixels. After digital magnification, the pixels from row 201 to row 400 of the left image are used to generate a cropped left image, and the pixels from row 201 to row 400 of the right image are used to generate a cropped right image. Thus, the vertical alignment is preserved.

[0032] In one embodiment, the left image sensor and the right image sensor are not the same image sensor. In this case, the captured left image and the captured right image are first calibrated and aligned vertically before the digital enlargement process. For example, suppose the left image captured by the left image sensor has 800 (horizontal, columns) × 600 (vertical, rows) pixels, while the right image captured by the right image sensor has 400 (horizontal) × 300 (vertical) pixels. The left and right images are first aligned vertically. For example, row numbers 0, 200, 400, and 600 of the left image may correspond to row numbers 0, 100, 200, and 300 of the right image, respectively. After digital enlargement, a subset of pixels from row 200 to row 400 of the left image and a subset of pixels from row 100 to row 200 of the right image are used. Thus, the vertical alignment is maintained.

[0033] Figure 5 shows a schematic diagram of digital magnification using a binocular vertical alignment holding configuration 500, where the magnified left image and the magnified right image are vertically aligned, thereby improving 3D visualization. Digital magnification using the binocular vertical alignment holding configuration 500 includes vertical alignment of the zoomed right image 510b with the zoomed left image 510a, thereby improving 3D visualization.

[0034] Digital magnification while preserving the overlapping area of ​​both eyes. Figure 6 shows a schematic diagram of a digitally enlarged stereoscopic image while maintaining the vertical alignment configuration 600. When digital enlargement is applied, the binocular overlap between the cropped left image and the cropped right image gradually decreases. The digitally enlarged stereoscopic image while maintaining the vertical alignment configuration 600 includes vertical alignment of the digitally enlarged left image 610b with the digitally enlarged right image 610a. For example, at a 2.3x enlargement, the binocular overlap decreases from 75% to 50%, and the 3D visualization decreases. At a 5.3x enlargement, the binocular overlap decreases from 75% to 0%. Maintaining vertical alignment without preserving the binocular overlap may result in a gradual decrease in binocular overlap with each digital enlargement.

[0035] After performing a first digital magnification at a first digital magnification level on the first image captured by the first image sensor 330b and the second image captured by the second image sensor 330a, the controller 310 may maintain the binocular overlap generated by adjusting the cropping of the first and second images to satisfy an overlap threshold. In one embodiment, a fixed value for the binocular overlap, such as 80%, 90%, or 100%, is maintained during the digital magnification process. In another embodiment, a numerical range for the binocular overlap, such as 60% to 90%, is maintained during the digital magnification process.

[0036] The controller 310 may perform a second digital magnification at a second digital magnification level on the first image captured by the first image sensor 330a and the second image captured by the second image sensor 330b. The second digital magnification level is increased from the first digital magnification level. When the controller 310 performs the second digital magnification at the second digital magnification level, it may maintain the binocular overlap generated after performing the first digital magnification at the first digital magnification level on the first and second images.

[0037] After performing the previous digital magnification on the first and second images at their respective previous digital magnification levels, the controller 310 may maintain the binocular overlap and vertical alignment determined when performing the first digital magnification on the first and second images at the first digital magnification level. After performing the first digital magnification on the first and second images at the first digital magnification level for each subsequent digital magnification level, the controller 310 may continue to maintain the binocular overlap and vertical alignment determined by adjusting the cropping of the first and second images to satisfy the overlap threshold. Each subsequent digital magnification level is an increase from each previous digital magnification level. For example, the overlap threshold may be satisfied if the binocular overlap includes a 75% overlap between the first and second images and is maintained for each subsequent digital magnification at each subsequent digital magnification level. In one embodiment, each subsequent digital magnification from the previous magnification level (e.g., an increase from 1x to 2x, and an increase from 2x to 4x) may be a recursive function.

[0038] Controller 310 may perform a first digital magnification at a first digital magnification level on the non-concentric portions of the first image and the non-concentric portions of the second image. The non-concentric portions of the first and second images are the parts of the first and second images that are different from the centers of the first and second images. Controller 310 may adjust the cropping of the first and second images to provide a binocular overlap between the non-concentric portions of the first image and the non-concentric portions of the second image. The binocular overlap between the non-concentric portions of the first image and the non-concentric portions of the second image satisfies an overlap threshold specified as a fixed value or range. Controller 310 may continue to crop the non-concentric portions of the first image and the non-concentric portions of the second image for each subsequent digital magnification at each subsequent digital magnification level. The binocular overlap between the non-concentric portions of the first image and the non-concentric portions of the second image is maintained from the first digital magnification at the first digital magnification level.

[0039] The non-concentric portions of the first image and the non-concentric portions of the second image can be resized for display to the user. In one embodiment, at each magnification level, the first center of the cropping of the non-concentric portion of the first image and the second center of the cropping of the non-concentric portion of the second image are determined by the system 300. In one embodiment, the first center of the cropping is fixed to a specific portion of the first image, and the second center of the cropping at each magnification level is determined based on the position of the corresponding first center of the cropping and the target binocular overlap. It should be understood that in some embodiments, the digital magnification of the left image or the right image may be concentric at one or more magnification levels. For example, the digital magnification of the left image may be concentric, while the digital magnification of the right image may be non-concentric in order to maintain the binocular overlap.

[0040] In one embodiment, the left and right image sensors are identical. The image sensors may use the same type of image lens, including an autofocus lens. The left and right image sensors may be positioned and calibrated before the digital magnification process so that the captured left and captured right images are vertically aligned. The digital magnification process preserves vertical alignment and binocular overlap (e.g., 80%). For example, assume that the left and right images each have 800 (horizontal, columns) × 600 (vertical, rows) pixels. After digital magnification, a cropped left image is generated using pixels from row 201 to row 400 and column 401 to column 600 of the left image, and a cropped right image is generated using rows 201 to row 400 and column 201 to column 400 of the right image. This cropping may produce sufficient binocular overlap (e.g., 80%). Non-concentric cropping in digital magnification combined with resizing may allow magnification while preserving both binocular overlap and vertical alignment. Similarly, as the system increases to a higher level of digital magnification, further non-concentric cropping is performed in combination with resizing on at least one of the images (e.g., the left or right image) to allow magnification while preserving both binocular overlap and vertical alignment.

[0041] In another example, machine learning algorithms are used to determine the center of cropping for the left image, the center of cropping for the right image, or the center of both during the digital magnification process. In one embodiment, machine learning-based object recognition and localization (e.g., recognizing a surgical field, or recognizing a surgical instrument, or recognizing tissue) can determine at least one center of cropping. For example, based on the left image, an operating table is recognized and localized, and its position within the operating table (e.g., the center of gravity) is assigned as the center of cropping for the left image, and based on the center of cropping for the left image and the desired binocular overlap to be maintained, the center of cropping for the right image is calculated.

[0042] In one embodiment, supervised learning can be implemented. In another embodiment, unsupervised learning can be implemented. In yet another embodiment, reinforcement learning can be implemented. It should be understood that feature learning, sparse dictionary learning, anomaly detection, and association rules can also be implemented. Various models can be implemented for machine learning. In one embodiment, artificial neural networks are used. In another embodiment, decision trees are used. In yet another embodiment, support vector machines are used. In yet another embodiment, Bayesian networks are used. In yet another embodiment, genetic algorithms are used.

[0043] In yet another example, neural networks, convolutional neural networks, or deep learning are used for object recognition, image classification, object localization, image segmentation, image registration, or a combination thereof. Neural network-based systems are often advantageous for the tasks of image segmentation, recognition, and registration.

[0044] In one example, a U-Net with a reduction path and an expansion path is used. The reduction path has consecutive convolutional layers and a maximum pooling layer. The expansion path performs upconversion and may have a convolutional layer. The convolutional layer before the output maps the feature vector to the required number of target classes in the final segmentation output. In one example, a V-Net is implemented for image segmentation to separate organs or tissues of interest (e.g., vertebrae). In one example, an autoencoder-based deep learning architecture is used for image segmentation to separate organs or tissues of interest. In one example, backpropagation is used to train the neural network.

[0045] In yet another example, deep residual learning is performed for image recognition, image segmentation, or image registration. The residual learning framework is used to facilitate network training. Multiple layers are implemented to learn residual functions that reference the layer inputs, rather than learning non-reference functions. One example of a network that performs deep residual learning is the deep residual network, or ResNet.

[0046] In another embodiment, a Generative Adversarial Network (GAN) is used for image recognition, or image segmentation, or image registration. In one example, a GAN performs image segmentation to separate organs or tissues of interest. In a GAN, a generator is implemented via a neural network to model a transformation function that takes a random variable as input and follows a target distribution when trained. A discriminator is simultaneously implemented via another neural network to distinguish between generated data and true data. In one example, the first network attempts to maximize the final classification error between generated data and true data, and the second network attempts to minimize the same error. After iterations of the training process, both networks may improve.

[0047] In yet another example, an ensemble method is used, employing multiple learning algorithms to achieve better predictive performance. In one embodiment, a Bayesian optimal classifier is used. In another embodiment, bootstrap aggregation is used. In yet another embodiment, boosting is used. In yet another embodiment, Bayesian parameter averaging is used. In yet another embodiment, Bayesian model integration is used. In yet another embodiment, model bucketing is used. In yet another embodiment, stacking is used. In yet another embodiment, the random forest algorithm is used. In yet another embodiment, the gradient boosting algorithm is used.

[0048] The controller 310 can determine the distances from the target to the first image sensor 330b and the second image sensor 330a. Based on the distances of the first image sensor 330b and the second image sensor 330a from the target, the controller 310 can perform cropping of the first and second images so as to maintain vertical alignment and binocular overlap for each digital magnification level.

[0049] In another embodiment, the system allows the user to determine the center of cropping for the left image, the center of cropping for the right image, or the center of both, for the digital enlargement process. If there are multiple users, each may have their own settings.

[0050] The display 320 may include one of a plurality of wearable displays that display a resized cropped first image and a resized cropped second image, and then, after digital enlargement is performed, display a 3D image of the target including the binocular overlap of the first and second images that have been vertically aligned to satisfy an overlap threshold. In one embodiment, the first image sensor 330b and the second image sensor 330a may be positioned close to the display 320 for the user to perform a surgical procedure on the target, which is a patient. In another embodiment, the first image sensor 330b and the second image sensor 330a may be positioned near the display 320 for the user to perform a surgical procedure on the target, which is a patient. In yet another example, the first image sensor 330b and the second image sensor 330a may be positioned on a stand, not adjacent to the display 320. It should be understood that the stand may be motorized or robotic. The display 320 may be a 3D monitor, a 3D projector, or a 3D projector with a coupler, used in conjunction with 3D glasses (e.g., polarizer or active shutter glasses).

[0051] The present invention discloses a method for digitally enlarging an image while preserving the binocular overlap. In one embodiment, cropping of the left image and cropping of the right image are performed by a controller 310 while preserving the binocular overlap. For example, if the original binocular overlap of the original left and right images is 75%, the left image to be cropped and the right image to be cropped can be cropped by the controller 310 so that the binocular overlap of the cropped images is also 75%.

[0052] In one embodiment, a left image sensor 330b having a left lens 340b worn by the user can capture a left image. A right image sensor 330a having a right lens 340a worn by the user can capture a right image. The left and right images can be provided to a controller 310. The controller 310 can calculate a left crop function that specifies how to crop the left image and a right crop function that specifies how to crop the right image. The left and right crop functions preserve the overlap of both eyes and the vertical alignment of both eyes. The controller 310 can crop the left image to produce a cropped left image using the left crop function that preserves the overlap of both eyes and the vertical alignment of both eyes. The controller 310 can crop the right image to produce a cropped right image using the right crop function that preserves the overlap of both eyes and the vertical alignment of both eyes.

[0053] Controller 310 may resize the cropped left image to generate a cropped and resized left image. Controller 310 may resize the cropped right image to generate a cropped and resized right image. A user-worn display 320 may display the cropped and resized left image to the user's left eye. Display 320 may display the cropped and resized right image to the user's right eye. In one embodiment, display 320 may be a near-eye 3D display. In another embodiment, display 320 may be a 3D monitor, a 3D projector, or a 3D projector with a coupler used in conjunction with 3D glasses (e.g., polarizer or active shutter glasses).

[0054] Figure 7 shows a schematic diagram of the retention of the binocular overlap and binocular vertical alignment configuration 700. The left cropped image and the right cropped image have 75% binocular overlap and vertical alignment at 2.3x, 5.3x, and 12x magnification, respectively, which may provide the user with an improved 3D visualization experience and depth perception. The retention of the binocular overlap and binocular vertical alignment configuration 700 includes the right cropped image 710b and the left cropped image 710a.

[0055] In another embodiment, the digital magnification method further includes an additional condition to be satisfied: namely, that the left cropped image shares the same geometric center as the original left image. The right cropped image may be calculated by the controller 310 and generated by the controller 310 based on the cropping of the left cropped image, while preserving binocular overlap and binocular vertical alignment. The advantages of this implementation are that the digital magnification process can be coaxial along the center (optical axis) of the left image, and the progress of digital magnification can be aligned with the user's left eye line of sight. Alternatively, the cropped right image may share the same center as the original right image. The left cropped image may be calculated by the controller 310 and generated by the controller 310 based on the position and cropping of the right cropped image, while preserving binocular overlap and binocular vertical alignment.

[0056] In another embodiment, the acceptable binocular overlap of the cropped image may be specified as a range rather than a specific numerical value. For example, the binocular overlap between the cropped left and right images may be specified to be within the range of 60% to 90%. In digital magnification, any numerical value between 60% and 90% may be deemed sufficient. The left image sensor 330b, having a left lens 340b mounted by the user controller 310, can capture the left image by using the acceptable range of binocular overlap as a guideline for cropping the left and right images. The right image sensor 330a and right lens 340a, mounted by the user, can capture the right image. The left and right images may be provided to the controller 310.

[0057] Controller 310 may calculate a left crop function that specifies how to crop the left image and a right crop function that specifies how to crop the right image. The left crop function and the right crop function may preserve the vertical alignment of the two eyes. The left crop function and the right crop function may preserve the binocular overlap as specified by an acceptable range of binocular overlap, such as 60% to 90%. Controller 310 may crop the left image to produce a cropped left image using the left crop function that preserves the binocular overlap and the vertical alignment of the two eyes. Controller 310 may crop the right image to produce a cropped right image using the right crop function that preserves the binocular overlap and the vertical alignment of the two eyes. Controller 310 may resize the cropped left image to produce a cropped and resized left image. Controller 310 may resize the cropped right image to produce a cropped and resized right image. Display 320 may display the cropped and resized left image to the user's left eye. Display 320 may display the cropped and resized right image to the user's right eye.

[0058] In another embodiment, the left lens 340b and the right lens 340a may be zoom lenses. By changing the focal length and angle of view of the zoom lenses, optical zoom may be possible. Thus, optical zoom may be used in combination with the digital magnification method described above. For example, 5.3x digital magnification may be used in combination with 2x optical zoom (totaling 10.6x magnification). It should be understood that the level of digital magnification may be continuous (e.g., fine increments over a range, e.g., magnification at any magnification level between 2x and 7x) or the level of magnification may be discrete (2x, 2.5x, 3x, 4x, 6x, 7x, etc.). In another embodiment, the controller 310 may transmit the magnified left and / or right images to another 3D display device for visualization. The 3D display device may be a wearable display, a monitor, a projector, a projector with a coupler, a passive 3D monitor with 3D polarizing glasses, an active 3D monitor with active shutter 3D glasses, or a combination thereof. In yet another embodiment, the controller 310 may transmit the enlarged left image and / or right image to another computer for visualization, storage, and broadcasting. In yet another embodiment, the controller 310 may record the enlarged left image and / or enlarged right image.

[0059] In yet another embodiment, the controller 310 may apply computer vision and / or image processing techniques to the magnified left image and / or the magnified right image. Additional computer vision analysis enables decision assistance, object recognition, image registration, and object tracking. For example, deep learning and neural networks may be used. In yet another embodiment, the near-eye 3D display 320 may display other medical image data (e.g., CT, MRI, ultrasound, nuclear medicine, surgical navigation, fluoroscopy, etc.) to the user, and the other medical image data is superimposed on the magnified left image and / or the magnified right image. It should be further understood that more than three image sensors may be used in the system. In one example, if there are more than three image sensors, at any given time, only two image sensors are selected to participate in the digital magnification process (e.g., three color sensors with three lenses). In the case of multiple image sensors and image lenses, it should be understood that multiple sets of two of these sensors may be calibrated against each other in separate processes.

[0060] In one embodiment, only one image sensor is used. This image sensor functions as both a left image sensor 330a and a right image sensor 330b. In another embodiment, a 3D scanning unit is used, which includes a projector and an image sensor, similar to a 3D scanner. In this way, a 3D scan can be generated. The 3D scanning unit may use epipolar geometry for the 3D scan. By using different virtual viewpoints and projection angles, virtual left and virtual right images can be generated based on the 3D scan. The aforementioned digital magnification process can be applied to the virtual left and virtual right images.

[0061] Devices and systems for digital augmentation and 3D augmented reality display. Figure 8 shows a schematic diagram of a physical embodiment of the digital magnification wearable device configuration 800. The digital magnification wearable device configuration 800 includes a right image sensor 330a, a left image sensor 330b, a right lens 340a, a left lens 340b, a right near-eye display 320a, a left near-eye display 320b, and an eyeglass frame 350. It should be understood that the wearable frame may be in the form of a head-mounted device instead of an eyeglass frame. It should be understood that the controller 310 may be a microcontroller, computer, FPGA, or ASIC. The digital magnification wearable device configuration 800 can perform digital magnification while maintaining binocular overlap and binocular vertical alignment.

[0062] In one embodiment, the digital magnifying wearable device configuration 800 may further include transparent plastic or glass surrounding the left near-eye display 320b and the right near-eye display 320a. For example, the digital magnifying wearable device configuration 800 may use a compact offset configuration, thereby making only a portion of the area in front of each eye opaque and the other portion transparent. In one example, the central portion of the area in front of each eye is opaque and the peripheral portion is transparent. In this way, a user such as a surgeon / dentist can look around the digital display near their eyes and see the patient with unobstructed natural vision. In one embodiment, the digital magnifying wearable device configuration 800 may further include prescription glasses so that myopia, hyperopia, and astigmatism can be corrected.

[0063] In another embodiment, the digital magnification wearable device configuration 800 may include an optical see-through configuration. Both near-eye 3D displays 320(a-b) are transparent or translucent. In one embodiment, the image sensor 330(a-b) may be a pair of color image sensors. Thus, the digital magnification wearable device configuration 800 can digitally magnify a stereoscopic color image and display it in 3D to the user on the near-eye 3D display 320(a-b). In one example, the left and right lenses 340(a-b) are lenses with a fixed focal length. In another example, the left and right lenses 340(a-b) are zoom lenses with a variable focal length. In yet another example, the color image sensor may be a complementary metal-oxide-semiconductor (CMOS) image sensor. In yet another example, the color image sensor may be a charge-coupled device (CCD) image sensor. In one example, the left and right color image sensors are combined with an autofocus lens that enables autofocus.

[0064] In one embodiment, only one image sensor is used. This image sensor functions as both a left image sensor 330a and a right image sensor 330b. In another embodiment, a 3D scanning unit is used, which includes a projector and an image sensor, similar to a 3D scanner. In this way, a 3D scan can be generated. The 3D scanning unit may use epipolar geometry for the 3D scan. By using different virtual viewpoints and projection angles, virtual left and virtual right images can be generated based on the 3D scan. The aforementioned digital magnification process may be applied to the virtual left and virtual right images. The 3D scanning unit may use visible wavelengths, infrared wavelengths, ultraviolet wavelengths, or a combination thereof.

[0065] The aforementioned 3D scanning unit may project dynamic projection patterns to facilitate 3D scanning. Some examples of dynamic patterns are binary codes, stripe boundary codes, and Miere patterns. In one embodiment, a binary codeword is represented by a series of black and white stripes. If black represents 1 and white represents 0, then a series of 0s and 1s at any given position can be encoded by a dynamic projection pattern, and the binary dynamic projection pattern can be captured and decoded by an image sensor and lens to reconstruct a binary codeword (e.g., 10100011) that encodes the position. Theoretically, N binary patterns can generate 2N different codewords for each dimension of the image (x-dimension or y-dimension). Similarly, binary coding can be extended to N-bit coding. For example, instead of the binary case where only 1s and 0s are represented by black and white, N-bit integers can be represented by intensities in between. For example, in the case of a 2-bit coding system, there are 2 × 2 = 4 different possibilities. When the maximum intensity is I, 0, 1, 2, and 3 can be represented by I, 2 / 3 × I, 1 / 3 × I, and 0, respectively. In other examples, dynamic stripe boundary code-based projection or dynamic moiré code-based projection may be performed.

[0066] In another embodiment, dynamic Fourier transform profilometry may be performed by a 3D scanning unit. In one embodiment, a periodic signal is generated to carry frequency domain information, including spatial frequency and phase. By inverse Fourier transforming only the fundamental frequency, the main phase values ​​in the range of -π to π are obtained. After spatial or temporal phase unwrapping (a process to remove 2π discontinuities and generate a continuous map), the actual 3D shape of the patient's anatomical structure can be reconstructed. Fourier transform profilometry is less affected by the patient's out-of-focus images and is therefore a suitable technique for intraoperative 3D scanning. Similarly, π-shift modified Fourier transform profilometry, in which a π-shift pattern is added to enable 3D scanning, can also be performed intraoperatively.

[0067] In another example, DC images may be used in a 3D scanning unit along with Fourier transform profilometry. By capturing the DC component, DC-modified Fourier transform profilometry can improve the quality of intraoperative 3D scanning. In yet another example, N-step phase-shift Fourier transform profilometry may be performed intraoperatively. It should be understood that the more steps (N) selected, the higher the accuracy of the 3D scanning. For example, 3-step phase-shift Fourier transform profilometry may be performed to enable high-speed intraoperative 3D scanning. It should be understood that periodic patterns such as trapezoidal, sinusoidal, or triangular patterns may be used in Fourier transform profilometry for intraoperative 3D scanning. It should also be understood that windowed Fourier transform profilometry, 2D Fourier transform profilometry, or wavelet Fourier transform profilometry can also be performed with the aforementioned devices and systems. It should be understood that in modified Fourier transform profilometry, multiple frequencies (e.g., dual frequencies) of the periodic signal may be used so that phase unwrapping is optional during intraoperative 3D scanning. The dynamic Fourier transform profilometry and modified Fourier transform profilometry described herein can improve the quality of 3D scanning of patients. Improved 3D scanning enhances image registration between intraoperative 3D scans and preoperative images (e.g., MRI and CT), thereby improving surgical navigation.

[0068] In yet another embodiment, the aforementioned 3D scanning unit performs Fourier transform profilometry or modified Fourier transform profilometry in combination with binary codeword projection. Fourier transform profilometry and binary codeword projection may be performed sequentially, simultaneously, or in combination thereof. The combined method may improve the accuracy of the 3D scan, but it will impair the speed of the 3D scan.

[0069] In another embodiment, the aforementioned projector may include at least one lens configured to defocus the projected pattern. The defocusing process by the lens is analogous to the convolution of a Gaussian filter on a binary pattern. As a result, the defocused binary pattern may create a periodic pattern similar to a sinusoidal pattern.

[0070] In another example, dithering techniques are used to generate high-quality periodic fringe patterns by binarizing higher-order bit fringe patterns (e.g., 8 bits), such as sinusoidal fringe patterns. In one example, ordered dithering is performed, and for example, a Bayer matrix may be used to enable ordered dithering. In another example, error diffusion dithering is performed, and for example, Floyd-Steinberg (FS) dithering or minimized mean error dithering may be performed. It should be understood that in some cases, dithering techniques may be performed in combination with defocusing techniques to improve the quality of intraoperative 3D scanning.

[0071] In another example, the aforementioned projector may generate statistical patterns. For instance, the projector may generate a pseudo-random pattern containing multiple dots. The position of each corresponding dot in the pseudo-random pattern may be predetermined by the projector. The projector may project the pseudo-random pattern onto a patient or target. The position of each corresponding dot in the pseudo-random pattern is projected onto the corresponding position on the patient / target. An image sensor may acquire intraoperative 2D images of multiple object points related to the patient / target in order to calculate 3D topography.

[0072] The controller 310 may associate each object point associated with the patient captured by the image sensor with a corresponding dot in a pseudo-random pattern projected onto the patient / target by the projector, based on the position of each corresponding dot predetermined by the projector. Based on the association of each object point with each position of each corresponding dot in the pseudo-random pattern predetermined by the projector, the controller 310 may convert the 2D image into a 3D scan of the patient / target. In one example, the projector may include one or more end-emitting lasers, at least one collimating lens, and at least one diffractive optical element. The end-emitting lasers and diffractive optical element may be controlled by the controller 310 to generate a pattern desirable for a particular 3D scanning application.

[0073] It should be understood that near-eye 3D displays may include LCD (liquid crystal) microdisplays, LED (light-emitting diode) microdisplays, organic LED (OLED) microdisplays, liquid crystal on silicon (LCOS) microdisplays, retinal scanning displays, virtual retinal displays, optical see-through displays, video see-through displays, convertible video optical see-through displays, wearable projection displays, and projection displays. In another example, the digital magnification wearable device configuration 800 may further include a light source for surgical field illumination. In one example, the light source is based on one or more light-emitting diodes (LEDs). In another example, the light source is based on one or more laser diodes with waveguides or optical fibers. In another example, the light source has a diffuser. In another example, the light source has a non-coherent light source such as an incandescent lamp. In yet another example, the light source has a coherent light source such as a laser diode and a phosphorescent material in film or volume form. In yet another embodiment, the light source is attached to a surgical instrument to illuminate a body cavity.

[0074] In another embodiment, the image sensor 330(a-b) is a pair of monochrome sensors. The system further includes at least one fluorescence emission filter. Thus, the digital magnification surgical loupe configuration can digitally magnify stereoscopic fluorescence images and display them in 3D to the user on the near-eye 3D display 320(a-b). The system further includes a light source capable of providing excitation light to the surgical field. It should also be understood that the light source may include laser light, light-emitting diodes (LEDs), incandescent lamps, projector lamps, arc lamps such as xenon, xenon-mercury, or metal halide lamps, and coherent or incoherent light sources. In one example, the light source includes one or more white LEDs with a low-pass filter (e.g., a 775 nm short-pass filter) and one or more near-infrared LEDs with a band-pass filter (e.g., an 830 nm band-pass filter). In another example, the light source includes one or more white LEDs with a low-pass filter (e.g., a 775 nm short-pass filter) and one or more near-infrared LEDs with a long-pass filter (e.g., an 810 nm long-pass filter). In one example, the light source may be controlled by sensors, such as an inertial measurement unit, to switch the lights on and off.

[0075] In another embodiment, the digital magnification wearable device configuration 800 includes at least two color image sensors, at least two monochrome image sensors, at least two beam splitters, and at least two narrowband filters. The monochrome image sensors, color sensors, and beam splitters are optically aligned on both sides (left and right) so that the left color image is aligned with the left monochrome image and the right color image is aligned with the right monochrome image. It should be understood that the beam splitters may be cube beam splitters, plate beam splitters, pellicle beam splitters, dichroic beam splitters, or polarizing beam splitters. It should be understood that the optical design may be a foldable configuration using mirrors.

[0076] In another example, the digital magnification wearable device configuration 800 includes a light source with an additional spectral filter. The digital magnification wearable device configuration 800 may be used to capture a narrowband reflectance or fluorescence image, digitally magnify the image, and display it to the user in 3D with a desired binocular overlap. For example, the light source may be a plurality of white LEDs and near-infrared LEDs (770 nm), and the spectral filter may be an 800 nm short-pass filter. In another embodiment, the device further includes additional sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, magnetometer, proximity sensor, microphone, force sensor, and ambient light sensor. In one example, the light source may be controlled by a sensor such as an inertial measurement unit to switch the light on and off. In another example, system 300 may be controlled by a sensor such as an inertial measurement unit and / or proximity sensor to switch system 300 on and off. Some examples of types of proximity sensors are photoelectric sensors, inductive sensors, capacitive sensors, and ultrasonic sensors.

[0077] In one embodiment, the digital augmentation wearable device configuration 800 further includes at least one microphone. System 300 may record audio data, such as dictation. System 300 uses the microphone to capture the audio data, performs speech recognition on the controller 310, and enables voice control of System 300. In one embodiment, voice control may include adjustment of the augmentation level (e.g., from 3x to 5x). In one example, a microphone array or multiple microphones are used, and the system may triangulate sound sources for multiple purposes, such as noise cancellation and voice control of multiple nearby devices. System 300 may distinguish one user from another based on the triangulation of the voice / audio signals. In yet another embodiment, the digital augmentation wearable device configuration 800 further includes tracking hardware, such as optical tracking hardware and electromagnetic tracking hardware. In yet another embodiment, the digital augmentation wearable device configuration 800 further includes communication hardware that enables wireless or wired communication, such as Wi-Fi, Bluetooth, cellular communication, Ethernet®, LAN, wireless communication protocols compatible with operating rooms, and infrared communication. Therefore, the device can stream the magnified data and / or original image data captured by the image sensor to another device, computer, or mobile device. In yet another embodiment, the lens 340(a-b) in the digital magnifying wearable device configuration 800 includes an autofocus lens.

[0078] In yet another embodiment, the lenses 340(a-b) within the digital magnifying wearable device configuration 800 are autofocus lenses, but the digital magnifying wearable device configuration 800 can focus the lenses upon user request. For example, upon user request via an input device or voice control, the lenses are focused upon user request. Therefore, autofocus is not activated unless requested by the user, thus avoiding unnecessary autofocus during surgical procedures. In one example, the focus settings for the left lens 340b and the right lens 340a are always the same. For example, to avoid the left lens focusing on a different focal plane than the right lens, the setting for focusing the left lens 340b and the setting for the right lens 340a are set to be the same.

[0079] In yet another embodiment, the digital magnification wearable device configuration 800 may further include additional input devices such as foot pedals, wired or wireless remote controls, one or more buttons, touch screens, microphones with voice control, and gesture control devices such as Microsoft Kinect. It should be understood that the controllers may be usable or disposable. It should be understood that sterile sheets or wraps may be placed around the input devices. In yet another embodiment, the digital magnification wearable device configuration 800 may display medical images such as MRI (magnetic resonance imaging) image data, computed tomography (CT) image data, positron emission tomography (PET) image data, single-photon emission computed tomography (SPECT), PET / CT, SPECT / CT, PET / MRI, gamma scintigraphy, radiography, and ultrasound. In yet another embodiment, the digital magnification wearable device configuration 800 may include digital storage hardware that enables recording of magnified data, and / or original image data from an image sensor, and / or audio data, and / or other sensor data.

[0080] Image stabilization In one embodiment, electronic image stabilization (EIS) is performed by controller 310. Controller 310 shifts the electronic image from frame to frame of the left video captured by the left camera and the right video captured by the right camera, to a degree sufficient to cancel out motion. During digital magnification, EIS uses pixels outside the boundaries of the cropped area to provide a buffer for motion. In one embodiment, subsequent frames may be tracked using optical flow or other image processing methods to detect and correct oscillating motion. In another embodiment, a feature matching image stabilization method may be used. Image features may be extracted via SIFT, SURF, ORB, BRISK, neural networks, etc.

[0081] In another example, optical image stabilization (OIS) is implemented. In one embodiment, OIS is located in lenses 340a and 340b. For example, using springs and mechanical mounting bases, the movement of the image sensors may be smoothed or canceled out. In one embodiment, image sensors 330a and 330b may be moved to cancel out the movement of the camera.

[0082] In yet another example, mechanical image stabilization (MIS) is implemented. A gimbal may be used for MIS. In one example, MIS is achieved by attaching a gyroscope to the system. The gyroscope is an external gyroscope (gimbal) that stabilizes image sensors 330a and 330b.

[0083] Stereoscopic calibration System 300 may require stereoscopic calibration to enable accurate 3D digital magnification. In one example, a single calibration is performed on the left and right sensors (by repeatedly capturing similar calibration patterns, such as fiducial or chessboard), based on the fact that the initial homography transformation and cropping are applied to a pair of images to achieve highly accurate alignment between the two images performed, after mechanical fixing to achieve vertical calibration. This is similar to finding the epipolar geometry between the two sensors and calibrating the two frames into a single plane to obtain (1) the same scale of the captured geometry with virtually the same focal length, (2) the same peripheral alignment of the captured scene by distortion removal, and (3) the same vertical alignment of the captured frame by homography (projection) transformation. The new calibrated frame (corrected frame) can then be used in the subsequent digital 3D magnification and visualization process, as described above.

[0084] Ergonomic calibration In one embodiment, ergonomic calibration may be performed on system 300 using one or more IMUs, one on the image sensor axis and the second on the display axis. Two important objectives are achieved when capturing and displaying digital images, and the headset is horizontally aligned to the center of the forehead (single IMU reading and correction). This requires that each corresponding eye has a symmetrical mechanical position (left sensor 330a for the left eye, right sensor 330b for the right eye). It also helps to maintain binocular overlap between the digitally magnified image captured and overlapped at the centers of the two image sensors and the centers of the two eyes perceived by natural vision around the display (by comparing and aligning the two IMUs).

[0085] Autofocus and on-demand autofocus Autofocus can be achieved via mechanical structures such as motors / actuators or via liquid lenses. In one example, controller 310 may perform a luminance evaluation by means of a Sobel filter, or a similar filter that extracts edges and high-frequency features from the left and / or right images, to find high-contrast images, high-frequency values, etc. The autofocus lens may test a wide range of focus (coarse focus) to find the coarse focus, and then perform a narrower range of focus (fine focus) based on the vicinity of the coarse focus. In one example, the right lens 340a and the left lens 340b may be assigned to the two ends of the focus range and progress toward the center. Once the optical focus value is found, both lenses are assigned the same or similar value to avoid the two lenses focusing on different image planes.

[0086] In another example, the controller 310 may perform calibration and use a disparity map to determine the working distance to a desired object. The controller 310 may use a previously calibrated frame to extract a partial or complete disparity or depth map. The controller 310 may then use a region or point of interest in a specific part of the image to evaluate the distance (working distance) to the desired object or working plane, and use that distance to determine an appropriate autofocus value from either a distance-dependent equation or a predetermined look-up table (LUT).

[0087] Additional methods for maintaining binocular overlap during digital magnification The binocular overlap can be defined as a variable for working distance and magnification level. By detecting and calculating the patient / target working distance from image sensors 330a and 330b, the controller 310 can define the distance to the point of interest or the average working distance of the region of interest, and then define an appropriate value for the binocular overlap between binocular visions to achieve proper 3D visualization from either a distance-dependent equation or a predetermined lookup table (LUT). In one example, the distance may be estimated using calibration and disparity maps to find the distance. A partial or complete disparity map or depth map (these two are related but have different values) is extracted using a previously calibrated frame. The controller 310 can then use the region of interest or point of interest in a specific part of the image to extract the distance to the desired object or working plane (working distance). In another example, the controller 310 may use the autofocus values ​​of the left autofocus lens and / or the right autofocus lens to embed the working distance.

[0088] controller The controller 310 comprises the hardware and software necessary to implement the method described herein. In one embodiment, the controller 310 includes a computer-readable medium containing processor-executable instructions configured to implement one or more of the techniques presented herein. An exemplary embodiment of a computer-readable medium or computer-readable device includes a computer-readable medium such as an SSD, CD-R, DVD-R, flash drive, or hard disk drive platter on which computer-readable data is encoded. This computer-readable data, such as binary data containing at least one of 0 or 1, also includes a set of computer instructions configured to operate according to one or more of the principles described herein. In some embodiments, the set of computer instructions is configured to perform a method such as, for example, at least some of the exemplary methods described herein. In some embodiments, the set of computer instructions is configured to implement a system such as, for example, at least some of the exemplary systems described herein. Many such computer-readable mediums configured to operate according to the techniques presented herein can be devised by those skilled in the art.

[0089] The following description provides a brief and general description of computing environments suitable for implementing one or more embodiments of the provisions described herein. Examples of computing devices include, but are not limited to, graphics processing unit (GPU) server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), and media players), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, microcontrollers, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and distributed computing environments including any of the systems or devices described above. In one embodiment, a controller may use a heterogeneous computing configuration.

[0090] While not mandatory, the examples are described in the general context of "computer-readable instructions" being executed by one or more computing devices. Computer-readable instructions can be distributed via computer-readable media. Computer-readable instructions can be implemented as program components such as functions, objects, application programming interfaces (APIs), and data structures that perform specific tasks or implement specific abstract data types. Typically, the functionality of computer-readable instructions can be combined or distributed as desired in various environments.

[0091] In one embodiment, the system comprises a computing device configured to implement one or more embodiments provided herein. In one configuration, the computing device includes at least one processing unit and one memory unit. Depending on the exact configuration and type of the computing device, the memory unit may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory), or any combination of the two. In other embodiments, the computing device may include additional features and / or functions. For example, the computing device may also include additional storage (e.g., removable and / or non-removable), including but not limited to cloud storage, magnetic storage, optical storage, etc. In one embodiment, the storage may contain computer-readable instructions for implementing one or more embodiments provided herein. The storage may also store other computer-readable instructions for implementing operating systems, application programs, etc. Computer-readable instructions may be loaded into memory for execution by the processing unit, for example.

[0092] As used herein, the term “computer-readable media” includes computer storage media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any way or technique for storing information such as computer-readable instructions or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Versatile Disk (DVD) or other optical storage devices, or any other media that can be used to store desired information and can be accessed by computing devices.

[0093] A computing device may also include communication connections that enable the computing device to communicate with other devices. Communication connections include, but are not limited to, modems, network interface cards (NICs), integrated network interfaces, radio frequency transmitters / receivers, infrared ports, USB connections, or other interfaces for connecting a computing device to other computing devices. Communication connections may include wired or wireless connections. Communication connections may transmit and / or receive communication media.

[0094] A computing device may include input devices such as keyboards, mice, pens, voice input devices, touch input devices, infrared cameras, depth cameras, touchscreens, video input devices, and / or any other input devices. Output devices such as one or more displays, speakers, printers, and / or any other output devices may also be included in a computing device. Input and output devices may be connected to the computing device via wired connections, wireless connections, or any combination thereof. In one embodiment, an input or output device from another computing device may be used as an input or output device for a computing device.

[0095] The components of computing device 6712 may be connected by various interconnections, such as buses. Such interconnections may include Peripheral Component Interconnects (PCI) such as PCI Express, Universal Serial Bus (USB), FireWire (IEEE 1394), and optical bus structures. In another embodiment, the components of the computing device may be interconnected by a network. For example, memory may consist of multiple physical memory units located in different physical locations and interconnected by a network.

[0096] Those skilled in the art will understand that storage devices used to store computer-readable instructions may be distributed across a network. For example, a computing device accessible over a network may store computer-readable instructions for implementing one or more embodiments provided herein. A computing device may access another computing device and download some or all of the computer-readable instructions for execution. Alternatively, the first computing device may download some of the computer-readable instructions as needed, or some instructions may be executed on the first computing device and some on the second computing device.

[0097] This specification provides various operations of the embodiments. In one embodiment, one or more of the operations described may constitute a computer-readable instruction stored in one or more computer-readable media, which, when executed by a computing device, causes the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as meaning that these operations are not necessarily dependent on that order. Alternative orderings will be understood by those skilled in the art who benefit from this description. Furthermore, it will be understood that not all operations are necessarily present in every embodiment provided herein. Also, it will be understood that not all operations are necessary in some embodiments.

[0098] conclusion It should be understood that the section describing embodiments for carrying out the invention, rather than the abstract section, is intended to be used to interpret the claims. The abstract section may describe one or more exemplary embodiments, not all embodiments of the present disclosure, and is therefore not intended to limit the scope of the present disclosure and the accompanying claims in any way.

[0099] In the above, this disclosure is described using functional building blocks that demonstrate the implementation of the specified functions and their relationships. The boundaries of these functional building blocks are arbitrarily defined herein for the sake of explanation. Alternative boundaries may be defined, as long as the specified functions and their relationships are adequately performed.

[0100] Those skilled in the art will see that various modifications may be made to the form and details without departing from the spirit and scope of this disclosure. Therefore, this disclosure should not be limited by any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A system for generating a 3D image from a captured image of a target, such that when digital enlargement is performed on the captured image, the generated 3D image of the target is maintained after digital enlargement, A first image sensor configured to capture a first image of the target, A second image sensor configured to capture a second image of the target, Performing digital magnification on the first image captured by the first image sensor and the second image captured by the second image sensor, Cropping the first image and the second image such that the first portion of the target captured by the first image sensor overlaps with the second portion of the target captured by the second image sensor, wherein the first portion of the target overlaps with the second portion of the target. The cropping of the first and second images is adjusted to provide a binocular overlap between the first and second portions of the target, wherein the binocular overlap between the first and second images is an overlap threshold, and when the overlap threshold is met, a 3D image of the target is obtained that is displayed to the user after the digital magnification is performed. Commanding the display to show the user the cropped first image and the cropped second image including the overlapping portion of both eyes, wherein the displayed cropped first image and cropped second image are to be displayed to the user as a 3D image with digital enlargement. A controller configured to perform the following actions A system that includes these features.

2. The aforementioned controller The cropped first image is resized to the original size of the first image captured by the first image sensor, and the cropped second image is resized to the original size of the second image captured by the second image sensor, wherein the resized cropped first image and the resized cropped second image include the overlapping portion of the first and second images. Commanding the display to show the user the resized cropped first image and the resized cropped second image including the overlapping portion of both eyes, wherein the displayed resized cropped first image and the resized cropped second image display the 3D image to the user in the digitally enlarged form. The system according to claim 1, further configured to perform the following:

3. The aforementioned controller Cropping the first image captured by the first image sensor and the second image captured by the second image sensor in order to vertically align the overlapping portion of the first portion of the target and the second portion of the target, wherein when the first plurality of vertical coordinates of the cropped first image are aligned with the corresponding vertical coordinates from the second plurality of coordinates of the cropped second image, the cropped first image is in a state where it is vertically aligned with the cropped second image. The cropping of the first and second images is adjusted to provide a binocular overlap between the first and second portions of the target, such that the binocular overlap between the first and second images is vertically aligned to satisfy the overlap threshold in order to generate a 3D image of the target that is displayed to the user after the digital magnification is performed. The system according to claim 2, further configured to perform the following:

4. The aforementioned controller The binocular overlap portion generated by performing a first digital magnification at a first digital magnification level on the first image captured by the first image sensor and the second image captured by the second image sensor, and then adjusting the cropping of the first and second images to satisfy the overlap threshold, Performing a second digital magnification at a second digital magnification level on the first image captured by the first image sensor and the second image captured by the second image sensor, wherein the second digital magnification level is increased from the first digital magnification level. When performing the second digital magnification, the binocular overlap portion generated after performing the first digital magnification at the first digital magnification level on the first and second images is maintained. The system according to claim 3, further configured to perform the following:

5. The aforementioned controller After performing the previous digital magnification on the first image and the second image at their respective previous digital magnification levels, the overlapping portion of the binoculars and the vertical alignment determined when performing the first digital magnification on the first image and the second image at the first digital magnification level are maintained. The method involves performing the first digital magnification on the first image and the second image at the first digital magnification level for each subsequent digital magnification level, and then adjusting the cropping of the first image and the second image to satisfy the overlap threshold, thereby continuing to maintain the binocular overlap and vertical alignment determined by this adjustment, wherein each subsequent digital magnification level is increased from each previous digital magnification level. The system according to claim 4, further configured to perform the following:

6. The aforementioned controller Performing a first digital enlargement at a first digital enlargement level on the non-concentric portions of the first image and the non-concentric portions of the second image, wherein the first digital enlargement is performed such that the non-concentric portions of the first image and the second image are portions of the first image and the second image that are different from the centers of the first image and the second image. The cropping of the first and second images is adjusted to provide a binocular overlap between the non-concentric portion of the first image and the non-concentric portion of the second image, such that the binocular overlap between the non-concentric portion of the first image and the non-concentric portion of the second image satisfies the overlap threshold. The method involves continuing to crop the non-concentric portions of the first image and the non-concentric portions of the second image for each subsequent digital enlargement at each subsequent digital enlargement level, such that the binocular overlap between the non-concentric portions of the first image and the non-concentric portions of the second image is maintained from the first digital enlargement at the first digital enlargement level. The system according to claim 4, further configured to perform the following:

7. The aforementioned controller Determining the distance between the first image sensor and the second image sensor from the target, Based on the distances of the first and second image sensors from the target, the cropping of the first and second images is performed for each digital magnification level, maintaining the vertical alignment and the binocular overlap. The system according to claim 4, further configured to perform the following:

8. The system according to claim 4, further comprising one of a plurality of wearable displays that displays the resized cropped first image and the resized cropped second image, and after the digital enlargement is performed, displays the 3D image of the target including the binocular overlap portion of the first image and the second image that have been vertically aligned to satisfy the overlap threshold.

9. The system according to claim 4, further comprising a display configured to display the resized cropped first image and the resized cropped second image, thereby displaying the 3D image of the target including the binocular overlap portion of the first and second images that have been vertically aligned to satisfy the overlap threshold after the digital enlargement has been performed.

10. The system according to claim 5, wherein the overlap threshold is satisfied if the binocular overlap includes a 75% overlap between the first image and the second image and is maintained for each subsequent digital magnification at each subsequent digital magnification level.

11. A method for generating a 3D image from a captured image of a target, such that when digital enlargement is performed on the captured image, the generated 3D image of the target is maintained after digital enlargement, The first step of capturing a first image of the target with the first image sensor, The steps include capturing a second image of the target with the second image sensor, The controller performs digital magnification on the first image of the target captured by the first image sensor and the second image of the target captured by the second image sensor. A step of cropping the first image and the second image such that a first portion of the target captured by the first image sensor overlaps with a second portion of the target captured by the second image sensor, wherein the first portion of the target partially or completely overlaps with the second portion of the target. A step of adjusting the cropping of the first image and the second image so as to provide a binocular overlap between the first and second portions of the target, wherein the binocular overlap between the first and second images is an overlap threshold, and when the overlap threshold is met, a 3D image of the target is obtained that is displayed to the user after the digital magnification is performed. A step of commanding a display to show the user the cropped first image and the cropped second image including the overlapping portion of both eyes, wherein the displayed cropped first image and cropped second image display the 3D image to the user with digital enlargement. Methods that include...

12. A step of resizing the cropped first image to the original size of the first image captured by the first image sensor, and resizing the cropped second image to the original size of the second image captured by the second image sensor, wherein the resized cropped first image and the resized cropped second image include the binocular overlap portion of the first image and the second image. A step of commanding the display to display to the user the resized cropped first image and the resized cropped second image including the binocular overlap, wherein the displayed resized cropped first image and the resized cropped second image display the 3D image to the user in the digitally enlarged state. The method according to claim 11, further comprising:

13. A step of cropping the first image captured by the first image sensor and the second image captured by the second image sensor in order to vertically align the overlapping portion of the first portion of the target and the second portion of the target, wherein when each vertical coordinate of the cropped first image is aligned with the corresponding vertical coordinate of the cropped second image, the cropped first image is in a state where it is vertically aligned with the cropped second image. A step of adjusting the cropping of the first and second images to provide a binocular overlap between the first and second portions of the target, wherein the binocular overlap between the first and second images is vertically aligned to satisfy the overlap threshold in order to generate a 3D image of the target that is displayed to the user after the digital magnification is performed. The system according to claim 12, further comprising:

14. The steps include: performing a first digital magnification at a first digital magnification level on the first image captured by the first image sensor and the second image captured by the second image sensor, and then maintaining the binocular overlap portion generated by adjusting the cropping of the first and second images to satisfy the overlap threshold; A step of performing a second digital magnification at a second digital magnification level on the first image captured by the first image sensor and the second image captured by the second image sensor, wherein the second digital magnification level is increased from the first digital magnification level. When performing the second digital magnification, the binocular overlap portion generated after performing the first digital magnification at the first digital magnification level on the first and second images is maintained. The method according to claim 13, further comprising:

15. After performing the previous digital magnification on the first image and the second image at the respective previous digital magnification level, the steps include maintaining the binocular overlap and the vertical alignment determined when performing the first digital magnification on the first image and the second image at the first digital magnification level, A step of maintaining the binocular overlap and vertical alignment determined by adjusting the cropping of the first and second images to satisfy the overlap threshold, after performing the first digital magnification at the first digital magnification level on the first and second images for each subsequent digital magnification at each subsequent digital magnification level, wherein each subsequent digital magnification level is increased from each previous digital magnification level. The method according to claim 14, further comprising:

16. A step of performing a first digital enlargement at a first digital enlargement level on the non-concentric portions of the first image and the non-concentric portions of the second image, wherein the non-concentric portions of the first image and the second image are portions of the first image and the second image that are different from the centers of the first image and the second image. A step of adjusting the cropping of the first image and the second image so as to provide a binocular overlap between the non-concentric portion of the first image and the non-concentric portion of the second image, wherein the binocular overlap between the non-concentric portion of the first image and the non-concentric portion of the second image satisfies the overlap threshold, A step of continuing to capture the non-concentric portions of the first image and the non-concentric portions of the second image for each subsequent digital magnification at each subsequent digital magnification level, wherein the binocular overlap between the non-concentric portions of the first image and the non-concentric portions of the second image is maintained from the first digital magnification at the first digital magnification level. The method according to claim 14, further comprising:

17. The steps include determining the distance between the first image sensor and the second image sensor from the target, The steps include performing the cropping of the first and second images based on the distances of the first and second image sensors from the target, so as to maintain the vertical alignment and the binocular overlap for digital magnification at the digital magnification level, and The method according to claim 14, further comprising:

18. The method according to claim 14, further comprising displaying the resized cropped first image and the resized cropped second image on a wearable display, and after the digital enlargement is performed, displaying the 3D image of the target including the binocular overlap portion of the first image and the second image that have been vertically aligned to satisfy the overlap threshold.

19. The method according to claim 18, further comprising the step of positioning the first image sensor and the second image sensor on the wearable display for a user to perform a surgical procedure on a target, which is a patient.

20. The method according to claim 15, comprising the step of satisfying the overlap threshold if the binocular overlap includes the overlap between the first image and the second image and is maintained for each subsequent digital enlargement at each subsequent digital enlargement level.

Images