Systems and methods for generating a vascular representation in mixed reality / virtual reality

Abstract

A medical analysis system features an ultrasound probe communicably coupled to a console, the console including a processor and a memory. The memory includes augmented reality ("AR") anatomy representation logic, which can include representation logic having: (i) visualization logic configured to capture information associated with a plurality of sub-images at different layers of an ultrasound image of an anatomical element; (ii) virtual slice positioning logic configured to position and orient each sub-image based on usage parameters during emission of ultrasound signals from the ultrasound probe in order to capture the ultrasound image; (iii) virtual object combination logic configured to combine each sub-image to form a virtual representation of the anatomical element; and / or (iv) virtual object display logic configured to present the virtual representation of the anatomical element in an AR environment.

Description

[0001] Priority

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 063,709, filed August 10, 2020, which is incorporated by reference in its entirety into the present application. TECHNICAL FIELD

[0003] The present application relates to the field of medical devices, and more particularly to systems and methods for generating vascular representations in mixed reality / virtual reality. BACKGROUND

[0004] In the past, clinicians have relied on various guidance systems, such as ultrasound systems, for helping to capture and present images of vessels (e.g., veins, arteries, etc.). However, traditional ultrasound systems only provide Doppler results to construct objects, where such objects are visual images without any data features associated with these images. Thus, diagnoses regarding vessel health are based solely on manual inspection of low resolution images, which can result in an unacceptable level of inaccurate diagnoses.

[0005] Accordingly, there is a need for a system that produces mixed reality and / or virtual reality images using artificial intelligence. SUMMARY

[0006] A medical analysis system is disclosed herein, including an ultrasound probe and a console communicatively coupled to the ultrasound probe. The console includes “augmented reality” (“AR”) anatomy representation logic. The AR anatomy representation logic is configured to initiate capture of information associated with a plurality of sub-images at different longitudinal positions of an ultrasound image of an anatomical element. The AR anatomy representation logic is further configured to longitudinally position and orient each sub-image based on a usage parameter during transmission of ultrasound signals in order to capture the ultrasound image. Finally, the AR anatomy representation logic is configured to combine each sub-image to form a virtual representation of the anatomical element for presentation in an AR environment.

[0007] In some embodiments, the usage parameter includes a speed of movement of the ultrasound probe during transmission of ultrasound signals in order to capture the ultrasound image.

[0008] In some embodiments, the usage parameter includes a direction of movement of the ultrasound probe during transmission of ultrasound signals in order to capture the ultrasound image.

[0009] In some embodiments, the AR environment includes mixed reality. The mixed reality includes the virtual representation of the anatomical element positioned above a real-world environment, the real-world environment including a real depiction of a patient body portion having the anatomical element.

[0010] In some embodiments, the anatomical element is a vessel within an arm or leg of a patient.

[0011] In some embodiments, the console further comprises a communication interface configured to provide the presentation of the virtual object to the AR headset.

[0012] Also disclosed herein is a medical analysis system comprising an ultrasound probe and a console communicatively coupled to the ultrasound probe. The console comprises a processor and a memory. The memory comprises AR anatomy representation logic comprising logic selected from the group consisting of visualization logic, virtual slice positioning logic, virtual object combination logic, and virtual object display logic, provided that at least two of the foregoing are selected. The visualization logic is configured to capture information associated with a plurality of sub-images at different layers of an ultrasound image of an anatomical element. The virtual slice positioning logic is configured to position and orient each sub-image based on a usage parameter during emission of ultrasound signals from the ultrasound probe for capturing the ultrasound image. The virtual object combination logic, when executed by the processor, is configured to combine each sub-image to form a virtual representation of the anatomical element. The virtual object display logic, when executed by the processor, is configured to present the virtual representation of the anatomical element in an AR environment.

[0013] In some embodiments, the usage parameter comprises a speed of movement of the ultrasound probe during emission of the ultrasound signals for capturing the ultrasound image.

[0014] In some embodiments, the usage parameter comprises a direction of movement of the ultrasound probe during emission of the ultrasound signals for capturing the ultrasound image.

[0015] In some embodiments, the AR environment comprises a mixed reality. The mixed reality comprises the virtual representation of the anatomical element positioned above a real-world environment comprising a real depiction of a patient body part having the anatomical element.

[0016] In some embodiments, the console further comprises a communication interface to provide the presentation of the virtual object to the AR headset.

[0017] In some embodiments, the anatomical element is a vessel.

[0018] In some embodiments, the virtual object display logic is configured to present the virtual representation of the anatomical element as an overlay over an image or a series of images.

[0019] In some embodiments, the image comprises an ultrasound image and the series of images comprises a video of a real-world environment.

[0020] In some embodiments, the visualization logic and the virtual slice positioning logic are implemented within the ultrasound probe. Additionally, the virtual object combination logic and the virtual object display logic are executed by a processor and implemented within the console.

[0021] In some embodiments, the visualization logic, the virtual slice positioning logic, the virtual object combination logic, and the virtual object display logic are implemented as software executed by a processor within the console.

[0022] The present invention also discloses a method comprising an information capturing operation, a positioning and orienting operation, and a combining operation. The information capturing operation comprises initiating capturing information associated with a plurality of sub-images at different longitudinal positions of an ultrasound image of an anatomical element. The positioning and orienting operation comprises longitudinally positioning and orienting each of the plurality of sub-images based on a usage parameter during emission of ultrasound signals from an ultrasound probe in order to capture the ultrasound image. The combining operation comprises combining each of the sub-images to form a virtual representation of the anatomical element for presentation in an AR environment.

[0023] In some embodiments, the usage parameter comprises a speed of movement of the ultrasound probe during emission of the ultrasound signals in order to capture the ultrasound image. Optionally, the usage parameter comprises a direction of movement of the ultrasound probe during emission of the ultrasound signals in order to capture the ultrasound image.

[0024] In some embodiments, the AR environment comprises a mixed reality. The mixed reality comprises the virtual representation of the anatomical element positioned over a real world environment comprising a real depiction of a body part of a patient having the anatomical element.

[0025] In some embodiments, the anatomical element is a vessel within a patient.

[0026] These and other features of embodiments of the present invention will become more apparent from the following description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] A more particular description of the disclosure will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope. The exemplary embodiments of the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0028] Figure 1 is an exemplary block diagram of a medical analysis system having AR anatomical representation logic to generate virtual objects overlaid ultrasound images;

[0029] Figure 2This is the first illustrative implementation of a medical analysis system, including the AR anatomical representation logic deployed therein;

[0030] Figure 3A It is a perspective view of a virtual object that overlays the captured ultrasound image of the blood vessels;

[0031] Figure 3B It is a perspective view of a virtual object that overlays the captured ultrasound image of the blood vessels;

[0032] Figure 3C yes Figure 3A An illustrative implementation scheme for virtual objects;

[0033] Figure 3D It is by Figure 2 The ultrasound probe captures to generate Figure 3C A perspective view of multiple slice images of the blood vessels of a virtual object;

[0034] Figure 4 This is a second illustrative implementation of a medical analysis system, including AR anatomical representation logic deployed within the ultrasound probe and console that form the medical analysis system;

[0035] Figure 5 It is by Figure 1 An exemplary operating method for implementing a medical analysis system. Detailed Implementation

[0036] Referring now to the accompanying drawings, in which the same structures will have the same reference numerals. It should be understood that the drawings are illustrations and schematic representations of exemplary embodiments of the invention, and are neither restrictive nor necessarily drawn to scale.

[0037] Regarding the terminology used herein, it should be understood that these terms are intended to describe specific embodiments and do not limit the scope of the concepts presented herein. Ordinal numbers (e.g., first, second, third, etc.) are sometimes used to distinguish or identify different components or operations and do not set any order or quantity restrictions. For example, the “first,” “second,” and “third” components or operations do not necessarily appear in this order, and the specific embodiment including these components or operations is not necessarily limited to these three components or operations. Similarly, labels such as “left,” “right,” “top,” “bottom,” “front,” “back,” etc., are used for convenience and are not intended to imply, for example, any particular fixed position, orientation, or direction. Rather, such markings reflect relative position, orientation, or direction. Unless the context clearly indicates otherwise, the singular form of the article includes plural reference.

[0038] The terms "logic" and "component" refer to hardware and / or software configured to perform one or more functions. As hardware, logic (or a component) can include circuitry having data processing and / or storage functionality. Examples of such circuitry can include, but are not limited to, a processor, a programmable gate array, a microcontroller, an application specific integrated circuit, a combinational circuit, etc. Additionally, or in conjunction with the above hardware circuitry, logic (or a component) can be software in the form of one or more software modules, which can be configured to operate as its corresponding circuitry. A software module can include, for example, an executable application, a daemon application, an application programming interface ("API"), a subroutine, a function, a process, a routine, a source code, or even one or more instructions. A software module can be stored in any type of suitable non-transitory storage medium (such as, for example, programmable circuitry), semiconductor memory, non-persistent memory such as volatile memory (e.g., any type of random access memory "RAM"), persistent memory such as non-volatile memory (e.g., read only memory "ROM", power backed RAM, flash memory, phase change memory, etc.), a solid state drive, a hard disk drive, an optical disk drive, or a portable storage device.

[0039] With respect to "alternative reality," the term "alternative reality" can refer to virtual reality, augmented reality, and mixed reality, unless the context dictates otherwise. "Virtual reality" includes virtual content in a virtual environment, which can be fanciful or a simulation of a real world. "Augmented reality" and "mixed reality" include virtual content in a real world environment, such as a real depiction of a patient body part including anatomical elements. Augmented reality includes virtual content in a real world environment, but the virtual content is not necessarily anchored in the real world environment. For example, the virtual content can be information overlaying the real world environment. The information can change as the real world environment changes due to time or environmental conditions in the real world environment, or the information can change as the augmented reality user moves through the real world environment; however, the information still overlays the real world environment. Mixed reality includes virtual content that is anchored in every dimension of the real world environment. For example, the virtual content can be a virtual object that is anchored in the real world environment. The virtual object can change as the real world environment changes due to time or environmental conditions in the real world environment, or the virtual object can change as the user moves through the real world environment to adapt to the perspective of the mixed reality user. The virtual object can also change according to any interaction with the user or another real world or virtual subject. The virtual object remains anchored in the real world environment unless the mixed reality user, or another real world or virtual subject, moves the virtual object to another location in the real world environment. Mixed reality does not exclude the aforementioned information that overlays the real world environment described with reference to augmented reality.

[0040] In the following description, the terms "or" and "and / or," as used herein, are to be interpreted as inclusive or meaning any one or any combination. For example, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C." There is an exception to this definition in that the term "or" in its ordinary sense, as used in a phrase such as "A or B" does not mean "A, B, or A and B" in the absence of a definition stating to the contrary.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs.

[0042] SUMMARY

[0043] Briefly, the embodiments disclosed herein relate to a medical analysis system for representing a region of a patient's body for analysis. One of the embodiments can involve monitoring the advancement of a medical component (e.g., a needle, introducer, catheter, etc.) by way of sound waves (ultrasound), for example. As disclosed, in some embodiments, the medical analysis system can include an ultrasound imaging system and an AR headset for conducting the analysis, where the ultrasound imaging system includes AR anatomy representation logic.

[0044] More specifically, the ultrasound imaging system includes an ultrasound probe and a console, which can be configured to include the AR anatomy representation logic or a portion thereof. The ultrasound probe is configured to emit ultrasound signals (sound waves) into a patient and receive back echoes of the ultrasound signals (sound waves) from the patient by way of a piezoelectric sensor array or an array of capacitive micromachined ultrasonic transducers ("CMUTs"). According to one embodiment of the disclosure, the ultrasound probe can receive commands from the console to capture information associated with multiple "slices" of an anatomical element (e.g., a vessel, tissue, etc.) during an ultrasound scan; i.e., information associated with multiple (two or more) sub-images of an ultrasound image of the anatomical element, where the sub-images are captured transverse to a longitudinal axis of the anatomical element. Each slice constitutes information associated with a two-dimensional ("2D") or three-dimensional ("3D") planar sub-image of the anatomical element, where multiple slices are overlaid to collectively reproduce a 3D representation of the anatomical element. Optionally, as another embodiment of the disclosure, the ultrasound probe can include visualization logic that automatically captures ultrasound scan information regarding individual slices of an image associated with the anatomical element and provides the ultrasound scan information to the console along with results of the piezoelectric sensor array or the array of CMUTs.

[0045] The console features an electronic circuit including a memory and a processor configured to transform the ultrasound signals of the echoes to produce ultrasound image segments corresponding to anatomical structures of the patient. These ultrasound image segments can be combined to form ultrasound frames for display. Additionally, according to one embodiment of the disclosure, the AR anatomy representation logic can be deployed as hardware, software, or a combination of hardware and software. For example, when deployed as software, the AR anatomy representation logic can include visualization logic configured to issue commands to the ultrasound probe to capture multiple "slices" of anatomical element images (e.g., vessels, arteries, etc.) during an ultrasound scan. The AR anatomy representation logic can further include virtual slice positioning logic to adjust the orientation and positioning of each image slice, virtual object combination logic to combine the imaged anatomical elements in an order, and virtual object display logic to present the virtual object along with the ultrasound imaging object.

[0046] More specifically, for example, as the ultrasound probe scans and moves along the ultrasound imaging region to capture a vessel, the visualization logic controls the capture of information associated with the vertically oriented portions (slices) of the ultrasound images (hereinafter "slice images") and returns the slice images to the visual slice positioning logic. The virtual slice positioning logic is configured to determine the longitudinal orientation and positioning of each slice image based at least in part on the direction and speed of the ultrasound probe in use, where such information is provided to the virtual object combination logic. The virtual object combination logic is configured to form the virtual object by organizing the slice images longitudinally and overlaying the virtual object laterally over the ultrasound images. For example, each visualization represented by a slice image would be positioned close to an adjacent slice image to construct the virtual object, such as a vessel virtual object, in a longitudinal direction.

[0047] Thereafter, the virtual object display logic is configured to display the common slice images as the anatomical element in the virtual context (i.e., as a virtual object within a virtual reality field of view or a mixed reality field of view, or as a 3D model of the vessel).

[0048] The alternate reality headset includes a display screen coupled to a headset frame having an electronic circuit including a memory and a processor. The display screen can be configured such that a wearer of the alternate reality headset is able to view a patient through the display screen. The display screen is configured to display a virtual anatomical object corresponding to ultrasound image segments over the patient.

[0049] In some embodiments, the ultrasound probe is configured with a pulsed wave Doppler imaging mode for transmitting and receiving ultrasound signals. The console is configured to capture ultrasound imaging frames according to the pulsed wave Doppler imaging mode, combine the ultrasound imaging frames and an aggregation function, and segment the ultrasound imaging frames or the aggregated ultrasound imaging frames into ultrasound image segments using an image segmentation function.

[0050] In some embodiments, when the AR anatomy representation logic is initiated, the console is configured to generate, by the virtual object composition logic, a virtual object as an aggregate of ultrasound image segments overlaid by a set of virtualizations (image slices). The console is configured to send the virtual anatomy object to the alternate reality headset for display over the patient.

[0051] Medical analysis system architecture

[0052] Reference Figure 1 is shown. According to this embodiment of the disclosure, the medical analysis system 100 includes an ultrasound imaging system 110 and an alternate reality AR headset 140. The ultrasound imaging system 110 includes a console 120 and an ultrasound probe 130, with the ultrasound imaging system 110 featuring AR anatomy representation logic 150. The AR anatomy representation logic 150 can be configured to generate virtual objects that are presented in an AR environment (such as virtual reality, augmented reality, or mixed reality), with the virtual objects overlaying images (e.g., ultrasound images) produced by the ultrasound imaging system 110 or a series of images (e.g., a video) associated with a real-world environment (e.g., a video including a patient or a body part of a patient, a body structure, etc.). The virtual objects can be viewable through the AR headset 140, or without the AR headset 140 on a display of the console 120. Optionally, it is contemplated that a magnetic field imaging system can be deployed in place of the ultrasound imaging system 110 described herein. It will be appreciated that the components and functionality of the console 120 described with reference to the ultrasound imaging system 110 are contemplated to apply to a magnetic field imaging system or similar system.

[0053] Nonetheless, in some embodiments of the medical analysis system 100, at least a portion of the functionality of the AR anatomy representation logic 150 can be deployed within the AR headset 140 in place of the console 120. Here, the AR headset 140 or another component operating in coordination with the AR headset 140 can act as or perform the functionality of the console (e.g., perform processing).

[0054] More specifically, as Figure 2As shown, a first illustrative embodiment of the medical analysis system 100 is shown, including an ultrasound probe 130 connected to a console 120, which includes AR anatomy representation logic 150. Here, the console 120 is characterized by electronic circuitry including a memory 212 and one or more processors 214 configured to convert ultrasound signals of echoes according to ultrasound conversion logic 216 to produce ultrasound frames and ultrasound image segments therefrom corresponding to anatomical elements (e.g., structures) of a patient. The console 120 is configured to operate in a first mode to capture ultrasound imaging frames (i.e., ultrasound images on a frame-by-frame basis) in the memory 212 according to a pulsed wave Doppler imaging mode of the ultrasound probe 130, to combine the ultrasound imaging frames and an aggregate function of the ultrasound conversion logic 216, and to segment the ultrasound imaging frames or the aggregated ultrasound imaging frames into ultrasound image segments. In a second mode of operation, however, the console 120 can be configured to convert the ultrasound image segments into virtual anatomical objects (hereinafter “virtual objects”) based on the operation of certain components within the AR anatomy representation logic 150. The console 120 is configured to send the virtual objects to the AR headset 140 by way of a wired or wireless communication interface 218 for display over an image (e.g., an ultrasound image) or series of images (e.g., a video of a real-world environment).

[0055] The console 120 includes a number of components of the medical analysis system 100, and the console 120 can take any of a variety of forms to house the number of components. The one or more processors 214 and the memory 212 (e.g., electrically erasable programmable read-only memory “EEPROM” or flash memory, among other non-volatile memory) of the console 120 are configured for controlling various functions of the medical analysis system 100, such as executing the AR anatomy representation logic 150 during operation of the medical analysis system 100. The console 120 also includes a digital controller or analog interface 220, and the digital controller or analog interface 220 is in communication with the one or more processors 214 and other system components to manage interface connections between the ultrasound probe 130, the AR headset 140, and other system components.

[0056] The console 120 also includes ports 222 for connecting with additional components, such as optional components 224, including printers, storage media, keyboards, etc. The ports 222 can be implemented as Universal Serial Bus ("USB") ports, although other types of ports or combinations of port types can be used, as well as other interfaces or connections described herein. The console 120 can include a power connection 226 to enable operable connection with an external power source 228. An internal power source 230 (e.g., a disposable battery or rechargeable battery) can also be used with or without the external power source 228. A power management circuit 232 is included in the digital controller or analog interface 220 of the console 120 to regulate power usage and distribution.

[0057] The display 234 can be a liquid crystal display ("LCD") that is integrated into the console 120 and used to display information to the clinician during a procedure. For example, the display 234 can be used to display an ultrasound image of a target in-vivo portion of a patient captured by the ultrasound probe 130. Additionally or alternatively, the display 234 can be used to display virtual objects over the ultrasound image without the need for the AR headset 140. The virtual objects would provide a more detailed virtual representation of in-vivo anatomical elements (e.g., vessels, tissue, etc.) of the patient being imaged.

[0058] Optionally, the display 234 can be separate from the console 120 rather than integrated into the console 120; however, such a display would be distinct from the display screen of the AR headset 140. The console 120 can also include a console button interface 236. In combination with the control buttons on the ultrasound probe 130, the clinician can use the console button interface 236 to immediately invoke a desired mode on the display 234 for use by the clinician. For example, as described above, two modes of operation can include a first mode (e.g., an ultrasound mode) and a second mode (e.g., an AR augmented mode).

[0059] The ultrasound probe 130 is configured to emit ultrasound signals to a patient and receive ultrasound signals from echoes from the patient through a piezoelectric sensor array 238 or a CMUT array. The ultrasound probe 130 can be configured with continuous wave or pulsed wave imaging modes. For example, the ultrasound probe 130 can be configured with the aforementioned pulsed wave Doppler imaging mode for emitting and receiving ultrasound signals.

[0060] The ultrasound probe 130 also includes a button and memory controller 240 for managing the ultrasound probe 130 and its button operations. The button and memory controller 240 can include non-volatile memory, such as EEPROM. The button and memory controller 240 is in operative communication with an ultrasound probe interface 242 of the console 120, where the ultrasound probe interface 242 includes piezoelectric input / output ("I / O") components 244 (or CMUT input / output ("I / O") components for engaging with a CMUT array) for engaging with the piezoelectric sensor array 238 of the ultrasound probe 130, and "button + memory" input / output components 246 for engaging with the button and memory controller 240 of the ultrasound probe 130. Thus, the operational mode of the ultrasound imaging system 110 can be controlled at the ultrasound probe 130 (via the button and memory controller 240) and / or at the console 120 (via the console button interface 236).

[0061] As Figure 2 As further shown, the AR anatomy representation logic 150 includes visualization logic 250, virtual slice positioning logic 260, virtual object combination logic 270, and virtual object display logic 280. Here, according to an embodiment of the disclosure, the visualization logic 250 can be configured to issue commands to the ultrasound probe 130 during an ultrasound scan to capture information associated with a plurality of "slices" of an image of an anatomical element, resulting in a visualization (e.g., a slice image of the anatomical element captured by sound waves). Thus, information associated with each frame (slice) of an ultrasound image can be captured at different prescribed time periods during an ultrasound scan, where the slice image is generated from sound waves emitted by the ultrasound probe 130 at different longitudinal portions or locations of the ultrasound image as the waves propagate and return through the anatomical element. The visualization logic 250 can use the information associated with the ultrasound frames to generate "virtual" slice images that, when aggregated, provide a virtual representation of the anatomical element captured by the ultrasound image. Figure 1 As further shown, the AR anatomy representation logic 150 includes visualization logic 250, virtual slice positioning logic 260, virtual object combination logic 270, and virtual object display logic 280. Here, according to an embodiment of the disclosure, the visualization logic 250 can be configured to issue commands to the ultrasound probe 130 during an ultrasound scan to capture information associated with a plurality of "slices" of an image of an anatomical element, resulting in a visualization (e.g., a slice image of the anatomical element captured by sound waves). Thus, information associated with each frame (slice) of an ultrasound image can be captured at different prescribed time periods during an ultrasound scan, where the slice image is generated from sound waves emitted by the ultrasound probe 130 at different longitudinal portions or locations of the ultrasound image as the waves propagate and return through the anatomical element. The visualization logic 250 can use the information associated with the ultrasound frames to generate "virtual" slice images that, when aggregated, provide a virtual representation of the anatomical element captured by the ultrasound image.

[0062] Optionally, according to another embodiment of the disclosure, the visualization logic 250 can be configured to generate slice images based on data associated with each ultrasound frame generated, where an aggregation of the ultrasound frames constitutes an ultrasound image. From the data associated with each ultrasound frame, the visualization logic 250 is configured to generate a virtual representation of a portion of the anatomical element captured by the ultrasound image.

[0063] Virtual slice positioning logic 260 is configured to determine the position / orientation of each slice image generated by visualization logic 250, based at least in part on the orientation and velocity of the ultrasound probe 130 during use. For example, the position of the virtual slice image above the ultrasound image and along at least the x and y axes of the virtual object can be determined based on the movement velocity of the ultrasound probe 130. Similarly, the position of the virtual slice image above the ultrasound image along any one or all of the x, y, or z axes can be determined based on the orientation of the ultrasound probe 130.

[0064] Virtual object combinational logic 270 is communicatively connected to virtual slice positioning logic 260. Based on the positioning of the slice images, virtual object combinational logic 270 is configured to vertically position each slice image at a defined location to generate a virtual object, wherein the virtual object is an aggregation of the positioned slice images. As an illustrative example, each slice image is arranged vertically adjacent to adjacent slice images to construct a virtual object, such as a vascular virtual object.

[0065] Virtual object display logic 280 is communicatively connected to virtual object combinational logic 270. Here, virtual object display logic 280 is configured to display aggregated slice images as virtual objects, which represent the analyzed anatomical elements within a virtual context. The virtual context may include, but is not limited to, a virtual reality view, a mixed reality view, or a 3D model of the anatomical elements.

[0066] Reference Figure 3A and 3B A perspective view of a virtual object 310 is drawn on the display 330, horizontally overlaying the captured ultrasound image 300 of the blood vessel 320. Specifically, Figure 3B Provided with Figure 3A A different, deeper focus, in which the cross-sections of virtual object 310 and nearby virtual objects (e.g., blood vessels) are displayed, showing their diameters. Information such as blood vessel diameter is useful for planning medical procedures involving catheter insertion. The display can be deployed as part of a console, or as a separate display, positioned in an AR environment (e.g., a mixed reality environment), in which display 330 is positioned near a portion of the patient's arm 340 to be diagnosed.

[0067] like Figure 3C and 3D As shown, virtual object 310 can represent a portion of the captured vascular 320, where... Figure 2 An ultrasound probe 130 captures multiple slice images 3501-350N (N≥1) of an ultrasound image 300 (slices taken transversely to the central axis of image 300) to generate a virtual object 310. Here, each of the slice images 3501-350N is... Figure 2The visualization logic 250 is based on capturing different sub-images from ultrasound images 300 and is... Figure 2 The virtual slice positioning logic 260 generates the virtual object 310 by positioning the slice images 3501-350N at a defined position in a 3D region (xyz axis). Such positioning can be performed on a two-dimensional region (xz axis or yz axis), while the virtual object combination logic 270 is configured to generate the virtual object 310 by positioning the slice images 3501-350N at a defined position in a 3D region (xyz axis), where the virtual object 310 is an aggregation of the positioned slice images 3501-350N, such as... Figure 3D As shown.

[0068] Now for reference Figure 4 A second illustrative embodiment of a medical analysis system 100 including AR anatomical representation logic 150 is shown, wherein the logic is deployed within an ultrasound probe 130 and a console 120. Here, the medical analysis system 100 includes a console 120 and an ultrasound probe 130; however, for this embodiment in the disclosed text, visualization logic 250 and virtual slice localization logic 260 are deployed within the ultrasound probe 130, while virtual object composition logic 270 and virtual object display logic 280 are deployed within the console 120. The operability of this decentralized embodiment of the AR anatomical representation logic 150 is related to… Figure 1 The AR anatomical representation logic 150 is consistent with the operability.

[0069] method

[0070] refer to Figure 5 This shows the result of Figure 1 and 4 An exemplary method of operation performed by a medical analysis system. According to an embodiment of the disclosed text, an ultrasound imaging system is activated (operation 500), and sound waves are emitted from an ultrasound probe into an area of ​​the patient's body (hereinafter referred to as the "ultrasound area"), thereby visualizing anatomical elements (operation 510). Based on this visualization, AR anatomical representation logic captures information associated with multiple "slices" of the anatomical element images during ultrasound scanning (operation 520). Each slice image may be associated with information about a sub-image or frame (layer) of the ultrasound image, and thus, each slice image is a virtualization of a portion of the ultrasound image of the anatomical element captured at different times.

[0071] After a plurality of slices of an anatomical element image is generated during the ultrasound scan, the AR anatomical representation logic determines a positioning of the slice images (operation 530) and an arrangement (combination) of the slice images based on specific use parameters associated with the ultrasound probe to generate a virtual object (operation 540). The positioning / combination can be for lateral (xz-axis or yz-axis) arrangement of the slice images within a virtual representation of the anatomical element (corresponding to the virtual object). The use parameters can include, but are not limited to, a speed and / or direction of the ultrasound probe movement over the ultrasound region.

[0072] Thereafter, the AR anatomical representation logic is configured to display the virtual object in the alternate reality (operation 550), such as an overlay on the ultrasound image, where the virtual object can better appear when viewed using the AR headset. However, the virtual object can be presented on a display such that the virtual object is also visible without the AR device.

[0073] Embodiments of the application can be implemented in other specific forms without departing from the spirit of the disclosure. It should be understood that the described embodiments are merely illustrative, and not restrictive. Therefore, the scope of the embodiments is defined not by the description above, but by the claims that follow. All changes within the meaning and range of equivalency of the claims are intended to be embraced therein.

Claims

1. A medical analysis system, characterized by comprising: an ultrasound probe; and a console communicatively coupled to the ultrasound probe, the console including a processor and a memory, wherein the memory includes alternative reality anatomy representation logic, the alternative reality anatomy representation logic including: (i) visualization logic configured to capture information associated with a plurality of sub-images at different layers of an ultrasound image of an anatomical element; (ii) virtual slice positioning logic configured to position and orient each sub-image based on usage parameters during the emission of ultrasound signals from the ultrasound probe in order to capture the ultrasound image, the usage parameters including a speed of movement of the ultrasound probe over an ultrasound region; (iii) virtual object combination logic, when executed by the processor, configured to combine each of the plurality of sub-images to form a virtual representation of the anatomical element; and (iv) virtual object display logic, when executed by the processor, configured to present the virtual representation of the anatomical element in an alternative reality environment on a display screen of an alternative reality headset.

2. The medical analysis system of claim 1, wherein, The speed of movement of the ultrasound probe over the ultrasound region determines a location of a virtual slice image over the ultrasound image.

3. The medical analysis system of claim 1, wherein, The alternative reality environment includes a mixed reality in which the virtual representation of the anatomical element is positioned over a real-world environment that is a true depiction of a patient body portion that includes the anatomical element.

4. The medical analysis system of claim 1, wherein, The console further includes a communication interface to provide the presentation of the virtual representation to the alternative reality headset.

5. The medical analysis system of claim 1, wherein, The anatomical element is a vessel.

6. The medical analysis system of claim 1, wherein, The virtual object display logic is configured to present the virtual representation of the anatomical element as an overlay over one or more ultrasound images.

7. The medical analysis system of claim 6, wherein, The virtual object display logic is configured to present the overlay over a series of ultrasound images that includes a video of a real-world environment.

8. The medical analysis system of claim 1, wherein, The visualization logic and the virtual slice positioning logic are implemented within the ultrasound probe, and the virtual object combination logic and the virtual object display logic are executed by the processor and implemented within the console.

9. The medical analysis system of claim 1, wherein, The visualization logic, the virtual slice positioning logic, the virtual object combination logic, and the virtual object display logic are implemented as software executed by the processor within the console.

10. The medical analysis system of claim 5, wherein, The virtual representation of the anatomical element is displayed with a diameter that is useful for a medical procedure.

11. The medical analysis system of claim 10, wherein, The medical procedure is a catheterization.

12. A method characterized by, comprising: capturing information associated with a plurality of sub-images at different longitudinal positions of an ultrasound image of an anatomical element; longitudinally positioning and orienting each of the plurality of sub-images based on usage parameters occurring during the emission of ultrasound signals in order to capture the ultrasound image, the usage parameters including a speed of movement of an ultrasound probe over an ultrasound region; combining each of the plurality of sub-images to form a virtual representation of the anatomical element for presentation in an alternative reality environment; and and presenting the virtual representation of the anatomical element in the alternative reality environment on a display screen of an alternative reality headset.

13. The method of claim 12, wherein, the use parameters include a direction of movement of the ultrasound probe during emission of the ultrasound signals in order to capture the ultrasound image.

14. The method of claim 12, wherein, the alternative reality environment includes a mixed reality in which a virtual representation of the anatomical element is positioned above a real-world environment, the real-world environment being a true depiction of a body part of a patient comprising the anatomical element, and the anatomical element being a vessel in the patient.

Images