Apparatus, method, and system for displaying mixed reality-based ultrasound image

Abstract

Disclosed are an apparatus, a method, and a system for displaying a mixed reality-based ultrasound image. The method for displaying a mixed reality-based ultrasound image, according to an embodiment of the present invention, may comprise the steps of: obtaining an ultrasound image captured by an ultrasound device and positioning information about a probe of the ultrasound device; generating, on the basis of color information about the ultrasound image, a composite image in which transparencies of a target region and a background region included in the ultrasound image are adjusted to be distinguished from each other; and outputting the composite image according to an output position determined by using the positioning information.

Description

Mixed reality-based ultrasound image display device, method, and system

[0001] This application relates to a mixed reality-based ultrasound image display device, method, and system. For example, this application relates to an AR-based image playback technique for two-dimensional ultrasound images in a mixed reality space.

[0002] Ultrasound imaging works by magnifying the reflection of sound waves transmitted through the skin and creating an image. Therefore, the practitioner only sees the exterior of the body, and the image is a cross-section of tissue within a certain depth of the skin. This requires continuous, real-time overlay of the downward-facing view with the image's cross-section. Furthermore, because the affected area and the ultrasound image display are typically not within the practitioner's line of sight, the practitioner must constantly turn their head to alternate between the two areas, creating a challenge.

[0003] Due to these technical characteristics, ultrasound imaging results vary depending on the practitioner's technical proficiency, requiring years of training to master. A major learning challenge arises from the discrepancy between the practitioner's actions of placing the ultrasound probe on the affected area, identifying and imaging the abnormal area, and the image displayed on the playback screen.

[0004] In other words, the most challenging learning curve in ultrasound procedures lies in achieving a real-time understanding of the scope and anatomy of ultrasound images, which are linked to the positioning of the ultrasound probe. In other words, spatial cognitive ability is required to mentally visualize the positioning of ultrasound images and understand anatomical structures, which presents a significant hurdle for beginners.

[0005] As a solution to the above-mentioned difficulties, a technique can be considered to directly project the results of the ultrasound image onto the affected area to align the positions. This technique directly reproduces the image obtained from the ultrasound diagnosis onto the affected area. However, due to the nature of the ultrasound diagnosis in which the procedure is performed while lying on a bed, the image projection can only be done downwards from the ceiling. In addition, due to the nature of the image projection using a projector, since it is only a direct transfer of the monitor of the ultrasound machine, the operator can only check the image rotated by 90 degrees regardless of the direction of the ultrasound probe.

[0006] Meanwhile, Augmented Reality (AR) is a technology that performs various analyses on images captured on mobile devices, etc., and reproduces new images that match the characteristics of the object. Users can check the newly created images on the surface of the object through the screen of the mobile device, and the system in the mobile device analyzes the object in real time and updates the image.

[0007] However, existing AR technologies are primarily optimized for users who constantly watch their mobile devices using the front camera, and do not account for situations where users create an image and then move their viewpoint to observe the same object. On the other hand, in the case of an augmented reality-based head-mounted display (AR HMD), when the AR software starts, the entire space in which the HMD is located is recognized and objects are created within the virtual space. Since the system tracks the HMD's movements in real time, AR images can be viewed from various angles.

[0008] However, even in the case of HMD systems that apply existing AR technology, the image may appear transparent when observing the opposite side of an already created AR image. Therefore, when outputting an AR image using an ultrasound image, when using a movable probe of a probe to reproduce an image of a patient's organs in AR space, there may be cases where the reproduced image cannot be observed in 3D due to the limitations of AR technology when the probe moves.

[0009] The technology underlying this application is disclosed in Korean Patent Publication No. 10-1231926.

[0010] The present invention is intended to solve the problems of the prior art described above, and to provide a mixed reality-based ultrasound image display device, method, and system that can resolve the problem of observing an augmented reality-based image generated from an ultrasound image from various angles.

[0011] However, the technical tasks to be achieved by the embodiments of the present invention are not limited to the technical tasks described above, and other technical tasks may exist.

[0012] As a technical means for achieving the above-described technical task, a mixed reality-based ultrasound image display method according to one embodiment of the present application may include a step of obtaining an ultrasound image photographed by an ultrasound device and positioning information of a probe of the ultrasound device, a step of generating a synthetic image in which transparency of a target area and a background area included in the ultrasound image are adjusted to be mutually distinguishable based on color information of the ultrasound image, and a step of outputting the synthetic image according to an output position determined using the positioning information.

[0013] In addition, the step of generating the synthetic image may include a step of specifying a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image according to the color information as a pixel corresponding to the target area, a step of specifying a pixel that does not satisfy the threshold condition as a pixel corresponding to the background area, and a step of upwardly adjusting the transparency of the pixel corresponding to the background area.

[0014] In addition, the step of outputting the composite image may output multiple composite images generated in correspondence with each of multiple ultrasound images taken multiple times at different locations by fixing them to correspond to each of the output locations in the three-dimensional space and overlapping them.

[0015] In addition, the step of generating the synthetic image may include a step of generating a mirror image in which color information of at least some of the plurality of pixels is converted based on a reference point set for the ultrasound image.

[0016] Additionally, the step of outputting the composite image may output the mirror image by overlapping it.

[0017] In addition, the step of outputting the synthetic image may output the synthetic image by superimposing it on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe.

[0018] Additionally, the outputting step may superimpose the synthetic image onto the reference image displayed on the screen of a head-mounted display worn by the user.

[0019] Additionally, the acquiring step may include a step of correcting the position information using the result of identifying the target area of ​​the user who gripped the probe on the screen.

[0020] Meanwhile, a mixed reality-based ultrasound image display service providing device according to one embodiment of the present invention may include a receiving unit that obtains an ultrasound image captured by an ultrasound device and positioning information of a probe of the ultrasound device, a processing unit that generates a synthetic image in which the transparency of a target area and a background area included in the ultrasound image are adjusted to be mutually distinguishable based on color information about the ultrasound image, and a transmitting unit that outputs the synthetic image according to an output position determined using the positioning information.

[0021] In addition, the processing unit may specify a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image according to the color information as a pixel corresponding to the target area, specify a pixel that does not satisfy the threshold condition as a pixel corresponding to the background area, and upwardly adjust the transparency of the pixel corresponding to the background area.

[0022] In addition, the transmitting unit can output multiple composite images generated in response to multiple ultrasound images photographed multiple times at different locations by fixing them to correspond to each of the output locations in the three-dimensional space and overlapping them.

[0023] In addition, the processing unit can generate a mirror image by converting color information of at least some of the plurality of pixels based on a reference point set for the ultrasound image.

[0024] Additionally, the transmitting unit can output the mirror images by overlapping them.

[0025] In addition, the transmitter can output the composite image by superimposing it on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe.

[0026] Additionally, the transmitter can superimpose the synthetic image onto the reference image displayed on the screen of a head-mounted display worn by the user.

[0027] Additionally, the receiving unit can correct the position information using the result of identifying the target area of ​​the user who has grasped the probe on the screen.

[0028] The above-described problem-solving methods are merely exemplary and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, additional embodiments may be included in the drawings and detailed description of the invention.

[0029] According to the above-described means for solving the problem of the present invention, a mixed reality-based ultrasound image display device, method, and system can be provided that can solve the problem of observing an augmented reality-based image generated from an ultrasound image from various angles.

[0030] According to the solution to the aforementioned problem of the present invention, considering the characteristic of head-mounted displays (AR HMDs) such as HoloLens and Magic Leap supporting three-dimensional recognition and reconstruction of a target space using dual external imaging cameras, when an ultrasound image corresponding to a two-dimensional image is placed in a three-dimensional space and observed, an image that can be viewed from multiple angles can be reproduced by considering the observation position and viewpoint of the user (observer).

[0031] However, the effects that can be obtained from this center are not limited to the effects described above, and other effects may exist.

[0032] Figure 1 is a schematic diagram of a mixed reality-based ultrasound image display system according to one embodiment of the present invention.

[0033] Figure 2 is a drawing showing an example of a mixed reality-based screen in which a synthetic image is superimposed on a reference image and output through a user terminal.

[0034] Figure 3 is a conceptual diagram for explaining a synthetic image including a virtual three-dimensional image corresponding to a photographed object.

[0035] Figure 4a is a drawing showing an example of a fixing member for combining a probe and a tracking device.

[0036] Figure 4b is a drawing showing an example of a user's typical probe gripping posture for performing ultrasonic probing.

[0037] Figure 4c is a drawing showing a holder system for an ultrasonic probe disclosed in the present invention and an example of a synthetic image output using the same.

[0038] Figure 5 is a conceptual diagram for explaining a pixel-by-pixel transparency adjustment process using color information of an ultrasound image.

[0039] Figure 6 is a conceptual diagram for explaining an augmented reality-based output method that uses mirror images to respond to various viewing directions.

[0040] FIG. 7 is a drawing showing an example of an output of a synthetic image corresponding to an ultrasound image probed using a phantom containing a predetermined object, as an experimental example linked to an ultrasound image display system based on mixed reality according to one embodiment of the present invention.

[0041] Figure 8 is a schematic diagram of a mixed reality-based ultrasound image display service providing device according to one embodiment of the present invention.

[0042] Figure 9 is a flowchart of an operation of a mixed reality-based ultrasound image display method according to one embodiment of the present invention.

[0043] Figure 10 is a detailed operation flowchart of a process for generating a synthetic image by a mixed reality-based ultrasound image display service providing device according to one embodiment of the present invention.

[0044] Below, with reference to the attached drawings, embodiments of the present invention are described in detail to facilitate easy implementation by those skilled in the art. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity, and similar reference numerals have been used throughout the specification to indicate similar elements.

[0045] Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" or "indirectly connected" with another element in between.

[0046] Throughout this specification, when it is said that a member is located “on,” “above,” “upper,” “lower,” “lower” or “lower” another member, this includes not only cases where the member is in contact with the other member, but also cases where another member exists between the two members.

[0047] Throughout this specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0048] This application relates to a mixed reality-based ultrasound image display device, method, and system. For example, this application relates to an AR-based image playback technique for two-dimensional ultrasound images in a mixed reality space.

[0049] In addition, this center is about a system that increases the efficiency of intuitive examination and screening by reproducing the results of existing ultrasound images to match the subject's affected area using mixed reality equipment and linking them with the positioning of the probe, thereby increasing intuitive understanding of lesions without secondary interpretation of anatomy.

[0050] Figure 1 is a schematic diagram of a mixed reality-based ultrasound image display system according to one embodiment of the present invention.

[0051] Referring to FIG. 1, a mixed reality-based ultrasound image display system (10) (hereinafter referred to as “display system (10)”) according to one embodiment of the present invention may include a mixed reality-based ultrasound image display service providing device (100) (hereinafter referred to as “service providing device (100)”) according to one embodiment of the present invention, an ultrasound device (200), a user terminal (300), a tracking device (400), and an input device (500).

[0052] The service providing device (100), the ultrasound device (200), the user terminal (300), the tracking device (400), and the input device (500) can communicate with each other through a network (20). The network (20) refers to a connection structure that enables information exchange between each node, such as terminals and servers, and examples of such a network (20) include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

[0053] In the description of the embodiments of the present invention, the ultrasound device (200) may refer to equipment that emits sound waves to obtain information about organs located under the skin of a subject to be photographed (e.g., a patient, etc.) and interprets the reflected signals to display a cross-sectional image. In addition, such an ultrasound device (200) is provided in a size that can be held in the hand of a user (practitioner) and includes a probe (210) having a sensor section provided at the bottom. Due to the characteristics of such a device, the ultrasound image obtained varies depending on the angle at which the probe (210) contacts the body of the subject to be photographed or the strength of the pressure applied. Accordingly, the user must accurately assess the anatomical characteristics of the human body and the condition of the patient, apply an appropriate amount of force to the affected area to obtain an image, and operate the equipment in various ways to instantly recognize an area determined to be abnormal and obtain a more detailed image. Therefore, years of diverse training and experience are required to accurately identify a lesion, and the accuracy of the examination varies depending on the skill of the practitioner.

[0054] Furthermore, conventional ultrasound devices have the limitation that the equipment that displays (plays back) the acquired ultrasound image is placed in a separate external location, unrelated to the probe (210) operated by the operator. In other words, in the case of conventional ultrasound devices, the operator must move the affected area with his or her hand while observing the image acquired in real time, and must track the exact location by alternating between the image and the affected area. Therefore, prior anatomical knowledge and the sense of touch of the hand on the affected area also become important.

[0055] The user terminal (300) may be any type of wireless communication device, such as, for example, a smartphone, a smart pad, a tablet PC, a PCS (Personal Communication System), a GSM (Global System for Mobile communication), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), an IMT (International Mobile Telecommunication)-2000, a CDMA (Code Division Multiple Access)-2000, a W-CDMA (W-Code Division Multiple Access), or a Wibro (Wireless Broadband Internet) terminal.

[0056] In particular, according to one embodiment of the present invention, the user terminal (300) may be an AR / VR device capable of reproducing the ultrasound image based on the mixed reality of the present invention. Specifically, the AR / VR device may refer to a head-mounted display (HMD) terminal. In this regard, the user terminal (300) may output a synthetic image based on an ultrasound image received from the service providing device (100) by overlaying it on a reference image described below.

[0057] In addition, in the description of the embodiment of the present invention, the tracking device (400) may be a device that is placed relative to the probe (210) of the ultrasonic device (200) and measures position information of the probe (210) and transmits it to the service providing device (100). More specifically, the position information measured by the tracking device (400) may include position information, angle (tilt) information, movement speed information, path information, etc. of the probe (210) in three-dimensional space.

[0058] That is, the tracking device (400) is configured with an optical camera and inertial specific equipment to track the position of the probe (210) of the ultrasonic device (200) in real time and can be attached to the probe (210), and after attachment, the translational and rotational directions of the probe (210) and the tracking device (400) are aligned so that positional information including movement measurement values ​​of the tracking device (400) can be interpreted as positional information according to the movement of the probe (210).

[0059] Meanwhile, in the description of the embodiment of the present invention, the tracking device (400) may be positioned relative to the probe (210) of the ultrasonic device (200) as a separate device (for example, by being coupled and connected to the probe (210) using a predetermined fixing member (for example, a screw-hole coupling structure, etc.)) so that the service providing device (100) may receive the position information measured by the tracking device (400) and use the same to obtain the position information of the probe (210) of the ultrasonic device (200), but is not limited thereto.

[0060] As another example, depending on the implementation example of the present invention, the tracking device (400) may refer to a sensor module such as an inertial sensor that is built into (mounted) the probe (210) to measure the position information of the probe (210) and transmit it to the service providing device (100).

[0061] In addition, in the description of the embodiment of the present invention, the input device (500) may be a device provided to apply a predetermined user input for setting and applying details applied in the process of converting an ultrasound image acquired from an ultrasound device (200) for a service providing device (100) into a two-dimensional (2D) or three-dimensional (3D) synthetic image (synthetic ultrasound image) that matches the positioning information of a probe (210) and playing it back through a user terminal (300).

[0062] For example, according to one embodiment of the present invention, the input device (500) may be provided in the form of a pedal (foot pedal) as illustrated in FIG. 1, and may be a device that determines whether a user input is accepted based on the pressure that the user (the operator) applies to the pedal, but is not limited thereto. As another example, the input device (500) may be provided in various types so as to be able to accept a wide range of user inputs, such as having an operating unit including a button, or identifying the arm, hand, face, etc. of the user (the operator) and recognizing gestures, or recognizing user input based on the voice of the user (the operator). For reference, the input device (500) may be otherwise referred to as an 'interaction device', etc.

[0063] Below, the specific functions and operations of the display system (10) will be described in detail.

[0064] The service providing device (100) can obtain an ultrasound image captured by an ultrasound device (200). For example, the service providing device (100) may receive a real-time ultrasound image captured by a probe (210) from the ultrasound device (200).

[0065] In addition, the service providing device (100) can obtain position information of the probe (210) of the ultrasonic device (200). According to one embodiment of the present invention, the service providing device (100) can receive position information of the probe (210) from a tracking device (400) placed with respect to the probe (210) or position information of the tracking device (400) that can be replaced as position information of the probe (210).

[0066] In addition, the service providing device (100) can convert the acquired ultrasound image into a synthetic image (synthetic ultrasound image) for outputting based on mixed reality (MR) through the user terminal (300).

[0067] For reference, in the description of the embodiments of the present invention, 'mixed reality' is a term that includes augmented reality (AR), which adds virtual information based on reality, and augmented virtuality (AV), which adds real information to a virtual environment, and refers to a technology that supports a more realistic experience of various digital information by combining reality and virtuality to create a new environment in which real and virtual objects coexist and allowing the user to interact with the environment in real time, and in relation to this, the display system (10) disclosed in the present invention is designed to overcome the problems of the conventional ultrasound device (200) by adding (combining) a synthetic ultrasound image ('synthetic image') obtained by converting an ultrasound image captured using a probe (210) to an actual image ('reference image') existing in an actual space where an ultrasound examination or ultrasound-guided procedure is performed, so that the practitioner (user) can recognize that the ultrasound image is displayed adjacent to the end (e.g., the lower end) of the probe (210).

[0068] In addition, the service providing device (100) can transmit the converted synthetic image to the user terminal (300), so that the synthetic image implemented on the reference image that is captured to include the probe (210) or the subject of capture can be output.

[0069] In this regard, FIG. 2 is a drawing exemplifying a screen based on mixed reality in which a synthetic image is superimposed on a reference image and output through a user terminal.

[0070] Referring to FIG. 2, the service providing device (100) captures a reference image (real image, I) that includes a probe on the screen of a user terminal (300), particularly, a head mounted display (HMD, 300) worn by the user. A) is a synthetic image (virtual image, I) that converts the original ultrasound image into a virtual ultrasound image based on the position information of the probe (210), the position information of the head-mounted display (AR HMD) worn by the user, and the field of view information. B ) can be displayed by overlapping, a synthetic image (I) is created by taking into account the position information of the probe (210). B ) is the reference image (I A ) can determine the size, position and direction of the overlay.

[0071] Specifically, according to one embodiment of the present invention, the service providing device (100) can calculate the display size, position, and direction of the synthetic image based on the position information of the probe (210) so that the synthetic image can be overlapped adjacent to the end region of the probe appearing in the reference image.

[0072] Here, the fact that the synthetic image is adjacent to the end region of the probe may mean that the synthetic image is arranged to extend downward based on the extension direction of the probe (210) from the lower surface of the probe (210) that contacts the skin of the subject of the probe (patient, etc.) being photographed, as illustrated in FIG. 2.

[0073] In addition, the service providing device (100) can determine the output (overlapping) size, position, and direction of the synthetic image so that the synthetic image is output in a direction perpendicular to the contact surface, taking into account the extension direction of the contact surface where the probe (210) and the target area of ​​the photograph are in contact based on the position information of the probe (210). Accordingly, the user can intuitively understand how the actual scan direction of the ultrasound image changes depending on the degree to which the user manipulates the probe (210) (e.g., tilts it or changes its position), by displaying the synthetic image perpendicular to the target area so as to correspond to the scan direction of the ultrasound image acquired using the probe (210).

[0074] In addition, according to one embodiment of the present invention, the service providing device (100) can obtain position information and field of view information of a head mounted display (user terminal, 300), and correct the position where the synthetic image is overlapped on the reference image based on the obtained position information of the HMD and field of view information of the user (operator).

[0075] In addition, according to one embodiment of the present invention, the service providing device (100) may receive a user input for adjusting the position of the synthetic image from at least one of the user terminal (300) and the input device (500), and operate to adjust the overlapping position of the synthetic image with respect to the reference image based on the received user input for position adjustment.

[0076] In other words, if there is a discrepancy between the actual position of the probe (210) that can be confirmed on the display screen where the reference image and the synthetic image are superimposed and the position where the synthetic image is displayed, the service providing device (100) can receive a user input for correcting the output (overlapping) position of the synthetic image through the input device (500) or the user terminal (300).

[0077] Figure 3 is a conceptual diagram for explaining a synthetic image including a virtual three-dimensional image corresponding to a photographed object.

[0078] Referring to FIG. 3, the service providing device (100) can display a synthetic image converted from an ultrasound image photographed at a specific time / position at a predetermined fixed position on a reference image continuously based on a capture input (specifically, a user input for capturing a synthetic image overlaid on a reference image) applied to at least one of a user terminal (300) and an input device (500), and can display a synthetic image including a virtual 3D image corresponding to a photographed object based on a plurality of synthetic images or a sequence of synthetic images continuously acquired in time series through the user terminal (300).

[0079] Specifically, the service providing device (100) receives a user input for capturing a synthetic image, and displays a fixed image in which the synthetic image is continuously maintained at a predetermined fixed position on a reference image based on the user input, and outputs a fixed image with a predetermined transparency applied to the synthetic image so as not to obstruct the view of a practitioner (user) looking at the affected area of ​​a subject (patient, etc.) being photographed.

[0080] In addition, the service providing device (100) can extract 3D boundary information for an object photographed by an ultrasound device based on color information of each of the plurality of fixed images in a state where fixed images are individually displayed at multiple different fixed locations based on user input (capture input) authorized multiple times.

[0081] Additionally, the service providing device (100) can update the synthetic image to include a virtual 3D image corresponding to the photographed object based on the extracted 3D boundary information.

[0082] That is, the service providing device (100) records the playback position of a synthetic image converted from an ultrasound image by a signal sent through an interaction device (input device, 500) such as a button, gesture, pedal, etc., and when an input value is continuously transmitted several times or more or for several hours or more, the service providing device (100) processes the images remembered at each position into an image with transparency and displays it as a fixed image, and compares the color information such as the gray scale of each pixel forming the fixed image, extracts points corresponding to a difference greater than a certain level, and then connects them to extract the outline of the photographed object (e.g., blood vessels, muscles, fascia, organs, etc.), accumulate them in real time, and generate a volume object (Volume Object) to update and output a synthetic image that can be confirmed three-dimensionally, thereby assisting the user in three-dimensionally understanding the structure (blood vessels, muscles, fascia, etc.) on the ultrasound image.

[0083] In addition, the service providing device (100) may provide a function for adjusting the degree of grayscale, which serves as a reference for extraction, through control via interaction. More specifically, the service providing device (100) may variably adjust an extraction reference value (e.g., a grayscale value, etc.) for extracting a plurality of fixed images based on a user's control input (e.g., gesture-based slide control, etc.) authorized via an input device (500).

[0084] That is, the display system (10) disclosed in this invention can be equipped with a function of fixing the location of the affected area in space by remembering the location of the affected area through interaction using an input device (500) such as a pedal, and of accumulating images of a plurality of consecutive ultrasound images and converting them into a three-dimensional model.

[0085] Meanwhile, in the above description, the display system (10) mainly described an embodiment in which an ultrasound image is displayed in an augmented (overlapping) manner on the screen of a head-mounted display (AR HMD) worn by a user. However, the display system (10) disclosed in the present application may include a playback device (not shown) that can receive an ultrasound image obtained as a result of an examination and play back and record the same, in addition to a display module that is in charge of the image output function of the main body of the ultrasound device (200) according to an implementation example of the present application, and can output the ultrasound image to an external display device through a separate cable or the like. In this regard, the display system (10) may include an image conversion cable that connects the image output terminal of the ultrasound device (200) and the playback device, a conversion module that converts the transmitted image signal so that it can be viewed on the playback device, and software for playing back and recording the resulting image.

[0086] Figure 4a is a drawing showing an example of a fixing member for combining a probe and a tracking device.

[0087] Referring to FIG. 4a, a fixing member (220) for coupling a probe (210) and a tracking device (400) has a hole having the same width as a screw hole drilled in the probe (210) so that it can be fixed without any gap or movement with the equipment that tracks the movement without interfering with the action of holding the probe (210) with the hand of the prober, and can be fastened by tightening a bolt that fits the screw hole, but is not limited thereto.

[0088] Meanwhile, in the case of a method of extracting 3D boundary information for a photographed object (organ, etc.) of an ultrasound device (200) as described above and updating and outputting a synthetic image to include a virtual 3D image, the synthetic image can be generated in a position close to the tracking device (400) within the Hololens. However, a still 2D video image (e.g., PNG format, etc.) system has a plurality of pixels inside a shape of a predetermined shape and size (e.g., a rectangle, etc.), and has image information including (R, G, B, (a)) values ​​in each pixel. Therefore, if a 2D image of a predetermined shape is reproduced in space, when a plurality of images overlap in the user's field of view, the image in front of the user may cover the image behind, which may interfere with the recognition of the 3D shape.

[0089] In consideration of this, the service providing device (100) disclosed in the present invention extracts color information for all pixels of an acquired ultrasound image, as described in detail below, selectively outputs only specific pixels based on a pre-specified threshold condition (e.g., threshold brightness, etc.), and adjusts transparency upward for unselected pixels (e.g., converts to full transparency) so that an image placed relatively farther back with respect to the user's line of sight in a plurality of images is not obscured.

[0090] In this regard, since ultrasound images are usually in the form of black and white images (binary images), and in ultrasound images, the probed photographed object (e.g., major organs of the body, etc.) is mostly expressed in a relatively bright color, the threshold condition for selecting a specific pixel can be set so that a pixel whose color information (e.g., at least one of R, G, B values) exceeds a preset threshold brightness is selected.

[0091] That is, when an ultrasound image is input, the service providing device (100) selects at least one of the R, G, and B values ​​as color information of the ultrasound image, determines the brightness thereof, selects each pixel (distinguishing between a target area and a background area) according to a preset (e.g., user-specified, etc.) threshold brightness, and then converts the transparency of pixels that are not selected. For reference, according to the implementation example of the present application, the threshold brightness described above may be set to a gray scale-based value.

[0092] To explain more specifically, first, the service providing device (100) can obtain an ultrasound image captured by an ultrasound device (200) and position information of a probe (210) of the ultrasound device (200).

[0093] Meanwhile, according to one embodiment of the present invention, the service providing device (100) can correct the position information of the probe (210) by using the result of identifying the target area of ​​the user who has grasped the probe (210) on the screen of the head mounted display (AR HMD) worn by the user.

[0094] In this regard, FIG. 4b is a drawing showing an example of a user's typical probe gripping posture for performing ultrasonic probing, and FIG. 4c is a drawing showing an example of a holder system for an ultrasonic probe disclosed in the present invention and a synthetic image output using the same.

[0095] Referring to FIG. 4b, when performing an ultrasound probe, the user typically holds the probe lightly, and at this time, the position of each finger of the probe may not be uniform depending on the posture of holding the probe, etc., but when applying the holder system for the probe (210) disclosed in the present invention as shown in FIG. 4c, the posture of the user holding the holder system is guided to maintain a relatively specific posture, so that the position of each finger of the probe can be fixed compared to a conventional, typical probe.

[0096] Accordingly, the service providing device (100) tracks in real time the target area (e.g., the position of the tip of the right finger, the index finger, etc.) of the user (probe) that is always fixed on the input reference image (e.g., the screen of the head-mounted display (AR HMD) worn by the user) using a tracking technology (e.g., hand-tracking technology, etc.), and then moves the holder and the image playback system to a position designated by a specific input of the probe, thereby eliminating the position accumulation error of the position information of the probe (210) that may occur during the continuously performed ultrasound examination.

[0097] Accordingly, if a discrepancy occurs between the actual probe observed on the screen of a head-mounted display (AR HMD) and the probe position according to the ultrasound image provided to the AR Display, the user (probe) can, by applying user input, correct the positioning information acquired in response to the ultrasound image to match the actual finger position, actual probe placement position, etc., as necessary.

[0098] In addition, the service providing device (100) can generate a synthetic image in which the transparency of the target area and the background area included in the acquired ultrasound image are adjusted to be distinguished based on color information about the acquired ultrasound image.

[0099] More specifically, in the description of the embodiments of the present invention, the 'target region' may mean an area corresponding to a photographed object to be detected within an acquired ultrasound image (e.g., a major body organ to be observed), and, in distinction thereto, the 'background region' may mean an area that does not correspond to a photographed object and for which information about the shape, color, etc. of the area does not need to be provided to the user.

[0100] Fig. 5 is a conceptual diagram illustrating a pixel-by-pixel transparency adjustment process using color information of an ultrasound image. Specifically, Fig. 5 (a) illustrates multiple pixels of an ultrasound image, Fig. 5 (b) illustrates a state in which transparency for pixels corresponding to a background area is selectively adjusted, and Fig. 5 (c) illustrates a state in which multiple composite images with pixel-by-pixel transparency adjustments applied are accumulated and output in a three-dimensional space.

[0101] Referring to FIG. 5, the service providing device (100) can specify a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image as a pixel corresponding to the target area based on color information of the ultrasound image.

[0102] In addition, the service providing device (100) can specify pixels that do not meet a preset threshold condition among a plurality of pixels forming the ultrasound image based on color information of the ultrasound image as pixels corresponding to the background area.

[0103] In addition, the service providing device (100) can upwardly adjust the transparency of pixels corresponding to the background area. Accordingly, even if multiple synthetic images generated from each of multiple ultrasound images are output in a three-dimensional space through the user terminal (300) as illustrated in (c) of FIG. 5, the phenomenon of the synthetic image located at the rear being covered by the synthetic image located relatively forward depending on the user's viewing direction can be prevented.

[0104] In addition, the service providing device (100) can output the generated synthetic image through a user terminal (300) or the like according to the output position determined using the position information of the acquired probe (210).

[0105] For example, the service providing device (100) can output an augmented reality (AR)-based image that superimposes a synthetic image on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe (210) of the ultrasound device (200).

[0106] In this regard, the service providing device (100) may output a synthetic image by superimposing it on a reference image displayed on the screen of a head mounted display (AR HMD) worn by the user.

[0107] Specifically, the service providing device (100) can output multiple composite images generated in response to multiple ultrasound images captured multiple times at different locations by fixing them to correspond to each output location in a three-dimensional space and overlapping them.

[0108] Figure 6 is a conceptual diagram for explaining an augmented reality-based output method that uses mirror images to respond to various viewing directions.

[0109] Referring to FIG. 6, the service providing device (100) can generate a mirror image (AR Mirror) by converting color information of at least some of a plurality of pixels based on a reference point set for an ultrasound image, and output the processed mirror image by superimposing it on a previously generated synthetic image.

[0110] In this regard, mobile-based AR technology can place newly generated augmented reality images to fit the target by considering the shape and location of the target in that way, but the image seen from the user's perspective appears as a transparent image with some normal RGB values, whereas due to the characteristics of augmented reality image playback, when the image is observed from the opposite side, only a blank scene with no information is observed.

[0111] Therefore, even if an HMD-based augmented reality method is adopted, there is a problem in that no information can be obtained when an image collected using a probe (210) of an ultrasound device (200) is played back in an augmented reality space, and then the probe (210) is fixed and moved to the opposite side to observe the image.

[0112] In consideration of this, the service providing device (100) disclosed in the present invention rotates and copies an ultrasound image corresponding to a 2D image and superimposes it on a synthetic image, but acquires all pixel information of the corresponding ultrasound image so that the left and right are not reversed, and then replaces the pixel information centered on a reference point of the corresponding ultrasound image (e.g., the vertical center point of the ultrasound image, etc.) to re-input color information to generate an image with a 'mirror' effect rather than a simple rotation, and superimposes this on the previously generated synthetic image to display it, thereby solving the problem of observing an augmented reality-based image generated from an ultrasound image from various angles.

[0113] FIG. 7 is a drawing showing an example of an output of a synthetic image corresponding to an ultrasound image probed using a phantom containing a predetermined object, as an experimental example linked to an ultrasound image display system based on mixed reality according to one embodiment of the present invention.

[0114] Referring to FIG. 7, a synthetic image generated from an ultrasonic image detected by an ultrasonic device (200) of an object located inside a phantom through a service providing device (100) disclosed in the present invention is superimposed on a reference image to match the actual location of the object, thereby enabling a user to intuitively confirm the shape, location, etc. of the object.

[0115] Figure 8 is a schematic diagram of a mixed reality-based ultrasound image display service providing device according to one embodiment of the present invention.

[0116] Referring to FIG. 8, the service providing device (100) may include a receiving unit (110), a processing unit (120), and a transmitting unit (130).

[0117] The receiving unit (110) can obtain an ultrasound image captured by the ultrasound device (200) and position information of the probe (210) of the ultrasound device (200).

[0118] According to one embodiment of the present invention, the receiver (110) can correct the position information of the probe (210) by using the result of identifying the target area of ​​the user who has grasped the probe (210) on the screen of the head mounted display (AR HMD) worn by the user.

[0119] The processing unit (120) can generate a synthetic image in which the transparency of the target area and background area included in the acquired ultrasound image is adjusted to be distinguished based on color information about the acquired ultrasound image.

[0120] Specifically, the processing unit (120) can specify a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image based on the color information of the ultrasound image as a pixel corresponding to the target area.

[0121] In addition, the processing unit (120) can specify pixels that do not meet a preset threshold condition among the plurality of pixels forming the ultrasound image based on the color information of the ultrasound image as pixels corresponding to the background area.

[0122] Additionally, the processing unit (120) can upwardly adjust the transparency of pixels corresponding to the background area.

[0123] In addition, according to one embodiment of the present invention, the processing unit (120) can generate a mirror image (AR Mirror) in which color information of at least some of a plurality of pixels is converted based on a reference point set for an ultrasound image.

[0124] The transmitter (130) can output a synthetic image generated by the processor (120) based on the output position determined using the acquired position information of the probe (210) through a user terminal (300), etc.

[0125] For example, the transmitter (130) can output an augmented reality (AR)-based image that superimposes a synthetic image on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe (210) of the ultrasound device (200).

[0126] In this regard, the transmitter (130) may output a synthetic image by superimposing it on a reference image displayed on the screen of a head-mounted display (AR HMD) worn by the user.

[0127] Specifically, the transmitter (130) can output multiple composite images generated in response to multiple ultrasound images captured multiple times at different locations by fixing them to correspond to each output location in the three-dimensional space and overlapping them.

[0128] Additionally, the transmitter (130) can output a processed mirror image by superimposing it on the generated composite image.

[0129] Below, we will briefly review the operating flow of the present invention based on the detailed description above.

[0130] Figure 9 is a flowchart of an operation of a mixed reality-based ultrasound image display method according to one embodiment of the present invention.

[0131] The mixed reality-based ultrasound image display method illustrated in FIG. 9 can be performed by the service providing device (100) described above. Therefore, even if omitted below, the description of the service providing device (100) can be equally applied to the description of the mixed reality-based ultrasound image display method.

[0132] Referring to FIG. 9, in step S11, the receiving unit (110) can obtain an ultrasonic image captured by the ultrasonic device (200) and position information of the probe (210) of the ultrasonic device (200).

[0133] According to one embodiment of the present invention, in step S11, the receiver (110) can correct the position information of the probe (210) by using the result of identifying the target area of ​​the user who has gripped the probe (210) on the screen of the head mounted display (AR HMD) worn by the user.

[0134] Next, in step S12, the processing unit (120) can generate a synthetic image in which the transparency of the target area and the background area included in the acquired ultrasound image are adjusted to be distinguished based on the color information for the acquired ultrasound image.

[0135] In addition, according to one embodiment of the present invention, in step S12, the processing unit (120) can generate a mirror image (AR Mirror) in which color information of at least some of a plurality of pixels is converted based on a reference point set for the ultrasound image.

[0136] Next, in step S13, the transmitter (130) can output the composite image generated through step S12 according to the output position determined using the acquired position information of the probe (210).

[0137] For example, in step S13, the transmitter (130) can output an augmented reality (AR)-based image in which a synthetic image is superimposed on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe (210) of the ultrasound device (200).

[0138] In this regard, in step S13, the transmitter (130) may output a synthetic image by superimposing it on a reference image displayed on the screen of a head-mounted display (AR HMD) worn by the user.

[0139] Specifically, in step S13, the transmitter (130) can output multiple composite images generated in correspondence to each of multiple ultrasound images captured multiple times at different locations by fixing them to correspond to each output location in the three-dimensional space and overlapping them.

[0140] Additionally, in step S13, the transmitter (130) can output the processed mirror image by superimposing it on the generated composite image.

[0141] In the above description, steps S11 to S13 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. Furthermore, some steps may be omitted as needed, and the order of the steps may be changed.

[0142] Figure 10 is a detailed operation flowchart of a process for generating a synthetic image by a mixed reality-based ultrasound image display service providing device according to one embodiment of the present invention.

[0143] The process of generating a synthetic image illustrated in FIG. 10 can be performed by the service providing device (100) described above. Therefore, even if omitted below, the description of the service providing device (100) can be equally applied to the description of FIG. 10.

[0144] Referring to FIG. 10, in step S121, the processing unit (120) can specify a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image as a pixel corresponding to the target area based on color information of the ultrasound image.

[0145] Next, in step S122, the processing unit (120) can specify a pixel that does not meet a preset threshold condition among a plurality of pixels forming the ultrasound image according to the color information of the ultrasound image as a pixel corresponding to the background area.

[0146] Next, in step S123, the processing unit (120) can upwardly adjust the transparency of the pixels corresponding to the background area.

[0147] In the above description, steps S121 to S123 may be further divided into additional steps or combined into fewer steps, depending on the implementation example of the present invention. Furthermore, some steps may be omitted as needed, and the order of the steps may be changed.

[0148] A mixed reality-based ultrasound image display method according to one embodiment of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The above hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0149] Additionally, the above-described mixed reality-based ultrasound image display method can also be implemented in the form of a computer program or application executed by a computer and stored in a recording medium.

[0150] The above description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0151] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. In a mixed reality-based ultrasound image display method, A step of obtaining an ultrasound image captured by an ultrasound device and positioning information of a probe of the ultrasound device; A step of generating a synthetic image in which the transparency of a target area and a background area included in the ultrasound image are adjusted to be mutually distinguishable based on color information of the ultrasound image; and A step of outputting the synthetic image according to the output position determined using the above position information, A display method comprising:

2. In paragraph 1, The steps of generating the above synthetic image are: A step of specifying a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image according to the color information as a pixel corresponding to the target area, and specifying a pixel that does not satisfy the threshold condition as a pixel corresponding to the background area; and a step of upwardly adjusting the transparency of pixels corresponding to the above background area; A display method comprising:

3. In paragraph 2, The step of outputting the above composite image is: A display method comprising: fixing and superimposing a plurality of synthetic images generated in response to each of a plurality of ultrasound images photographed multiple times at different locations so as to correspond to each of the output locations in a three-dimensional space.

4. In paragraph 3, The steps of generating the above synthetic image are: A step of generating a mirror image by converting color information of at least some of the plurality of pixels based on a reference point set for the ultrasound image; A display method comprising:

5. In paragraph 4, The step of outputting the above composite image is: A display method for outputting the above mirror images by overlapping them.

6. In paragraph 1, The step of outputting the above composite image is: A display method for outputting a composite image by superimposing the composite image on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe.

7. In paragraph 6, The above output step is, A display method comprising superimposing the synthetic image onto the reference image displayed on the screen of a head mounted display worn by a user.

8. In paragraph 7, The above obtaining steps are: A step of correcting the position information by using the result of identifying the target area of ​​the user who has grasped the probe on the above screen. A display method comprising:

9. In a device providing ultrasound image display service based on mixed reality, A receiving unit that obtains an ultrasound image captured by an ultrasound device and positioning information of a probe of the ultrasound device; A processing unit that generates a synthetic image by adjusting the transparency of a target area and a background area included in the ultrasound image to be mutually distinguishable based on color information about the ultrasound image; and A transmitter that outputs the synthetic image according to the output position determined using the above position information; A service providing device including:

10. In paragraph 9, The above processing unit, A service providing device that specifies a pixel that satisfies a preset threshold condition among a plurality of pixels forming the ultrasound image according to the color information as a pixel corresponding to the target area, specifies a pixel that does not satisfy the threshold condition as a pixel corresponding to the background area, and upwardly adjusts the transparency of the pixel corresponding to the background area.

11. In paragraph 10, The above transmitter, A service providing device that generates multiple synthetic images corresponding to each of multiple ultrasound images taken multiple times at different locations and outputs them by fixing them to correspond to each of the output locations in a three-dimensional space.

12. In paragraph 11, The above processing unit, Generating a mirror image by converting color information of at least some of the plurality of pixels based on a reference point set for the above ultrasound image, The above transmitter, A service providing device that outputs the above mirror images by overlapping them.

13. In paragraph 9, The above transmitter, A service providing device that outputs the composite image by superimposing it on a reference image that is photographed to include at least one of the subject of the ultrasound image and the probe.

14. In paragraph 13, The above transmitter, A service providing device that superimposes the synthetic image onto the reference image displayed on the screen of a head mounted display worn by a user.

15. In paragraph 14, The above receiver, A service providing device that corrects the position information by using the result of identifying the target area of ​​the user who has grasped the probe on the above screen.

Images