Mixed reality-based ultrasound image output system and method

Abstract

The present invention includes: a data receiving unit for receiving an ultrasound image and a bio-signal of a subject; a first content generation unit for generating first mixed-reality (MR) content by matching the ultrasound image to a specific object; a second content generation unit for generating second MR content by matching the bio-signal to a predetermined space; and an output device that displays MR content generated by the first content generation unit or the second content generation unit, wherein the first content generation unit recognizes a probe shape of an ultrasound diagnosis device through a camera provided in the output device, detects movement of the probe shape, and changes an output position of the first MR content.

Description

Mixed reality-based ultrasound image output system and method

[0001] The present invention relates to a mixed reality-based ultrasound image output system and method, and more particularly, to a mixed reality-based ultrasound image output system and method that aligns and outputs an ultrasound image on an examination area using a device that displays in an AR or VR manner.

[0002] Ultrasound refers to sound waves with a frequency of 20,000 Hz or higher, which is higher than the human audible range. Ultrasound imaging (ultrasonography) is the process of creating images of the differences in reception generated through diffusion, reflection, absorption, and scattering of sound waves at various boundaries such as internal organs, bones, muscle tissue, and blood of the human body.

[0003] Ultrasound-guided biopsy (US-guided biopsy) is a procedure that involves inserting a thin needle into a mass or localized lesion under ultrasound guidance to obtain a tissue sample. US-guided biopsy can help pinpoint the lesion, increasing accuracy and reducing false negatives. Consequently, US-guided biopsy is widely used in clinical practice.

[0004] Ultrasound-guided radiofrequency ablation (US-guided RF ablation) is a treatment method that uses heat generated by inserting a needle-shaped electrode into a tumor under ultrasound guidance and then passing an electric current through it, thereby burning the tumor. US-guided RF ablation offers the advantage of real-time guidance, allowing for rapid and accurate insertion of the electrode into the tumor, compared to CT (Computerized Tomography) or MRI (Magnetic Resonance Imaging) imaging.

[0005] Ultrasound-guided biopsy can cause significant fatigue, as the operator must hold the ultrasound probe for extended periods and frequently turn their head to compare and confirm the ultrasound image with the biopsy site. Because the ultrasound monitor and biopsy site cannot be viewed simultaneously, ultrasound-guided biopsy is more difficult to perform. It also requires a high level of expertise and skill, making it a procedure that only experienced professionals can perform. Furthermore, in actual examination environments, the ultrasound equipment is often placed far away from the patient's bed and the numerous pieces of equipment in the operating room, which can hinder ultrasound-guided biopsy.

[0006] Ultrasound-guided radiofrequency ablation (USA) treatment times can be longer as the size of the tumor to be treated gradually decreases with early detection, making it more difficult to locate the tumor using ultrasound imaging. Because ultrasound-guided radiofrequency ablation requires intense concentration during treatment, it can be difficult to timely monitor physical risk factors such as blood pressure, heart rate, and oxygen saturation.

[0007] Therefore, there is a need for a mixed reality-based ultrasound image output system and method to overcome the limitations of ultrasound-guided biopsy and ultrasound-guided radiofrequency ablation therapy.

[0008] The purpose of the present invention is to enable a practitioner to simultaneously view a biopsy site and ultrasound through a wearable output device, and to superimpose an ultrasound image and a lesion site on the biopsy site.

[0009] In addition, the present invention seeks to display a physical danger signal in the direction of the wearer's gaze through an output device that can be worn by the operator.

[0010] Additionally, the present invention seeks to synchronize input signals to ensure the reliability of output information.

[0011] In order to achieve the above object, the present invention comprises: a data receiving unit that receives an ultrasound image and a bio-signal of a subject; a first content generating unit that generates first MR (Mixed-reality) content by aligning the ultrasound image with a specific object; a second content generating unit that generates second MR content by aligning the bio-signal with a specific space; and an output device that displays the MR content generated by the first content generating unit or the second content generating unit; wherein the first content generating unit recognizes the probe shape of an ultrasound diagnostic device through a camera provided in the output device, and detects movement of the probe shape to change the output position of the first MR content.

[0012] Preferably, the method may further include a lesion detection unit that detects a lesion area in the ultrasound image using artificial intelligence learned from an ultrasound image data set in which the lesion area is annotated.

[0013] Preferably, the first content generation unit can change the first MR content so that the lesion area detected by the lesion detection unit is distinguished from other areas.

[0014] Preferably, the first content generation unit can change the MR content so that the lesion area is not distinguished from other areas when it is confirmed through the ultrasound image that the needle has contacted the lesion area.

[0015] Preferably, the device may further include a synchronization unit that synchronizes the ultrasound image, the bio-signal, and the lesion area so that the ultrasound image and the bio-signal measured at the same time can be output, and the lesion area detected at the same time can be displayed.

[0016] Preferably, the synchronization unit may include a delay synchronization module that synchronizes based on the most delayed signal among the ultrasound image, the bio-signal, and the lesion area when the lesion area detected by the lesion detection unit is smaller than a preset reference value; and a prediction synchronization module that synchronizes based on the most advanced signal among the ultrasound image, the bio-signal, and the lesion area when the lesion area detected by the lesion detection unit is larger than a preset reference value.

[0017] Preferably, the prediction synchronization module can synchronize another signal to the most advanced signal using extrapolation.

[0018] Preferably, the output device includes a first camera that photographs the wearer's forward direction and a second camera that photographs the wearer's face, wherein the first camera can recognize the probe shape of the ultrasonic diagnostic device, and the second camera can track the wearer's gaze direction.

[0019] Preferably, the output device can display the second MR content along the wearer's gaze direction tracked by the second camera.

[0020] Preferably, the output device can be controlled by the movement of the wearer's eyes or pupils captured by the second camera.

[0021] In addition, the present invention includes a data receiving step of receiving an ultrasound image and a bio-signal of a subject; a first content generating step of generating first MR (Mixed-reality) content by aligning the ultrasound image with a specific object; a second content generating step of generating second MR content by aligning the bio-signal with a specific space; and a step of displaying the MR content generated in the first content generating step or the second content generating step on an output device; and another feature of the first content generating step is that the shape of a probe of an ultrasound diagnostic device is recognized through a camera provided in the output device, and the movement of the probe shape is detected to change the output position of the first MR content.

[0022] The present invention has the advantage of reducing operator fatigue by allowing the biopsy site and ultrasound to be viewed simultaneously, and increasing accuracy and stability by allowing the ultrasound image and lesion site to be superimposed on the biopsy site.

[0023] In addition, the present invention has the advantage of ensuring patient safety by displaying a physical danger signal in the direction of vision through an output device that can be worn by the operator.

[0024] In addition, the present invention has the advantage of ensuring the reliability of output information by synchronizing input signals and preventing the needle from being inserted into a different part.

[0025] In addition, the present invention can increase the freedom of the operator by providing an environment in which the equipment can be controlled without touching.

[0026] Figure 1 shows a configuration diagram of a mixed reality-based ultrasound image output system according to an embodiment of the present invention.

[0027] Figure 2 shows a configuration of a stand-alone mixed reality-based ultrasound image output system according to an embodiment of the present invention.

[0028] Figure 3 shows a configuration of a mixed reality-based ultrasound image output system using a smartphone tethering method according to an embodiment of the present invention.

[0029] FIG. 4 is a diagram illustrating the relationship between the size of MR content output to an output device according to an embodiment of the present invention and the probe angle.

[0030] Figure 5 shows a configuration diagram of a lesion detection unit according to an embodiment of the present invention.

[0031] Figure 6 shows a process in which a lesion detection unit according to an embodiment of the present invention detects a lesion using artificial intelligence.

[0032] Fig. 7 shows a configuration diagram of a synchronization unit according to an embodiment of the present invention.

[0033] FIG. 8 is a diagram illustrating a synchronization method of a delay synchronization module according to an embodiment of the present invention.

[0034] FIG. 9 is a diagram illustrating a synchronization method of a prediction synchronization module according to an embodiment of the present invention.

[0035] Figure 10 shows a screen displayed through an output device according to an embodiment of the present invention.

[0036] A data receiving unit that receives ultrasound images and bio-signals of a subject;

[0037] A first content generation unit that generates first MR (Mixed-reality) content by aligning the above ultrasound image with a specific object;

[0038] A second content generation unit that generates second MR content by aligning the above biosignals to a certain space; and

[0039] An output device that displays MR content generated by the first content generation unit or the second content generation unit;

[0040] The above first content creation unit,

[0041] A mixed reality-based ultrasound image output system characterized in that it recognizes the probe shape of an ultrasound diagnostic device through a camera equipped in the output device, detects movement of the probe shape, and changes the output position of the first MR content.

[0042] Hereinafter, the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. The same reference numerals in each drawing indicate components that perform substantially the same functions.

[0043] The purpose and effects of the present invention can be naturally understood or made clearer by the following description, and the purpose and effects of the present invention are not limited solely by the following description. Furthermore, in describing the present invention, if a detailed description of known technologies related to the present invention is deemed to unnecessarily obscure the gist of the present invention, such detailed description will be omitted.

[0044] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the description of the invention, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0045] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0046] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0047] When interpreting components, even if there is no explicit description, it is interpreted as including the margin of error. When describing temporal relationships, for example, when temporal continuity is described with phrases such as "after," "following," "next to," or "before," this also includes cases where the relationship is not continuous, unless "immediately" or "directly" is used.

[0048] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

[0049] Fig. 1 illustrates a configuration diagram of a mixed reality-based ultrasound image output system (100) according to an embodiment of the present invention. Referring to Fig. 1, the mixed reality-based ultrasound image output system (100) may include a data receiving unit (110), a first content generating unit (120), a second content generating unit (130), a lesion detection unit (140), a synchronization unit (150), and an output device (160).

[0050] The mixed reality-based ultrasound image output system (100) can reduce operator fatigue by allowing the biopsy site and ultrasound to be viewed simultaneously, and can increase accuracy and stability by allowing the ultrasound image and lesion site to be superimposed on the biopsy site. The mixed reality-based ultrasound image output system (100) can output the subject's bio-signals together with the ultrasound image through an AR / VR / MR device or glasses, etc. The mixed reality-based ultrasound image output system (100) can analyze bio-signals, etc. and display the derived body risk signals in the operator's line of sight through an output device (AR / VR / MR device or glasses, etc.) worn by the operator so that the operator can immediately recognize them, thereby ensuring the patient's safety.

[0051] The mixed reality-based ultrasound image output system (100) has the advantage of ensuring the reliability of output information by synchronizing input signals and preventing needle insertion into different parts due to signals output at different points in time. The mixed reality-based ultrasound image output system (100) can increase the freedom of the operator by providing an environment in which equipment can be controlled without touching.

[0052] The mixed reality-based ultrasound image output system (100) can be provided in a smartphone tethering method and a stand-alone method.

[0053] FIG. 2 illustrates a configuration of a stand-alone mixed reality-based ultrasound image output system according to an embodiment of the present invention. Referring to FIG. 2, the mixed reality-based ultrasound image output system (100) can be operated independently without a separate user terminal (smartphone, laptop, etc.). The stand-alone method can perform the computing operations required for implementing mixed reality on the output device (160) itself. The stand-alone method has the advantage of convenient usability because there is no need to carry a user terminal.

[0054] Specifically, the stand-alone method can measure the ultrasound image and biosignal of a subject using an ultrasound device and a biosignal device, and then transmit them to a medical device data processing server. Here, the medical device data processing server may refer to the data receiving unit (110) of the present invention. The medical device data processing server can provide patient biosignal change information and ultrasound images to an AI analysis server (module) and an output device.

[0055] The AI ​​analysis server can output AI analysis results, which may refer to lesion detection. In other words, the AI ​​analysis server may refer to the lesion detection unit (140) of the present invention. The AI ​​analysis server can provide the AI ​​analysis results to an MR HMD (Head Mounted Display) or glasses via the Internet.

[0056] MR HMD or glasses can receive patient bio-signal change information and ultrasound images from a medical device data processing server via the Internet, and using the received data, can create first MR (Mixed-reality) content by aligning the ultrasound image with a specific object, and can create second MR content by aligning the bio-signals with a specific space. That is, in the stand-alone method, the first content generation unit (120) and the second content generation unit (130) can be included in the output device (160).

[0057] FIG. 3 illustrates a configuration of a mixed reality-based ultrasound image output system using a smartphone tethering method according to an embodiment of the present invention. Referring to FIG. 3, the mixed reality-based ultrasound image output system (100) can be connected to and operated by a smartphone. The smartphone tethering method can be performed on the smartphone without the output device (160) itself performing the computing operations required for implementing mixed reality. The smartphone tethering method has the advantage of being lightweight and generating less heat because the output device (160) does not have a processor or battery for computing.

[0058] Specifically, the smartphone tethering method can transmit patient bio-signal change information, ultrasound images, and AI analysis results to a smartphone via the Internet, and the smartphone can use the transmitted data to align the ultrasound images with a specific object to create first MR (Mixed-reality) content, and align the bio-signals with a specific space to create second MR content. That is, the smartphone tethering method may not include the first content generation unit (120) and the second content generation unit (130) within the output device (160).

[0059] The data receiving unit (110) can receive ultrasound images and bio-signals of a subject. The data receiving unit (110) can be connected to an ultrasound diagnostic device and a bio-signal measuring device by wire or wirelessly to receive ultrasound images and bio-signals in real time. The data receiving unit (110) can transmit the received ultrasound images and bio-signals to a first content generating unit (120), a second content generating unit (130), or a lesion detecting unit (140). When the data receiving unit (110) is connected wirelessly to another component, it can use a broad mobile communication network such as CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), or Wi-Fi. When the data receiving unit (110) is connected wirelessly to another component, it can use a short-range wireless communication such as Bluetooth.

[0060] The first content generation unit (120) can generate first MR (Mixed-reality) content by aligning an ultrasound image with a specific object. Here, the specific object may refer to a body part that can be directly superimposed 1:1 on the anatomical structure of the location to be measured by the ultrasound image.

[0061] The first content generation unit (120) can correct the first MR content by interacting with the ultrasound diagnostic device, the data receiving unit (110), or the output device (160). That is, the first content generation unit (120) can correct the first MR content so that the ultrasound image can be accurately aligned with a specific object through the shape of the probe of the ultrasound diagnostic device, the distance between the operator's eyes and the output device (160), etc.

[0062] Traditionally, tracking patterns (markers, QR codes, etc.) attached to the ultrasound diagnostic device's probe were analyzed to calculate the position where the ultrasound image should be aligned. However, when performing a biopsy or treatment using an ultrasound image, the practitioner must continuously move the probe to accurately acquire the required ultrasound image. During this process, the tracking pattern attached to the probe may become distorted or invisible. Technically, even if the tracking pattern is tilted or rotated by just 30 degrees, the camera cannot recognize it.

[0063] Calculating the matching position using a tracking pattern in this way has limitations. Therefore, the first content generation unit (120) according to an embodiment of the present invention can recognize the probe shape of the ultrasound diagnostic device through a camera equipped in the output device (160), detect the movement of the probe shape, and change the output position of the first MR content.

[0064] The first content generation unit (120) can output the first MR content in the direction in which the front of the probe faces through the recognized probe shape, and preferably, can change the output position of the first MR content so that the ultrasound image directly overlaps 1:1 with the anatomical structure of the location to be measured in the direction in front of the probe based on the part where the probe and the subject's body come into contact.

[0065] Ultrasound devices output ultrasound waves in a straight line perpendicular to the probe and visualize the reflected signals. Therefore, an ultrasound image is an image of a body part perpendicular to the probe. Therefore, outputting the first MR content so that the ultrasound image directly overlaps the anatomical structure of the location being measured on a 1:1 basis means that the output device will create a sense that the ultrasound image is physically located beneath the probe.

[0066] FIG. 4 is a diagram illustrating the relationship between the size of MR content output to an output device according to an embodiment of the present invention and the probe angle. Referring to FIG. 4 (a), when the probe outputs ultrasound perpendicularly to the subject's body, the ultrasound image area output within the practitioner's AR glasses can be confirmed. If the practitioner is wearing AR glasses, the measurement area under the probe is calculated to look like from the position of the practitioner's eyes wearing the AR glasses and output to the AR glasses. Therefore, if the MR image is output to the practitioner so as to be directly superimposed 1:1 on the anatomical structure of the location to be measured, the 3D location of the actual lesion inside the body can be accurately identified, allowing the needle to be inserted accurately into that location. The size of the ultrasound image output to the AR glasses can be determined through the calculation of how the measurement area under the probe looks like to the practitioner's eyes.

[0067] Referring to Figure 4 (b), when the probe outputs ultrasound at an obtuse angle relative to the subject's body, the ultrasound image area displayed within the operator's AR glasses can be confirmed. In this case, the viewing angle for the measurement area beneath the probe is greater than when the probe outputs ultrasound perpendicular to the subject's body, and accordingly, the size of the ultrasound image area displayed within the AR glasses also increases.

[0068] Referring to (c) of Fig. 4, when the probe outputs ultrasound at an acute angle to the subject's body, the ultrasound image area displayed within the operator's AR glasses can be confirmed. In this case, the viewing angle for the measurement area beneath the probe becomes smaller than when the probe outputs ultrasound perpendicular to the subject's body, and accordingly, the size of the ultrasound image area displayed within the AR glasses also becomes smaller.

[0069] As examined through (a) to (c) of Fig. 4, the size of the ultrasound image displayed on the output device may vary depending on the positional relationship between the output device worn by the practitioner and the probe, such as the angle of the probe and the distance between the practitioner and the probe. However, if the size of the displayed ultrasound image is excessively small, the practitioner may have difficulty performing the procedure while viewing the ultrasound image. To solve this, as shown in Fig. 4 (d), if the size of the ultrasound image area displayed in the output device is below a certain level, a separate ultrasound image may be displayed within the output device. Therefore, the practitioner can safely perform the procedure through the separately displayed ultrasound image even if the ultrasound image is displayed excessively small depending on the positional relationship between the output device and the probe. In addition, a separate ultrasound image may be displayed in a specific area of ​​the AR display according to the practitioner's control.

[0070] The first content generation unit (120) determines the location where the ultrasound image will be enhanced not by detecting a tracking pattern, but by determining it through the shape of the probe, so that a reliable ultrasound image can be output even when the probe moves during the procedure.

[0071] In the case of conventional technologies using tracking patterns, the tracking pattern had to be attached to a probe, etc., in order to enhance ultrasound images through an MR HMD or glasses. The mixed reality-based ultrasound image output system (100) according to an embodiment of the present invention utilizes the shape of the probe without using a tracking pattern, and therefore can be applied to existing ultrasound diagnostic devices by simply equipping modular equipment without separate installation.

[0072] The first content generation unit (120) can change the first MR content so that the lesion area detected by the lesion detection unit (140) can be distinguished from other areas. Although the operator can distinguish the lesion in the ultrasound image, it is very difficult to distinguish the lesion area from other areas. Therefore, currently, whether or not to detect the lesion is determined based on the operator's ability. The first content generation unit (120) can emphasize the outer edge of the lesion area detected by the lesion detection unit (140) or emphasize it by changing the color of the inside of the lesion area. If the user wants to check the inside of the lesion according to the user's settings, the first content generation unit (120) can emphasize the outer edge of the detected lesion area. If the user wants to check the size of the lesion according to the user's settings, the first content generation unit (120) can emphasize the lesion area by changing the color of the inside of the lesion area.

[0073] If the first content generation unit (120) confirms that the needle has contacted the lesion area through ultrasound imaging, it can modify the MR content so that the lesion area is indistinguishable from other areas. Even if the practitioner has identified the lesion area, accurately inserting a needle into the lesion area to acquire lesion tissue or treat the lesion is a very difficult skill. Therefore, the first content generation unit (120) can visually notify the practitioner that the lesion area has been contacted by restoring the highlighted lesion area to be identical to other areas when the needle has accurately pierced the lesion area.

[0074] The second content generation unit (130) can generate second MR content by aligning the biosignal to a certain space. Here, the certain space means a space where the operator can check the output biosignal while focusing on the ultrasound image and performing a biopsy or treatment. Preferably, it can mean the gaze direction of the operator's right or left eye. The biosignal can include an electrocardiogram (ECG), an electroencephalogram (EEG), an electromyogram (EMG), a galvanic skin response (GSR), a skin temperature (SKT), a photoplethysmography (PPG), a pulse rate (PR), blood pressure, oxygen saturation, etc.

[0075] The second content generation unit (130) can determine that a dangerous situation has occurred for the subject if the bio-signal exceeds the reference value and can provide a warning or notification to the operator. The second content generation unit (130) can modify the second MR content so that the screen on which the bio-signal is output blinks or flashes red when a bio-signal abnormality occurs.

[0076] Fig. 5 shows a configuration diagram of a lesion detection unit (140) according to an embodiment of the present invention. Referring to Fig. 5, the lesion detection unit (140) may include an input unit (141), a preprocessing unit (142), a learning unit (143), a lesion prediction unit (144), and a postprocessing unit (145).

[0077] The lesion detection unit (140) can detect lesion areas in ultrasound images using artificial intelligence trained on an ultrasound image data set in which lesion areas are annotated. Specifically, the lesion detection unit (140) can detect lesion areas in ultrasound images using a semantic segmentation model. The semantic segmentation model is a process of dividing a digital image into multiple sets of pixels, and through segmentation, the representation of the image can be simplified and converted into something easy to interpret. The lesion detection unit (140) can use a CNN-based model as the semantic segmentation model.

[0078] The input unit (141) can receive ultrasound images of normal and patient groups for learning, testing, or actual lesion detection. The input unit (141) can receive ultrasound images from the data receiving unit (110). The input unit (141) can transmit the received ultrasound images to the preprocessing unit (142).

[0079] The preprocessing unit (142) can preprocess the ultrasound image received from the input unit (141) to be advantageous for learning or detection. The preprocessing unit (142) can perform preprocessing in a way that normalizes pixel values. The preprocessing unit (142) can perform preprocessing to adjust the size and resolution of the ultrasound image received so that it can be input into the artificial neural network. The ultrasound image used for learning may contain the patient's personal information, which may cause security and personal information leakage issues. Therefore, the preprocessing unit (142) according to the present invention can perform preprocessing to de-identify the patient information in the ultrasound image and metadata.

[0080] The learning unit (143) can train artificial intelligence using preprocessed ultrasound images with lesion regions annotated. The artificial intelligence trained here may be a CNN-based or SOTA architecture-based semantic segmentation model.

[0081] A convolutional neural network (CNN) is a type of deep learning model inspired by the structure of the visual cortex of animals, designed to process data with grid patterns, such as images. A convolutional neural network typically includes convolutional layers, pooling layers, and fully connected layers. Convolutional and pooling layers can exist repeatedly within a neural network, and input data can be transformed into output through these layers. Convolutional layers utilize kernels (or masks) to extract features. The element-wise product between each element of the kernel and the input value is computed at each location and summed to obtain the output, which is called a feature map. This process can be repeated, applying multiple kernels to form any number of feature maps. In a convolutional neural network, convolutional and pooling layers perform feature extraction, while fully connected layers map the extracted features to the final output, such as a classification operation.

[0082] Convolutional neural networks, such as neural networks, can be trained to minimize output errors. Separate from the forward propagation process, which extracts values ​​from the input layer to the output layer, backpropagation occurs within the neural network, calculating the error between the input training data and the corresponding neural network output values ​​and updating the weights of the nodes in each layer to reduce this error. The training process in convolutional neural networks can be summarized as finding the kernel that extracts the output values ​​with the lowest error based on the given training data. The kernel is the only parameter that is automatically learned during the training of the convolutional layer. On the other hand, in convolutional neural networks, kernel size, number of kernels, padding, etc. are hyperparameters that must be set before the training process begins. Therefore, convolutional neural network models can be distinguished based on the kernel size, number of kernels, and number of convolutional and pooling layers.

[0083] The lesion prediction unit (144) can predict and detect the lesion area from the subject's ultrasound image using the artificial intelligence learned in the learning unit (143).

[0084] The post-processing unit (145) can perform post-processing to distinguish the lesion area detected by the lesion prediction unit (144) from other areas. This is the same function as that performed by the first content generation unit (120) described above.

[0085] Figure 6 illustrates a process for detecting lesions using artificial intelligence by a lesion detection unit (140) according to an embodiment of the present invention. Referring to Figure 6, the lesion detection unit (140) receives an ultrasound image, inputs it into a CNN (Convolutional Neural Network), and then inputs it into a high-performance Semantic Segmentation model to obtain a binary mask that distinguishes the lesion prediction area from other areas. Thereafter, the lesion detection unit (140) can select the lesion area from the lesion prediction area.

[0086] Fig. 7 illustrates a configuration diagram of a synchronization unit (150) according to an embodiment of the present invention. Referring to Fig. 7, the synchronization unit (150) may include a delay synchronization module (151) and a prediction synchronization module (152).

[0087] The synchronization unit (150) can synchronize the ultrasound image, the bio-signal, and the lesion area so that the ultrasound image and bio-signal measured at the same time can be output, and the lesion area detected at the same time can be displayed. The synchronization unit (150) plays an important role in enabling the operator to perform biopsy and treatment based on information at the same time. If the ultrasound image and the lesion area are measured at different times, the operator expects that there will be a lesion in the displayed lesion area, but in reality, there may be no lesion at that location because it is a lesion area at a different time.

[0088] Thus, the reason why synchronization of each signal is necessary is because the time required for the ultrasound image, biosignal, or lesion area to be constructed, detected, or transmitted varies. The synchronization unit (150) can ensure stability and reliability by synchronizing the ultrasound image, biosignal, or lesion area displayed to the operator via the output device (160) with those measured at the same time.

[0089] Fig. 8 is a diagram illustrating a synchronization method of a delay synchronization module according to an embodiment of the present invention. Referring to Fig. 8, the delay synchronization module (151) can synchronize based on the most delayed signal among the ultrasound image, biosignal, and lesion area when the lesion area detected by the lesion detection unit (140) is smaller than a preset reference value.

[0090] Specifically, the lesion area, ultrasound image, and biosignal can be delayed by T1, T2, and T3, respectively, so that the signals can arrive (T1>T2>T3). At this time, if synchronization is performed based on the lesion area with the longest delay time, synchronization is easy because measured values ​​already exist at the synchronization reference time. That is, the ultrasound image can be synchronized with the lesion area by outputting a signal from a time point earlier than Tp2, and the biosignal can be synchronized with the lesion area by outputting a signal from a time point earlier than Tp3. This synchronization method has the advantage of high reliability and accuracy because measured values ​​already exist, but has the disadvantage of a long delay time.

[0091] Fig. 9 is a diagram illustrating a synchronization method of a prediction synchronization module (152) according to an embodiment of the present invention. Referring to Fig. 9, the prediction synchronization module (152) can synchronize based on the most advanced signal among the ultrasound image, biosignal, and lesion area when the lesion area detected by the lesion detection unit (140) is larger than a preset reference value.

[0092] Specifically, if synchronization is performed based on the biosignal with the shortest delay time, there is a problem that the lesion area and the ultrasound image do not have data that has reached the synchronization reference point. Therefore, the prediction synchronization module (152) can synchronize another signal to the most advanced signal using extrapolation. Extrapolation refers to a method used when obtaining a value that exceeds the limit because the available data is limited. In another embodiment, the prediction synchronization module (152) can predict data up to the point in time of the most advanced signal using artificial intelligence.

[0093] That is, the prediction synchronization module (152) can synchronize with the biosignal by predicting the signal at a time point ahead by Tf1 and outputting the lesion area, and predicting the signal at a time point ahead by Tf2 and outputting the ultrasound image. This synchronization method has the advantage of short delay time because it synchronizes at the shortest delay time, but has the disadvantage of low reliability and accuracy because it predicts and outputs values ​​that have not yet been measured.

[0094] Accordingly, the mixed reality-based ultrasound image output system (100) according to an embodiment of the present invention can use a delay synchronization module (151) with high accuracy when the detected lesion is small, and can use a prediction synchronization module (152) with short delay time when the detected lesion is large. The delay synchronization module (151) and the prediction synchronization module (152) can also be selected according to user settings.

[0095] Fig. 10 illustrates a screen displayed through an output device (160) according to an embodiment of the present invention. Referring to Fig. 10, the output device (160) can display MR content generated by the first content generation unit (120) or the second content generation unit (130). The output device (160) may be a device that can be worn by a practitioner.

[0096] The output device (160) may be an AR / VR / MR device, glasses, or HMD capable of displaying the first MR content and the second MR content in an augmented manner. The output device (160) may include a processor and a battery to perform the computing operations required for implementing mixed reality. To reduce weight, the output device (160) may not include a processor and a battery to perform the computing operations.

[0097] The output device (160) may include a first camera that captures the wearer's forward direction and a second camera that captures the wearer's face. The first camera can recognize the probe shape of the ultrasound diagnostic device, and the second camera can track the wearer's gaze direction. The first camera may be used to recognize the probe shape in the aforementioned first content generation unit (120).

[0098] The output device (160) can display second MR content along the wearer's gaze direction tracked by the second camera. As described above, the second MR content can be composed of biosignals, and the status of the biosignals must be detectable even when the operator is focusing on the ultrasound image. Accordingly, the output device (160) can detect the gaze direction of the wearer's left or right eye using the second camera and display the second MR content near the corresponding gaze direction.

[0099] The output device (160) can be controlled by the movement of the wearer's eyes or pupils captured by the second camera. The practitioner holds an ultrasound probe in one hand for ultrasound diagnosis and a needle in the other hand for biopsy or treatment. Therefore, in order to control the output device (160), the practitioner must put down the probe or needle he or she is holding or request an assistant to do so. However, there are limitations in controlling the output device (160) frequently or in the event of an emergency where the output device (160) must be quickly controlled. Therefore, the output device (160) can be controlled by the blinking of the wearer's eyes or the movement of the pupils, thereby providing the practitioner with a high degree of freedom. Here, control may mean changing the UI, turning the first MR content or the second MR content on / off, etc.

[0100] Another embodiment of the present invention, a mixed reality-based ultrasound image output method, may include a data receiving step, a first content generating step, a second content generating step, a lesion detection step, a synchronization step, and a displaying step.

[0101] The data receiving step can receive ultrasound images and biosignals of a subject from an ultrasonic diagnostic device and a biosignal measuring device. The data receiving step refers to an operation performed in the data receiving unit (110) described above.

[0102] The first content generation step can generate first MR (Mixed-reality) content by aligning an ultrasound image with a specific object. The first content generation step can recognize the probe shape of the ultrasound diagnostic device through a camera equipped in the output device, and detect the movement of the probe shape to change the output location of the first MR content. The first content generation step refers to the operation performed by the first content generation unit (120) described above.

[0103] The second content generation step can generate second MR content by aligning biosignals to a specific space. The second content generation step refers to the operation performed in the second content generation unit (130) described above.

[0104] The lesion detection step can detect lesion areas in ultrasound images using artificial intelligence trained on an ultrasound image data set in which lesion areas are annotated. The lesion detection step refers to the operation performed in the lesion detection unit (140) described above.

[0105] The synchronization step can synchronize the ultrasound image, biosignal, and lesion area so that the ultrasound image and biosignal measured at the same time can be output, and the lesion area detected at the same time can be displayed. The synchronization step refers to the operation performed in the synchronization unit (150) described above.

[0106] The displaying step may display the MR content generated in the first content generation step or the second content generation step on an output device. The displaying step refers to an operation performed on the aforementioned output device (160).

[0107] While the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims described below but also by all changes or modifications derived from the claims and equivalent concepts.

[0108] [Explanation of symbols]

[0109] 100: Mixed Reality-Based Ultrasound Image Output System

[0110] 110: Data receiving unit 120: First content generating unit

[0111] 130: Second content generation unit 140: Lesion detection unit

[0112] 141: Input section 142: Preprocessing section

[0113] 143: Learning section 144: Lesion prediction section

[0114] 145: Post-processing unit 150: Synchronization unit

[0115] 151: Delayed Synchronization Module 152: Predictive Synchronization Module

[0116] 160: Output device

[0117] The present invention aims to provide a method and system for recognizing context of a digital slide image, to which a contextual image expression method that is invariant to rotational transformation and can reflect the surrounding context is applied.

Claims

1. A data receiving unit that receives ultrasound images and bio-signals of the subject; A first content generation unit that generates first MR (Mixed-reality) content by aligning the above ultrasound image with a specific object; A second content generation unit that generates second MR content by aligning the above biosignals to a certain space; and An output device that displays MR content generated by the first content generation unit or the second content generation unit; The above first content creation unit, A mixed reality-based ultrasound image output system characterized in that it recognizes the probe shape of an ultrasound diagnostic device through a camera equipped in the output device, detects movement of the probe shape, and changes the output position of the first MR content.

2. In paragraph 1, A mixed reality-based ultrasound image output system further comprising a lesion detection unit that detects a lesion area in the ultrasound image using artificial intelligence learned from an ultrasound image data set in which the lesion area is annotated.

3. In paragraph 2, The above first content creation unit, A mixed reality-based ultrasound image output system characterized in that the first MR content is changed so that the lesion area detected by the lesion detection unit is distinguished from other areas.

4. In paragraph 3, The above first content creation unit, A mixed reality-based ultrasound image output system characterized in that, when it is confirmed through the ultrasound image that the needle has contacted the lesion area, the MR content is changed so that the lesion area is not distinguished from other areas.

5. In paragraph 2, A mixed reality-based ultrasound image output system characterized by further including a synchronization unit that synchronizes the ultrasound image, the bio-signal, and the lesion area so that the ultrasound image and bio-signal measured at the same time can be output and the lesion area detected at the same time can be displayed.

6. In paragraph 5, The above synchronization unit, A delay synchronization module that synchronizes the most delayed signal among the ultrasound image, the biosignal, and the lesion area when the lesion area detected by the lesion detection unit is smaller than a preset reference value; and A prediction synchronization module that synchronizes based on the most advanced signal among the ultrasound image, the biosignal, and the lesion area when the lesion area detected by the lesion detection unit is larger than a preset reference value; A mixed reality-based ultrasound image output system characterized by including:

7. In paragraph 6, The above prediction synchronization module, A mixed reality-based ultrasound image output system characterized by synchronizing another signal to the most advanced signal using extrapolation.

8. In paragraph 1, The above output device is, It includes a first camera that photographs the wearer's forward direction and a second camera that photographs the wearer's face, The above first camera, Recognize the probe shape of the ultrasound diagnostic device, The above second camera, A mixed reality-based ultrasound image output system characterized by tracking the wearer's gaze direction.

9. In paragraph 8, The above output device is, A mixed reality-based ultrasound image output system characterized in that the second MR content is displayed along the wearer's gaze direction tracked by the second camera.

10. In paragraph 8, The above output device is, A mixed reality-based ultrasound image output system characterized in that it is controlled by the movement of the wearer's eyes or pupils captured by the second camera.

11. Data receiving step for receiving ultrasound images and bio-signals of the subject; A first content generation step of generating first MR (Mixed-reality) content by aligning the above ultrasound image with a specific object; A second content generation step of generating second MR content by aligning the above biosignals to a certain space; and A step of displaying MR content generated in the first content generation step or the second content generation step on an output device; The above first content creation step is: A mixed reality-based ultrasound image output method characterized in that the probe shape of an ultrasound diagnostic device is recognized through a camera provided in the output device, and the movement of the probe shape is detected to change the output position of the first MR content.

Images