Method and apparatus for realizing operation of object on basis of hand movement in mixed reality

Abstract

According to various embodiments, there may be provided a method and apparatus for realizing an operation of an object on the basis of hand movement in a mixed reality, the method comprising the steps of: realizing a three-dimensional reconstruction model generated for a specific patient in a mixed reality space; detecting an insertion object provided to function with the three-dimensional reconstruction model and a hand of a user manipulating the insertion object; determining movement of the insertion object corresponding to hand movement of the user; and realizing the movement of the insertion object inserted into the three-dimensional reconstruction model.

Description

Method and device for implementing object motion based on hand movements in mixed reality

[0001] The present disclosure relates to a method and a device for implementing motion of an object based on hand movements in mixed reality.

[0002] The number of cancer patients has been steadily increasing in recent years, and the incidence of cancer among young people in particular has been increasing significantly.

[0003] In this regard, invasive tube insertion, which involves inserting equipment such as central venous catheter insertion into the patient's body to treat lesions, is mainly performed as a method of chemotherapy.

[0004] In order to minimize side effects of this invasive tube insertion procedure, it is important that the tip is accurately positioned in a specific location inside the body. The length of the tube to be inserted during the intubation surgery varies from patient to patient, so X-ray or C-arm images are taken several times during the procedure to confirm that the tip is in the correct position.

[0005] In other words, the number of times a patient is exposed to radiation greatly depends on the skill of the medical staff, which causes variations in radiation exposure, making it difficult to consistently apply the principles of medical radiation optimization.

[0006] In addition, in order to improve the proficiency of medical personnel in invasive tube insertion, anatomical models for various lesions need to be provided for practice, but the types and numbers of models provided for practice are extremely limited.

[0007] As a way to solve these problems, there is a need to provide an educational model for invasive tube insertion or a method and device that can assist medical staff in performing invasive tube insertion.

[0008] According to various embodiments, a method and device for acquiring hand movements in a mixed reality space and implementing motions of objects in response thereto can be provided.

[0009] The problems to be solved through various embodiments of the present invention are not limited to the problems mentioned above, and other problems not mentioned above will be clearly understood by those skilled in the art from the description below.

[0010] According to one embodiment, a method for implementing an object's motion based on hand movements in mixed reality can be provided, including: implementing a three-dimensional reconstruction model generated for a specific patient in a mixed reality space; detecting an insertion object provided to function with the three-dimensional reconstruction model and a user's hand manipulating the insertion object; determining a movement of the insertion object corresponding to a movement of the user's hand; and implementing a movement of the insertion object inserted into the three-dimensional reconstruction model.

[0011] Here, the method for implementing an object's motion based on hand movements in the mixed reality may further include the step of implementing in the mixed reality space a step of detecting at least a part of the actual body of the specific patient in the mixed reality space; and a step of aligning the 3D reconstruction model with at least a part of the actual body of the specific patient.

[0012] Here, the matching step may be performed based on a body surface virtual landmark formed outside the 3D reconstruction model and a body surface real landmark formed outside at least a part of the actual body.

[0013] Here, in the step of implementing the movement of the inserted object or before performing the step of implementing the movement of the inserted object, a step of projecting an internal image of the three-dimensional reconstruction model onto at least a part of the actual body may be further included.

[0014] Here, the internal image of the three-dimensional reconstruction model may be an image in which at least some elements of blood vessels, organs, tissues, nerves, and bones are implemented.

[0015] Here, when a virtual insertion object is detected for the insertion object, the projecting step determines that at least a part of the tip of the virtual insertion object and the body of the virtual insertion object are inserted into the skin configured in the 3D reconstruction model or the skin of the actual body of the patient based on the movement of the virtual insertion object, and projects the internal image of the 3D reconstruction model, and displays the movement of the inserted part of the virtual insertion object including the tip of the virtual insertion object through the internal image.

[0016] Here, when a real insertion object is detected for the insertion object, the projecting step may determine, based on the movement of the real insertion object, that at least a part of the tip of the real insertion object and the body of the real insertion object are inserted into the skin configured in the 3D reconstruction model or the skin of the actual body of the patient, project the internal image of the 3D reconstruction model, predict the movement of the inserted part of the real insertion object including the tip of the real insertion object, and display the movement of the inserted part of the real insertion object through the internal image.

[0017] Here, the step of implementing the movement of the inserted object may display the tip of the inserted object and at least a part of the body of the inserted object by moving forward, backward, or rotating according to the function of the inserted object corresponding to the movement of the user's hand.

[0018] Here, the step of implementing the movement of the inserted object may further include the step of determining whether the state of at least a portion of the tip of the inserted object and the body of the inserted object predicted according to the implemented movement of the inserted object satisfies a preset guideline; and, if the guideline is determined not to be satisfied, the step of displaying information on the unsatisfied condition element of the guideline and the influence according to the unsatisfied condition element in the mixed reality space.

[0019] Here, the above guidelines can be configured to reflect the state of the body into which the insertion object is inserted or is expected to move.

[0020] Here, the three-dimensional reconstruction model may be a fluoroscopic image of at least a part of the body of the specific patient generated by segmenting at least some elements of skin, blood vessels, organs, tissues, and bones based on at least some of the CT images, MRI images, and ultrasound images acquired for the specific patient, and reconstructing and combining the segmented at least some elements in three dimensions.

[0021] According to another embodiment, a device for implementing an object's motion based on hand movements in mixed reality can be provided, including: an object implementation unit for implementing a three-dimensional reconstruction model generated for a specific patient in a mixed reality space and implementing a movement of an inserted object inserted into the three-dimensional reconstruction model; an object detection unit for detecting the inserted object and a user's hand manipulating the inserted object, the inserted object being provided to function with the three-dimensional reconstruction model; and a function processing unit for determining a movement of the inserted object corresponding to the user's hand movement.

[0022] Here, the object detection unit can detect at least a part of the actual body of the specific patient in the mixed reality space, and align the 3D reconstruction model with at least a part of the actual body of the specific patient to implement the 3D reconstruction model in the mixed reality space.

[0023] Here, the object implementation unit can perform the alignment based on a body surface virtual landmark formed outside the 3D reconstruction model and a body surface real landmark formed outside at least a part of the actual body.

[0024] Here, the object implementation unit can project an internal image of the 3D reconstruction model onto at least a part of the actual body.

[0025] Here, the internal image of the three-dimensional reconstruction model may be an image in which at least some elements of blood vessels, organs, tissues, nerves, and bones are implemented.

[0026] Here, when the object detection unit detects a virtual insertion object for the insertion object, the object implementation unit determines, based on the movement of the virtual insertion object, that at least a part of the tip of the virtual insertion object and the body of the virtual insertion object are inserted into the skin configured in the 3D reconstruction model or the skin of the actual body of the patient, the object implementation unit projects the internal image of the 3D reconstruction model and displays the movement of the inserted part of the virtual insertion object including the tip of the virtual insertion object through the internal image.

[0027] Here, when the object detection unit detects a real insertion object for the inserted object, the object implementation unit determines that at least a part of the tip of the real insertion object and the body of the real insertion object are inserted into the skin configured in the 3D reconstruction model or the skin of the actual body of the patient based on the movement of the real insertion object, and projects the internal image of the 3D reconstruction model, predicts the movement of the inserted part of the real insertion object including the tip of the real insertion object, and displays the movement of the inserted part of the real insertion object through the internal image.

[0028] Here, the object implementation unit can implement the movement of the insertion object by displaying the tip of the insertion object and at least a part of the body of the insertion object moving forward, backward, or rotating according to the function of the insertion object corresponding to the movement of the user's hand.

[0029] Here, the device for implementing the motion of an object based on hand movement in the mixed reality may further include a motion determination unit that determines whether the state of at least a part of the tip of the inserted object and the body of the inserted object predicted according to the movement of the implemented inserted object satisfies a preset guideline, and, if it determines that the guideline is not satisfied, displays information on the unsatisfied condition element of the guideline and the influence according to the unsatisfied condition element in the mixed reality space.

[0030] Here, the above guidelines can be configured to reflect the state of the body into which the insertion object is inserted or is expected to move.

[0031] Here, the three-dimensional reconstruction model may be a fluoroscopic image of at least a part of the body of the specific patient generated by segmenting at least some elements of skin, blood vessels, organs, tissues, and bones based on at least some of the CT images, MRI images, and ultrasound images acquired for the specific patient, and reconstructing and combining the segmented at least some elements in three dimensions.

[0032] According to the above, the device for implementing the motion of an object based on hand movements has the effect of providing an environment in which the position and movement of the device can be accurately confirmed without using radiation when performing an invasive tube insertion surgery that inserts the device into the patient's body.

[0033] According to various embodiments, a device for implementing an object's motion based on hand movements provides a surgical assistance system that anatomically accurately displays biological elements inside a patient's body by projecting a three-dimensional reconstruction model created by reflecting anatomical elements of a part of the patient's body onto the patient, thereby reducing the risk of accidents in invasive tube insertion surgery that inserts equipment into the patient's body.

[0034] According to various embodiments, a device for implementing motion of an object based on hand movements can provide an educational opportunity for treating various cases of diseases using invasive tube insertion surgery by providing realistic three-dimensional reconstruction models of the internal body of various patients.

[0035] According to various embodiments, a device for implementing an object's motion based on hand movements can provide a surgical environment that reduces the physical burden on a patient by providing visual and auditory information about the insertion depth, insertion direction, and insertion angle of a tube inserted into a patient's body based on hand movements.

[0036] FIG. 1 is a schematic diagram illustrating a configuration of a device that implements object motion based on hand movements in mixed reality according to one embodiment.

[0037] FIG. 2 is a schematic diagram illustrating the operation of a processing unit for implementing the operation of an object based on hand movement in mixed reality according to one embodiment, divided by function.

[0038] FIG. 3 is a schematic diagram illustrating the external configuration of a device that implements object motion based on hand movements in mixed reality according to one embodiment.

[0039] FIG. 4 is a flowchart illustrating a flow of operations in which a device according to one embodiment performs processing of an object in a mixed reality space.

[0040] FIG. 5 is a diagram illustrating a flow of operations for implementing a three-dimensional reconstruction model by a device according to one embodiment.

[0041] FIG. 6 is a diagram illustrating a flow of detailed operations for implementing a three-dimensional reconstruction model by a device according to one embodiment.

[0042] FIG. 7 is a diagram illustrating a flow of operations for implementing movement of an inserted object in a device according to one embodiment.

[0043] FIG. 8 is a drawing for explaining an operation of implementing a three-dimensional reconstruction model in a device according to one embodiment.

[0044] FIG. 9 is a drawing illustrating an operation of aligning a three-dimensional reconstruction model to a corresponding body part of a patient in a device according to one embodiment.

[0045] FIG. 10 is a drawing illustrating an operation of implementing movement of an object inserted into a patient's body according to hand movement in a device according to one embodiment.

[0046] FIG. 11 is a drawing for explaining an operation of determining joints and fingertips from a user's hand identified by the device.

[0047] FIG. 12 is a drawing for explaining an operation of implementing movement of an inserted object in response to hand movement in a device according to one embodiment.

[0048] FIG. 13 is a drawing for explaining another operation for implementing movement of an inserted object in response to hand movement in a device according to one embodiment.

[0049] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0050] The terms used in the examples are for the purpose of description only and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood to not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0051] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0052] In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

[0053] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the embodiment. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms.

[0054] When a component is described as being 'connected', 'coupled' or 'connected' to another component, it should be understood that the component may be directly connected or connected to that other component, but that there may also be other components 'connected', 'coupled' or 'connected' between the components.

[0055] Additionally, when it is described that a component is 'connected' or 'connected' to another component by communication, it should be understood that the component may be connected or connected to that other component by wireless or wired communication, but that other components may be 'connected' or 'connected' between the components.

[0056] Components included in one embodiment and components with common functions will be described using the same names in other embodiments. Unless otherwise stated, the descriptions given in one embodiment may also apply to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

[0057] The 'data' processed by the device and / or the device can be expressed in terms of 'information'. Here, information can be used as a concept that includes data.

[0058] The present disclosure relates to a method and a device for implementing motion of an object based on hand movements in mixed reality.

[0059] More specifically, a method and device for implementing a 3D reconstruction model created for the inside of a part of a patient's body in mixed reality and an object inserted into the patient's body in a mixed reality space, detecting a user's hand in the mixed reality space, and implementing movement of the object in the 3D reconstruction model based on the hand movement manipulating the object can be described.

[0060] To this end, a device (hereinafter, “device”) that implements object motion based on hand movements in mixed reality may be provided as a device for processing mixed reality (MR). However, without limitation thereto, the device may also be provided as a device for processing virtual reality (VR) or augmented reality (AR).

[0061] These devices can implement a mixed reality space that overlaps at least partly with the real space, and can implement some objects existing in the real world through the mixed reality space, or implement some objects created in the virtual space into the real world through the output unit (display or lens) of the device.

[0062] Here, the real space for implementing the mixed reality space may be a real space seen through the lens of the device, and / or a real space captured through a camera and then output through a display.

[0063] A mixed reality space may include a space created based on virtual reality, augmented reality, and / or mixed reality in a real space or virtual space implemented as described above.

[0064] Mixed reality spaces, or virtual spaces, can encompass three-dimensional (3D) space and the metaverse of 3D space. However, mixed reality spaces can also encompass virtual reality of 2D space, as well as 3D space or the metaverse of 3D space.

[0065] In describing various embodiments, the terms mixed reality or mixed reality space may be used as expressions that include spatiotemporal meanings implemented through devices such as virtual reality, augmented reality, and / or mixed reality.

[0066] Hereinafter, various embodiments of the present disclosure will be described with reference to the drawings. Since the drawings attached to this specification serve to further understand the technical idea of ​​the present invention together with the detailed description of the invention, the present invention should not be interpreted as being limited only to matters described in such drawings.

[0067] FIG. 1 is a schematic diagram illustrating a configuration of a device for implementing object motion based on hand movements in mixed reality according to one embodiment. FIG. 2 is a schematic diagram illustrating the operation of a processing unit for implementing object motion based on hand movements in mixed reality according to one embodiment, dividing the operation by function. FIG. 3 is a schematic diagram illustrating an external configuration of a device for implementing object motion based on hand movements in mixed reality according to one embodiment.

[0068] Referring to FIG. 1, the device (100) may be configured to include a processing unit (110), a storage unit (120), a camera (130), a communication unit (150), and an input / output unit (140).

[0069] Here, the device (100) is illustrated as including a communication unit (150), but is not limited thereto, and the communication unit (150) may be selectively included / excluded during the design and / or manufacturing stages.

[0070] The processing unit (110) can perform a function according to a control command received from an input device through at least one program (e.g., an application, a tool, a plug-in, etc., hereinafter referred to as an object processing program). At this time, the object processing program can be stored in the storage unit (120) of the device (100) and / or in the storage unit of an external device connected to the device (100).

[0071] The processing unit (110) can obtain control commands from an input device that is not connected via electricity or communication, based on an object processing program. To this end, the processing unit (110) can detect the user's hand via a camera (130) and determine the movement of at least one object manipulated by the user's hand.

[0072] Here, the object manipulated by the user's hand may be an actual object or a virtual object implemented through virtual reality.

[0073] To be more specific, the object may be a specific object within the scope of medical devices inserted into a patient's body in a medical environment, such as a catheter, cannula, endoscope, tube, or needle.

[0074] However, without being limited thereto, the object may be configured to include a user's hand captured through a camera (130), a patient's body, a 3D reconstruction model related to a specific patient implemented in a mixed reality space, etc.

[0075] At this time, objects inserted into the patient's body can be expressed as inserted objects.

[0076] The processing unit (110) can detect an inserted object from an image captured by a camera (130) or implement a virtual inserted object in a mixed reality space based on user input.

[0077] The processing unit (110) can detect a motion of manipulating an object with a hand detected through the camera (130) for an object photographed through the camera (130), predict the movement of the object based on the movement of the hand, and implement the movement of the object in a mixed reality space.

[0078] To explain in more detail, in a situation where a part of an object is inserted into a patient's body, the processing unit (110) can predict the movement of the object, particularly the movement of at least a part of the object inserted into the patient's body, based on the hand movement of the user manipulating the object, and implement the movement of at least a part of the object through a mixed reality space.

[0079] Additionally, for a target that implements the movement of an object, the processing unit (110) can implement a 3D reconstruction model that has been previously generated in response to a specific body of a patient in a mixed reality space. The processing unit (110) can implement the movement of the object in the implemented 3D reconstruction model.

[0080] To this end, the processing unit (110) may be configured to include an object implementation unit (201), an object detection unit (203), a function processing unit (205), and an operation determination unit (207), as illustrated in FIG. 2.

[0081] According to various embodiments of the present disclosure, at least some of the processing unit (110), and the object implementation unit (201), object detection unit (203), function processing unit (205), and operation determination unit (207) constituting the processing unit may be configured through at least one controller (or processor) and / or configured to operate as a mechanical control device.

[0082] The storage unit (120) can store various data processed by at least one component (e.g., processing unit (110)) of the device (100) and / or image data captured through the camera (130). The data (and / or image data) can include, for example, an object processing program (or software) and input data or output data related thereto.

[0083] The storage unit (120) may include an artificial intelligence algorithm including at least some of an artificial neural network algorithm, a blockchain algorithm, a deep learning algorithm, a regression analysis algorithm, and mechanisms, operators, language models, and big data related thereto for performing object processing.

[0084] For example, the storage unit (120) may include an algorithm for detecting an actual inserted object from an image captured by a camera or implementing a virtual inserted object, identifying a user's hand, and detecting the movement of the hand manipulating the detected or implemented inserted object, and / or an algorithm for performing a specified operation (e.g., regression analysis, big data analysis) in relation thereto, in order to process the movement of the object.

[0085] The storage unit (120) can store data for determining and processing control and operation of devices through data received through the input / output unit (140) and / or the communication unit (150).

[0086] The operations described for the storage unit (120) are processed by the processing unit (110), and data for processing related operations, data being processed, processed data, preset data, etc. can be stored in the storage unit (120) as a database.

[0087] Data stored in the storage unit (120) can be changed, modified, deleted, and / or new data can be created by the processing unit (110) based on an administrator input of the device (100) or an administrator input of an administrator device (or user device) connected to the device (100).

[0088] The storage unit (120) can store device setting information of the device (100). The device setting information may be setting information for at least some of the functions of the device (100).

[0089] The storage unit (120) may store user information for at least one patient. Each user information may store at least some of user identification information (e.g., identification, ID), a password, and user-customized setting information. The user-customized setting information may be setting information for at least some of the control rights and / or functions of the device (100), and may be set and stored based on administrator input.

[0090] Additionally, the storage unit (120) may store at least one image matched with user identification information, for example, at least some images among CT images, MRI images, and ultrasound images of a specific body.

[0091] In addition, the storage unit (120) can store a three-dimensional reconstruction model for a specific body generated based on at least some images among CT images, MRI images, and ultrasound images.

[0092] The storage unit (120) may be configured to include volatile memory or non-volatile memory.

[0093] Referring to FIG. 3, the camera (130) may be configured to include at least one camera that is equipped to capture an image in a range identical or similar to the user's field of view as a FOV (field of view) area while the device (100) is worn on the user's head.

[0094] At this time, the camera (130) may be configured to include a depth camera or a 3D camera capable of capturing and processing 3D information. To this end, the camera (130) may be configured to include two or more lenses, or two or more cameras may be configured to have the same or similar FOV areas.

[0095] The input / output unit (140) may include at least one output unit (e.g., a display, a lens, or a projector) for outputting an object processing program and / or a virtual reality (VR) image, an augmented reality (AR) image, or a mixed reality (MR) image based thereon.

[0096] According to various embodiments, implementing an object in a mixed reality space may be a case where a processing unit (110) outputs a 3D processed object to a display (or through a display) so that the object appears to actually exist in the real space.

[0097] However, it is not limited to this, and implementing an object in a mixed reality space may be done by projecting the object into the real space as a three-dimensional graphic as if the object actually exists in the real space.

[0098] In addition, the input / output unit (140) may further include at least some of an input unit (not shown) such as a microphone for inputting a control command through the user's voice, a button for inputting a control command through touch or click, and an output unit (not shown) for outputting data such as a speaker and a driving unit.

[0099] For example, the input / output unit (140) may include at least one microphone for acquiring a user's voice or a specific sound from the surrounding environment of the device (100). In this case, the microphone may acquire the operating sounds of keys constituting a physical keyboard (e.g., typing sounds).

[0100] According to various embodiments, the camera (130) may also be configured as an element of the input / output unit (140).

[0101] Referring to FIG. 1, the input / output unit (140) is illustrated as being directly connected to the processing unit (110), but may also be connected through the communication unit (150).

[0102] The communication unit (150) can support the establishment of a wired communication channel between the device (100) and at least one other electronic device (e.g., a manager device or a server), the establishment of a wireless communication channel, and the performance of communication through the established communication channel.

[0103] The communication unit (150) may be operated dependently or independently of the processing unit (110) and may include one or more communication processors that support wired or wireless communication. According to one embodiment, the communication unit (150) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module).

[0104] The communication unit (150) can communicate with an external electronic device (or external device) via a short-range communication network such as Bluetooth, BLE (Bluetooth Low Energy), WiFi, WiFi direct, IrDA (infrared data association), ZigBee, UWB, RF (Radio Frequency) and / or a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., LAN or WAN).

[0105] According to various embodiments of the present invention, the device (100) may include at least a portion of the functions of all information and communication devices, including mobile communication terminals, multimedia terminals, wired terminals, fixed terminals, and internet protocol (IP) terminals.

[0106] The device (100) is a device that processes at least some virtual objects, and FIG. 3 is a schematic drawing of a device that implements object motion based on hand movement in mixed reality according to one embodiment.

[0107] Referring to FIG. 3, the device (100) may be configured to include an HMD (head mounted display) device that is worn on the user's head (or scalp) and provides mixed reality (or augmented reality, virtual reality) through vision.

[0108] According to FIG. 3, the device (100) is illustrated as an example of a VR device, but is not limited thereto, and an AR device or an MR device may also be included in the scope of the device (100).

[0109] The device (100) can detect an actual inserted object from an image captured through a camera (130), or implement a virtual inserted object in a mixed reality space.

[0110] In addition, the device (100) can detect a user's hand from an image captured through a camera (130), detect the movement of the hand that manipulates the inserted object, and predict the movement of the inserted object based on the detected movement and implement it in a mixed reality space.

[0111] Hereinafter, a method for processing an object in a mixed reality space of a device will be described in detail with reference to FIGS. 4 to 13. In this regard, FIG. 4 is a flowchart illustrating an operation flow for processing an object in a mixed reality space of a device according to an embodiment. FIG. 5 is a diagram illustrating an operation flow for implementing a 3D reconstruction model by a device according to an embodiment. FIG. 6 is a diagram illustrating a detailed operation flow for implementing a 3D reconstruction model by a device according to an embodiment. FIG. 7 is a diagram illustrating an operation flow for implementing the movement of an inserted object by a device according to an embodiment. FIG. 8 is a diagram explaining an operation for implementing a 3D reconstruction model by a device according to an embodiment. FIG. 9 is a diagram explaining an operation for matching a 3D reconstruction model to a corresponding body part of a patient by a device according to an embodiment. FIG. 10 is a diagram illustrating an operation for implementing the movement of an object inserted into a patient's body according to hand movement by a device according to an embodiment. FIG. 11 is a diagram explaining an operation for determining joints and fingertips from a user's hand identified by the device. FIG. 12 is a diagram illustrating an operation for implementing movement of an inserted object in response to hand movement in a device according to one embodiment. FIG. 13 is a diagram illustrating another operation for implementing movement of an inserted object in response to hand movement in a device according to one embodiment.

[0112] First, referring to FIG. 4, the object implementation unit (201) can implement (401) a 3D reconstruction model generated for a specific patient in a mixed reality space. To this end, the object implementation unit (201) can form a mixed reality space based on an image captured through a camera (130).

[0113] At this time, the image captured by the camera (130) may be an image composed of a single frame, or an image (or images) composed of multiple frames. In addition, the image acquired by the camera (130) may be an image captured at a specified time interval or captured in real time.

[0114] Referring to FIG. 8, in more detail, when the device (100) is worn on the user's head, the camera (130) may be configured to capture an image in a range identical to or similar to the user's field of view (81) as a FOV (field of view) area.

[0115] At this time, the object implementation unit (201) can set a range identical or similar to the FOV area captured by the camera (130) as a mixed reality area and implement a 3D reconstruction model (801) for the patient within the mixed reality space.

[0116] Here, the 3D reconstruction model (801) may be a 3D model in which at least some elements of the skin, blood vessels, organs, tissues, and bones within the skin of the body part are implemented as an anatomical image created for a specific patient's body part.

[0117] In this regard, although not shown in the drawing, the processing unit (110) may further include a 3D modeling unit (not shown) for generating a 3D reconstruction model as described above.

[0118] To be more specific, the 3D modeling unit may acquire at least some of the previously captured CT images, MRI images, and ultrasound images to create a 3D reconstruction model (801) of at least a portion of the body of a specific patient.

[0119] The 3D modeling unit can distinguish at least some elements of skin, blood vessels, organs, tissues, nerves, and skeleton (or bones) based on the acquired images, and reconstruct and combine the distinguished at least some elements in 3D to create a 3D reconstruction model for at least a part of the body.

[0120] At this time, the 3D modeling unit can obtain various landmarks from the acquired images and place elements such as skin, blood vessels, organs, tissues, nerves, and bones based on the positions (or relative positions) of the acquired landmarks.

[0121] Here, the landmarks obtained from the images may include at least some of the features existing on the human body surface, such as nipples, navels, collarbones, moles, and scars, and / or at least some of the human internal devices, such as the heart, liver, and lungs.

[0122] As described above, the generated 3D reconstruction model can be generated to enable 3D viewing of internal elements of a body part according to user input.

[0123] Returning to FIG. 4 again, the object implementation unit (201) can implement a 3D reconstruction model generated as described above for a specific body part of a specific patient based on user input into a mixed reality space.

[0124] According to various embodiments, the object implementation unit (201) can implement a 3D reconstruction model in a mixed reality space by aligning it with the patient's body.

[0125] For example, the object detection unit (203) can detect (501) at least a part of the actual body of the specific patient in a mixed reality space (e.g., an image captured through a camera (130). Thereafter, the object implementation unit (201) can align (503) the 3D reconstruction model with at least a part of the actual body of the specific patient.

[0126] At this time, the object implementation unit (201) can compare at least one real landmark detected from the body surface of the patient photographed through the camera (130) with at least one virtual landmark detected from the body surface of the 3D reconstruction model to determine whether the patient detected from the mixed reality space and the patient who is the target of the 3D reconstruction model are the same.

[0127] If the object implementation unit (201) determines that the patient detected from the mixed reality space and the patient who is the target of the 3D reconstruction model are the same, the object implementation unit (201) can match the 3D reconstruction model to the corresponding body part of the actual patient by matching the actual landmark detected from the actual patient in the mixed reality space with the virtual landmark detected from the 3D reconstruction model.

[0128] For example, referring to FIG. 9, the object implementation unit (201) can match the navel, clavicle, and nipples detected from the chest of a patient (901) detected from the mixed reality space with the navel, clavicle, and nipples detected from the 3D reconstruction model (801) to project the 3D reconstruction model (801) of the patient's chest onto the chest of the actual patient (901).

[0129] To this end, the object implementation unit (201) may perform an operation of aligning a 3D reconstruction model with a patient's body using at least one deep learning model. At this time, the deep learning model for alignment may be pre-trained to perform alignment of the 3D reconstruction model with the patient's body based on landmarks and biometric elements.

[0130] At this time, when the 3D reconstruction model is aligned with the patient's body, the object implementation unit (201) can project and display an image of the inside of the patient's body (e.g., chest) that constitutes the 3D reconstruction model onto the actual patient's body, as illustrated in FIG. 9. However, the object implementation unit (201) can also project and display an image of the patient's body surface, excluding the image of the inside of the body, from the 3D reconstruction model onto the patient's body.

[0131] The generated 3D reconstruction model is configured in a state in which the shape and arrangement of the internal biological elements as well as the body surface of the patient are identical to the patient's condition, and as described above, the object implementation unit (201) can accurately display the shape and position of the internal biological elements of the patient's body (901) by projecting the 3D reconstruction model (801) onto the patient's body in a state aligned with the patient's body.

[0132] At this time, if the object implementation unit (201) confirms that the inserted object (e.g., an actual inserted object or a virtual inserted object) has been inserted into the patient's body (e.g., an actual patient's body or a virtual body of an implemented 3D reconstruction model), the object implementation unit (201) can project an image of the patient's body's interior constituting the 3D reconstruction model onto the actual patient's body and display it. This will be described in more detail later.

[0133] Returning to FIG. 4 again, the object detection unit (203) can detect (403) an insertion object equipped to function with a 3D reconstruction model and a user's hand manipulating the insertion object.

[0134] For example, the object detection unit (203) can detect an inserted object equipped to function as a 3D reconstruction model from a mixed reality space and a user's hand manipulating the inserted object.

[0135] For example, referring to FIG. 10, the object detection unit (203) can detect an actual inserted object (1010) from an image captured by a camera (130) in a mixed reality space, and call up information about the actual inserted object, such as specifications and movement of the actual inserted object.

[0136] To explain in more detail, the object detection unit (203) can identify an actual inserted object based on at least some external features of the detected object (1010), such as shape, length, thickness, etc., and call up information about the identified actual inserted object.

[0137] However, without limitation thereto, based on user input, the object implementation unit (201) may implement a specific virtual insertion object in the mixed reality space. At this time, the virtual insertion object implemented in the mixed reality space is implemented to perform the same or similar function as the corresponding real insertion object, and may be implemented in the mixed reality space so as to be operable based on the user's hand movements.

[0138] In addition, the object detection unit (203) can detect the user's hand (1001) from an image captured by a camera (130) in a mixed reality space and obtain the movement of the detected hand.

[0139] In this regard, referring to FIGS. 10 and 11, the object detection unit (203) can obtain multiple image frames by photographing the user's hand located within the FOV area of ​​the mixed reality space using the camera (130). The object detection unit (203) can recognize the hand from the multiple image frames.

[0140] For example, the object detection unit (203) can detect an object (1001) from an image acquired through a camera (130) and identify the object (1001) as the left hand (1101) based on the shape and movement of the detected object (1001).

[0141] Although not shown in FIG. 10, as described above, when the object detection unit (203) detects the user's right hand as an object from an image acquired through the camera (130), it will be able to identify the object as the right hand based on the shape and movement of the detected object.

[0142] As described above, the object detection unit (203) can distinguish and recognize the user's left hand (1101) and right hand (1111) based on at least some of the shape of the hand, for example, the angle of the hand, the position of the thumb, and the length of the fingers.

[0143] In addition, when a hand holding an inserted object (1010) is detected, the object detection unit (203) can distinguish and recognize the user's left hand (1101) and right hand (1111) based on at least some of the angle of the hand holding the inserted object (1010), the position of the thumb, and the length of the fingers.

[0144] At this time, the object detection unit (203) can determine the position of each joint of the plurality of fingers constituting the hand and the position of each end point of the plurality of fingers based on the shape of the identified hand (1101, 1111).

[0145] To explain in more detail, the object detection unit (203) can identify the user's hand (1101, 1111) from an image acquired through the camera (130), detect the back of the hand (or palm) constituting the hand and at least some of the plurality of fingers connected to the back of the hand based on the shape of the hand, and determine the positions of each of the joints of the plurality of fingers and the positions of the endpoints of each of the plurality of fingers.

[0146] To explain in more detail, the object detection unit (203) can obtain three-dimensional position coordinate values ​​(e.g., at least some of L0 to L20 and R0 to R20) including depth values ​​for the end points of each of the plurality of joints and fingers included in the recognized hand (at least some of 1101 and 1111).

[0147] Here, 'joint' refers to a part where multiple bones in the hand are connected to each other, and may refer to one or more parts of the fingers, hand, etc., or the palm.

[0148] The object detection unit (203) can obtain a position coordinate value (L0) for a wrist joint of the left hand (1101) and can obtain position coordinate values ​​(L1, L5, L9, L13, L17) for a plurality of joints included in the palm. The object detection unit (203) can obtain position coordinate values ​​for three joints for each of five fingers.

[0149] For example, in the case of the left hand (1101), the object detection unit (203) can obtain position coordinate values ​​of L2, L3, and L4 for the thumb, position coordinate values ​​of L6, L7, and L8 for the index finger, position coordinate values ​​of L10, L11, and L12 for the middle finger, position coordinate values ​​of L14, L15, and L16 for the ring finger, and position coordinate values ​​of L18, L19, and L20 for the little finger.

[0150] At this time, the object detection unit (203) can determine the position coordinate values ​​of the last joint among the position coordinate values ​​of each finger (e.g., L4 of the thumb, L8 of the index finger, L12 of the middle finger, L16 of the ring finger, L20 of the little finger) as the end point position (or end point position coordinate value) of each finger.

[0151] Likewise, the object detection unit (203) can obtain a position coordinate value (R0) for a wrist joint for the right hand (1111) and position coordinate values ​​(R1, R5, R9, R13, R17) for a plurality of joints included in the palm. The object detection unit (203) can obtain position coordinate values ​​for three joints for each of five fingers.

[0152] For example, in the case of the right hand (1111), the object detection unit (203) can obtain position coordinate values ​​of R2, R3, and R4 for the thumb, position coordinate values ​​of R6, R7, and R8 for the index finger, position coordinate values ​​of R10, R11, and R12 for the middle finger, position coordinate values ​​of R14, R15, and R16 for the ring finger, and position coordinate values ​​of R18, R19, and R20 for the little finger.

[0153] At this time, the object detection unit (203) can determine the position coordinate values ​​of the last joint among the position coordinate values ​​of each finger (e.g., R4 of the thumb, R8 of the index finger, R12 of the middle finger, R16 of the ring finger, R20 of the little finger) as the end point position (or end point position coordinate value) of each finger.

[0154] The object detection unit (203) can obtain three-dimensional position coordinate values ​​including position coordinate values ​​of the X-axis, Y-axis, and Z-axis on a three-dimensional rectangular coordinate system for each of the 21 joints in each hand (1001 or 1011).

[0155] As described above, the object detection unit (203) can determine the end point position (L4, L8, L12, L16, L20, R4, R8, R12, R16, or R20) of each finger based on the joint position coordinate values ​​of each finger and the movement of the joint position coordinate values.

[0156] At this time, the object detection unit (203) can predict the invisible joint position or endpoint position based on the joint position coordinate values ​​and movement of the remaining fingers when the position of at least one finger tip is not visible because it is covered by another finger or another object.

[0157] As described above, the object detection unit (203) is shown and described as acquiring position coordinate values ​​(L0 to L20, or R0 to R20) for 21 joints included in each of the user's hands (e.g., left hand or right hand), but this is according to one example, and the number of multiple joints is not limited to 21.

[0158] In addition, the object implementation unit (201) can display a virtual hand (1011) for detecting and expressing the movement of joints and end points of the hand identified by the object detection unit (203) by matching it with the actual hand (1001).

[0159] In describing various embodiments of the present disclosure, the expression of a hand may include the meaning of at least some of the joints that constitute the hand, for example, the joints of the wrist and fingers (or fingertips).

[0160] Returning to FIG. 4 again, the function processing unit (205) can determine (405) the movement of the inserted object corresponding to the movement of the user's hand.

[0161] According to the above-described description, the inserted object detected by the object detection unit (203) through the mixed reality space may be an actual inserted object or a virtual inserted object selected according to user input.

[0162] Here, the expression that the object detection unit (203) detects a virtual inserted object through a mixed reality space may mean that the object implementation unit (201) implements a virtual inserted object in the mixed reality space.

[0163] However, without being limited to this, the expression that the object detection unit (203) detects a virtual inserted object through a mixed reality space may also mean that the object detection unit (203) detects a virtual inserted object implemented in the mixed reality space by the object implementation unit (201).

[0164] The object detection unit (203) can track a hand detected in a mixed reality space and obtain information about hand movements. For example, the object detection unit (203) can detect hand movements that grasp and manipulate an inserted object.

[0165] According to an example, the object detection unit (203) may determine that the user has grasped the inserted object with his / her hand when the inserted object is located between the preset fingertips of the hand in the mixed reality space, for example, the thumb tip and the index finger tip, and the distance between the thumb tip and the index finger tip is less than the preset distance.

[0166] The function processing unit (205) can predict the movement of an inserted object manipulated by the user's hand based on the detected hand movement.

[0167] Here, if the inserted object is an actual inserted object, the function processing unit (205) can compare the movement of the actual inserted object detected from an image captured by the camera (130) with the movement of the actual inserted object predicted based on the user's hand movement, and correct the movement of the predicted actual inserted object based on the movement of the actual inserted object detected from the image.

[0168] At this time, the function processing unit (205) can predict the movement of the actual inserted object by reflecting the correction algorithm in predicting the movement of the actual inserted object after correction in response to the user's hand movement.

[0169] Based on this, the function processing unit (205) can predict the movement of at least a part of the actual inserted object that is not visible and inserted inside the patient's body based on the user's hand movement, even if at least a part of the actual inserted object is inserted inside the patient's body.

[0170] According to another embodiment, when the inserted object is a virtual inserted object, the function processing unit (205) can predict the movement of the virtual inserted object in the mixed reality space based on the hand movement manipulating the virtual inserted object (e.g., the user's hand movement detected from an image captured through the camera (130).

[0171] Referring to FIG. 10, regarding the movement of the inserted object, the 3D reconstruction model (801) may be generated using at least one physiological tool related to the physiological characteristics of the biological element.

[0172] For example, the physiological tools may include at least some of Physiome Engine, OpenSim, Virtual Cell Model (VCell), BioGears, HeartFlow, Blood Simulations, and LungSim.

[0173] That is, at least some of the various biological elements, such as the skin and blood vessels (1020) of the patient (901) implemented in the mixed reality space as part of the 3D reconstruction model, can be predicted and implemented with at least some of the physiological functions and characteristics.

[0174] In this regard, the function processing unit (205) can predict the movement of the inserted object in response to the movement of the user's hand by considering the interaction with the patient's body and / or the 3D reconstruction model.

[0175] Hereinafter, the operation of implementing the movement of the inserted object (1010) can be described in more detail with reference to FIGS. 10, 12, and 13.

[0176] First, referring to FIGS. 10 and 12, the object detection unit (203) can detect the hand (1201) of the user manipulating the inserted object (1010) and the movement of the hand (1201).

[0177] Here, V Refer is the unit vector, V, of the direction in which the tip (1011) of the insertion object (1010) should be inserted. Index is a unit vector for the movement of a specific part of the hand (1201) that holds the inserted object (1010), θ Tolerance is V Refer Wow V Index The angle between din-forward can be defined as a distance vector along which the tip (1011) of the insertion object (1010) is inserted, and din-backward can be defined as a distance vector along which the tip (1011) of the insertion object (1010) is withdrawn.

[0178] Additionally, S is the location where the tip (1011) of the insertion object (1010) is inserted into the body on the body surface (1220) of the patient (901), C Puncture C is the location where the tip (1011) of the insertion object (1010) inserted into the body of the patient (901) is inserted into the blood vessel (1020). Target The tip (1011) of the inserted object (1010) may be defined as a target location to which it should move along the blood vessel (1020).

[0179] Here, V Index A specific part defined in can be assumed to be the end point of a specific finger, for example, the end point of the index finger (e.g., the index finger of the hand (1201)) that gripped the inserted object (1010). However, V IndexThe specific part defined in is not limited to the tip of the index finger, but may be set to the tip of at least one of various fingers, or may include at least a part of the joint of at least one finger, and / or the joint of the wrist.

[0180] For example, the object detection unit (203) is V Index Based on the hand (1201) the first position of a specific part (C t ) in the second position (C t-1 ) and the distance vector (d) to Tolerance can be detected. Afterwards, the function processing unit (205) detects the movement distance vector (d) and θ Tolerance For example, movement of the tip of the inserted object (1010) based on V Refer And the movement (din-forward or din-backward) of the tip (1011) of the inserted object (1010) can be predicted.

[0181] In addition, the function processing unit (205) detects the V detected by the object detection unit (203). Index Based on this, the rotation of the tip (1011) of the inserted object (1010) can also be predicted.

[0182] In this regard, the function processing unit (205) can calculate the final position of the movement of the tip (1011) of the inserted object (1010) based on the mathematical expression (1) below.

[0183] (1)

[0184] As described above, the function processing unit (205) can predict the movement of the inserted object (1010) (and / or the movement of the tip (1011)) based on the movement of the hand (1201) detected by the object detection unit (203).

[0185] In addition, the function processing unit (205) can correct the final position of the tip (1011) of the predicted insertion object (1010) according to the state of the organ (e.g., blood vessel) into which the insertion object (1010) is inserted. For example, the function processing unit (205) can obtain the state (e.g., elasticity of the blood vessel) of the blood vessel into which the insertion object (1010) is inserted and / or the surroundings of the predicted tip (1011) position. If the function processing unit (205) confirms that the elasticity of the vestibule obtained for the blood vessel exceeds a preset reference elasticity (e.g., the average value of the blood vessel elasticity), the function processing unit (205) can multiply the movement distance of the tip (1011) predicted for the insertion object (1010) by a preset variable (e.g., 0.8) and apply the variable to predict the final position of the tip (1011) of the insertion object (1010) based on the predicted movement distance.

[0186] The organ (or inside the body) into which the inserted object (1010) is inserted is exemplified as a blood vessel, but is not limited thereto, and may be configured to include at least some of various elements that make up the body, such as skin, nerves, bones, large intestine, and small intestine.

[0187] In addition, the object detection unit (203) and / or the function processing unit (205) may detect movement of at least a part of the body of the inserted object (1010) connected to the tip (1011) as well as movement of the tip of the inserted object (1010), and may predict movement of at least a part of the body of the inserted object (1010) based on the detected information.

[0188] And, as shown in FIGS. 12 and 13, the object implementation unit (201) can implement (407) the movement of the object inserted into the 3D reconstruction model.

[0189] To explain in more detail, the object implementation unit (201) can implement the movement of an inserted object (1010) moving inside a blood vessel (1020) of a 3D reconstruction model (801) according to the prediction result of the function processing unit (205).

[0190] Here, the movement of the insertion object (1010) implemented by the object implementation unit (201) may include at least a portion of the movement of the tip (1011) of the insertion object (1010) and the movement of the body of the insertion object (1010). In addition, unless otherwise specified in the following description, the expression of the tip may include at least a portion of the tip and the body.

[0191] Referring to FIG. 12, the object implementation unit (201) generates an image of at least a part of the inserted object (1010) inserted into the body of the patient (901) according to the movement of the tip (1011) of the inserted object (1010) predicted by the function processing unit (203) in response to the movement of the hand (1201), and based on the generated image, implements the movement of the tip (1011) of the inserted object (1010) in a 3D reconstruction model (801) (e.g., inside a blood vessel (1020)) aligned with the body of the patient (901).

[0192] At this time, the object implementation unit (201) moves the tip (1011) of the insertion object (1010) to the target position (C) according to the function of the insertion object corresponding to the user's hand movement. Target ) can be displayed to move forward, backward, or turn in the direction.

[0193] Based on this, referring to FIG. 13, the object implementation unit (201) detects hand movements (e.g., hand movements based on unit vectors) (1311, 1313, 1315, 1317) for each section, and / or predicts tip movements of the inserted object corresponding to hand movements detected for each section (e.g., tip movements of the inserted object based on unit vectors) (D OCPE ) can implement the tip movement of the inserted object.

[0194] The object implementation unit (201) can project an internal image (e.g., a body surface or skin internal image, a spherical image) of the 3D reconstruction model (801) onto the patient's body when implementing at least a part of the insertion object (1010) inserted into the patient's body through the 3D reconstruction model (801) at the time when the tip (1011) of the insertion object (1010) is inserted into the patient's body (901).

[0195] According to the above, it was explained that the object implementation unit (201) can project the internal image constituting the 3D reconstruction model (801) onto the body of the patient (901) at the time of implementing the 3D reconstruction model (801) aligned with the body of the patient (901) in the mixed reality space.

[0196] However, without being limited thereto, the object implementation unit (201) may implement the movement of the inserted object (1010) (or the movement of the tip (1011) of the inserted object (1010)) without projecting the internal image constituting the 3D reconstruction model (801) at the time of implementing the 3D reconstruction model (801) aligned with the body of the patient (901) in the mixed reality space. In this state, if the object implementation unit (201) detects that the tip (1011) of the inserted object (1010) is inserted into the body (or inside the skin) of the patient (901), the object implementation unit (201) may project the internal image (e.g., a body surface or inside the skin image, a perspective image) of the 3D reconstruction model (801) onto the patient's body at the time of insertion.

[0197] Here, if the insertion object is a virtual insertion object, the object implementation unit (201) can project an internal image of the 3D reconstruction model onto the body of the patient (901) by determining, based on the movement of the virtual insertion object, that the tip of the virtual insertion object has been inserted into the inside of the skin configured in the 3D reconstruction model or the inside of the skin of the actual body of the patient.

[0198] As described above, the object implementation unit (201) can project (601) an internal image of the 3D reconstruction model onto at least a part of the actual body, either in the step of implementing the movement of the inserted object or before performing the step of implementing the movement of the inserted object.

[0199] In addition, the object implementation unit (201) can accurately display the position and movement of the insertion object (1010) by implementing at least a part (e.g., at least a part including the tip (1011)) of the insertion object (1010) that is inserted into the body of the patient (901) and is not visible through an internal image projected onto the body of the patient (901).

[0200] According to various embodiments, the motion determination unit (207) determines (701) whether the state of the tip of the inserted object predicted according to the movement of the inserted object implemented satisfies a preset guideline, and if it determines that the guideline is not satisfied, it can display (703) the unsatisfied condition elements of the guideline and the predicted information according to the unsatisfied condition elements in the mixed reality space.

[0201] To this end, the storage unit (120) may be configured to have a simulation including guidelines for medical measures to be performed on the patient (e.g., surgery, procedure, examination, or invasive tube insertion for treatment) using a three-dimensional reconstruction model (801).

[0202] Here, the guideline may be configured to include at least one guide information configured as conditional elements including at least one specific location, a risk level of the specific location (whether the specific location is a risk area), and at least some of an entry angle, a bending angle, and a speed for at least a part of the tip and the body at the specific location, in order to guide the movement of at least a part of the tip and the body of the inserted object.

[0203] Additionally, the guidelines may be configured to reflect the condition (e.g., elasticity) of the body (e.g., at least a portion of blood vessels, skin, nerves, bones, colon, small intestine, etc.) into which the insertion object (1010) is inserted or is expected (or determined) to move.

[0204] For example, when a simulation for central venous catheter insertion using a 3D reconstruction model (801) is set, the object implementation unit (201) displays a puncture location (S) for inserting an insertion object (1010) on the body surface of the 3D reconstruction model (801), and a puncture location (C) of a blood vessel for inserting a tip (1011) of an insertion object (1010) inserted into the body of a patient (901) into the interior of a blood vessel (1020). Puncture ), the location of the puncture site (C) of the vessel from the puncture site (S) Puncture ) and / or the location of perforation of the blood vessel (C Puncture ) from the target position (C Target ), guide information including the entry angle and / or speed information of the tip (1011) for at least one specific location can be displayed as a guideline on the internal image to guide the insertion object (1010) including at least a part of the movement path to the tip (1011).

[0205] More specifically, the guide information may be configured to include information about at least a portion of the entry angle and speed of at least a portion of the tip (1011) and the body, with respect to a curved portion that bends by a preset angle or more during the predicted (or implemented) movement path of the tip and / or body of the insertion object (1010).

[0206] At this time, the guide information may be set by dividing it into items such as an incision item that guides the incision location for inserting the tube by planning the puncture site (S) before surgery, an insertion item that guides the puncture location and the final position (e.g., CA junction) where the tube should be located and / or the insertion direction vector by determining the puncture location and the final position, a deflection item that guides the curved area where blood vessels, the colon, the urethra, etc. in the human body are bent during insertion of the insertion object (1010), a threshold that can provide feedback on the insertion speed, angle, location, etc. that should be considered during surgery, a decision parameter item that guides points such as Feature Engineering, etc., depending on the diameter, size, length, etc. of the tube, and / or when the user moves (or manipulates) the tube using a thin object such as tweezers.

[0207] At this time, the object implementation unit (201) can display guide information (and / or guidelines based on the guide information) in a three-dimensional reconstruction model (801) in stages according to the position of the tip (1011) of the inserted object (1010).

[0208] The object implementation unit (201) can display guide information regarding the set tip entry angle, speed, etc. around the position of the tip (1011) when displaying preset guide information through the 3D reconstruction model (801).

[0209] The motion judgment unit (207) determines whether to puncture the patient's (901) body surface at a designated location based on the movement of the tip (1011) of the inserted object (1010) implemented in the mixed reality space, or whether the inserted tip (1011) through the puncture location (S) follows the designated guideline to the puncture location (C) of the blood vessel. Puncture) and / or the tip (1011) inserted into the blood vessel (1020) follows the designated guidelines to reach the target location (C Target ) can be monitored to see if it moves.

[0210] At this time, the motion judgment unit (207) can predict the speed of the tip (1011) according to the movement of the insertion object (1010) while monitoring the movement of the tip (1011) of the insertion object (1010) corresponding to the hand movement.

[0211] Based on this, the motion judgment unit (207) can predict not only whether the movement of the tip (1011) deviates from the designated path (guide path) according to the guideline, but also a situation such as bending (or twisting, 1013, 1015, hereinafter referred to as bending) of the tube connected to the tip (1011), as illustrated in FIG. 10.

[0212] At this time, the motion judgment unit (207) can predict the possibility of damage to the blood vessel based on the speed of the tip (1011) at the time when the position of the tip (1011) deviates from the guide path, and display the deviation position and prediction information in a designated area of ​​the mixed reality space.

[0213] In addition, the motion judgment unit (207) can display correct information about hand movement, for example, information about at least some of the movement direction, rotation, and speed of the finger holding the inserted object (1010), in case of deviation from the guide path, through a designated area of ​​the mixed reality space or provide guidance through auditory information.

[0214] According to the above, the device for implementing the motion of an object based on hand movements has the effect of providing an environment in which the position and movement of the device can be accurately confirmed without using radiation when performing an invasive tube insertion surgery that inserts the device into the patient's body.

[0215] According to various embodiments, a device for implementing an object's motion based on hand movements provides a surgical assistance system that anatomically accurately displays biological elements inside a patient's body by projecting a three-dimensional reconstruction model created by reflecting anatomical elements of a part of the patient's body onto the patient, thereby reducing the risk of accidents in invasive tube insertion surgery that inserts equipment into the patient's body.

[0216] According to various embodiments, a device for implementing motion of an object based on hand movements can provide an educational opportunity for treating various cases of diseases using invasive tube insertion surgery by providing realistic three-dimensional reconstruction models of the internal body of various patients.

[0217] According to various embodiments, a device for implementing an object's motion based on hand movements can provide a surgical environment that reduces the physical burden on a patient by providing visual and auditory information about the insertion depth, insertion direction, and insertion angle of a tube inserted into a patient's body based on hand movements.

[0218] According to the detailed description, the functions of the various embodiments described as being performed by the device (100) are operations processed through the processing unit (110) of the device (100), and can be performed organically connected with components of the device (100) and / or devices connected to the device (100).

[0219] Although the embodiments described above have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the various embodiments.

[0220] For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted by other components or equivalents.

[0221] In particular, when explaining with reference to a flow chart, it is described as comprising multiple steps and executing the steps sequentially in a specified order, but it is not necessarily limited to the described order.

[0222] In other words, it is also possible to apply the embodiment by changing or deleting at least some of the steps described in the flowchart and executing them, or by adding at least one step, and it is also possible to apply the embodiment by executing one or more steps in parallel. In other words, the steps are not necessarily limited to operating in a chronological order, and should be included in various embodiments of the present disclosure.

[0223] Therefore, other implementations, other embodiments, and equivalents to the claims should also fall within the scope of the claims described below.

Claims

1. A step of implementing a 3D reconstruction model created for a specific patient into a mixed reality space; A step of detecting an insertion object equipped to function with the above 3D reconstruction model and a user's hand manipulating the insertion object; a step of determining the movement of the inserted object corresponding to the movement of the user's hand; and A method for implementing motion of an object based on hand movements in mixed reality, comprising: a step of implementing motion of the inserted object inserted into the three-dimensional reconstruction model; 2. In paragraph 1, The step of implementing in the mixed reality space comprises the steps of detecting at least a part of the actual body of the specific patient in the mixed reality space; and A method for implementing motion of an object based on hand movements in mixed reality, further comprising: a step of aligning the three-dimensional reconstruction model to at least a part of the actual body of the specific patient.

3. In paragraph 2, The above-mentioned matching step is a method for implementing an object's motion based on hand movements in mixed reality, based on a body surface virtual landmark formed outside the 3D reconstruction model and a body surface real landmark formed outside at least a part of the actual body.

4. In paragraph 2, A method for implementing motion of an object based on hand movements in mixed reality, further comprising: a step of projecting an internal image of the three-dimensional reconstruction model onto at least a part of the actual body, in a step of implementing motion of the inserted object or before performing the step of implementing motion of the inserted object.

5. In paragraph 4, A method for implementing an object's motion based on hand movements in mixed reality, wherein the internal image of the above 3D reconstruction model is an image in which at least some elements of blood vessels, organs, tissues, nerves, and bones are implemented.

6. In paragraph 4, If a virtual insertion object is detected for the above insertion object, The projecting step projects the internal image of the 3D reconstruction model when it is determined based on the movement of the virtual insertion object that at least a portion of the tip of the virtual insertion object and the body of the virtual insertion object are inserted into the inside of the skin configured in the 3D reconstruction model or the inside of the skin of the actual body of the patient. A method for implementing an object's motion based on hand movements in mixed reality, wherein the motion of an inserted part of the virtual inserted object, including a tip of the virtual inserted object, is displayed through the internal image.

7. In paragraph 4, If a real inserted object is detected for the above inserted object, The projecting step projects the internal image of the 3D reconstructed model when it is determined based on the movement of the real insertion object that at least a part of the tip of the real insertion object and the body of the real insertion object are inserted into the skin formed in the 3D reconstructed model or the skin of the actual body of the patient. A method for implementing motion of an object based on hand movements in mixed reality, wherein the motion of an inserted portion of the real-world inserted object including a tip of the real-world inserted object is predicted, and the motion of the inserted portion of the real-world inserted object is displayed through the internal image.

8. In paragraph 1, A method for implementing the movement of an object based on hand movements in mixed reality, wherein the step of implementing the movement of the inserted object is to display the tip of the inserted object and at least a part of the body of the inserted object by moving forward, backward, or rotating according to the function of the inserted object corresponding to the movement of the user's hand.

9. In paragraph 1, The step of implementing the movement of the above-described inserted object comprises: a step of determining whether the state of at least a part of the tip of the inserted object and the body of the inserted object predicted according to the implemented movement of the inserted object satisfies a preset guideline; and In case the above guideline is determined to be not satisfied, the step of displaying information on the unsatisfied condition element of the above guideline and the influence according to the unsatisfied condition element in the mixed reality space is further included; The above guidelines are a method for implementing the motion of an object based on hand movements in mixed reality, which is configured to reflect the state of the body into which the inserted object is inserted or is expected to move.

10. In paragraph 1, The above 3D reconstruction model is, A method for implementing the motion of an object based on hand movements in mixed reality, wherein at least some elements of skin, blood vessels, organs, tissues, and bones are distinguished based on at least some of CT images, MRI images, and ultrasound images acquired for the specific patient, and at least some of the distinguished elements are reconstructed in three dimensions and combined to create a perspective image of at least a part of the body of the specific patient.

11. An object implementation unit that implements a 3D reconstruction model generated for a specific patient in a mixed reality space and implements the movement of an inserted object inserted within the 3D reconstruction model; An object detection unit that detects the insertion object and the user's hand manipulating the insertion object, which are provided to function with the three-dimensional reconstruction model; and A device for implementing motion of an object based on hand movement in mixed reality, comprising a function processing unit that determines the motion of the inserted object corresponding to the user's hand movement.

12. In paragraph 11, A device for implementing an object's motion based on hand movements in mixed reality, wherein the object detection unit detects at least a part of the actual body of the specific patient in the mixed reality space, aligns the 3D reconstruction model with at least a part of the actual body of the specific patient, and implements the 3D reconstruction model in the mixed reality space.

13. In paragraph 12, The above object implementation unit is a device that implements the movement of an object based on hand movements in mixed reality, based on a body surface virtual landmark formed outside the 3D reconstruction model and a body surface real landmark formed outside at least a part of the actual body.

14. In paragraph 12, The above object implementation unit is a device that implements the movement of an object based on hand movements in mixed reality by projecting an internal image of the 3D reconstruction model onto at least a part of the actual body.

15. In paragraph 14, A device for implementing the motion of an object based on hand movements in mixed reality, wherein the internal image of the above 3D reconstruction model is an image in which at least some elements of blood vessels, organs, tissues, nerves, and bones are implemented.

16. In paragraph 14, If the above object detection unit detects a virtual insertion object for the above insertion object, The object implementation unit, based on the movement of the virtual inserted object, determines that at least a part of the tip of the virtual inserted object and the body of the virtual inserted object are inserted into the inside of the skin configured in the 3D reconstructed model or the inside of the skin of the actual body of the patient, and projects the internal image of the 3D reconstructed model, and displays the movement of the inserted part of the virtual inserted object, including the tip of the virtual inserted object, through the internal image, thereby implementing the movement of the object based on hand movement in mixed reality.

17. In paragraph 14, If the above object detection unit detects a real inserted object for the inserted object, The object implementation unit, based on the movement of the physical insertion object, determines that at least a part of the tip of the physical insertion object and the body of the physical insertion object are inserted into the inside of the skin configured in the 3D reconstruction model or the inside of the skin of the actual body of the patient, projects the internal image of the 3D reconstruction model, predicts the movement of the inserted part of the physical insertion object including the tip of the physical insertion object, and displays the movement of the inserted part of the physical insertion object through the internal image, which is a device for implementing the movement of an object based on hand movement in mixed reality.

18. In paragraph 11, A device for implementing an object's motion based on hand movements in mixed reality, wherein the object implementation unit implements the movement of the inserted object by displaying the tip of the inserted object and at least a part of the body of the inserted object as moving forward, backward, or rotating according to the function of the inserted object corresponding to the movement of the user's hand.

19. In paragraph 11, A motion determination unit is further included, which determines whether the tip of the inserted object predicted according to the movement of the inserted object implemented above and the state of at least a part of the body of the inserted object satisfy a preset guideline, and, if it is determined that the guideline is not satisfied, displays information on the unsatisfied condition element of the guideline and the influence according to the unsatisfied condition element in the mixed reality space. The above guidelines are a device for implementing the motion of an object based on hand movements in mixed reality, which is configured to reflect the state of the body into which the inserted object is inserted or is expected to move.

20. In paragraph 11, The above 3D reconstruction model is, A device for implementing the motion of an object based on hand movements in mixed reality, which is a perspective image of at least a part of the body of the specific patient, created by distinguishing at least some elements of skin, blood vessels, organs, tissues, and bones based on at least some of the CT images, MRI images, and ultrasound images acquired for the specific patient, and reconstructing and combining the distinguished at least some elements in three dimensions.

Images