Display device, system and mixed reality equipment for three-dimensional virtual model of organism

Abstract

The present application discloses a display device and system for a three-dimensional virtual model of a living organism, and a mixed reality device. The display device includes: a construction module, which is used to construct a first virtual model including surface features and internal features of the living organism based on two-dimensional scanning data of the living organism and image data of the living organism collected during surgery; a calibration module, which is used to calibrate the first virtual model based on a second virtual model of the living organism to obtain a calibrated first virtual model; and a display module, which is used to determine the position and posture of the calibrated first virtual model in a real scene using the image data of the surface markers, so as to display the calibrated first virtual model and the real scene through the mixed reality device.

Description

Technical Field

[0001] The present application relates to the field of medical data processing technology, and in particular to a display device and system for a three-dimensional virtual model of a biological body, and a mixed reality device. Background Art

[0002] Mixed reality (MR) technology is to superimpose computer-generated images with the real world, display images in virtual space in real space, and allow the human eye to see virtual objects in the real world. The most well-known is the optical perspective head-mounted display HoloLens, which uses real-time positioning and map construction technology to perform real-time spatial positioning, and provides multiple interactive functions such as voice and gestures, so that users can have an immersive experience.

[0003] With the development of mixed reality (MR) technology, some researchers have tried to apply it to surgical navigation and remote surgery. At present, the patient's preoperative computed tomography (CT) data and other scanning data are usually used to generate a three-dimensional virtual model of the patient's internal structure, and the position and posture of the entity (such as organs or bones, etc.) corresponding to the three-dimensional virtual model are determined during the operation, so that the three-dimensional virtual model and the real entity in the surgical scene can be fused and displayed in front of the doctor through the mixed reality (MR) device worn by the doctor, so that the doctor has a pair of "perspective eyes". At present, it is usually necessary to manually fix an infrared reflective ball that can be positioned by an optical tracking device on the entity (such as a physical bone) corresponding to the three-dimensional virtual model to determine the entity's position and posture, thereby realizing the fusion display of the three-dimensional virtual model and the entity.

[0004] However, doctors have found through long-term clinical practice that this solution cannot achieve accurate fusion display of virtual images and patient lesion areas during use. It relies on manual adjustment of the position of the virtual image to complete alignment, is highly dependent on network signal quality, and is time-consuming. Most importantly, it cannot guarantee accurate display and real-time changes. In particular, if the infrared reflective ball is directly fixed on the entity, invasive operation is required (such as cutting the skin to expose bones or organs). Orthopedic surgery is complex and delicate. Doctors must accurately identify and locate tissue structures such as joints, bones, ligaments, muscles and nerves. At the same time, they must also consider factors such as operation time, bleeding volume, trauma area, and patient pain. Based on the reality of narrow surgical field of view and limited operating space, invasive operations have extremely high technical requirements for doctors. In addition, manual configuration of the infrared reflective ball depends on the subjective visual perception of the operator (doctor or nurse). It is not only time-consuming, but often cannot achieve accurate matching of the three-dimensional virtual model with the entity. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned related technologies, the purpose of the present application is to provide a display device and system for a three-dimensional virtual model of a biological body, as well as a mixed reality device, so as to overcome the technical problems existing in the above-mentioned related technologies, such as invasive operation caused by displaying three-dimensional virtual models based on infrared reflective balls and the inability to achieve accurate matching of the three-dimensional virtual model with the entity.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a display device for a three-dimensional virtual model of a living organism, comprising: a construction module, which is used to construct a first virtual model including surface features and internal features of the living organism based on two-dimensional scanning data of the living organism and image data of the living organism collected during surgery; wherein the two-dimensional scanning data includes surface markers of the target area of ​​the living organism and scanning data of the internal tissue, and the image data of the living organism includes image data of the surface of the target area of ​​the living organism and the surface markers; a calibration module, which is used to calibrate the first virtual model based on a second virtual model of the living organism to obtain a calibrated first virtual model; wherein the second virtual model is pre-constructed based on the preoperative CT data of the living organism, and the second virtual model includes the internal features of the living organism; a display module, which is used to determine the position and posture of the calibrated first virtual model in a real scene using the image data of the surface markers, so as to display the calibrated first virtual model and the real scene through a mixed reality device.

[0007] According to a second aspect of the present application, there is provided a display system for a three-dimensional virtual model of a living organism, comprising: a computer device, for constructing a first virtual model including surface features and internal features of the living organism based on two-dimensional scanning data and image data of the living organism collected during surgery; and calibrating the first virtual model based on a second virtual model of the living organism to obtain a calibrated first virtual model; wherein the two-dimensional scanning data includes surface markers of a target area of ​​the living organism and scanning data of internal tissues, the image data of the living organism includes image data of the surface of the target area of ​​the living organism and the surface markers, the second virtual model is pre-constructed based on preoperative CT data of the living organism and the second virtual model includes the internal features of the living organism; a mixed reality device, connected to the computer device, for determining the position and posture of the calibrated first virtual model in a real scene using the image data of the surface markers, so as to display the calibrated first virtual model and the real scene through the mixed reality device.

[0008] According to a third aspect of the present application, there is provided a mixed reality device, comprising: a sensing device for collecting image data of a living body; a storage device for storing at least one program; a display device for displaying a virtual model in a fusion with a real scene; a processing device connected to the sensing device, the storage device and the display device, and configured to call the at least one program from the storage device and implement, when executed: constructing a first virtual model including surface features and internal features of the living body based on two-dimensional scanning data of the living body and image data of the living body collected during surgery; wherein the two-dimensional scanning data includes surface markers of a target area of ​​the living body and scanning data of internal tissues, and the image data of the living body includes image data of the surface of the target area of ​​the living body and the surface markers; calibrating the first virtual model based on a second virtual model of the living body to obtain a calibrated first virtual model; wherein the second virtual model is pre-constructed based on CT data of the living body before surgery and the second virtual model includes the internal features of the living body; determining the position and posture of the calibrated first virtual model in the real scene using the image data of the surface markers, so as to display the calibrated first virtual model and the real scene through the mixed reality device.

[0009] In summary, the display device and system of the three-dimensional virtual model of a living organism, and the mixed reality device provided by the present application construct a first virtual model through the two-dimensional scanning data of the living organism and the image data of the living organism collected during the operation, thereby ensuring that the surface features and internal features of the first virtual model are consistent with the actual state / actual posture of the target area of ​​the living organism during the operation, and after calibrating the first virtual model using the second virtual model constructed before the operation, the size of the internal features in the calibrated first virtual model is consistent with the size of the real internal features of the living organism. Furthermore, by using the surface markers configured on the body surface, the calibrated first virtual model and the real scene can be fused and displayed without invasive operation, so as to achieve accurate matching of the three-dimensional virtual model and the entity, thereby facilitating the application of clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features and advantages of the invention of the present application can be better understood by referring to the exemplary embodiments and drawings described in detail below. The drawings are briefly described as follows.

[0011] Figure 1 Shown is a schematic diagram of the structure of a display device for a three-dimensional virtual model of a living organism in one embodiment of the present application.

[0012] Figure 2a Shown is a side view of a human left foot with a body surface marker configured on it in one embodiment of the present application.

[0013] Figure 2bShown is a top view of a human left foot with a body surface marker configured thereon in one embodiment of the present application.

[0014] Figure 3a Shown is a schematic diagram of a lateral X-ray of a human left foot in one embodiment of the present application.

[0015] Figure 3b Shown is a schematic diagram of an AP X-ray of a human left foot in one embodiment of the present application.

[0016] Figure 4 Shown is a schematic diagram of a first virtual model constructed in one embodiment of the present application.

[0017] Figure 5 Shown is a schematic diagram of a second virtual model in one embodiment of the present application.

[0018] Figure 6 Shown is a flowchart of a mixed reality device displaying a calibrated first virtual model in one embodiment of the present application.

[0019] Figure 7a Shown is a schematic diagram of the relative posture relationship between feature points on the first metatarsal and the second phalanx of the thumb in the first virtual model in one embodiment of the present application.

[0020] Figure 7b Shown is a schematic diagram of the relative posture relationship between feature points on the first metatarsal and the second phalanx of the thumb in the second virtual model in one embodiment of the present application.

[0021] Figure 7c Shown is a schematic diagram of the relative posture relationship between feature points on the first metatarsal and the second phalanx of the thumb in the first virtual model after calibration in one embodiment of the present application.

[0022] Figure 8 Shown is a schematic structural diagram of a display device for a three-dimensional virtual model of an organism in another embodiment of the present application.

[0023] Fig. 9 Shown is a schematic structural diagram of a display device for a three-dimensional virtual model of an organism in yet another embodiment of the present application.

[0024] Fig.10 Shown is a schematic diagram of the structure of a display system for a three-dimensional virtual model of an organism in one embodiment of the present application.

[0025] Fig.11 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application.

[0026] Fig.12 Shown is a schematic diagram of the structure of a mixed reality device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following is a description of the embodiments of the present application by specific specific embodiments, and those familiar with the technology can easily understand the advantages of the present application and the technical effects that can be achieved by the content disclosed in this specification. In the following description, some embodiments can refer to the drawings pointed to. It should be understood that other embodiments without drawings can also be used, and specific steps, modules or units, electrical and operational changes can be made without departing from the spirit and scope of the present application. The detailed description below should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims published by the present application. The terms used herein are only for describing specific embodiments and are not intended to limit the present application.

[0028] Although in some instances the terms first, second, etc. are used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object, and are not used to define the order, timing, priority or importance of multiple objects. For example, a first virtual model can be referred to as a second virtual model, and similarly, a second virtual model can be referred to as a first virtual model without departing from the scope of the various described embodiments. Both the first virtual model and the second virtual model are describing virtual models, but unless the context clearly indicates otherwise, they are not the same virtual model.

[0029] Furthermore, as used in this article, the singular forms "one", "an" and "the" are intended to also include plural forms, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprising", "including" indicate the presence of the described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. For example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, the term "and / or" that may be used hereinafter, describes the association relationship of associated objects, indicating that three kinds of relationships may exist, for example, A and / or B, which can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / ", if not otherwise specified, generally represents that the associated objects before and after are a kind of "and / or" relationship. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0030] Some nouns or terms used in each embodiment of the present application are explained below, and the nouns or terms are also used as a part of the content of the invention. It will be appreciated by those skilled in the art that, unless otherwise defined, all terms including technical terms and scientific terms used herein have the same meaning as the general understanding of the technicians in the field to which the present application belongs. It should also be understood that those terms such as those defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and unless specifically defined as herein, will not be interpreted with an idealized or overly formal meaning.

[0031] As used herein, the term "computing device" refers to any programmable computer system known or to be developed in the future. In a preferred embodiment, the computer will be coupled to a network. The computer system can be configured with processor executable software instructions to perform the processes described herein.

[0032] As used herein, the term "module" is intended to refer to a computer-related entity, whether the computer-related entity is hardware, a combination of hardware and software, software, or software in execution. For example, a module can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. In an example, both an application running on a server and the server can be modules. One or more modules can reside within a process and / or a thread of execution, and modules can be localized on one computer and / or distributed between two or more computers.

[0033] As used herein, the term "pose" includes position and posture, and the position can be represented by the numerical values ​​on the three coordinate axes (x, y, z) of the three-dimensional space coordinate system. For example, the position of the calibrated first virtual model in the real scene can be represented by the numerical values ​​on the three coordinate axes of the world coordinate system. The posture can be represented by the three rotation angles (pitch, yaw, roll) corresponding to the three coordinate axes (x, y, z) of the three-dimensional space coordinate system. For example, the posture of the calibrated first virtual model in the real scene can be represented by the rotation angles (pitch, yaw, roll) of the calibrated first virtual model around the three coordinate axes (x, y, z) of the world coordinate system.

[0034] Real scenes refer to the physical world that people can sense and / or interact with without the help of electronic systems. Real scenes include physical items or physical objects or real objects such as physical organisms and physical surgical instruments included in hospital operating rooms. People can directly sense and / or interact with real scenes, such as through vision, touch, hearing, taste and smell.

[0035] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A virtual reality (VR) environment includes multiple virtual objects that a person can sense and / or interact with. For example, a hospital bed, surgical instruments, and a computer-generated image representing an avatar of a person are examples of virtual objects.

[0036] In contrast to virtual reality (VR) environments, which are designed to be based entirely on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments that are designed to include sensory input from the real scene, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality (MR) environment is anything between a fully physical environment at one end and a virtual reality (VR) environment at the other end, but not including either end.

[0037] A mixed reality (MR) device is a device that can display a virtual object (such as a calibrated virtual model) in a fusion manner with a real scene, and further, a user can interact with the virtual object. For example, the MR device is a head-mounted MR device (such as MR glasses).

[0038] In view of what is described in the background technology, the display of three-dimensional virtual models based on infrared reflective balls provided in the prior art will lead to invasive operations and the inability to achieve accurate alignment of the three-dimensional virtual model with the entity. For example, the current method of aligning the virtual image and the lesion area based on HoloLens is to fix a rigid positioning target with an infrared reflective ball on a surgical tool or a bony landmark, and at the same time, an external optical tracking system is required to track the position information of the reflective infrared ball on the positioning target in the coordinate system. This requires manual adjustment of the positioning target to complete the position alignment, which depends on the subjective visual perception of the surgeon. It is not only time-consuming, but also unable to achieve accurate alignment. Moreover, when the spatial position of the positioning target moves, the optical tracking system needs to recalculate the position information. In real surgical scenarios, the patient's limbs are often unable to be completely still, and the surgeon needs to perform operations during the operation, and thus real-time tracking cannot be achieved. Furthermore, in the process of aligning the three-dimensional virtual model with the entity, if the optical path is blocked or the positioning target is suddenly displaced, the image is prone to jitter, affecting the surgeon's surgical operation.

[0039] In addition, in the existing fusion display of three-dimensional virtual models and entities, in addition to display devices such as HoloLens glasses, an optical tracking system is also required. The optical tracking system needs to occupy a certain space in the operating room, and the optical signal can only be transmitted up to 2 meters when in use. When the distance is too far, the picture may disappear. The display device is extremely dependent on the quality of the network signal. When the signal is weak, there will be inaccurate matching and matching delays, which will manifest as picture jitter, unclear or disappearing virtual images, and thus affect the progress of the operation. In addition to the purchase of an optical tracking system, the aforementioned solution also requires the purchase of a rigid positioning target. The reflective infrared ball configured on it is a disposable consumable and cannot be recycled. Therefore, a new positioning target needs to be purchased for each operation, thereby increasing the cost of the operation.

[0040] Furthermore, since the three-dimensional virtual model is generated before the operation, directly displaying the three-dimensional virtual model generated before the operation during the operation may cause a large error between the three-dimensional virtual model and the entity due to different human body postures, and thus it cannot be accurately matched with the entity and cannot be widely used in clinical operations. For example, the three-dimensional virtual model before the operation is a human foot model, and the human foot model is a model in which the foot surface is in a straight state, but the patient's foot bends during the operation, the positions of multiple bones in the foot will also change accordingly, and the relative position relationship between the skin and bones of the human foot may also change, resulting in the human foot model generated before the operation not matching the physical human foot during the operation. As a result, the human foot model generated before the operation is difficult to accurately match with the physical human foot and difficult to be widely used in clinical operations.

[0041] The present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. The technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments and technical effects obtained by ordinary technicians in this field without creative work should belong to the scope of protection of this application. The "one implementation method", "implementation method" or similar words mentioned in the whole text of this specification mean that the specific features, structures or characteristics described together with the implementation method are included in at least one implementation method of the present application. Therefore, in the whole text of this specification, the appearance of the phrases "in one implementation method", "in an implementation method" and similar words may (but not necessarily) refer to the same implementation method.

[0042] The present application discloses a display device for a three-dimensional virtual model of a living organism, wherein the display device for the three-dimensional virtual model of a living organism constructs a first virtual model through two-dimensional scanning data of the living organism and image data of the living organism collected during surgery, thereby ensuring that the surface features and internal features of the first virtual model are consistent with the real posture / real state of the target area of ​​the living organism during surgery, and calibrating the first virtual model using a second virtual model constructed before surgery so that the sizes of the internal features in the calibrated first virtual model are consistent with the sizes of the real internal features of the living organism, and further, using surface markers configured on the body surface, the position and posture of the calibrated first virtual model in a real scene can be determined without invasive operation, and then the calibrated first virtual model and the real scene are fused and displayed, so as to achieve accurate matching of the three-dimensional virtual model with the entity and facilitate application in clinical surgery.

[0043] In some embodiments, the display device of the three-dimensional virtual model of the organism can be configured entirely in the mixed reality device, or can be configured in a distributed manner in the mixed reality device and one or more computer devices that communicate / network with the mixed reality device. This application does not limit the configuration method of the display device of the three-dimensional virtual model of the organism, as long as the first virtual model can be generated, the first virtual model can be calibrated, and the calibrated first virtual model and the real scene can be displayed through the mixed reality device by means of mutual communication.

[0044] See also Figure 1 , which is a schematic diagram of the structure of a display device for a three-dimensional virtual model of an organism in one embodiment of the present application. As shown in the figure, the display device 1 for the three-dimensional virtual model of an organism includes a construction module 10, a calibration module 11, and a display module 12.

[0045] The construction module 10, calibration module 11, and display module 12 may be implemented in software that is run by different types of processors. For example, a module of executable code may include one or more physical or logical blocks of computer instructions organized as objects, procedures, or functions. However, the executable files of the modules do not have to be physically located together, but may include different commands stored in different locations that, when logically connected together, comprise the module and achieve the specified goals of the module.

[0046] Of course, a module of executable code can be one or many instructions, and can even be distributed in several different code segments, distributed in different programs, and distributed in multiple storage devices. Similarly, operational data can be identified and shown in the module, and the operational data can be embodied in any suitable form and organized in any suitable type of data structure. The operational data can be collected as a single data set, or can be distributed in different locations (including different storage devices), and can exist at least partially only as electrical signals in a system or network. When a module or a portion of a module is implemented in software, the software portion is stored on one or more computer-readable media.

[0047] The construction module 10 is used to construct a first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data of the organism and the image data of the organism collected during the operation. Specifically, the construction module 10 obtains the two-dimensional scanning data of the organism collected during the operation and the image data of the organism collected during the operation, and constructs the first virtual model based on the obtained two-dimensional scanning data and image data of the organism. Wherein, the organism includes a human body or an animal body. In the following embodiments, the organism is temporarily described by taking the left foot of a human body as an example. The operation refers to the process in which the organism is located on a support platform in the operating room of a hospital to prepare for or start an operation. The support platform is, for example, an operating bed, a hospital bed, an examination bed, etc.

[0048] Before collecting the two-dimensional scanning data and image data during the operation, at least one body surface marker is also configured on the body surface (body surface) of the target area of ​​the biological body. The target area includes the part to be operated on in the body and the body surface corresponding to the part to be operated on in the body. For example, the part to be operated on is one or more of the bones of the left foot of the human body, and the corresponding target area can be the entire left foot. Figure 2a and Figure 2b , Figure 2a The display is a side view of a human left foot with a body surface marker configured thereon in one embodiment of the present application. Figure 2b Shown is a top view of a human left foot with surface markers in one embodiment of the present application. As shown in the figure, before the two-dimensional scanning data and image data are collected during surgery, a surface marker T1 and T2 are respectively configured on the surface of the ankle and instep of the patient's left foot.

[0049] In one embodiment, the body surface marker includes a physical object that can be detected by a transmission scanning device. The physical object can be attached to the body surface of the target area or configured on the body surface of the target area by wearing or the like. The number of the physical objects can be one or more. The physical object can be configured to include a spherical structure, a linear structure, and / or a planar structure. For example, the physical object is configured as a QR code sticker, and for example Figure 2a and Figure 2b The physical object illustrated in the figure is a circular patch. In another embodiment, the body surface marker is configured as a pattern mark that can be detected by a transmission scanning device. Specifically, a user (such as a doctor) can draw a pattern mark on the body surface of the target area using a material that can be detected by a transmission scanning device. The pattern mark may include points, lines, and / or surfaces. The material for drawing the pattern mark is related to the type of the transmission scanning device. For example, the transmission scanning device is an X-ray fluoroscopy device, and the material for drawing the pattern mark may be a fluorescent powder for X-ray fluoroscopy. It should be noted that the present application does not limit the shape or structure of the body surface marker, and the material for drawing the pattern mark. It only requires that the body surface marker be distinguished from human tissue or bones and can be recognized by a transmission scanning device.

[0050] In one embodiment, after completing the configuration of the body surface markers, the user (such as a doctor or a nurse) uses a transmission scanning device to collect two-dimensional scanning data of the organism during the operation. Specifically, during the operation, the user uses the transmission scanning device to see through the target area of ​​the organism to collect the two-dimensional scanning data, and then transmits or imports the two-dimensional scanning data to a mixed reality device or a computer device configured with the building module, so that the building module obtains the collected two-dimensional scanning data. In this embodiment, the transmission scanning device is an X-ray transmission scanning device, for example, the X-ray perspective scanning device is a C-arm X-ray machine. Configuring the transmission scanning device as an X-ray transmission scanning device can reduce the radiation to the organism during the operation.

[0051] In one embodiment, the collected two-dimensional scanning data of the organism includes the surface markers of the target area of ​​the organism and the scanning data of the internal tissue. In one embodiment, the transmission scanning device is an X-ray transmission scanning device, and the collected two-dimensional scanning data of the organism includes the frontal X-ray data and / or lateral X-ray data of the target area of ​​the organism. Figure 3a and Figure 3b , Figure 3a It is a schematic diagram of a lateral X-ray of a human left foot in one embodiment of the present application, Figure 3bShown is a schematic diagram of an AP X-ray of a human left foot in one embodiment of the present application. As shown in the figure, the lateral X-ray and the AP X-ray obtained after the human left foot is transmitted and scanned by an X-ray transmission scanning device include not only the image features of the surface markers of the target area of ​​the human left foot (T1', T2'), but also the image features of the internal tissues of the human left foot. These image features are used as scanning data for the subsequent construction of a first virtual model including the surface features and the internal features of the human left foot.

[0052] In one embodiment, the two-dimensional scanning data of the organism includes the body surface markers of the target area of ​​the organism, the coordinate data (pixel coordinates) of the internal tissues, and the light intensity data (pixel value) corresponding to the coordinate data. For example, if the two-dimensional scanning data is the two-dimensional scanning data of the left foot, then the two-dimensional scanning data includes the body surface markers of the left foot, the internal tissues (bones, organs, etc.), the coordinate data of the body surface, and the light intensity data corresponding to the coordinate data. It should be noted that the bones and internal organs in the body absorb X-rays to different degrees, and thus the light intensity data corresponding to the coordinate data of the bones and internal organs in the body are also different. In this way, the body surface markers, internal tissues (bones, organs, etc.), and the body surface can be distinguished according to the difference in light intensity data.

[0053] In one embodiment, the image data of the organism is collected in real time by the mixed reality device during the operation, for example, it is captured in real time by a sensor device in the mixed reality device, and the sensor device will be described in detail later. For another example, the image data of the organism can also be captured in real time by a camera device configured in a hospital operating room, and the camera device transmits the image data captured in real time to a mixed reality device or a computer device configured with the building module. In a specific embodiment, a user (such as a doctor) wears the mixed reality device during the operation, and the sensor device of the mixed reality device obtains the image data of the organism in real time. In the following embodiments, the image data of the organism is captured in real time by a mixed reality device during the operation. It should be noted that if the target area of ​​the organism is fixed during the operation, the target area will not change, and the image data of the organism can also be captured once by the mixed reality device during the operation.

[0054] In one embodiment, the image data of the organism includes image data of the body surface and body surface markers of the target area of ​​the organism. Specifically, the image data of the organism includes coordinate data (pixel coordinates) of the body surface and body surface markers of the target area and light intensity data (pixel values) corresponding to the coordinate data. The light intensity data may include only single-channel light intensity data or multi-channel light intensity data. In other words, the image data of the organism may be color image data or black-and-white image data. It should be noted that since the image data is not scanned data obtained based on transmission scanning, the image data does not include image data of the in vivo tissue of the target area.

[0055] After the construction module 10 obtains the two-dimensional scanning data and image data of the organism collected during the operation, it constructs a first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data and image data of the organism. The surface features of the organism include the three-dimensional features of the external contour of the target area, and may further include the three-dimensional features of the external tissues of the target area. For example, when the target area is the foot, the surface features of the organism include the external contour of the foot, and the three-dimensional features of external tissues such as the nails on the foot. The internal features include the three-dimensional features of the internal tissues of the target area, such as the internal bones and internal organs. For example, when the target area is the foot, the internal features include the three-dimensional features of multiple foot bones in the foot. Furthermore, in one embodiment, the first virtual model may also include the three-dimensional features of surface markers. For example, please refer to Figure 4 , which is a schematic diagram of a first virtual model constructed in one embodiment of the present application. As shown in the figure, the first virtual model of the foot includes a surface feature A and an internal feature B of the foot. The surface feature A includes a three-dimensional feature of the external contour of the foot, and the internal feature B includes a three-dimensional feature of the bones of the foot. In this way, the surface features and internal features of the first virtual model constructed using the two-dimensional scanning data and image data of the organism collected during the operation are consistent with the actual state or actual posture of the target area during the operation.

[0056] Furthermore, in one embodiment, the first virtual model also includes the three-dimensional features of the body surface markers. It should be noted that the three-dimensional features of the body surface markers may be included in the first virtual model or may not be included in the first virtual model. In the embodiment where the three-dimensional features of the body surface markers are not included in the first virtual model, the construction module 10 only records the positions and postures of the body surface markers on the first virtual model to reduce the amount of data of the first virtual model.

[0057] In one embodiment, the transparency of the first virtual model is 0.5, so that the first virtual model can be prevented from completely blocking the real scene, thereby improving the safety of the operation.

[0058] In one embodiment, the construction module 10 directly inputs the two-dimensional scanning data of the organism and the collected image data of the organism into a pre-trained model construction network to construct the first virtual model. The model construction network can be a three-dimensional recurrent reconstruction neural network (3D-R2N2 neural network), a convolutional neural network, a deep neural network, or a generative adversarial network. The training method of the model construction network will be described in detail later.

[0059] In another embodiment, before constructing the first virtual model, the construction module 10 also performs feature point detection and matching on the acquired two-dimensional scanning data and image data to determine the first body surface feature point and the second body surface feature point in the two-dimensional scanning data and the image data that represent the same body surface feature point in the organism, and identify the body feature point in the two-dimensional scanning data. In this embodiment, the construction module 10 can construct a first virtual model containing the body surface features and body features of the organism based on the coordinate data of the first body surface feature point and the body feature point in the two-dimensional scanning data, and the coordinate data of the second body surface feature point in the image data of the organism. For example, the two-dimensional scanning data and the coordinate data of the first body surface feature point and the body feature point corresponding thereto, and the image data and the coordinate data of the second body surface feature point corresponding thereto are input into a pre-trained model construction network to improve the accuracy and efficiency of model construction. The feature point is a point with unique local properties. For example, the feature point is a point, a corner point, a bifurcation point, etc. where the pixel value changes dramatically. In one example, the construction module 10 obtains the first body surface feature points, the second body surface feature points, and the body feature points using a feature point detection and matching algorithm. Examples of the feature point detection and matching algorithm include a scale-invariant feature transform algorithm (SIFT algorithm), an ORB (Oriented FAST and Rotated BRIEF) algorithm, a feature detection and matching algorithm based on deep learning, and the like.

[0060] After the construction module 10 constructs the first virtual model, since the first virtual model is not directly constructed based on the real-time scanned CT data or the real-time scanned X-ray data, the shape and size of the internal structure of the constructed first virtual model may be biased. To this end, the calibration module 11 calibrates the first virtual model according to the second virtual model of the organism to obtain the calibrated first virtual model.

[0061] The second virtual model is pre-constructed based on the preoperative CT data of the organism, and the second virtual model includes the in-vivo features of the organism. Figure 5, which is a schematic diagram of a second virtual model in one embodiment of the present application. As shown in the figure, the second virtual model is a virtual model of the foot, and the virtual model includes the in vivo features of the foot (the shape, size, and posture of each foot bone). Since the CT data includes two-dimensional tomography data / two-dimensional slice data of multiple layers inside the organism, the second virtual model constructed based on the CT data has a high accuracy. In other words, the second virtual model is credible and consistent with the real organism.

[0062] In one embodiment, see Figure 6 , which is a flowchart of a mixed reality device displaying a calibrated first virtual model in one embodiment of the present application. As shown in the figure, before the operation, a CT perspective scanning device is first used to perform a perspective scan on the organism to obtain the preoperative CT data, and then a three-dimensional reconstruction is performed based on a preset medical three-dimensional reconstruction method and the preoperative CT data to construct the second virtual model, and the second virtual model is transmitted to the calibration module 11. The preset medical three-dimensional reconstruction method is exemplified by a surface rendering algorithm (such as a Marching Cubes algorithm) or a volume rendering algorithm. The volume rendering algorithm includes a ray casting algorithm, a shearing-deformation algorithm, a frequency domain volume rendering algorithm, a snowball algorithm, etc.

[0063] In one embodiment, during the construction of the second virtual model, the preoperative CT data may be preprocessed such as denoising, and then the 3D model may be constructed using the preprocessed CT data and a 3D model construction method in the medical field such as a surface rendering algorithm or a volume rendering algorithm. Furthermore, the constructed second virtual model may be post-processed such as repairing defects, adjusting transparency, and displaying sections.

[0064] In one embodiment, the calibration module 11 inputs the second virtual model and the first virtual model into a pre-trained registration network to obtain a calibrated first virtual model output by the registration network. The registration network is exemplified by a spatiotemporal transform neural network (STN network), a convolutional neural network (CNN network), etc. The training method of the registration network will be described in detail later.

[0065] In another embodiment, the calibration module 11 calibrates the first virtual model according to the coordinate data of the feature points of the in-vivo features in the second virtual model to obtain the calibrated first virtual model. In one example, the coordinate data of multiple (two or more) feature points on the same rigid in-vivo feature are used to calibrate the coordinate data of the corresponding multiple feature points in the first virtual model to obtain the calibrated first virtual model. For example, the coordinate data of multiple (two or more) feature points on the same rigid in-vivo feature are used to determine the size and shape of the rigid in-vivo feature, and the coordinate data of the corresponding multiple feature points in the first virtual model are calibrated using the determined size and shape, so that the shape and size of the corresponding rigid in-vivo feature in the calibrated first virtual model are consistent with the shape and size of the rigid in-vivo feature of the second virtual model.

[0066] For another example, the coordinate data of multiple (two or more) feature points on the same rigid in-vivo feature are used to determine the spacing between the feature points, and the coordinate data of the corresponding multiple feature points in the first virtual model are calibrated using the determined spacing, so that the spacing between the corresponding feature points in the calibrated first virtual model is consistent with the spacing between the feature points of the second virtual model. Among them, the rigid in-vivo feature refers to an in-vivo feature whose shape and size will not change with the movement of the organism, such as the femur, navicular bone, first metatarsal of the thumb, second metatarsal of the thumb, and first metatarsal in the foot bones. It should be noted that when calibrating, the relative position relationship between different rigid in-vivo features is not changed to avoid the change of the relative position relationship and the actual posture of the target area during the operation.

[0067] In one example, see Figures 7a to 7c , Figure 7a The diagram shows the relative position relationship between the feature points on the first metatarsal bone and the second phalanx of the thumb in the first virtual model in one embodiment of the present application. Figure 7b The diagram shows the relative position relationship between the feature points on the first metatarsal bone and the second phalanx of the thumb in the second virtual model in one embodiment of the present application. Figure 7c The diagram shows the relative position relationship between the feature points on the first metatarsal and the second phalanx of the thumb in the first virtual model after calibration in one embodiment of the present application, as shown in FIG. Figure 7a As shown, according to the coordinate data of the two characteristic points (210, 211) on the first metatarsal 21 and the coordinate data of the characteristic point 220 on the second phalanx 22 of the thumb, it can be determined that the distance between the two characteristic points (210, 211) on the first metatarsal 21 is C, and the distance between the characteristic point 220 and the characteristic point 210 on the second phalanx 22 of the thumb is D. Similarly, Figure 7b As shown, it can be based on Figure 7bThe corresponding coordinate data determines that the distance between the feature points (210, 211) is C1, and the distance between the feature point 220 and the feature point 210 is D1. Based on the distances C1 and D1, Figure 7a The coordinate data of the feature point 211 and the feature point 220 shown in FIG. 1 are corrected to obtain the coordinate data of the three feature points in the first virtual model after calibration without changing the relative posture relationship between the first metatarsal bone and the second phalanx of the thumb, as shown in FIG. Figure 7c As shown, the distance between the feature point 220 and the feature point 210 in the calibrated first virtual model is D1, and the distance between the feature points (210, 211) in the calibrated first virtual model is C1.

[0068] After the calibration module 11 obtains the calibrated first virtual model, the display module 12 uses the image data of the body surface markers to determine the position of the calibrated first virtual model in the real scene, so as to display the calibrated first virtual model and the real scene through a mixed reality device.

[0069] In one embodiment, the display module 12 determines the posture of the body surface marker in the real scene based on the image data of the body surface marker, and then determines the posture of the calibrated first virtual model in the real scene according to the determined posture of the body surface marker in the real scene and the posture of the three-dimensional features of the body surface marker in the calibrated first virtual model on the calibrated first virtual model, so as to display the calibrated first virtual model and the real scene through a mixed reality device. It should be noted that when the mixed reality device is displayed, depending on the different configurations of the mixed reality device, the displayed real scene can be an image of the real scene presented by the mixed reality device or a real scene observed by the user through the mixed reality device.

[0070] In one example, if Figure 6 As shown, the image data of the organism is collected by using a mixed reality device. Specifically, after the organism is configured with a body surface marker during the operation, the two-dimensional scanning data of the organism is collected by the transmission scanning device, and the image data of the organism is collected by the mixed reality device. The calibration module 11 can determine the posture of the body surface marker in the real scene based on the image data of the body surface marker, that is, the posture of the body surface marker relative to the mixed reality device, and then determine the posture of the calibrated first virtual model in the real scene based on the posture of the body surface marker in the real scene and the posture of the three-dimensional feature of the body surface marker on the calibrated first virtual model, that is, the posture of the calibrated first virtual model relative to the mixed reality device, and then the mixed reality device can fuse the calibrated first virtual model and the real scene for display, so that the user can see both the real scene through the mixed reality device and the calibrated first virtual model in the target area of ​​the organism.

[0071] In one embodiment, see Figure 8 , which is a schematic diagram of the structure of a display device for a three-dimensional virtual model of an organism in another embodiment of the present application, as shown in the figure, the display device 1 for a three-dimensional virtual model of an organism also includes a sharing module 13, and the sharing module 13 is used to send the display data of the mixed reality device to a sharing device connected to the mixed reality device for display by the sharing device; in this way, the user can observe the display content seen by the user (such as the surgeon) wearing the mixed reality device through the sharing device. The sharing device is exemplified by a computer device or a mixed reality device.

[0072] Furthermore, the sharing module also includes displaying the calibrated first virtual model configured with annotation information sent by the shared device belonging to the whitelist in the mixed reality device and all shared devices. In this way, in the scenario of remote surgery, the remote doctor wearing the shared device configured as a mixed reality device can interact with the displayed calibrated first virtual model, for example, configuring annotation information on the displayed calibrated first virtual model, and then the doctor in the operating room can observe the annotation information annotated by the remote doctor in real time. Among them, the annotation information is exemplified by surgical path, puncture point, etc.

[0073] In one embodiment, see Fig. 9 , which is a structural schematic diagram of a display device for a three-dimensional virtual model of an organism in another embodiment of the present application, as shown in the figure, the display device 1 of the three-dimensional virtual model also includes an updating module 14, and the updating module 14 is used to update the image data of the organism, so that the construction module 10 updates the first virtual model, so that the calibration module 11 calibrates the updated first virtual model and the display module 12 displays the calibrated updated first virtual model and the real scene through a mixed reality device. In this way, it is ensured that when the posture of the target area of ​​the organism changes, the calibrated updated first virtual model displayed by the mixed reality device matches the current posture of the target area, ensuring that the three-dimensional virtual model and the organism can be accurately matched, and then can be widely used in actual clinical surgery. The posture of the target area refers to the shape or state presented by the target area. For example, the target area is a foot, and the posture of the foot includes but is not limited to toes together, toes open, the instep is straight, the instep is bent, the toes are bent, and the toes are straight.

[0074] In a specific embodiment, the mixed reality device collects image data of the organism in real time, the update module 14 updates the image data of the organism when determining that the image data of the organism has changed, the construction module 10 updates the first virtual model based on the updated image data of the organism, the calibration module 11 calibrates the updated first virtual model using the second virtual model to obtain the calibrated updated first virtual model, and then the display module 12 determines the position of the calibrated updated first virtual model in the real scene based on the updated image data of the body surface markers, so as to display the calibrated updated first virtual model and the real scene through the mixed reality device. In one example, the update module 14 determines whether the image data of the organism has changed based on the image data of the organism acquired in real time and a preset judgment rule. Among them, the preset judgment rule is that a change in the position, posture, and / or posture of the target area of ​​the organism means a change in the image data of the organism. For example, if a person's foot changes from a straight state to a bent state, it can be determined that the posture of the target area has changed. For another example, if a person's foot moves from the left side of the operating table to the right side, it can be determined that the position of the target area has changed. For another example, if a person's foot changes from being parallel to the operating table to being at an angle to the operating table, it can be determined that the posture of the target area has changed.

[0075] In one embodiment, the construction module 10 further determines whether the part to be operated on and the preset area around the part to be operated on have changed based on the updated image data of the organism. If no change has occurred, the construction module does not update the first virtual model. Otherwise, the construction module updates the first virtual model based on the updated image data of the organism. In this way, the real state of the part to be operated on can be viewed in real time through the virtual model, and the extra calculation caused by frequent updates of the first virtual model can be avoided.

[0076] In order to obtain the trained model building network mentioned above, in one embodiment, the display device of the three-dimensional virtual model further includes a first training module (not shown), which is used to train the network parameters of the model building network using the first sample data until the loss function of the model building network converges to a preset threshold to obtain the trained model building network. For the sake of distinction, in the following embodiments, the preset threshold used in training the model building network is referred to as the first preset threshold, and the preset threshold used in training the registration network is referred to as the second preset threshold.

[0077] In one embodiment, the first sample data includes a third virtual model (real sample) pre-constructed using CT data of the body surface and body of the organism, and the corresponding two-dimensional scanning data and image data of the organism. In other words, the first sample data includes multiple groups of two-dimensional scanning data, image data, and pre-constructed third virtual model belonging to the same organism; further, the first sample data also includes pre-determined first body surface feature points, second body surface feature points, and body feature points in the two-dimensional scanning data to increase the speed of training. In the following embodiments, for ease of explanation, the first sample data includes multiple groups of two-dimensional scanning data, image data, and pre-constructed third virtual model belonging to the same organism as an example.

[0078] In one embodiment, a first preset tag is pre-set in the two-dimensional scanning data and the image data respectively, and the first preset tag is set with tag information for indicating a paired relationship. In other words, the first preset tag is set with tag information for indicating whether the two-dimensional scanning data and the image data belong to the same organism.

[0079] In one embodiment, the model building network is configured as a generative adversarial network, and the model building network includes a generator and a discriminator. During the training process, the first training module inputs the two-dimensional scanning data and image data of multiple organisms in the first sample data and random noise into the generator so that the generator can generate multiple first virtual models, and the first training module inputs the third virtual model (real sample) corresponding to the multiple organisms and the first virtual model generated by the generator into the discriminator so that the discriminator can output a discrimination result; wherein the discrimination result includes whether the input model is a real sample or a generated sample, and the probability value corresponding to the real sample or the generated sample; further, the first training module calculates the loss function of the generative adversarial network according to the discrimination result, the generated first virtual model, and the third virtual model, and adjusts the model parameters of the generative adversarial network until the loss function of the generative adversarial network reaches a first preset threshold. The first preset threshold value has a value range of 0.45-0.55. For example, the first preset threshold value is 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53, 0.54, or 0.55.

[0080] Among them, the loss function L1 of the generative adversarial network is L gen +L ce， L gen is the generation loss of the generator, L ce is the classification loss of the discriminator. In one embodiment, the classification loss L ce is the classification cross entropy loss, the generation loss Lgen is the 3D reconstruction loss. For example, the 3D reconstruction loss is −αy g log(y e )﹣(1﹣α)(1﹣y g )log(1﹣y e ), where α is the preset coefficient, y g is the 3D voxel of the third virtual model, y e is the three-dimensional voxel of the first virtual model generated.

[0081] It should be noted that the model building network is configured as a generative adversarial network to improve the building accuracy of the first virtual model. However, this is not limited to this. In other embodiments, the model building network can also be configured as any of the networks in the above examples.

[0082] In order to obtain the trained registration network mentioned above, in one embodiment, the display device also includes a second training module (not shown in the figure), and the second training module is used to train the network parameters of the registration network using the second sample data until the loss function of the registration network converges to a second preset threshold to obtain the trained registration network; wherein, the second sample data includes a second virtual model of the organism, and a corresponding first virtual model; further, the second virtual model and the first virtual model are correspondingly pre-set with a second preset label, and the second preset label is provided with label information for indicating whether the second virtual model and the first virtual model belong to the same organism.

[0083] In one embodiment, the second training module inputs a plurality of second virtual models and the corresponding first virtual models into the registration network and outputs the calibrated first virtual model. The second training module calculates the loss function of the registration network based on the in-vivo features of the calibrated first virtual model and the corresponding second virtual model, and adjusts the model parameters of the registration network until the loss function of the registration network converges to a second preset threshold. The loss function of the registration network is exemplified by a shape matching loss function or a geometric loss function. The shape matching loss function is exemplified by an intersection-over-union loss function (IoU loss function). The value range of the second preset threshold is 0-0.05, for example, the second preset threshold is 0, 0.01, 0.01, 0.01, 0.01, or 0.01.

[0084] In summary, the display device of the three-dimensional virtual model of an organism provided by the present application constructs a first virtual model through the two-dimensional scanning data of the organism and the image data of the organism collected during the operation, thereby ensuring that the surface features and internal features of the first virtual model are consistent with the actual state / actual posture of the target area of ​​the organism during the operation, which is conducive to the accurate matching of the three-dimensional virtual model with the entity; and after calibrating the first virtual model with the second virtual model constructed before the operation, the size of the internal features in the calibrated first virtual model is consistent with the size of the real internal features of the organism, further ensuring the accurate matching of the three-dimensional virtual model with the entity; and using the surface markers configured on the body surface, the calibrated first virtual model and the real scene can be fused and displayed without performing invasive operations on the organism to expose the internal structure, thereby finally achieving the accurate matching of the three-dimensional virtual model with the entity.

[0085] In addition, the display device of the three-dimensional virtual model of the organism provided by the present application can reduce the surgical cost of configuring the positioning target and can be convenient for clinical surgical applications; further, during the operation, as long as the mixed reality device detects the image data of the organism in real time, it can track the target area of ​​the organism in real time and display the calibrated first virtual model in real time, without the need to configure an optical tracking system that occupies a large space. Therefore, the solution of the present application will not have the problem of being unable to collect image data and the disappearance of the screen (disappearance of the virtual model) when the distance exceeds 2 meters, and there will be no signal delay, screen jitter and other problems caused by the need for signal transmission between more devices. Furthermore, it can also avoid the additional configuration cost caused by configuring the optical tracking system; in particular, the present application updates the image data of the organism in real time and displays the updated first virtual model after calibration through the update module, which can ensure that the displayed three-dimensional virtual model is consistent with the actual target area when the target area of ​​the organism changes during the operation, which is conducive to the smooth progress of the operation.

[0086] Although in the related art, a one-time scan of body surface contour data before surgery can be performed during surgery to construct a body surface contour model including markers, and the structural model including surface features and internal features constructed before surgery is fused with the body surface contour model to obtain a surgical guidance model including the internal features and the body surface contour model, and then the doctor determines the surgical path based on the surgical guidance model and the physical markers for surgical guidance; however, since the above scheme assumes that the target area of ​​the human body during surgery and before surgery will not change and the internal features before surgery are marked with information such as the surgical path guiding the surgery, the above scheme directly applies the internal features obtained before surgery to generate the surgical guidance model, resulting in the generated surgical guidance model being inconsistent with the posture of the target area during surgery due to changes in human body posture.

[0087] Compared with the above-mentioned related technologies, the present application can ensure that the displayed virtual model is consistent with the actual state / actual posture of the target area; since the above-mentioned related technologies are body surface contour data scanned once, the scheme cannot update the surgical guidance model according to the real-time state of the human body; while the present application can display the updated three-dimensional virtual model in real time based on the updated image data, ensuring that the three-dimensional virtual model is consistent with the state of the target area at every moment; and the surgical guidance model in the above-mentioned related technologies will not be displayed through a mixed reality device, resulting in the doctor still needing to constantly switch between the computer screen and the human body, thereby failing to efficiently guide the doctor's surgery; while the present application can display the calibrated first virtual model through a mixed reality device, without the need to constantly switch viewing angles, thereby improving surgical efficiency.

[0088] The present application also discloses a display system for a three-dimensional virtual model of an organism, wherein the display system is configured with the display device for the three-dimensional virtual model of an organism as described above. For example, each module of the display device for the three-dimensional virtual model of an organism is configured in a computer device and a mixed reality device in the display system so that the display system can realize the functions of each module.

[0089] See also Fig.10 , which is a structural schematic diagram of a display system of a three-dimensional virtual model of an organism in one embodiment of the present application, as shown in the figure, the display system includes a computer device 3 and a mixed reality device 4. The computer device 3 is communicatively connected to the mixed reality device 4 for data transmission between the mixed reality device 4 and the computer device 3. Specifically, the computer device 3 is used to construct a first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data of the organism collected during the operation and the image data of the organism; and to calibrate the first virtual model based on the second virtual model of the organism to obtain a calibrated first virtual model. The mixed reality device 4 is connected to the computer device 3, and is used to determine the position and posture of the calibrated first virtual model in the real scene using the image data of the surface markers, so as to display the calibrated first virtual model and the real scene through the mixed reality device 4.

[0090] In one embodiment, see Fig.11 , which is a schematic diagram of the structure of a computer device in one embodiment of the present application. As shown in the figure, the computer device 3 includes a storage device 30, a processing device 31 connected to the storage device 30, and an interface device 32.

[0091] In some embodiments, the storage device 30 is used to store at least one program, and the storage device 30 includes but is not limited to: read-only memory, random access memory, and non-volatile memory. For example, the storage device 30 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 30 may also include a memory away from one or more processing devices 31, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as the CPU and peripheral interfaces.

[0092] In one embodiment, the storage device 30 is also used to store the second virtual model or the collected two-dimensional scanning data of the organism, etc., so that the computer device 3 can construct and calibrate the first virtual model.

[0093] In some embodiments, the interface device 32 is used to communicate with the mixed reality device 4 to receive the image data of the organism obtained by the mixed reality device 4. The interface device 32 includes at least one interface unit, each of which is used to output a visual interface, receive a human-computer interaction event generated according to a user's operation, etc. For example, the interface device 32 includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the interface device 32 also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0094] In some embodiments, the interface device 32 is further used to receive display data synchronously transmitted by the mixed reality device 4. The display data includes the real scene and virtual objects seen by the user through the mixed reality device 4.

[0095] In some embodiments, the interface device 32 is further used to send the calibrated first virtual model to the mixed reality device 4 .

[0096] In some embodiments, the processing device 31 is connected to the storage device 30, and the processing device 31 is used to call the at least one program from the storage device 30 and construct a first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data of the organism and the image data of the organism collected during the operation; and calibrate the first virtual model based on the second virtual model of the organism to obtain a calibrated first virtual model; wherein, the method of constructing the first virtual model and calibrating the first virtual model is the same or similar to that described above, and will not be repeated here.

[0097] In some embodiments, the processing device 31 includes one or more processors. The processing device 31 can be operated to perform data read and write operations with the storage device 30. The processing device 31 includes one or more general-purpose microprocessors, one or more special-purpose processors (ASICs), one or more digital signal processors, one or more field programmable logic arrays (FPGAs), or any combination thereof. In one example, the special-purpose processor includes a graphics processor.

[0098] In one embodiment, if Fig.11 As shown, the computer device 3 further includes a display device 33. The display device 33 is, for example, a display or a touch screen of the computer device 3.

[0099] In the embodiment where the computer device 3 includes the display device 33, the processing device 31 is further used to display the display data synchronously transmitted by the mixed reality device 4 received by the interface device 32 on the display device 33. In this way, a user who does not wear a mixed reality device during the operation can also view the calibrated first virtual model and the real scene through the computer device 3.

[0100] In one embodiment, the mixed reality device 4 is a device that can merge virtual objects (such as a calibrated first virtual model or a calibrated updated first virtual model) and real scenes into reality; further, the user can also interact with the virtual objects. The mixed reality device 4 is exemplified by a head-mounted mixed reality device 4 (such as mixed reality glasses). In one embodiment, the mixed reality device 4 may include one or more sensors (such as depth sensors, cameras, etc.) for scanning or collecting information from a real scene (such as an operating room, etc.) and a circuit system for transmitting the collected information to another device (such as a server, workstation, desktop computer, laptop, tablet computer, smart phone, etc.). Further, the mixed reality device 4 may include a wireless interface for connecting to the Internet, a local wireless network, or another computing device.

[0101] In one embodiment, the mixed reality device 4 may also be used to receive audio / gesture input from the user. Such gestures or audio input may be spoken voice commands or recognized user gestures, which, when recognized by the mixed reality device 4, may cause the mixed reality device 4 to execute the corresponding command, and the gestures may be recognizable gestures described in the embodiments below. In one embodiment, the mixed reality device 4 is preferably lightweight and is constructed to avoid the use of heavy components, which may make the mixed reality device 4 uncomfortable to wear. For example, the mixed reality device 4 is mixed reality glasses, which are relatively light, and the user's head is less weighted when wearing the mixed reality glasses during surgery.

[0102] The mixed reality device 4 may be configured to determine a display area in a real scene for displaying a virtual object in the display area, so that the user can see the virtual object at the location of the display area. Further, the mixed reality device 4 may also provide image data and depth data to the mobile processor and receive display information from the mobile processor, which may be separate from the mixed reality device 4, such as in a computer device connected to the mixed reality device 4, as shown in FIG. Fig.10 discussed in more detail.

[0103] The mixed reality device 4 in the present application is capable of rendering virtual objects displayed on a display area to provide a mixed reality experience, which can facilitate interaction with a computing device and collaboration with other users.

[0104] See also Fig.12 , which is a schematic diagram of the structure of a mixed reality device in one embodiment of the present application. As shown in the figure, the mixed reality device 4 includes a sensing device 40, a storage device 41, a processing device 42, and a display device 43.

[0105] The sensor device 40 is used to collect image data of a biological body. In some embodiments, the sensor device 40 includes at least one sensor. In an embodiment where the sensor device 40 includes multiple sensors, the multiple sensors may all be sensors of the same type or sensors of different types.

[0106] In one embodiment, the sensing device 40 includes a camera, which is used to provide image data of the organism to the processing device 42. Further, the sensing device 40 may also include a depth sensor, which is used to provide depth data to the processing device 42, and then the processing device 42 can determine the distance of the object relative to the mixed reality device 4 according to the depth data.

[0107] In some embodiments, the storage device 41 is used to store at least one program. The at least one program is executable by the processing device 42. Here, the storage device 41 includes but is not limited to: read-only memory, random access memory, and non-volatile memory. For example, the storage device 41 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 41 may also include a memory away from one or more processing devices 42, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as the CPU and peripheral interfaces.

[0108] The display device 43 is used to integrate the virtual model with the real scene, so that the user can see both the virtual object and the real scene through the display device 43. For example, the display device 43 determines the display area of ​​the calibrated first virtual model based on the determined position of the calibrated first virtual model in the real scene, so as to integrate and display the calibrated first virtual model and the real scene.

[0109] In one embodiment, the display device 43 is a semi-transparent display device, through which the user can see the real scene, and the display device 43 can display a virtual model (e.g., a calibrated first virtual model) so that the virtual model appears to be anchored to the display area seen by the user through the display device 43. In this way, the user can view the real scene with a semi-transparent head-mounted display device, wherein objects in the real scene (such as a hospital bed or a living organism) are visible through the head-mounted display device, and the head-mounted display device also places the virtual model in the visible scene, and the virtual model can be anchored to or connected to the real-world object, and this embodiment provides a mixed reality experience.

[0110] In one embodiment, the display device 43 is a display device based on video synthesis technology. Specifically, the processing device 42 (such as a graphics processor) obtains the image information of the real scene collected by the sensor device 40, and then synthesizes the virtual model into the image of the real scene after information processing, and finally the processing device 42 controls the display device 43 to display the synthesized image to the user. In one example, the display device based on video synthesis technology includes a display.

[0111] In one embodiment, the display device 43 is a display device based on an optical element, for example, the display device 43 is a holographic display device. The processing device 43 controls the display device based on an optical element to first directly project the virtual model onto the optical element unit, and then send the virtual object into the human eye through the reflection of the optical element unit. At the same time, the picture in the real scene will directly enter the human eye through the optical element unit, so that the user can see the virtual object while seeing the real scene.

[0112] In one example, the optical element-based display device includes a display and an optical element unit. The optical element unit includes a semi-transparent and semi-reflective mirror, a holographic diffraction waveguide grating, or other suitable optical elements. In this embodiment, the processing device controls the display to display the virtual model at a preset position and in a preset posture, and the optical element unit is used to reflect the virtual model on the display into the field of view of the human eye and to transmit light from the real scene.

[0113] It should be noted that the description of the display device in the present application is limited to examples. In other embodiments, the display device may also be a display device including elements such as optical waveguides.

[0114] In some embodiments, the processing device 42 is connected to the storage device 41, the display device 43, and the sensor device 40, and is used to determine the position of the calibrated first virtual model in the real scene using the image data of the body surface marker when calling and executing the at least one program from the storage device 41, so as to display the calibrated first virtual model and the real scene through the mixed reality device 4.

[0115] In some embodiments, the processing device 42 includes one or more processors. The processing device 42 can be operative to perform data read and write operations with the storage device 41. The processing device 42 includes one or more general-purpose microprocessors, one or more special-purpose processors (ASICs), one or more digital signal processors, one or more field programmable logic arrays (FPGAs), or any combination thereof. In one example, the special-purpose processor includes a graphics processor.

[0116] In one embodiment, the mixed reality device 4 further includes an interface device (not shown), which is used to communicate with the computer device 3, and further, the interface device is also used to communicate with a shared device. The interface device includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the interface device also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0117] In one embodiment, the display system further includes a shared device (not shown) that is communicatively connected to the mixed reality device 4, and the shared device is used to synchronously display the display data of the mixed reality device 4. In this way, the user (a user in the operating room or a user wearing the shared device remotely) can observe the display content seen by the user wearing the mixed reality device 4 (e.g., the surgeon) through the shared device, which facilitates surgical cooperation, surgical teaching, remote surgery, etc.

[0118] In one embodiment, the display system (eg, computer device 3 ) may also receive a calibrated first virtual model configured with annotation information sent by a shared device belonging to the whitelist, so as to display it in the mixed reality device 4 and all shared devices.

[0119] This application also provides a mixed reality device, please continue to refer to Fig.12 The mixed reality device includes a sensing device 40, a storage device 41, a processing device 42, and a display device 43. The structures and functions of the sensing device 40, the storage device 41, and the display device 43 and the structure of the processing device 42 are the same or similar to those described above and will not be repeated here.

[0120] In one embodiment, the mixed reality device is used with a surgical navigation system to allow doctors to perform surgical operations. In this embodiment, the surgical navigation system is an electronic system that assists users in performing surgery by providing guidance information. The guidance information includes: graphics or data representing the relative position relationship between surgical instruments and the surgical site to be operated on, and / or surgical paths, etc., to provide real-time guidance information to users.

[0121] In one embodiment, the mixed reality device 4 is a device that can merge virtual objects (such as a calibrated first virtual model or a calibrated updated first virtual model) and real scenes into reality; further, the user can also interact with the virtual objects. The mixed reality device 4 is exemplified by a head-mounted mixed reality device 4 (such as mixed reality glasses). In one embodiment, the mixed reality device 4 may include one or more sensors (such as depth sensors, cameras, etc.) for scanning or collecting information from a real scene (such as an operating room, etc.) and a circuit system for transmitting the collected information to another device (such as a server, workstation, desktop computer, laptop, tablet computer, smart phone, etc.). Further, the mixed reality device 4 may include a wireless interface for connecting to the Internet, a local wireless network, or another computing device.

[0122] In one embodiment, the mixed reality device 4 may also be used to receive audio / gesture input from the user. Such gestures or audio input may be spoken voice commands or recognized user gestures, which, when recognized by the mixed reality device 4, may cause the mixed reality device 4 to execute the corresponding command, and the gestures may be recognizable gestures described in the embodiments below. In one embodiment, the mixed reality device 4 is preferably lightweight and is constructed to avoid the use of heavy components, which may make the mixed reality device 4 uncomfortable to wear. For example, the mixed reality device 4 is mixed reality glasses, which are relatively light, and the user's head is less weighted when wearing the mixed reality glasses during surgery.

[0123] The mixed reality device 4 may be configured to determine a display area in a real scene for displaying a virtual object in the display area, so that the user can see the virtual object at the location of the display area. Further, the mixed reality device 4 may also provide image data and depth data to the mobile processor and receive display information from the mobile processor, which may be separate from the mixed reality device 4, such as in a computer device connected to the mixed reality device 4, as shown in FIG. Fig.10 discussed in more detail.

[0124] The mixed reality device 4 in the present application can render virtual objects displayed on the display area to provide a mixed reality experience, which can facilitate interaction with computing devices and collaboration with other users. The processing device 42 is connected to the sensor device 40, the storage device 41 and the display device 43, and is used to call at least one program from the storage device 41 and execute it to build a first virtual model containing the surface features and internal features of the organism based on the two-dimensional scanning data of the organism and the image data of the organism collected during the operation; and calibrate the first virtual model based on the second virtual model of the organism to obtain a calibrated first virtual model; and use the image data of the body surface marker to determine the position of the calibrated first virtual model in the real scene, so as to display the calibrated first virtual model and the real scene through the mixed reality device. Among them, the method of constructing and calibrating the first virtual model, and displaying the calibrated first virtual model and the real scene through the mixed reality device is the same or similar to that described above, and will not be repeated here.

[0125] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] In summary, the display device and system of the three-dimensional virtual model of the organism disclosed in the present application, as well as the mixed reality device, construct a first virtual model through the two-dimensional scanning data of the organism and the image data of the organism collected during the operation, thereby ensuring that the surface features and internal features of the first virtual model are consistent with the actual posture of the target area of ​​the organism during the operation, and after calibrating the first virtual model using the second virtual model constructed before the operation, the size of the internal features in the calibrated first virtual model is consistent with the size of the real internal features of the organism. Furthermore, using the surface markers configured on the body surface, the calibrated first virtual model and the real scene can be fused and displayed without invasive operation, so as to achieve accurate matching of the three-dimensional virtual model and the entity, thereby facilitating the application of clinical surgery; further, the two-dimensional scanning data of the organism is configured as frontal X-ray data and lateral X-ray data, and the first virtual model is calibrated using the second virtual model, which not only reduces the impact of radiation on the organism but also ensures the accuracy of the displayed virtual model.

[0127] The above embodiments are merely illustrative of the inventive essence of the present application and the beneficial effects obtained, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the principles and scope of the present application. Therefore, all equivalent modifications or changes achieved by a person with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A display device for a three-dimensional virtual model of an organism, characterized in that: include: A construction module is used to construct a first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data of the organism and the image data of the organism collected during the operation; wherein the two-dimensional scanning data includes the surface marks of the target area of ​​the organism and the scanning data of the internal tissue, and the image data of the organism includes the image data of the surface of the target area of ​​the organism and the surface marks; the method of constructing the first virtual model including the surface features and internal features of the organism based on the two-dimensional scanning data and the image data of the organism collected during the operation includes: inputting the two-dimensional scanning data of the organism and the collected image data of the organism into a pre-trained model construction network to construct the first virtual model; A calibration module, used for calibrating the first virtual model according to the second virtual model of the organism to obtain a calibrated first virtual model; wherein the second virtual model is pre-constructed according to the preoperative CT data of the organism, and the second virtual model includes the in-vivo features of the organism; the method of calibrating the first virtual model according to the second virtual model of the organism to obtain the calibrated first virtual model includes: using the coordinate data of multiple feature points on the second virtual model that belong to the same rigid in-vivo feature as the rigid in-vivo feature in the first virtual model to calibrate the coordinate data of the corresponding multiple feature points in the first virtual model, so that the shape and size of the corresponding rigid in-vivo features in the calibrated first virtual model are consistent with the shape and size of the rigid in-vivo features of the second virtual model; or, inputting the second virtual model of the organism and the first virtual model into a pre-trained registration network to obtain the calibrated first virtual model output by the registration network; The display module is used to determine the position and posture of the calibrated first virtual model in the real scene by using the image data of the body surface marker, so as to display the calibrated first virtual model and the real scene through a mixed reality device.

2. The display device for a three-dimensional virtual model of an organism according to claim 1, characterized in that: The two-dimensional scanning data of the organism is collected during surgery by using a transmission scanning device to see through a target area of ​​the organism, wherein a body surface marker is configured on the body surface of the target area.

3. The display device of the three-dimensional virtual model of an organism according to claim 2, characterized in that: The two-dimensional scanning data of the organism includes frontal X-ray data and lateral X-ray data of the target area of ​​the organism.

4. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The body surface marker includes a physical object or a pattern mark that can be detected by a transmission scanning device.

5. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The image data of the organism is collected in real time by the mixed reality device during surgery.

6. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The method of calibrating the first virtual model according to the second virtual model of the organism to obtain the calibrated first virtual model includes: calibrating the first virtual model according to the coordinate data of the feature points of the in-vivo features in the second virtual model to obtain the calibrated first virtual model.

7. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The method of using the image data of the body surface marker to determine the position and posture of the calibrated first virtual model in the real scene, and displaying the calibrated first virtual model and the real scene through a mixed reality device includes: Determining the position and posture of the body surface marker in a real scene based on the image data of the body surface marker; According to the determined posture of the body surface marker in the real scene and the characteristics of the body surface marker in the calibrated first virtual model, the posture of the calibrated first virtual model in the real scene is determined to display the calibrated first virtual model and the real scene through a mixed reality device.

8. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The display device also includes a sharing module, which is used to send display data of the mixed reality device to a sharing device connected to the mixed reality device for display by the sharing device.

9. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The display device further comprises an updating module for updating the image data of the organism so that the construction module can update the first virtual model.

10. The display device of the three-dimensional virtual model of an organism according to claim 9, characterized in that: The updating module updates the image data of the organism when determining that the image data of the organism is changed.

11. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The display device also includes a first training module, which is used to train the network parameters of the model construction network using the first sample data until the loss function of the model construction network converges to a preset threshold to obtain the trained model construction network; wherein, the first sample data includes a third virtual model pre-constructed using CT data on the surface and inside of the organism, and the corresponding two-dimensional scanning data and image data of the organism.

12. The display device of the three-dimensional virtual model of an organism according to claim 1, characterized in that: The display device also includes a second training module, which is used to train the network parameters of the registration network using second sample data until the loss function of the registration network converges to a preset threshold to obtain the trained registration network; wherein the second sample data includes a second virtual model of the organism, and a corresponding first virtual model.

13. A display system for a three-dimensional virtual model of an organism, characterized in that: include: A computer device, used to construct a first virtual model including surface features and internal features of a living organism based on two-dimensional scanning data and image data of the living organism collected during surgery; and to calibrate the first virtual model based on a second virtual model of the living organism to obtain a calibrated first virtual model; wherein the two-dimensional scanning data includes surface markers of a target area of ​​the living organism and scanning data of internal tissues, the image data of the living organism includes image data of the surface of the target area of ​​the living organism and the surface markers, the second virtual model is pre-constructed based on CT data of the living organism before surgery and the second virtual model includes the internal features of the living organism; the method of constructing the first virtual model including surface features and internal features of the living organism based on the two-dimensional scanning data and image data of the living organism collected during surgery The method comprises: inputting the two-dimensional scanning data of the organism and the collected image data of the organism into a pre-trained model building network to build the first virtual model; the method of calibrating the first virtual model according to the second virtual model of the organism to obtain the calibrated first virtual model comprises: using the coordinate data of multiple feature points on the same rigid in-vivo feature as the rigid in-vivo feature in the first virtual model in the second virtual model to calibrate the coordinate data of the corresponding multiple feature points in the first virtual model, so that the shape and size of the corresponding rigid in-vivo feature in the calibrated first virtual model are consistent with the shape and size of the rigid in-vivo feature of the second virtual model; or, inputting the second virtual model of the organism and the first virtual model into a pre-trained registration network to obtain the calibrated first virtual model output by the registration network; A mixed reality device is connected to the computer device and is used to determine the position and posture of the calibrated first virtual model in the real scene by using the image data of the body surface markers, so as to display the calibrated first virtual model and the real scene through the mixed reality device.

14. A mixed reality device, characterized in that: include: A sensor device for collecting image data of a living organism; A storage device for storing at least one program; A display device, used for integrating the virtual model with the real scene for display; A processing device connected to the sensor device, the storage device and the display device, and configured to call the at least one program from the storage device and execute the program to achieve: Based on the two-dimensional scanning data of the organism and the image data of the organism collected during the operation, a first virtual model including the surface features and the internal features of the organism is constructed; wherein the two-dimensional scanning data includes the surface marks of the target area of ​​the organism and the scanning data of the internal tissue, and the image data of the organism includes the image data of the surface of the target area of ​​the organism and the surface marks; the method of constructing the first virtual model including the surface features and the internal features of the organism based on the two-dimensional scanning data and the image data of the organism collected during the operation includes: inputting the two-dimensional scanning data of the organism and the collected image data of the organism into a pre-trained model construction network to construct the first virtual model; The first virtual model is calibrated according to the second virtual model of the organism to obtain a calibrated first virtual model; wherein the second virtual model is pre-constructed according to the preoperative CT data of the organism and the second virtual model includes the in-vivo features of the organism; the method of calibrating the first virtual model according to the second virtual model of the organism to obtain the calibrated first virtual model includes: using the coordinate data of multiple feature points on the second virtual model that belong to the same rigid in-vivo feature as the rigid in-vivo feature in the first virtual model to calibrate the coordinate data of the corresponding multiple feature points in the first virtual model, so that the shape and size of the corresponding rigid in-vivo features in the calibrated first virtual model are consistent with the shape and size of the rigid in-vivo features of the second virtual model; or, inputting the second virtual model of the organism and the first virtual model into a pre-trained registration network to obtain the calibrated first virtual model output by the registration network; The position and posture of the calibrated first virtual model in the real scene are determined by using the image data of the body surface markers, so that the calibrated first virtual model and the real scene are displayed through a mixed reality device.

Images