MR equipment and medical imaging systems

Abstract

The present application discloses an MR device and a medical imaging system, wherein the MR device includes a camera, a display, and a processing device, wherein the processing device is connected to the camera and the display, and is used to call and execute a medical imaging program based on the imported medical data to achieve: superimposed presentation of a menu window and a 3D stereo model of biological tissue in the current real scene of the display interface, wherein the 3D stereo model incorporates layer images of multiple anatomical planes; dynamically adjust the spatial posture of the 3D stereo model in the current real scene based on the user's interactive operation with the menu window and / or the 3D stereo model, and dynamically display any layer image of any anatomical plane.

Description

Technical Field

[0001] The present application relates to the technical field of medical data processing, and in particular to an MR device and a medical imaging system. Background Art

[0002] At present, in medical scenarios such as preoperative preparation and operation, doctors often use the MR device they wear to view the 3D stereo model (or 3D virtual model) of the surgical site (such as the lesion site), so that they can intuitively perceive the surgical site inside the patient that is not directly visible to the human eye. For example, during the operation, the 3D stereo model of the surgical site is provided to assist the doctor in performing surgery on the patient to improve the safety of the operation. However, since the MR device often only superimposes the 3D stereo model with the real scene, the user cannot interact with the 3D stereo model, which reduces the user experience.

[0003] Therefore, how to present 3D stereo models in MR devices and provide good interactive physical examinations is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the shortcomings of the above-mentioned related technologies, the purpose of the present application is to provide an MR device and a medical imaging system to overcome the technical problems existing in the above-mentioned related technologies of how to present a 3D stereoscopic model in an MR device and provide good interactive physical examination.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect disclosed in the present application discloses an MR device, comprising: a camera device, for collecting current environmental data; a display device, for presenting in real time in its display interface a current real scene corresponding to the current environmental data; a processing device, connected to the camera device and the display device, for calling and executing a medical imaging program based on the imported medical data to achieve: superimposing a menu window and a 3D stereo model of biological tissue in the current real scene of the display interface, wherein the 3D stereo model incorporates layer images of multiple anatomical planes; dynamically adjusting the spatial posture of the 3D stereo model in the current real scene based on the user's interactive operations with the menu window and / or the 3D stereo model, and dynamically displaying any layer image of any anatomical plane.

[0006] In some examples of the first aspect, the processing device is also used to switch the 3D stereoscopic model to a first interaction mode or a second interaction mode based on a user's interaction operation, wherein the first interaction mode corresponds to the interaction operation for dynamically adjusting the spatial posture, and the second interaction mode corresponds to the interaction operation for dynamically displaying any layer image of any anatomical plane.

[0007] In some examples of the first aspect, the first interaction mode is configured such that the 3D stereoscopic model is presented in a manner of being surrounded by a bounding box, and the processing device hides the bounding box based on a user's interaction operation on the bounding box to switch to the second interaction mode.

[0008] In some examples of the first aspect, hiding the bounding box based on the user's interactive operation on the bounding box to switch to the second interaction mode includes: hiding the bounding box when detecting a gaze of the user on the bounding box and a confirmation operation on the bounding box.

[0009] In some examples of the first aspect, orientation information is marked on the bounding box or outside the bounding box to give a user an orientation prompt.

[0010] In some examples of the first aspect, the interactive operation of dynamically adjusting the spatial posture of the 3D stereoscopic model in the current real scene includes a drag operation and a rotation operation.

[0011] In some examples of the first aspect, the dynamically adjusting the spatial posture of the 3D stereo model in the current real scene includes: identifying a platform object in the current real scene to present the 3D stereo model as being placed on the platform object when it is detected that the user drags the 3D stereo model onto the platform object based on the interactive operation.

[0012] In some examples of the first aspect, the processing device is further used to present a model display option in the menu window to hide or display the entirety or a sub-organization of the 3D stereoscopic model based on a user's interactive operation of the model display option.

[0013] In some examples of the first aspect, the model display options include an overall display option, an overall hiding option, and a sub-organization display option, and the processing device displays the entire 3D stereo model based on the user's interactive operation on the overall display option, hides the entire 3D stereo model based on the user's interactive operation on the overall hiding option, and displays or hides the corresponding sub-organizations in the 3D stereo model based on the user's interactive operation on the sub-organization display option.

[0014] In some examples of the first aspect, the plurality of anatomical planes include a transverse plane, a coronal plane, and a sagittal plane.

[0015] In some examples of the first aspect, the processing device displays a corresponding layer image based on a user's dragging operation on any anatomical plane.

[0016] In some examples of the first aspect, the processing device is also used to present each anatomical plane option and its layer option in the menu window, so as to display the corresponding layer image of the corresponding anatomical plane based on the user's interactive operation of each anatomical plane option and its layer option, and to hide or display the corresponding layer image of the corresponding anatomical plane presented on the 3D stereoscopic model.

[0017] In some examples of the first aspect, the processing device presents a grayscale adjustment option in the menu window to adjust the contrast and / or brightness of the displayed layer image based on a user's interactive operation of the grayscale adjustment option.

[0018] In some examples of the first aspect, the processing device presents a region selection item in the menu window to display the selected region in the layer image based on a user's interactive operation on the region selection item.

[0019] In some examples of the first aspect, the processing device presents a color marking option in the menu window to present the sub-tissue in the displayed layer image in a color rendering manner based on the user's interactive operation on the color marking option.

[0020] In some examples of the first aspect, the color marking option is an overall marking option, and the overall marking plus option corresponds to an interactive operation of performing an overall color rendering display on all sub-tissues in the displayed layer image.

[0021] In some examples of the first aspect, different sub-tissues in the 3D stereoscopic model are presented in different colors.

[0022] In some examples of the first aspect, the MR device is configured as an Apple Vision head-mounted display device.

[0023] The second aspect of the present application discloses a medical imaging system, comprising an MR device as described in any example of the first aspect and a field of view sharing device communicatively connected to the MR device, wherein the field of view sharing device is used to synchronously display an interface displayed by a display device in the MR device.

[0024] In summary, the present application discloses an MR device and a medical imaging system, which superimposes a 3D stereo model (with layer images of multiple anatomical planes fused on the 3D stereo model) and a menu window on the current real scene of the display interface, and dynamically adjusts the spatial posture of the 3D stereo model in the current real scene based on the user's interactive operation with the menu window and / or the 3D stereo model, and dynamically displays any layer image of any anatomical plane. In this way, the user can not only change the spatial posture of the 3D stereo model to display the 3D stereo model in its desired posture, but also adjust the positions of the anatomical planes fused in the 3D stereo model to display the layer images at the corresponding positions, thereby providing the user with a good interactive experience while presenting rich imaging information. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The specific features of the invention involved in this application are shown in the attached claims. The features and advantages of the invention involved in this application can be better understood by referring to the exemplary embodiments and drawings described in detail below. The drawings are briefly described as follows:

[0026] Figure 1 Shown is a structural schematic diagram of an MR device in one embodiment of the present application.

[0027] Figure 2 Shown is a schematic diagram of an interface in which a 3D stereoscopic model in an embodiment of the present application is presented in a first interaction mode.

[0028] Figure 3 In one embodiment, the present application shows Figure 2 Schematic diagram of the interface after the spatial posture of the 3D stereo model is adjusted.

[0029] Figure 4 In one embodiment, the present application will Figure 3 A schematic diagram of an interface in which a 3D stereoscopic model presented in a first mode is switched to one presented in a second mode.

[0030] Figure 5 Shown is a schematic diagram of an interface in which a coronal layer image is fused onto a 3D stereoscopic model in one embodiment of the present application.

[0031] Figure 6 In one embodiment, the present application shows Figure 5 Schematic diagram of the interface after hiding the coronal layer image presented in FIG.

[0032] Figure 7 In one embodiment, the present application shows Figure 4 Schematic diagram of the interface showing layer images of multiple anatomical planes after the position of the coronal plane is adjusted.

[0033] Figure 8Shown is a schematic diagram of an interface presenting area selection items in one embodiment of the present application.

[0034] Fig. 9 In one embodiment, the present application will Figure 6 Schematic diagram of the 3D model after it is hidden.

[0035] Fig.10 The application is shown in one embodiment. Fig. 9 Schematic representation of the sub-organizations hidden in the 3D stereo model.

[0036] Fig.11 In one embodiment, the present application will Figure 6 Schematic diagram of the interface where a sub-organization is dragged out.

[0037] Fig.12 In one embodiment, the present application shows Figure 7 Schematic diagram of the interface for overall rendering of all sub-organizations of the layer image shown.

[0038] Fig.13 Shown is a schematic diagram of the structure of a medical imaging system in one embodiment of the present application. DETAILED DESCRIPTION

[0039] The following is a description of the embodiments of the present application by specific specific embodiments, and those familiar with the technology can easily understand the advantages of the present application and the technical effects that can be achieved by the content disclosed in this specification. In the following description, some embodiments can refer to the drawings pointed to. It should be understood that other embodiments without drawings can also be used, and specific steps, modules or units, electrical and operational changes can be made without departing from the spirit and scope of the present application. The detailed description below should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims published by the present application. The terms used herein are only for describing specific embodiments and are not intended to limit the present application.

[0040] Although in some instances the terms first, second or third, etc. are used in this document to describe various elements or parameters, these elements or parameters should not be limited by these terms. These terms are only used to distinguish one object or parameter from another object or parameter, and are not used to define the order, timing, priority or importance of multiple objects. For example, the first interaction mode may be referred to as the second interaction mode, and similarly, the second interaction mode may be referred to as the first interaction mode, without departing from the scope of the various described embodiments, the first interaction mode and the second interaction mode are both describing an interaction mode of the 3D stereo model, but unless the context clearly indicates otherwise, they are not the same interaction mode.

[0041] Furthermore, as used in this article, the singular forms "one", "an" and "the" are intended to also include plural forms, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprising", "including" indicate the presence of the described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. For example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, the term "and / or" that may be used hereinafter, describes the association relationship of associated objects, indicating that three kinds of relationships may exist, for example, A and / or B, which can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / ", if not otherwise specified, generally represents that the associated objects before and after are a kind of "and / or" relationship. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0042] Some nouns or terms used in each embodiment of the present application are explained below, and the nouns or terms are also used as a part of the content of the invention. It will be appreciated by those skilled in the art that, unless otherwise defined, all terms including technical terms and scientific terms used herein have the same meaning as the general understanding of the technicians in the field to which the present application belongs. It should also be understood that those terms such as those defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and unless specifically defined as herein, will not be interpreted with an idealized or overly formal meaning.

[0043] The medical image data described in the embodiments of the present application are medical image data of a living body, which are obtained by scanning the living body using a medical scanning device (e.g., a CT device, an MRI device, or an X-ray device, etc.), and include pixel coordinates (or coordinate data) and pixel values. For example, the medical image data are medical image data such as CT data, X-ray data, MRI data, and ultrasound data of the living body. Among them, the living body is exemplified as a human body or an animal body, and the medical image data is 3D image data. In one embodiment, the medical image data can be stored in a DICOM format. A 3D stereoscopic model of a biological tissue can be established based on the medical image data. For example, a 3D stereoscopic model of a human cervical spine can be established based on medical image data including the whole body or a part of a human body.

[0044] The layer image described in the embodiments of the present application is a two-dimensional medical image (also referred to as a tomographic image), which is a visualized image of medical image data or the processed medical image data described later. For example, arbitrary layer images of multiple anatomical planes of an organism can be generated based on the medical image data, and the multiple anatomical planes include a cross-section (or horizontal plane), a coronal plane, and a sagittal plane. Different pixel values ​​in the layer image generally represent different tissues in the organism. For example, the pixel value unit in the layer image is HU, the pixel value corresponding to fat is approximately -100HU to -50HU, and the pixel value corresponding to bone is +700HU to 3000HU.

[0045] The biological tissues described in the embodiments of the present application include components of the organism such as organs and systems in the organism. In the following embodiments, the organism is taken as an example of a human body, and the biological tissues include but are not limited to nerves, blood vessels, organs, bones, and / or muscles of the human body. In one example, the biological tissue includes the thoracic vertebrae of the human body, and further, the biological tissue may also include organs in the chest or abdomen near the thoracic vertebrae. In the following embodiments, the subtissue of the biological tissue is a partial tissue in the biological tissue, for example, the biological tissue is the thoracic vertebrae, and the subtissue of the thoracic vertebrae is the individual vertebrae constituting the thoracic vertebrae.

[0046] In an embodiment of the present application, a spatial posture (also referred to as a posture) includes a position and an attitude. The position is a coordinate position in a world coordinate system. The attitude is three rotation angles (pitch, yaw, roll) corresponding to three coordinate axes (x, y, z) in the world coordinate system. In one embodiment, the world coordinate system is determined based on the MR device. In one example, the world coordinate system is determined based on the coordinate system of the camera device. For example, the origin and coordinate axes of the coordinate system of the camera device when the MR device is turned on are used as the origin and coordinate axes of the world coordinate system. In the above embodiments, the description of the world coordinate system is limited to examples.

[0047] In an embodiment of the present application, the interactive operation may include operations such as dragging operations and rotating operations for adjusting the posture of virtual objects, and operations for triggering virtual objects, etc. Among them, the interactive operation can be implemented using one or more of recognizable gestures, recognizable voices, and recognizable eye movement states. The recognizable gestures may include finger clicks (such as single clicks or double clicks, etc.), taps, knocks, pushes, guides, pinches (such as multiple pinches, pinches while moving, pinches while rotating, pinches with both hands, etc.), flicks, turns, rotates, grabs, pulls, drags, palms open, draws (such as finger painting), forms shapes with fingers (such as "OK" gestures), and swipes, etc. The recognizable gestures can be recognizable at the apparent position of the virtual object, in the vicinity of the apparent position, in the direction of the apparent position (relative to the user), or in the user's field of vision. The recognizable voices include voices about display content (such as voices of "display selection menu window"), and voices about confirmation or selection, etc. The recognizable eye movement states include gazes, continuous blinking, etc. In the following embodiments, interactive operations for implementing different functions are collectively referred to as interactive operations, and those skilled in the art may select interactive operations for implementing different functions based on the description of the following embodiments.

[0048] As described in the background technology, in the prior art, users (such as doctors) cannot interact with the 3D stereoscopic model displayed by the MR device, which reduces the user experience. Taking the medical scene of preoperative preparation as an example, in the related technology, the position of the displayed 3D stereoscopic model can usually be changed based on the user's rotation operation and / or translation operation on the displayed 3D stereoscopic model to achieve interaction between the user and the 3D stereoscopic model.

[0049] However, since the reconstructed 3D stereoscopic model often cannot observe the internal conditions of biological tissues, and does not include biological tissues other than the surgical site that the user also needs to pay attention to, such as organs, in order to view the relevant information of biological tissues other than the surgical site, the user needs to use the MR device or remove the MR device to view the medical images in the medical imaging film or the medical images displayed on the display screen of the computer device in the real scene, and the user needs to additionally associate the medical images with the 3D stereoscopic model in the brain. This is inconvenient for the user to perform surgery or preoperative preparations, and reduces the user experience.

[0050] The present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. The technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments and technical effects obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of this application. "An implementation method", "implementation method" or similar words mentioned in the whole text of this specification mean that the specific features, structures or characteristics described together with the implementation method are included in at least one implementation method of the present application. Therefore, in the whole text of this specification, the appearance of the phrases "in one implementation method", "in an implementation method" and similar words may (but not necessarily) relate to the same implementation method.

[0051] In some embodiments, the present application provides an MR device and a medical imaging system, which superimposes a 3D stereo model (with layer images of multiple anatomical planes fused on the 3D stereo model) and a menu window on the current real scene of the display interface, and dynamically adjusts the spatial posture of the 3D stereo model in the current real scene based on the user's interactive operation with the menu window and / or the 3D stereo model, and dynamically displays any layer image of any anatomical plane. In this way, the user can not only change the spatial posture of the 3D stereo model to make the 3D stereo model displayed in the user's desired posture, but also adjust the positions of the anatomical planes fused in the 3D stereo model to display the layer images of the anatomical planes at the corresponding positions, thereby providing the user with a good interactive experience while presenting rich imaging information, so as to facilitate the user's surgery and preoperative communication, etc.

[0052] In one embodiment, the MR (Mixed Reality Glasses) device can overlay / superimpose virtual objects (such as the 3D stereoscopic model, layer images of anatomical planes, menu windows, import windows, etc. described below) on the real scene for display. In one example, the MR device can present the real scene and the virtual objects superimposed on the real scene through a display interface, so that the virtual objects viewed by the user through the MR device appear to be in the real scene.

[0053] In some embodiments, the MR device is a head-mounted MR device (eg, MR glasses or a MR helmet). For example, the MR device is configured as a head-mounted display device Apple Vision.

[0054] In one embodiment, see Figure 1 , which is a schematic diagram of the structure of an MR device in one embodiment of the present application. As shown in the figure, the MR device 1 includes an imaging device 10, a display device 11, and a processing device 12.

[0055] The camera device 10 is used to collect current environment data. In some embodiments, the camera device 10 includes at least one environment perception sensor, and correspondingly, the current environment data is the sensor data (such as image data) obtained by the environment perception sensor from detecting the current environment. In the embodiment where the camera device 10 includes multiple environment perception sensors, the multiple environment perception sensors can all be sensors of the same type or sensors of different types.

[0056] In one embodiment, the camera device 10 includes a visual sensor (such as a binocular camera, etc.), which is used to provide image data to the processing device 12, and then the processing device 12 can analyze the image data to estimate the position of the object in the image data relative to the MR device 1 by using triangulation analysis. Further, the camera device 10 may also include a depth sensor, which is used to provide the depth data of the object to the processing device 12, and then the processing device 12 can determine the distance of the object relative to the MR device 1 according to the depth data.

[0057] In one embodiment, in order to sense the motion state of the MR device 1 itself to ensure accurate superposition of virtual objects with real scenes, the MR device 1 also includes motion sensors, such as electronic compasses, accelerometers, gyroscopes, magnetic sensors, mechanical or electronic level sensors, and inertial sensors, etc. These sensors alone or in combination can provide the processing device 12 of the MR device 1 with motion data including the up / down / horizontal orientation and displacement of the MR device 1, thereby providing the user's head position / orientation.

[0058] The display device 11 is used to present the current real scene corresponding to the current environment data in real time in its display interface. For example, the current real scene presented in the display interface is an image or video of the current real scene.

[0059] In one embodiment, the processing device 12 determines the display parameters of the display device 11 for displaying images based on the current environment data, for example, the display parameters are the brightness, color and contrast of the display unit (such as a liquid crystal unit, an LED unit, etc.), and then the display device 11 presents the current real scene in its display interface based on the display parameters. In other examples, when an image processor is built in the display device 11, the display device 11 can also use the current environment data to determine the display parameters for presenting the current real scene in the display interface.

[0060] In one embodiment, the display device 11 is a display device based on video synthesis technology. Specifically, the processing device 12 superimposes the virtual object with the current environment data, and controls the display device 11 to display the superimposed image to the user, so that the user can see both the current real scene and the virtual object in the display interface. The display device 11 is exemplified as a display.

[0061] In one embodiment, please continue to refer to Figure 1 As shown in the figure, the MR device 1 also includes a storage device 13. The storage device 13 stores at least one program for the processing device 12 to call and execute, such as the medical imaging program described later. Furthermore, the storage device 13 also stores current environmental data, virtual objects, etc. Here, the storage device 13 includes but is not limited to: read-only memory, random access memory, non-volatile memory. For example, the storage device 13 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 13 may also include a memory away from one or more processing devices 12, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or a suitable combination thereof. The memory controller may control the access to the memory by other components of the device, such as the CPU and peripheral interfaces. In some other embodiments, the storage device 13 may not be a part of the MR device 1, but may be configured as a storage device on the cloud, server, or other device. The medical imaging program and the current environmental data are configured in the storage device, and may be communicated with, for example, by means of remote communication. Figure 1 The processing device 12 of the MR device 1 described in the embodiment transmits the stored data. In the following embodiments, the medical imaging program and the current environment data are stored in the storage device 13 included in the MR device 1 as an example for explanation.

[0062] In one embodiment, the processing device 12 is connected to the camera device 10 and the display device 11, and is used to implement step S110 and step S120 based on the imported medical data when calling and executing the medical imaging program. For example, the processing device 12 calls the medical imaging program from the storage device 13, and implements step S110 and step S120 based on the imported medical data.

[0063] In one embodiment, an import window is superimposed and presented in the real scene in the display interface of the display device 11 for importing medical data. In one example, the processing device 12 uses a spatial position at a preset distance in front of the user and at a preset height from the ground as the superimposed position of the import window, so as to superimpose and display the import window in the display interface based on the superimposed position of the import window, so that the import window seen by the user through the display interface looks like a position at a preset height from the ground in front of the user. In other examples, the processing device 12 superimposes and presents the import window in the current real scene of the display interface based on the user's interactive operation for indicating the display position of the import window. For example, the interactive operation is a recognizable gesture, and the processing device 12 uses the area where the recognizable gesture displayed by the user is located as the superimposed position of the import window, so as to superimpose and display the import window in the display interface based on the superimposed position of the import window, so that the import window seen by the user through the display interface looks like it is at the area where the recognizable gesture is located. It should be noted that after the import window is presented, when the user's field of view changes, its superimposed position remains unchanged, and the processing device can change the superimposed position of the import window based on the interaction between the user and the import window.

[0064] Further, the processing device 12 imports medical data based on the interaction between the user and the import window. For example, the import window includes an import area, and the user points to or clicks the import area with a finger to view the file packages corresponding to multiple medical data. After the user points to or clicks the file package to be displayed and points to or clicks the import area again, the import of medical data can be completed. The imported medical data can be stored in the storage device 13 or in a remote storage device (such as a field of view sharing device or server connected to the MR device 1).

[0065] The imported medical data is generated by a computer device (e.g., a server connected to the MR device 1 in communication, or a computer device connected to the server of the MR device 1 in communication, or a workbench connected to the MR device 1 in communication, etc.) based on medical image data. In one embodiment, the medical data includes the medical image data and model data of a 3D stereoscopic model of a biological tissue obtained based on the medical image data. The model data of the 3D stereoscopic model of the biological tissue can be generated directly using the medical image data or can be generated using the processed medical image data described later. Taking the example that the medical image data is whole-body CT data and the biological tissue is cervical vertebrae, the medical data includes whole-body CT data and model data of a 3D stereoscopic model of the cervical vertebrae obtained based on the whole-body CT data.

[0066] In another embodiment, the medical data also includes processed medical image data obtained by processing the medical image data. Specifically, the computer device performs image recognition on the medical image data in advance to identify the biological tissue in the medical image data, and uses the medical image data marked with the biological tissue as the processed medical image data. In one example, the biological tissue can be identified by color. In other words, the pixel value corresponding to the biological tissue part in the processed medical image data is color data. For example, the biological tissue is the heart, and the pixel value corresponding to the central part of the medical image data is processed into color data as the processed medical image data.

[0067] In one embodiment, the computer device may also perform image recognition on the medical image data in advance to identify multiple sub-tissues of biological tissues in the medical image data, and use the medical image data with the sub-tissues of biological tissues marked as processed medical image data. In one example, different sub-tissues of biological tissues may be marked with different colors. In other words, the pixel values ​​corresponding to different sub-tissues of biological tissues in the processed medical image data are different color data. For example, the medical image data is whole-body CT data, and the biological tissue is cervical vertebrae. The computer device processes the pixel values ​​corresponding to multiple vertebrae of the cervical vertebrae in the whole-body medical image data into different color data and uses them as processed medical image data. In this embodiment, after the data of different sub-tissues in the processed medical image data are processed into different color data, different sub-tissues in the layer image presented in the display interface in the following embodiments can be presented in different colors.

[0068] In an embodiment in which the model data of the 3D stereoscopic model of the biological tissue is generated using processed medical imaging data, different sub-tissues in the model data of the 3D stereoscopic model are marked with different colors, so that different sub-tissues in the 3D stereoscopic model of the biological tissue presented in the display interface in the subsequent embodiments are presented in different colors.

[0069] In one embodiment, the coordinate data in the model data of the 3D stereo model matches the coordinate data in the medical image data and / or the processed medical image data. In one example, the coordinate data in the model data of the 3D stereo model and the coordinate data in the medical image data and / or the processed medical image data are in the same coordinate system, so that the processing device 12 can directly fuse the model data of the 3D stereo model with the medical image data. In other examples, the coordinate data of the 3D stereo model and the coordinate data in the medical image data may also have a coordinate conversion relationship, and the coordinate conversion relationship is pre-stored in the medical data, so that the processing device can fuse the model data of the 3D stereo model with the medical image data.

[0070] In a specific embodiment, the computer device selects multiple points in the model data as matching points, and determines the coordinate transformation relationship between the model data and the processed medical image data (i.e., the coordinate transformation relationship between the model data and the medical image data) based on the coordinate data of the multiple matching points and the coordinate data of the points in the processed medical image data corresponding to the multiple matching points. Then, the processing device can use the coordinate transformation relationship to unify the coordinate data in the model data of the 3D stereo model, the medical image data, and the coordinate data in the processed medical image data in the same coordinate system and use the same coordinate system as the coordinate system of the medical data, or directly store the coordinate transformation relationship in the medical data. It should be noted that in other embodiments, when matching, the coordinate transformation relationship can also be determined using medical image data.

[0071] The processing device 12 may implement step S110 based on the imported medical data.

[0072] In step S110, the processing device 12 superimposes and presents a menu window and a 3D stereoscopic model of the biological tissue in the current real scene of the display interface, wherein the 3D stereoscopic model incorporates layer images of multiple anatomical planes.

[0073] The fusion refers to displaying the layer images of multiple anatomical planes and the 3D stereoscopic model in a unified coordinate system, so that the two-dimensional biological tissue in the layer image of the anatomical plane at a position is located in the 3D stereoscopic model, that is, the two-dimensional biological tissue in the layer image of the anatomical plane at a position appears to be hidden in the 3D stereoscopic model. For example, the 3D stereoscopic model of the thoracic spine is fused with layer images, and the two-dimensional thoracic spine in the layer image appears to be hidden in the 3D stereoscopic model of the thoracic spine. In one example, the acquired medical data is data pre-unified in the same coordinate system, and the processing device can directly display the 3D stereoscopic model and the layer images of multiple anatomical planes in the coordinate system of the medical data. In another example, the acquired medical data is data with a coordinate transformation relationship, and the processing device can display the 3D stereoscopic model and the layer images of multiple anatomical planes in a unified coordinate system based on the coordinate transformation relationship.

[0074] In an embodiment in which the menu window and the 3D stereoscopic model of the biological tissue are displayed for the first time after the medical data is imported, the processing device 12 can present the menu window and the 3D stereoscopic model of the biological tissue in the current real scene of the display interface after acquiring the medical data. In one embodiment, the superimposed position of the import window is used as the superimposed position of the menu window, so that the processing device 12 can superimpose and display the menu window in the display interface, so that the menu window seen by the user through the display interface looks like it is in the same position as the import window displayed previously. The superimposed position of the 3D stereoscopic model of the biological tissue for the first time is configured to be located near the superimposed position of the menu window, so that the 3D stereoscopic model of the biological tissue seen by the user through the display interface looks like it is near the menu window. For example, the superimposed position for the first time display is located in front of or on the left and right sides of the superimposed position of the menu window. In another embodiment, the processing device 12 may also utilize the user's interactive operation to determine the overlay position of the menu window and configure the overlay position of the initial display of the 3D stereoscopic model of the biological tissue to be located near the overlay position of the menu window, wherein the method of utilizing the user's interactive operation to determine the overlay position of the menu window is the same or similar to the method of utilizing the interactive operation to determine the overlay position of the import window as described above, and will not be repeated here.

[0075] In one embodiment, the 3D stereoscopic model of the biological tissue is fused with layer images of multiple anatomical planes, wherein the multiple anatomical planes include a transverse section, a coronal plane, and a sagittal plane. For example, the 3D stereoscopic model is fused with layer images of a transverse section, a coronal plane, and a sagittal plane. In an embodiment of the present application, the layer images of the multiple anatomical planes are fused into the 3D stereoscopic model, so when the spatial posture of the 3D stereoscopic model is adjusted, the multiple layer images will also be adjusted accordingly. It should be noted that the biological tissue included in the layer image may be the same as the biological tissue included in the medical image data or the processed medical image data, or may be less than the biological tissue included in the medical image data or the processed medical image data.

[0076] In one embodiment, different sub-tissues in the model data of the 3D stereoscopic model are marked with different colors, and accordingly, different sub-tissues in the presented 3D stereoscopic model are presented in different colors.

[0077] In one embodiment, in order to facilitate the overall adjustment of the spatial posture of the 3D stereoscopic model of the biological tissue displayed for the first time so that it can be displayed in the spatial posture expected by the user, the 3D stereoscopic model of the biological tissue displayed for the first time is presented in a first interactive mode. The first interactive mode is configured such that the 3D stereoscopic model is presented in a manner surrounded by a bounding box, and the bounding box is configured to be sufficient to surround the 3D stereoscopic model and the layer images fused thereon, and the 3D stereoscopic model presented in the first interactive mode can be moved as a whole to adjust the spatial posture as a whole. Please refer to Figure 2 , which is a schematic diagram of an interface in which a 3D stereoscopic model in an embodiment of the present application is presented in a first interactive mode. As shown in the figure, the processing device 12 displays the operating room scene in the current real scene of the display interface, and superimposes a menu window 2 and a 3D stereoscopic model 3 of a biological tissue in the current real scene, wherein the 3D stereoscopic model 3 of the biological tissue is fused with layer images of multiple anatomical planes (layer images 4a of the cross section, layer images 4b of the coronal plane, and layer images 4c of the sagittal plane), and the 3D stereoscopic model 3 of the biological tissue and the fused layer images (4a, 4b, 4c) can be surrounded by a boundary box 5. In other embodiments, the 3D stereoscopic model of the biological tissue displayed for the first time can also be presented in the second interactive model described later.

[0078] In one embodiment, in order to facilitate the user to dynamically adjust the spatial position of the 3D stereoscopic model, the bounding box or the outer side of the bounding box is marked with orientation information to give the user an orientation prompt. The orientation information corresponds to the orientation of the 3D stereoscopic model, in other words, the orientation information corresponds to the orientation of the real biological body. For example, Figure 2 As shown, the outside of the boundary box is marked with six orientation information: up, down, left, right, front, and back (located at the back side of the 3D stereo model), which correspond to the upper side, lower side, left side, right side, front side, and back side of the organism respectively.

[0079] In step S120, the processing device 12 dynamically adjusts the spatial position of the 3D stereo model in the current real scene based on the user's interactive operation with the menu window and / or the 3D stereo model, and dynamically displays any layer image of any anatomical plane.

[0080] In one embodiment, the processing device dynamically adjusts the spatial position of the 3D stereo model in the current real scene based on the interactive operation between the user and the 3D stereo model. The interactive operation for dynamically adjusting the spatial position of the 3D stereo model in the current real scene includes a drag operation and a rotation operation. The drag operation is used to dynamically adjust the position of the 3D stereo model in the current real scene, and then the processing device 12 will change the position of the 3D stereo model in the current real scene based on the drag direction and drag distance of the drag operation. The rotation operation is used to dynamically adjust the posture of the 3D stereo model in the current real scene, and then the processing device 12 will change the posture of the 3D stereo model in the current real scene based on the rotation direction and rotation angle corresponding to the rotation operation. In one example, the rotation operation is a recognizable gesture of both hands (such as a pinch gesture), and the two recognizable gestures (such as a pinch gesture) move relative to each other, wherein the processing device controls the 3D stereo model to rotate around the center of the 3D stereo model, and the processing device determines the rotation direction of the 3D stereo model around its center based on the direction of change of the line connecting the two hands in the recognizable gesture, and determines the rotation angle of the 3D stereo model around its center based on the change size of the line connecting the two hands in the recognizable gesture. For example, the processing device uses the initial line connecting the two hands in the recognizable gesture as a reference for calculating the rotation direction and rotation angle, and subsequently calculates the rotation direction and rotation angle of the 3D stereo model based on the change of the line connecting the two hands in the recognizable gesture and the initial line. The present application does not limit the interactive operation of controlling the posture of the 3D stereo model. In other embodiments, the processing device can also control the rotation of the 3D stereo model around its center based on the finger's sliding operation on the 3D stereo model.

[0081] See also Figure 2 and Figure 3 , Figure 3 In one embodiment, the present application shows Figure 2 The interface diagram after adjusting the spatial posture of the 3D stereo model shown in FIG. Figure 2 The 3D stereoscopic model 3 shown in the figure performs a drag operation corresponding to the left side and the upper side of the 3D stereoscopic model 3 and a rotation operation corresponding to the 3D stereoscopic model 3 rotating from the right side to the left side thereof. The processing device performs a drag operation and a rotation operation on the 3D stereoscopic model 3 based on the user's drag operation and the rotation operation. Figure 2 The spatial pose of the 3D stereo model shown is adjusted to Figure 3 The spatial pose shown, where Figure 3 The 3D stereo model 3 is Figure 2 The 3D stereoscopic model 3 in FIG. 1 is moved to the lower left and rotated from the right side to the left side around its own center.

[0082] In another embodiment, the processing device 12 dynamically adjusts the spatial position of the 3D stereo model in the current real scene based on the user's interaction with the menu window, wherein the menu window includes a position adjustment option, and the processing device 12 dynamically adjusts the 3D stereo model based on the user's interaction with the position adjustment option. In one example, the position adjustment option includes a position adjustment option and a posture adjustment option, wherein the position adjustment option includes an upward movement distance option, a downward movement distance option, a forward movement distance option, a backward movement distance option, a left movement distance option, and a right movement distance option; and the posture adjustment option includes a rotation direction option and a rotation angle option.

[0083] In one embodiment, the processing device 12 identifies the platform object in the current real scene so that when it is detected that the user drags the 3D stereo model onto the platform object based on the interactive operation, the 3D stereo model is presented as being placed on the platform object. Specifically, the processing device 12 identifies the platform object in the real scene based on the current environmental data collected by the camera device, and then dynamically changes the spatial posture of the 3D stereo model based on the user's interactive operation, and drags the 3D stereo model onto the platform object. When the 3D stereo model is dragged onto the platform object, the 3D stereo model is presented as being placed on the platform object. In other words, when the 3D stereo model is dragged onto the platform object, the user's interactive operation corresponds to the 3D stereo model passing through the platform object, and the processing device will not continue to adjust the superimposed position of the 3D stereo model downward, so that when the 3D stereo model is dragged onto the platform object, the 3D stereo model is always presented as being placed on the platform object. Among them, the platform object is exemplified by a desktop, a ground, and a wall.

[0084] In some embodiments, the processing device 12 is also used to switch the 3D stereo model to be presented in the first interactive mode or the second interactive mode based on the user's interactive operation. Specifically, the processing device 12 can switch the 3D stereo model presented in the first interactive mode to the 3D stereo model presented in the second interactive mode based on the user's interactive operation for mode switching, and can also switch the 3D stereo model presented in the second interactive mode to the 3D stereo model presented in the first interactive mode. The second interactive mode corresponds to the interactive operation for dynamically displaying any layer of images of any anatomical plane. Specifically, the bounding box of the 3D stereo model disappears in the second interactive mode, and then the processing device 12 can dynamically display any layer of images of any anatomical plane based on the user's interactive operation.

[0085] In a specific embodiment, the processing device 12 hides the bounding box based on the user's interactive operation on the bounding box to switch to the second interaction mode. Specifically, when the 3D stereo model is presented in the first interaction mode, for example, after the processing device 12 dynamically adjusts the spatial posture of the 3D stereo model presented in the first interaction mode according to the description in the above embodiment and adjusts the 3D stereo model to a desired spatial posture, the processing device 12 hides the bounding box based on the user's interactive operation on the bounding box to switch to the second interaction mode. In one example, the bounding box is hidden when the user's gaze on the bounding box and the confirmation operation on the bounding box are detected. For example, the confirmation operation is two consecutive pinch gestures, but it is not limited to this. In other examples, it can also be other recognizable gestures, or recognizable voices, etc.

[0086] See also Figure 3 and Figure 4 , Figure 4 In one embodiment, the present application will Figure 3 A schematic diagram of an interface in which a 3D stereoscopic model presented in a first mode is switched to a 3D stereoscopic model presented in a second mode. As shown in the figure, when the user's gaze is detected Figure 3 The bounding box 5 in the embodiment is hidden when the bounding box 5 is pinched twice to present Figure 4 The 3D solid model 3 shown is in a state where there is no outer bounding box.

[0087] In another specific embodiment, after the first interaction mode is switched to the second interaction mode, when the user needs to continue to move the 3D stereo model, the presentation mode of the 3D stereo model can also be switched from the second interaction mode to the first interaction mode, so as to continue to dynamically adjust the spatial position of the 3D stereo model. In one example, when the user's gaze on the 3D stereo model without a bounding box and the confirmation operation on the 3D stereo model are detected, the bounding box is displayed. The confirmation operation in this embodiment is the same or similar to the confirmation operation in the previous embodiment, and will not be repeated here.

[0088] It should be noted that, although the above-mentioned embodiment is described by taking the 3D stereoscopic model not being hidden as an example, in the embodiment where the 3D model is hidden, the processing device may also switch the 3D stereoscopic model to be presented in the first interaction mode or the second interaction mode based on the user's interactive operation. In the present embodiment, when the hidden 3D stereoscopic model is presented in the first interaction mode, only the fused layer image in the 3D stereoscopic model is displayed surrounded by a bounding box in the display interface.

[0089] In some embodiments, the processing device 12 dynamically displays any layer image of any anatomical plane based on the user's interactive operation with the menu window and / or the 3D stereoscopic model. Specifically, the processing device 12 can dynamically display layer images corresponding to anatomical planes at different positions based on the user's interactive operation. The cross-section is a horizontal plane in the front-back, left-right and right-hand directions of the 3D stereoscopic model, the coronal plane is a horizontal plane in the up-down, left-right and right-hand directions of the 3D stereoscopic model, and the sagittal plane is a horizontal plane in the up-down, front-back and right-hand directions of the 3D stereoscopic model. Please refer to Figure 4 , Figure 4 In the image 4a of the cross section, the processing device can dynamically display the cross section at different positions based on the user's interactive operation. For example, the cross section of the upper body of the organism at different heights can be dynamically displayed.

[0090] In one embodiment, the processing device 12 presents each anatomical plane option and its layer option in the menu window. Figure 4 As shown, the menu window 2 presents a cross-section option displayed with the text "cross-section", a coronal option displayed with the text "coronal plane", and a sagittal option displayed with the text "sagittal plane", and correspondingly displays layer options represented by plus and minus signs, and the numerical value displayed on the rear side of the anatomical plane corresponds to the position of the anatomical plane. Among them, the displayed numerical values ​​are exemplified by the positions of the cross-section, coronal, and sagittal planes in the coordinate system of the 3D stereoscopic model. Furthermore, the processing device 12 can display the corresponding layer images of the corresponding anatomical planes based on the user's interactive operations on each anatomical plane option and its layer options, and hide or display the corresponding layer images of the corresponding anatomical planes presented on the 3D stereoscopic model. Among them, the processing device can hide or display the corresponding layer images of the corresponding anatomical planes on the 3D stereoscopic model based on the user's interactive operations on each anatomical plane option, and the processing device can adjust the position of the anatomical plane based on the user's interactive operations on the layer options to display the layer images of the corresponding position, and specifically, the position of the anatomical plane can be dynamically adjusted to dynamically display the layer images of the anatomical planes at the corresponding positions. In one example, the interactive operations on each anatomical plane option and its layer option are exemplified by recognizable gestures such as clicking a corresponding option with a finger, pointing a finger at a corresponding option, and the like.

[0091] See also Figure 5 and Figure 6 , Figure 5 The interface diagram showing a 3D stereoscopic model fused with a coronal layer image in one embodiment of the present application is shown. Figure 6 In one embodiment, the present application shows Figure 5 The schematic diagram of the interface after the coronal layer image presented in the figure is hidden, as shown in the figure, the processing device Figure 5The interactive operation of the Chinese character "coronal plane" hides the layer image of the coronal plane in the 3D stereo model. Figure 6 Only the 3D stereo model is shown. Figure 5 In the , the eye pattern in front of the word "transverse" and the word "sagittal" is displayed with a slash, which corresponds to hiding the layer images of the transverse and sagittal planes, and the eye pattern in front of the word "coronal" is displayed without a slash, which corresponds to displaying the layer image of the coronal plane. Figure 4 and Figure 7 , Figure 7 In one embodiment, the present application shows Figure 4 A schematic diagram of an interface for displaying layer images of multiple anatomical planes after the position of the coronal plane is adjusted, and the processing device changes the image based on the user's interactive operation Figure 4 The spatial pose of the 3D stereo model 3 in Figure 7 The spatial pose shown will Figure 4 The 3D stereo model in 3 is hidden and the user points to it with his finger Figure 4 The plus sign after the word "coronal plane" is shown so that the position of the coronal plane is indicated by Figure 4 The 256 becomes Figure 7 342 in the above, and then the processing device is Figure 7 The positions of the transverse layer image 4a and the sagittal layer image 4c displayed in FIG. Figure 4 The positions of the transverse layer image 4a and the sagittal layer image 4c shown do not change. Figure 7 The mid-coronal image 4b ​​is relative to Figure 4 The position of the coronal slice image 4b ​​is shown to be shifted.

[0092] In another embodiment, the processing device 12 dynamically displays any layer image of any anatomical plane based on the interactive operation between the user and the 3D stereoscopic model. Among them, the dragging operation of the cross section by the user is exemplified by moving in the up-down direction after a single hand is in a pinching gesture, the dragging operation of the coronal plane by the user is exemplified by moving in the front-to-back direction after a single hand is in a pinching gesture, and the dragging operation of the sagittal plane by the user is exemplified by moving in the left-to-right direction after a single hand is in a pinching gesture, and then the processing device dynamically displays layer images of different cross-sectional positions when detecting that the single hand is in a pinching gesture and moving in the up-to-down direction, dynamically displays layer images of different coronal positions when detecting that the single hand is in a pinching gesture and moving in the front-to-back direction, and dynamically displays layer images of different sagittal positions when detecting that the single hand is in a pinching gesture and moving in the left-to-right direction.

[0093] By selectively hiding anatomical planes, the user can observe only the layer images of the anatomical planes that he or she wishes to view in detail, thereby preventing other anatomical planes from obstructing the anatomical planes that he or she wishes to observe. By dynamically adjusting the positions of anatomical planes, the user can dynamically observe the layer images of anatomical planes at different positions. In other words, in the process of dynamically switching the positions of anatomical planes, the user can continuously observe the layer images of anatomical planes at different positions that are displayed continuously.

[0094] In one embodiment, the processing device may present a grayscale adjustment option in the menu window to adjust the contrast and / or brightness of the displayed layer image based on the user's interactive operation of the grayscale adjustment option. The grayscale adjustment option includes, for example, a window width adjustment option and a window level adjustment option. The window width adjustment option corresponds to adjusting the contrast of the displayed layer image, and the window level adjustment option is used to adjust the brightness of the displayed layer image. The contrast and / or brightness of the layer image can also be adjusted by interactive operation of the window width adjustment option and / or the window level adjustment option. In this way, the user can not only view the layer image while viewing the 3D stereo model, but also highlight different biological tissues by adjusting the contrast and / or brightness of the layer image. For example, the CT value of the bone is high. When observing the bone, the window width can be set wider. In this way, although the contrast is poor, the overall density of the bone can be displayed to facilitate the observation of the bone. In one example, Figure 5 As shown, the window width adjustment options presented in the menu window include a plus option and a minus option behind the word "window width", and the processing device can adjust the contrast of the CT image based on the user's interactive operation of the plus option or the minus option behind the word "window width". The window level adjustment options presented in the menu window include a plus option and a minus option behind the word "window level", and the processing device can adjust the brightness of the CT image based on the user's interactive operation of the plus option or the minus option behind the word "window level". The numerical value behind the word "window width" represents the window width value of the current CT image, and the numerical value behind the word "window level" represents the window level value of the current CT image.

[0095] In one embodiment, the processing device presents a region selection item in the menu window to display the selected region in the layer image based on the user's interactive operation on the region selection item. The region selection item includes options for different regions and / or components in the organism, such as air options, blood options, abdomen options, brain options, chest options, bones options, lung options, etc. Figure 8 , Figure 8 The display is a schematic diagram of an interface showing a region selection item in an embodiment of the present application. As shown in the figure, the region selection item is configured to be presented in the form of a drop-down menu, and the user and Figure 8After the user interacts with the symbol “…” shown in the figure (for example, the user points to the symbol “…” with his finger), the processing device will present the area selection items shown in the figure in the menu window based on the interactive operation, and the area selection items include an air option displayed as the text “CT air”, a lung option displayed as the text “CT lung”, a brain option displayed as the text “CT brain”, a bone option displayed as the text “CT bone”, and an abdomen option displayed as the text “CT abdomen”. After the user interacts with the corresponding option, the processing device highlights the corresponding area in the layer image based on the interactive operation performed by the user with the corresponding option.

[0096] In one embodiment, the processing device is further used to present a model display option in the menu window to hide or display the entire 3D stereo model or a sub-organization of the 3D stereo model based on the user's interactive operation on the model display option. In one example, the model display option includes an overall display option, an overall hiding option, and a sub-organization display option, and the processing device displays the entire 3D stereo model based on the user's interactive operation on the overall display option, hides the entire 3D stereo model based on the user's interactive operation on the overall hiding option, and displays or hides the corresponding sub-organization in the 3D stereo model based on the user's interactive operation on the sub-organization display option.

[0097] See also Figure 6 and Fig. 9 , Fig. 9 In one embodiment, the present application will Figure 6 The schematic diagram of the 3D model after hiding is shown in the figure. As shown in the figure, the text "Show All" area in the upper right corner of the display interface is the overall display option, the text "Hide All" area in the display interface is the overall hiding option, and the rounded rectangular solid color area in the display interface (for example Figure 6 The medium blue area, yellow area, green area, red area, purple area, cinnabar area, pink area, and gray area are sub-organization display options, and the user points with his finger Figure 6 The text "Hide All" area, Figure 6 The entire 3D model will be hidden, that is, it will appear as follows Fig. 9 shown, and Fig. 9 The solid color area in the rounded rectangle also becomes hidden (i.e., the eye pattern on the front of the rounded rectangle changes to a state with a slash). The user points to the Fig. 9 In the area where the text "Show All" is clicked, the hidden 3D model will be re-presented, that is, Figure 6 shown.

[0098] See also Fig. 9 and Fig.10 , Fig.10The application is shown in one embodiment. Fig. 9 Schematic diagram of the sub-organizations hidden in the 3D stereo model, in Fig. 9 When the user points his finger to the yellow area, the yellow vertebrae in the 3D model will appear. Fig.10 As shown, the user Fig.10 When the yellow area is clicked again, the yellow vertebra in the 3D stereoscopic model will be hidden again. It should be noted that, although the above example uses the model display options including the overall display option, the overall hiding option, and the sub-tissue display option as an example for explanation, it is not limited to this. In other embodiments, the overall display option and the overall hiding option can also be configured as an option for controlling whether the entire 3D stereoscopic model is displayed.

[0099] When a layer image with an anatomical plane is fused and displayed in a 3D stereoscopic model, the 3D stereoscopic model can be hidden to avoid occluding the anatomical plane, so that the layer image fused in the 3D stereoscopic model can be seen more clearly, which is convenient for users to watch. Moreover, by separately displaying some sub-tissues in the 3D stereoscopic model, users can clearly see the status of the sub-tissues. For example, when the user selects the sub-tissue where the lesion is located for separate viewing, the user can have a clearer understanding of the condition of the lesion, thereby improving the success rate of the operation when the MR device is used during the operation.

[0100] In one embodiment, the processing device may also drag the sub-tissue out of the 3D stereoscopic model based on the user's interactive operation on the sub-tissue in the 3D stereoscopic model. The user's interactive operation on the sub-tissue in the 3D stereoscopic model is, for example, gazing at the sub-tissue and dragging the sub-tissue, and the dragging operation is, for example, moving one hand in a pinching gesture, such as moving the thumb and index finger in a pinching gesture. Figure 6 and Fig.11 , Fig.11 In one embodiment, the present application will Figure 6 Schematic diagram of the interface where the user gazes at the sub-organization in the Figure 6 While the user is moving the yellow vertebrae in the middle, the user moves one hand in a pinching gesture towards the user. The yellow vertebrae are then dragged out of the 3D model. The user stops moving the hand and the vertebrae are displayed. Fig.11 In the state shown, the yellow vertebra is separated from the 3D stereoscopic model and through the MR device, the yellow vertebra appears to be located close to the user. In this way, the user can not only observe the sub-tissues (such as lesion tissue) in the 3D stereoscopic model separately, but also drag the sub-tissue to the front for careful observation, so as to see the lesion tissue more clearly and improve the success rate of the operation.

[0101] In order to highlight the subtissues in the layer image, the processing device presents a color mark option in the menu window, so as to present the subtissues in the displayed layer image in a color rendering manner based on the user's interactive operation on the color mark option. Wherein, the subtissues in the displayed layer image that can be presented in a color rendering manner correspond to the tissues in the 3D stereoscopic model. For example, if the subtissues in the 3D stereoscopic model include multiple vertebrae of the thoracic spine, then the subtissues in the layer image that can be presented in a color rendering manner also include multiple vertebrae of the thoracic spine.

[0102] In one embodiment, the color marking option is an overall marking option, and the overall marking option performs an interactive operation of overall color rendering display on all sub-tissues in the displayed layer image. Specifically, the processing device performs an overall color rendering display on all sub-tissues in the displayed layer image based on the user's interactive operation on the overall marking option. In one example, the overall color rendering display corresponds to color rendering display of all sub-tissues in different colors, for example, after the overall color rendering display, the outline or the whole of each sub-tissue is displayed in a different color. Please refer to Figure 7 and Fig.12 , Fig.12 In one embodiment, the present application shows Figure 7 Schematic diagram of the interface for rendering all sub-organizations of the layer image as a whole, the user Figure 7 For example, the user points to the word "color mark" with a finger, and the processing device, based on the user's interactive operation on the word "color mark", Figure 7 The outline of each subtissue in the layer image of each anatomical plane is displayed in a different color, such as Fig.12 As shown in , the outline of each subtissue is rendered in different colors. In other embodiments, the color marking option can also be a subtissue marking option, and the number of the subtissue marking options is the same as the number of subtissues in the 3D stereo model, corresponding to color rendering of a single subtissue of the displayed layer image.

[0103] By presenting the sub-tissues in the layer image in a color rendering manner, it is convenient for the user to quickly locate the corresponding sub-tissue in the layer image, so as to improve the surgical efficiency, etc.

[0104] In some embodiments, the processing device 12 includes one or more processors. The processing device 12 can be operable to perform data read and write operations with the storage device 13. The processing device may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). For another example, it can also be an application-specific integrated circuit (ASIC), a discrete gate or transistor logic device, or a discrete hardware component, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0105] In some embodiments, the MR device 1 also includes a communication interface (not shown). The communication interface includes at least one interface unit, each of which is used to output a visual interface, receive human-computer interaction events generated according to the operation of a technician, etc. For example, the communication interface includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the communication device also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc. For example, the communication interface is used to transmit the display data of the MR device to the field of view sharing device described in the embodiments below, so that the field of view sharing device can synchronously display the display interface of the MR device.

[0106] This application also provides a medical imaging system, see Fig.13 , which is a schematic diagram of the structure of a medical imaging system in one embodiment of the present application. As shown in the figure, the medical imaging system includes an MR device 1 and a field of view sharing device 6.

[0107] The MR device 1 is the same as or similar to the MR device described in any of the above embodiments, and will not be described in detail herein.

[0108] The visual field sharing device 6 is in communication connection with the MR device 1, and is used to synchronously display the display interface displayed by the display device in the MR device 1. Specifically, the visual field sharing device 6 and the MR device 1 can be located in the same actual physical space or in different actual physical spaces.

[0109] In an embodiment where the field of view sharing device and the MR device are located in different actual physical spaces, the virtual objects (such as a menu window, a 3D stereoscopic model of biological tissue, and layer images of multiple anatomical planes fused in the 3D stereoscopic model) seen by a user (such as a doctor participating in a surgery remotely) through the field of view sharing device and the real scene and the virtual objects seen by the user (such as a doctor in an operating room) through the MR device are the same.

[0110] In an embodiment where the field of view sharing device and the MR device are located in the same actual physical space, the field of view sharing device may display the same content as field of view sharing devices located in different actual physical spaces. For example, the virtual objects and real scenes seen by the user through the field of view sharing device located in the same actual physical space are the same as the virtual objects and real scenes seen by the user through the MR device. The field of view sharing device may also superimpose and display the virtual objects in its perspective and the real scenes in its perspective; in other words, the virtual objects and real scenes respectively seen by the user (e.g., a doctor's assistant in an operating room) through the field of view sharing device and the user (e.g., a doctor in an operating room) through the MR device are the virtual objects and real scenes in their respective perspectives.

[0111] In one embodiment, the visual field sharing device is exemplified as an MR device, wherein the visual field sharing device configured as an MR device and Figure 1 The structures of the MR devices 1 shown are the same or similar and will not be described in detail here.

[0112] Furthermore, the medical imaging system further includes a computer device (not shown) that is communicatively connected to the MR device 1 and is used to determine medical data based on the medical imaging data and import the medical data into the MR device 1. The computer device is exemplified by a server that is communicatively connected to the MR device, or a computer device that is communicatively connected to the server of the MR device, or a workbench that is communicatively connected to the MR device, etc.).

[0113] In summary, the MR device and medical imaging system disclosed in the present application display a 3D stereoscopic model (with layer images of multiple anatomical planes fused on the 3D stereoscopic model) and a menu window superimposed on the current real scene of the display interface, and dynamically adjusts the spatial posture of the 3D stereoscopic model in the current real scene based on the user's interactive operation with the menu window and / or the 3D stereoscopic model, and dynamically displays any layer image of any anatomical plane. In this way, the user can not only change the spatial posture of the 3D stereoscopic model to make the 3D stereoscopic model displayed in its desired posture, but also adjust the positions of the anatomical planes fused in the 3D stereoscopic model to display the layer images at the corresponding positions, thereby providing the user with a good interactive experience while presenting rich image information.

[0114] The above embodiments are merely illustrative of the inventive essence of the present application and the beneficial effects obtained, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the principles and scope of the present application. Therefore, all equivalent modifications or changes achieved by a person with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An MR device, characterized in that: include: A camera device, used to collect current environment data; A display device, used to present in real time in its display interface a current real scene corresponding to the current environmental data; A processing device, connected to the camera device and the display device, is used to call and execute a medical imaging program based on the imported medical data to achieve: Superimposing a menu window and a 3D stereoscopic model of biological tissue in the current real scene of the display interface, wherein the 3D stereoscopic model is integrated with layer images of multiple anatomical planes; The spatial position of the 3D stereoscopic model in the current real scene is dynamically adjusted based on the user's interactive operation with the menu window and / or the 3D stereoscopic model, and any layer image of any anatomical plane is dynamically displayed.

2. The MR device according to claim 1, characterized in that The processing device is also used to switch the 3D stereoscopic model to a first interactive mode or a second interactive mode based on the user's interactive operation, the first interactive mode corresponding to the interactive operation used to dynamically adjust the spatial posture, and the second interactive mode corresponding to the interactive operation used to dynamically display any layer image of any anatomical plane.

3. The MR device according to claim 2, characterized in that The first interaction mode is configured such that the 3D stereoscopic model is presented in a manner of being surrounded by a bounding box, and the processing device hides the bounding box based on a user's interaction operation on the bounding box to switch to the second interaction mode.

4. The MR device according to claim 3, characterized in that Hiding the bounding box to switch to the second interaction mode based on the user's interaction operation on the bounding box includes: hiding the bounding box when detecting a gaze of the user on the bounding box and a confirmation operation on the bounding box.

5. The MR device according to claim 3, characterized in that The boundary box or the outer side of the boundary box is marked with position information to give the user a position prompt.

6. The MR device according to claim 1, characterized in that The interactive operation of dynamically adjusting the spatial position of the 3D stereoscopic model in the current real scene includes a dragging operation and a rotating operation.

7. The MR device according to claim 1, characterized in that The dynamically adjusting the spatial posture of the 3D stereo model in the current real scene includes: identifying a platform object in the current real scene so as to present the 3D stereo model as being placed on the platform object when it is detected that the user drags the 3D stereo model onto the platform object based on the interactive operation.

8. The MR device according to claim 1, characterized in that The processing device is further configured to present a model display option in the menu window to hide or display the entirety or a sub-organization of the 3D stereoscopic model based on a user's interactive operation on the model display option.

9. The MR device according to claim 8, characterized in that The model display options include an overall display option, an overall hiding option, and a sub-organization display option. The processing device displays the entire 3D stereo model based on the user's interactive operation on the overall display option, hides the entire 3D stereo model based on the user's interactive operation on the overall hiding option, and displays or hides the corresponding sub-organizations in the 3D stereo model based on the user's interactive operation on the sub-organization display option.

10. The MR device according to claim 1, characterized in that The multiple anatomical planes include a transverse plane, a coronal plane, and a sagittal plane.

11. The MR device according to claim 1, characterized in that The processing device displays a corresponding layer image based on a user's dragging operation on any anatomical plane.

12. The MR device according to claim 1 or 11, characterized in that The processing device is also used to present each anatomical plane option and its layer option in the menu window, so as to display the corresponding layer image of the corresponding anatomical plane based on the user's interactive operation on each anatomical plane option and its layer option, and to hide or display the corresponding layer image of the corresponding anatomical plane presented on the 3D stereoscopic model.

13. The MR device according to claim 1, characterized in that The processing device presents a grayscale adjustment option in the menu window to adjust the contrast and / or brightness of the displayed layer image based on a user's interactive operation on the grayscale adjustment option.

14. The MR device according to claim 1, characterized in that The processing device presents a region selection item in the menu window to display the selected region in the layer image based on a user's interactive operation on the region selection item.

15. The MR device according to claim 1, characterized in that The processing device presents a color marking option in the menu window to present the sub-tissue in the displayed layer image in a color rendering manner based on the user's interactive operation on the color marking option.

16. The MR device according to claim 15, characterized in that The color marking option is an overall marking option, and the overall marking plus option corresponds to an interactive operation of performing an overall color rendering display on all sub-tissues in the displayed layer image.

17. The MR device according to claim 1, characterized in that Different subtissues in the 3D stereoscopic model are presented in different colors.

18. The MR device according to claim 1, characterized in that The MR device is configured as a head-mounted display device Apple Vision.

19. A medical imaging system, characterized in that: It comprises the MR device as described in any one of claims 1 to 18 and a field of view sharing device communicatively connected to the MR device, wherein the field of view sharing device is used to synchronously display a display interface displayed by a display device in the MR device.

Images