A mixed reality augmented display system and method

Abstract

The application provides a kind of mixed reality augmented display system and method, the system includes: structured light module, processing module;Structured light module includes camera, projector;Processing module is configured as: obtaining the point cloud of target object collected by structured light module;The registration of target object point cloud and target object model, and the same coordinate system of structured light module and target object model is unified;The pose parameter of the image shooting model is input to the projected model in the same coordinate system and the pose parameter of the projector, and the projected image is generated;The projected image is sent to the structured light module, to be projected to the target object by the projector.The mapping relationship of model space and real space is determined quickly and high-precisionly, the projected image is generated by image acquisition model, and is displayed on the real target object by the projector of structured light module, to achieve "virtual and real combination", to intuitively carry out surgical planning, surgical navigation, multi-person discussion, teaching and the like.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a mixed reality augmented display system and method. Background Technology

[0002] Before treating a patient, it is necessary to develop an appropriate treatment plan.

[0003] A common practice in existing technologies is to directly observe medical image models on a screen and plan surgical procedures. The drawbacks of this method are: 1. Doctors can only adjust the medical image model on the screen, limiting their field of vision and hindering multi-person discussions; 2. The medical image model observed on the screen differs visually from the actual patient in the real space, making it less intuitive; 3. The medical image model does not directly correspond to the actual patient, making it difficult for doctors to accurately locate surgical procedures on the real patient.

[0004] Another common practice is for doctors to observe and adjust medical imaging models to ensure their angles closely match those of the actual patient (for comparison and spatial understanding). Then, surgical details are projected onto the patient's surface using a projector. The projector is adjusted by comparing the projection with the medical imaging model on the screen to optimize the projection, ensuring accurate placement of the surgical details. Surgical planning is then based on the projected details, such as planning the smallest and most accurate incision line based on the projected tumor or brain region, or planning needle entry points to avoid blood vessels based on the projected blood vessels. The drawbacks of this method are: 1. The projection process is complex and requires repeated manual adjustments; 2. The projection accuracy is relatively low.

[0005] In response, this invention proposes a mixed reality augmented display system and method to solve or at least partially solve the above-mentioned defects of the prior art.

[0006] The mixed reality augmented display system and method provided by the present invention have at least the following beneficial effects:

[0007] 1. It can display the virtual information that needs to be observed onto the real target object, achieving "virtual and real combination", which improves information density. Users can directly observe the target object (and the projected information on the target object) to intuitively carry out surgical planning, surgical navigation and other content, without the need for "spatial conversion" between medical imaging space and real space.

[0008] 2. It offers a wider field of view and more flexible observation angles, making it convenient for group discussions and teaching. Furthermore, the projected content can incorporate color and texture, enhancing the display effect.

[0009] 3. By capturing images of the model to be projected, the projector accurately projects the images onto the surface of the target object. There is no need to use tracking devices to detect whether the projection point meets expectations, nor is there a need to spend time adjusting the projector angle and correcting the projection effect. The projection efficiency is higher and the projection is more accurate.

[0010] 4. The structured light module has the dual function of collecting point clouds and projection. The processing module processes the data to generate the projected image. The whole system has a simple structure and low cost.

[0011] 5. Patient registration based on structured light modules has higher registration accuracy and efficiency, and correspondingly, the projection display accuracy and efficiency are also higher. Summary of the Invention

[0012] This invention provides a mixed reality augmented display system and method to address the shortcomings of existing technologies, such as the lack of intuitiveness in doctors' observation of observation models and surgical planning, and the low accuracy of locating surgical-related content on patients.

[0013] This invention provides a mixed reality augmented display system, comprising: a structured light module and a processing module;

[0014] The structured light module includes a camera and a projector;

[0015] The processing module is configured as follows:

[0016] Obtain the point cloud of the target object acquired by the structured light module;

[0017] The point cloud of the target object is registered with the target object model, and the structured light module and the target object model are unified to the same coordinate system; wherein, the target object model contains a model to be projected;

[0018] The model to be projected in the same coordinate system and the pose parameters of the projector are input into the image capture model to generate the image to be projected.

[0019] The image to be projected is sent to the structured light module so that it can be projected onto the target object through the projector.

[0020] According to a mixed reality augmented display system provided by the present invention, the system further includes a fixing module for fixing the structured light module at a desired position;

[0021] The fixing module can adopt any of the following structures: a fixed connection structure, or an installation structure that matches the installation position.

[0022] According to the present invention, a mixed reality augmented display system includes multiple sets of fixed modules and structured light modules, which can realize point cloud projection from multiple angles.

[0023] The present invention also provides a mixed reality augmented display system, comprising: a structured light module, a processing module, and a robotic arm;

[0024] The structured light module includes a camera and a projector;

[0025] The structured light module is mounted or integrated on the robotic arm;

[0026] The processing module is configured as follows:

[0027] Based on the current robotic arm posture, the structured light module and the target object model are unified to the same coordinate system; wherein, the target object model includes the model to be projected;

[0028] The model to be projected in the same coordinate system and the pose parameters of the projector are input into the image capture model to generate the image to be projected.

[0029] The image to be projected is sent to the structured light module so that it can be projected onto the target object through the projector.

[0030] According to a mixed reality augmented display system provided by the present invention, the processing module is further configured as follows:

[0031] The robotic arm is controlled to drive the structured light module to register, and the first transformation relationship between the robotic arm coordinate system and the model coordinate system is obtained.

[0032] Based on the current robotic arm posture, determine the second transformation relationship between the robotic arm coordinate system and the structured light coordinate system;

[0033] Based on the first transformation relationship and the second transformation relationship, the structured light module and the target object model are unified to the same coordinate system.

[0034] According to a mixed reality augmented display system provided by the present invention, the pose parameters of the projector in the same coordinate system include:

[0035] The coordinates of the projector in the same coordinate system, as well as the pitch angle and roll angle of the projector.

[0036] According to a mixed reality augmented display system provided by the present invention, the processing module is further configured as follows:

[0037] The projector's internal parameters, the model to be projected, and the projector's pose parameters are input into the image capture model to generate the image to be projected.

[0038] According to a mixed reality augmented display system provided by the present invention, the target object model is a patch model with color and / or texture.

[0039] According to a mixed reality augmented display system provided by the present invention, the image capture model is a callable interface generated based on a three-dimensional model processing framework.

[0040] The present invention also provides a mixed reality augmented display method for acquiring point clouds of target objects collected by a structured light module; wherein, the structured light module includes a camera and a projector;

[0041] The point cloud of the target object is registered with the target object model, and the structured light module and the target object model are unified to the same coordinate system; wherein, the target object model contains a model to be projected;

[0042] The model to be projected in the same coordinate system and the pose parameters of the projector are input into the image capture model to generate the image to be projected.

[0043] The image to be projected is sent to the structured light module so that it can be projected onto the target object through the projector.

[0044] A mixed reality augmented display method provided by the present invention further includes:

[0045] If movement of the target object is detected, an alarm action is executed; and / or,

[0046] If movement of the target object is detected, the steps of acquiring the point cloud of the target object collected by the structured light module and sending the image to be projected to the structured light module for projection onto the target object by the projector are re-executed.

[0047] The present invention also provides another mixed reality augmented display method, which unifies the structured light module and the target object model into the same coordinate system according to the current posture of the robotic arm; wherein, the structured light module includes a camera and a projector, the structured light module is installed or integrated at the end of the robotic arm, and the target object model includes a model to be projected;

[0048] The model to be projected in the same coordinate system and the pose parameters of the projector are input into the image capture model to generate the image to be projected.

[0049] The image to be projected is sent to the structured light module so that it can be projected onto the target object through the projector.

[0050] A mixed reality augmented display method provided by the present invention further includes:

[0051] If movement of the target object is detected, an alarm action is executed; and / or,

[0052] When the target object is detected to be moving, the robotic arm is controlled to register the structured light module, obtain the first transformation relationship between the robotic arm coordinate system and the model coordinate system, and re-execute the steps of unifying the structured light module and the target object model to the same coordinate system according to the current robotic arm posture, and sending the image to be projected to the structured light module for projection onto the target object through the projector.

[0053] According to a mixed reality augmented display method provided by the present invention, the movement of the target object is detected in the following manner:

[0054] The camera periodically acquires images, and when the cumulative pixel difference between two acquired images is greater than a preset threshold, it is determined that the target object has moved. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 This is one of the structural schematic diagrams of a mixed reality augmented display system provided by the present invention;

[0057] Figure 2 This is the second schematic diagram of a mixed reality augmented display system provided by the present invention;

[0058] Figure 3 This is the third schematic diagram of the structure of a mixed reality augmented display system provided by the present invention;

[0059] Figure 4 This is a rendering of a mixed reality augmented display system provided by the present invention;

[0060] Figure 5 This is a flowchart illustrating a mixed reality augmented display method provided by the present invention;

[0061] Figure 6 This is a flowchart illustrating another mixed reality augmented display method provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] The following is combined Figures 1-6 This invention describes a mixed reality augmented display system and method.

[0064] Combination Figure 1 A mixed reality augmented display system of the present invention will be described, such as Figure 1 As shown, the system includes a structured light module 100 and a processing module 200.

[0065] The structured light module 100 includes a camera 110 and a projector 120. The camera 110 and projector 120 have a fixed positional relationship, and their relative positions are calibrated before leaving the factory. The structured light module 100 can acquire 3D point clouds based on the structured light principle. Structured light point cloud acquisition is an active point cloud acquisition technology with advantages such as non-contact operation, large field of view, high precision (large amount of point cloud data), and good real-time performance. During point cloud acquisition, the structured light module 100 projects an coded 2D pattern onto the target object through the projector 120. This 2D pattern will be distorted due to the inconsistent height of the object's surface. The camera 110 captures the distorted pattern, and then decodes the pattern to obtain the correspondence between the object's surface and the projected pattern points. Based on the triangulation principle, the 3D point cloud of the target object's surface is then calculated. The structured light module 100 can also project onto the target object through the projector 120.

[0066] The processing module 200 interacts with the structured light module 100 for data processing. The processing module 200 can be integrated with the structured light module 100 or it can be an external independent device, such as a laptop, workstation, PDA (personal digital assistant), etc. The structured light module 100 can be placed in a desired location, such as on a desktop, operating table, or hospital bed. The structured light module 100 can also be fixed in the desired location by a fixing module, or it can be used in conjunction with a robotic arm to flexibly adjust its position. This invention does not limit the form, hardware form, or spatial location of the processing module 200.

[0067] Processing module 200 includes a memory and a processor, the processor being able to execute computer programs stored in the memory to achieve the following:

[0068] S11. Obtain the point cloud of the target object collected by the structured light module;

[0069] The processing module 200 receives point cloud data (i.e., target object point cloud) acquired from the structured light module 100 for the target object. This target object point cloud is, as understood, visually visible surface point cloud data of the target object, such as facial skin data, or tissue data exposed during craniotomy. The position of the structured light module 100 in real space (e.g., the structured light coordinate system) is known; accordingly, the position of the target object in real space can be determined based on the acquired target object point cloud.

[0070] S12. Register the point cloud of the target object with the model of the target object, and unify the structured light module 100 and the model of the target object into the same coordinate system; wherein, the model of the target object contains the model to be projected;

[0071] Specifically, the target object model is a 3D patch model corresponding to the target object. For example, it can be a 3D model generated from medical images of the target object (MRI images, functional MRI images, CT images, phase-contrast magnetic resonance angiography, etc.). Alternatively, if the target object is a physical demonstration model, the target object model is its corresponding 3D design model. The target object model contains the model to be projected, such as patch models of regions like tumors, blood vessels, and brain areas, or models of puncture points and incision lines (understandably, for point and line-type objects to be projected, they need to be "thickened" to obtain the corresponding patch model for easier subsequent processing and projection). Using patch models balances data processing efficiency with the image quality of the image to be projected. Furthermore, color rendering and texture addition can be applied to the patch model to optimize the final projection effect.

[0072] The target object point cloud in the structured light coordinate system contains surface information (visually visible to the target object), and the target object model in the model coordinate system also contains surface information (visually visible to the target object). Registration between the target object point cloud and the target object model determines the transformation relationship from the structured light coordinate system to the model coordinate system. Based on this transformation relationship, the structured light module 100 and the target object model can be unified to the same coordinate system, for example, unified to the model coordinate system in virtual space, unified to the structured light coordinate system in real space, or further transformed to the coordinate system of a hospital bed, operating table, etc., in real space, taking into account the real spatial position of the structured light module. The structured light coordinate system is established based on the structured light module 100, for example, using the camera coordinate system or projector coordinate system as the structured light coordinate system, or establishing a coordinate system at a specific location within the structured light module 100.

[0073] S13. Input the model to be projected in the same coordinate system and the pose parameters of the projector 120 into the image capture model to generate the image to be projected.

[0074] The structured light module is calibrated before leaving the factory, specifically the position of the projector 120 within it. For example, using the camera coordinate system as the structured light coordinate system, the calibration parameters are the relative positions of the projector 120 and the camera 110. Alternatively, a coordinate system can be established using the projector 120 as a reference, or with the midpoint between the camera 110 and the projector 120 as the origin. The position of the structured light module 100 within the aforementioned coordinate system is fixed. Since the structured light module 100 includes the projector 120, the position and attitude information of the projector 120 within this same coordinate system can be determined by combining the calibration parameters of the structured light module 100. The target object model contains the model to be projected; therefore, the coordinates of the model to be projected within this same coordinate system are also determinable.

[0075] The model to be projected in the same coordinate system and the pose parameters of the projector 120 are input into the image acquisition model. The image acquisition model can capture images of the model to be projected under the pose of the projector 120, generating a two-dimensional pixel image to be projected. The image acquisition model can acquire images of the three-dimensional model under the required pose and can also perform lighting and shadow processing to display the model to be projected more realistically.

[0076] S14. Send the image to be projected to the structured light module 100 so that it can be projected onto the target object through the projector 120.

[0077] The processing module 200 sends the image to be projected to the structured light module 100. The projector 120 of the structured light module 100 projects the image onto the surface of the target object. For example, the divided brain regions can be projected onto the patient's head to facilitate the planning of puncture points that avoid important brain regions. Alternatively, blood vessels can be projected onto the exposed brain tissue after craniotomy to avoid blood vessel areas during surgery and reduce the risk of bleeding. Another example is that the incision line can be projected onto the skin surface for auxiliary positioning. Yet another example is that multiple elements can be projected onto the patient simultaneously (such as blood vessels + brain regions, brain regions + incisions, blood vessels + incisions, brain regions + blood vessels + incisions, etc.).

[0078] Since the image capture model generates the image to be projected based on the pose parameters of the projector, the projector does not need to adjust the projection parameters. It can directly project the image to be projected onto the target object to accurately display the "model to be projected" at the corresponding position of the target object.

[0079] This embodiment of the mixed reality augmented display system uses a structured light module to acquire point clouds and register them with a target object model. This quickly and accurately determines the position of the target object's point cloud and the mapping relationship between the model space and the real space. By inputting the projector's pose information and the model to be projected into the image acquisition model in the same coordinate system, a projected image is generated efficiently. The virtual information ("projected model") is then displayed on the real target object, achieving a "virtual-real combination." Users do not need to switch their gaze between the target object and the screen, facilitating intuitive surgical planning, surgical navigation, multi-person discussions, and teaching. Furthermore, this embodiment eliminates the need for tracking devices to detect whether the projection point matches expectations, and also eliminates the need to adjust the projector angle or correct the projection effect, improving projection accuracy and efficiency. The structured light module has the dual function of point cloud acquisition and projection, reducing system costs.

[0080] Based on the above embodiments, in one embodiment, the system further includes a fixing module 300 for fixing the structured light module 100 at the desired position;

[0081] The fixed module 300 can adopt any of the following structures: a fixed connection structure, or an installation structure that matches the installation position.

[0082] Specifically, the fixing module 300 is used to fix the structured light module 100 at the required position. For example, based on the optimal projection range (distance) of the point cloud acquisition of the structured light module 100, and combined with the available location in the real space, the structured light module 100 is fixed at a suitable position. The structure of the fixing module 300 can be flexibly set according to cost, usage environment, and other requirements. The fixing module 400 can be a fixed connection structure, for example, referring to... Figure 2 The fixed connection structure can adopt a fixed support structure (tripod, support rod, folding bracket, etc.). Figure 2 (Only the fixed support structure of the tripod is shown.) For example, the fixed connection structure can be a fixed suspension structure to suspend the structure light module 100 in the required position (e.g., above the hospital bed). Furthermore, the suspension structure adopts a universal arm, which is connected to the structure light module 100 through a ball joint, so that the position of the structure light module 100 can be adjusted and fixed. The fixed module 300 can also be an installation structure that matches the installation position, such as an installation structure that matches the installation position on the hospital bed (such as a slot, clamp, etc.).

[0083] Based on any of the above embodiments, in one embodiment, the system includes multiple sets of fixed modules 300 and structured light modules 100, which can realize point cloud projection from multiple angles.

[0084] Specifically, a structured light module 100 can only project onto the target object from one angle at a time and display relevant content on the target object. By using multiple sets of fixed modules 300 and structured light modules 100, the projection blind spot can be reduced and the display effect on the target object can be optimized.

[0085] This invention also provides a hybrid display enhancement system. The hybrid reality augmented display system described below can be referred to in relation to the aforementioned hybrid display enhancement system. Figure 3 The system includes a structured light module 100, a processing module 200, and a robotic arm 400.

[0086] The structured light module 100 includes a camera 110 and a projector 120. The camera 110 and projector 120 have a fixed positional relationship, and their relative positions are calibrated before leaving the factory. The structured light module 100 can acquire 3D point clouds based on the structured light principle. Structured light point cloud acquisition is an active point cloud acquisition technology with advantages such as non-contact operation, large field of view, high precision (large amount of point cloud data), and good real-time performance. During point cloud acquisition, the structured light module 100 can project an coded 2D pattern onto the target object through the projector 120. The pattern will be distorted due to the inconsistent height of the object's surface. The camera 110 captures the distorted pattern, and then decodes the pattern to obtain the correspondence between the object's surface and the projected pattern points. Based on the triangulation principle, the 3D point cloud of the target object's surface is then calculated. Additionally, the structured light module 100 can also project onto the target object through the projector 120.

[0087] The structure light module 100 is installed or integrated on the robotic arm 400. The robotic arm 400 is at least a four-axis robotic arm, preferably a six-axis robotic arm. The robotic arm 400 can drive the structure light module 100 to move to the required position to collect the point cloud of the target object and project it onto the target object according to the control instructions of the processing module 200.

[0088] The processing module 200 communicates with the structured light module 100 and the robotic arm 400 to exchange data. The processing module 200 can be integrated with the aforementioned structured light module 100, or it can be an external independent device, such as a laptop, workstation, PDA, etc. The present invention does not limit the form or hardware form of the processing module 200.

[0089] Processing module 200 includes a memory and a processor. The processor can run computer programs stored in the memory to perform the following tasks:

[0090] S21. Based on the current posture of the robotic arm, unify the structured light module 100 and the target object model into the same coordinate system; wherein, the target object model contains the model to be projected;

[0091] Specifically, one feasible solution is: the structured light module 100 collects the point cloud of the target object under the current posture of the robotic arm, and then the processing module 200 directly registers the point cloud of the target object collected under the current posture of the robotic arm with the target object model, establishes the transformation relationship between the structured light coordinate system and the model coordinate system, thereby unifying the structured light module 100 and the target object model into the same coordinate system.

[0092] Another feasible approach is to pre-register the structured light system, establishing a transformation relationship between the robotic arm coordinate system and the model coordinate system. Specifically, the robotic arm 400 drives the structured light module 100 to acquire point clouds of the target object in one or more poses. The processing module 200 transforms the point clouds to the robotic arm coordinate system based on the robotic arm's acquisition posture (if point clouds are acquired from multiple poses, point cloud fusion is also required within the robotic arm coordinate system). Then, the point clouds in the robotic arm coordinate system are registered with the target object model, obtaining the transformation relationship between the robotic arm coordinate system and the model coordinate system. Subsequently, during projection in one or more poses, the processing module 200 determines the transformation relationship between the structured light coordinate system and the robotic arm coordinate system based on the current robotic arm (projection) posture, thereby unifying the structured light module 100 and the target object model into the same coordinate system.

[0093] Examples of the same coordinate system mentioned above include the model coordinate system in virtual space, the structured light coordinate system in real space, and the hospital bed coordinate system and operating table coordinate system in real space.

[0094] S22. Input the model to be projected in the same coordinate system and the pose parameters of the projector 120 into the image capture model to generate the image to be projected.

[0095] The structured light module 100 is calibrated before leaving the factory, specifically the position of the projector 120 within it. For example, using the camera coordinate system as the structured light coordinate system, the calibration parameters are the relative positions of the projector 120 and the camera 110. The position of the structured light module 100 within the aforementioned coordinate system is fixed. Since the structured light module 100 includes the projector 120, the position and attitude information of the projector 120 within this coordinate system can be determined by combining the calibration parameters of the structured light module 100. The target object model includes the model to be projected; therefore, the coordinates of the model to be projected within this coordinate system are also determinable.

[0096] The model to be projected in the same coordinate system and the pose parameters of the projector 120 are input into the image acquisition model. The image acquisition model can capture images of the model to be projected under the pose of the projector 120, generating a two-dimensional pixel image to be projected. The image acquisition model can also acquire images of the three-dimensional model under the required pose and perform lighting and shadow processing to generate a more realistic image to be projected.

[0097] S23. Send the image to be projected to the structured light module 100 so that it can be projected onto the target object through the projector 120.

[0098] The processing module 200 sends the image to be projected to the structured light module 100. The projector 120 of the structured light module 100 projects the image onto the surface of the target object. For example, the divided brain regions are projected onto the patient's head to facilitate the planning of puncture points that avoid important brain regions. Another example is projecting blood vessels onto the exposed brain tissue after craniotomy so that the blood vessel area can be avoided during the operation, reducing the risk of bleeding. Yet another example is projecting the incision line onto the skin surface for auxiliary positioning, etc.

[0099] Since the image capture model generates the image to be projected based on the pose parameters of the projector, the projector does not need to adjust the projection parameters. It can directly project the image to be projected onto the target object to accurately display the "model to be projected" at the corresponding position of the target object.

[0100] The mixed reality augmented display system in this embodiment uses a structured light module to collect point clouds and register them with a target object model. This quickly and accurately determines the position of the target object's point cloud and the mapping relationship between the model space and the real space. By inputting the projector's pose information in the same coordinate system and the model to be projected into the image acquisition model, a projected image is generated efficiently. The virtual information ("projected model") is then displayed on the real target object, achieving a "virtual-real combination." Users do not need to switch their gaze between the target object and the screen, facilitating intuitive surgical planning, surgical navigation, multi-person discussions, and teaching. In addition, this embodiment uses a robotic arm to drive the structured light module to collect and project point clouds from multiple angles, improving projection accuracy and flexibility, and reducing projection blind spots.

[0101] Based on any of the above embodiments, in one embodiment, the processing module is further configured as follows:

[0102] The robotic arm is controlled to drive the structured light module to register, and the first transformation relationship between the robotic arm coordinate system and the model coordinate system is obtained.

[0103] Based on the current robot arm posture, determine the second transformation relationship between the robot arm coordinate system and the structured light coordinate system;

[0104] Based on the first and second transformation relationships, the structured light module and the target object model are unified to the same coordinate system.

[0105] Specifically, the system can achieve one-time registration and multiple projections. Registration establishes a mapping relationship between the real space and the medical image space. The processing module 200 controls the robotic arm 400 to drive the structured light module 100 to acquire (structured light) point clouds of the target object in one or more poses. The processing module 200 transforms the point cloud to the robotic arm coordinate system according to the robotic arm's acquisition posture (if point clouds are acquired from multiple poses, point cloud fusion is also required in the robotic arm coordinate system). Then, the point cloud in the robotic arm coordinate system is registered with the target object model to obtain the transformation relationship between the robotic arm coordinate system and the model coordinate system. Then, when projecting in one or more subsequent poses, the processing module 200 determines the transformation relationship between the structured light coordinate system and the robotic arm coordinate system in the current robotic arm (projection) posture. Then, combined with the transformation relationship between the robotic arm coordinate system and the model coordinate system determined in the registration process, the transformation relationship between the structured light coordinate system and the model coordinate system is determined, thereby unifying the structured light module 100 and the target object model into the same coordinate system.

[0106] The mixed reality augmented display system in this embodiment can achieve one-time registration and multiple / multi-angle flexible projection, without having to perform the process of collecting point clouds and registering with medical images every time projection is performed, thus improving projection efficiency and projection flexibility.

[0107] Based on any of the above embodiments, in one embodiment, the pose parameters of the projector in the same coordinate system include:

[0108] The coordinates of the projector in the same coordinate system, as well as the projector's pitch and roll angles.

[0109] Specifically, the pose parameters of the projector in the same coordinate system include position data and attitude data. The position data is the coordinates of the projector in the same coordinate system, which affects the image acquisition distance of the image capturing model. The attitude data is the pitch angle and roll angle of the projector in the same coordinate system, which affects the image acquisition orientation of the image capturing model.

[0110] Based on any of the above embodiments, in one embodiment, the processing module is further configured as follows:

[0111] The projector's internal parameters and the model to be projected, along with the projector's pose parameters, are input into the image capture model to generate the image to be projected.

[0112] Specifically, the model to be projected is the object being imaged, the pose parameters of the projector and the position and orientation of the image being imaged, and the intrinsic parameters of the projector, which are the inherent properties of the image capturing model during the imaging process, including focal length, distortion coefficient, etc. By inputting the projector's intrinsic parameters and the model to be projected, along with the projector's pose parameters, into the image capturing model, the generated image to be projected becomes more consistent with the physical and spatial properties of the projector.

[0113] Based on any of the above embodiments, in one embodiment, the target object model is a patch model with color and / or texture.

[0114] Specifically, after establishing a medical image model, color and texture information can be added to facilitate the identification and differentiation of different regions and tissues after projection. For example, textures can be added to brain functional areas, different colors can be added to different brain functional areas, and blood vessels can be set to red, etc., and then subsequent processing can be performed to generate the image to be projected. Figure 4 This example of a projection rendering allows users to quickly distinguish different areas and organizations. Furthermore, using a patch model balances data processing efficiency with image quality of the image to be projected.

[0115] In this embodiment, by setting the target object model as a patch model with color and / or texture, the projection becomes more intuitive and the projection effect is optimized.

[0116] Based on any of the above embodiments, in one embodiment, the image capture model is a callable interface generated based on a 3D model processing framework.

[0117] Specifically, the image capture model is used to process 3D models and generate 2D pixel images (projection images) in the required pose. The image capture model can also perform lighting and shadow processing to make the generated projection images more realistic. The image capture model can be a callable interface generated based on a 3D model processing framework, such as VTK, Blender, Cinemachine, etc. Setting the image capture model as a callable interface can reduce data dependencies between different programs and improve application running efficiency.

[0118] This invention also provides a mixed reality augmented display method. The mixed reality augmented display method described below can be referred to in conjunction with the mixed reality augmented display systems described above. Figure 5 This is a flowchart illustrating a mixed reality augmented display method provided by the present invention, as shown below. Figure 5 As shown, the method includes:

[0119] S51. Acquire the point cloud of the target object collected by the structured light module; wherein, the structured light module includes a camera and a projector;

[0120] S52. Register the point cloud of the target object with the model of the target object, and unify the structured light module and the model of the target object into the same coordinate system; wherein, the model of the target object contains the model to be projected;

[0121] S53. Input the model to be projected in the same coordinate system and the pose parameters of the projector into the image capture model to generate the image to be projected.

[0122] S54. Send the image to be projected to the structured light module so that it can be projected onto the target object through the projector.

[0123] This embodiment of mixed reality augmented display acquires and registers the target object's point cloud with the target object model, quickly and accurately determining the position of the target object's point cloud and the mapping relationship between the model space and the real space. By inputting the projector's pose information and the model to be projected into the image acquisition model in the same coordinate system, the projected image is generated efficiently, displaying the virtual information ("projected model") onto the real target object, achieving "virtual-real integration." Users do not need to switch their gaze between the target object and the screen, facilitating intuitive surgical planning, surgical navigation, multi-person discussions, teaching, etc. In addition, this embodiment does not require tracking equipment to detect whether the projection landing point meets expectations, nor does it require adjusting the projector angle or correcting the projection effect, improving the accuracy and efficiency of projection. The structured light module has the dual function of point cloud acquisition and projection, reducing system costs.

[0124] Based on the above embodiments, in one embodiment, the method further includes:

[0125] If movement of the target object is detected, an alarm action is executed; and / or,

[0126] If movement of the target object is detected, the steps of acquiring the point cloud of the target object collected by the structured light module and sending the image to be projected to the structured light module are re-executed so that it can be projected onto the target object by the projector.

[0127] Specifically, if the target object moves, the generated image to be projected will contain errors, and the projection point will also change. In this case, an alarm action can be executed (audio and visual enhancement, text and icon prompts on the screen, etc.). The system can then determine whether to reproject based on the user's input. Furthermore, if movement of the target object is detected, automatic reprojection can be initiated, meaning the steps from "acquiring the target object point cloud collected by the structured light module" to "sending the image to be projected to the structured light module for projection onto the target object via the projector" are re-executed.

[0128] This embodiment ensures the projection effect even when the target object is moving.

[0129] Based on the above embodiments, in one embodiment, whether the target object has moved is detected in the following manner:

[0130] Images are periodically captured by a camera. When the cumulative pixel difference between two captured images exceeds a preset threshold, it is determined that the target object has moved.

[0131] This embodiment provides a convenient and quick way to determine whether the target object has moved, thus optimizing the projection effect.

[0132] This invention also provides another mixed reality augmented display method. The mixed reality augmented display method described below can be referred to in conjunction with the mixed reality augmented display systems described above. Figure 6 This is a flowchart illustrating another mixed reality augmented display method provided by the present invention, as shown below. Figure 6 As shown, the method includes:

[0133] S51. Based on the current posture of the robotic arm, unify the structured light module and the target object model into the same coordinate system; wherein, the structured light module includes a camera and a projector, the structured light module is installed or integrated on the robotic arm, and the target object model contains the model to be projected;

[0134] S52. Input the model to be projected in the same coordinate system and the pose parameters of the projector into the image capture model to generate the image to be projected.

[0135] S53. Send the image to be projected to the structured light module so that it can be projected onto the target object through the projector.

[0136] The mixed reality augmented display system in this embodiment uses a structured light module to collect point clouds and register them with a target object model. This quickly and accurately determines the position of the target object's point cloud and the mapping relationship between the model space and the real space. By inputting the projector's pose information in the same coordinate system and the model to be projected into the image acquisition model, a projected image is generated efficiently. The virtual information ("projected model") is then displayed on the real target object, achieving a "virtual-real combination." Users do not need to switch their gaze between the target object and the screen, facilitating intuitive surgical planning, surgical navigation, multi-person discussions, and teaching. In addition, this embodiment uses a robotic arm to drive the structured light module to collect and project point clouds from multiple angles, improving projection accuracy and flexibility, and reducing projection blind spots.

[0137] It is understandable that the above process is the projection step of the structured light module in the current pose of the robotic arm. There is a registration step before that. After registration, projection can be performed in one or more poses. The specific registration steps can be referred to the embodiment of the mixed reality augmented display system, which will not be repeated here.

[0138] Based on the above embodiments, in one embodiment, the method further includes:

[0139] If movement of the target object is detected, an alarm action is executed; and / or,

[0140] When the target object is detected to be moving, the robotic arm is controlled to register the structured light module, obtain the first transformation relationship between the robotic arm coordinate system and the model coordinate system, and re-execute the steps of unifying the structured light module and the target object model to the same coordinate system according to the current robotic arm posture, and sending the image to be projected to the structured light module so as to project it onto the target object through the projector.

[0141] Specifically, if the target object moves, the generated image to be projected will have errors, and the projection point will change. In this case, an alarm action can be executed (audio-visual prompts, text and icon prompts on the screen, etc.). Based on the user's input, it can be determined whether to re-register and re-project. Alternatively, if target object movement is detected, automatic re-registration and re-projection can be initiated. This involves controlling the robotic arm to register the structured light module, obtaining the first transformation relationship between the robotic arm coordinate system and the model coordinate system, and re-executing the steps from "unifying the structured light module and the target object model to the same coordinate system based on the current robotic arm posture (and the first transformation relationship)" to "sending the image to be projected to the structured light module for projection onto the target object via the projector."

[0142] This embodiment ensures the projection effect when the target object moves.

[0143] Based on the above embodiments, in one embodiment, whether the target object has moved is detected in the following manner:

[0144] Images are periodically captured by a camera. When the cumulative pixel difference between two captured images exceeds a preset threshold, it is determined that the target object has moved.

[0145] This embodiment provides a convenient and quick way to determine whether the target object has moved, thus optimizing the projection effect.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A mixed reality augmented display system, characterized by, Comprise: a structured light module, a processing module; the structured light module comprises a camera, a projector; the processing module is configured to: acquire a target object point cloud collected by the structured light module; register the target object point cloud with a target object model to obtain a conversion relationship between a structured light coordinate system and a model coordinate system, and unify the structured light module and the target object model to a same coordinate system according to the conversion relationship; wherein the target object model contains a to-be-projected model; the target object model is a three-dimensional patch model corresponding to a target object, and the to-be-projected model corresponds to a to-be-projected object in the target object model; input the to-be-projected model in the same coordinate system and a pose parameter of the projector into an image shooting model to generate a to-be-projected image; the image shooting model is used to shoot the to-be-projected model at the pose of the projector to generate the to-be-projected image; send the to-be-projected image to the structured light module to project the target object through the projector.

2. The mixed reality augmented display system of claim 1, wherein, Further comprise a fixing module for fixing the structured light module at a required position; the fixing module can adopt any one of the following structures: a fixed connection structure, a mounting structure matched with a mounting position.

3. The mixed reality augmented display system of claim 2, wherein, The system comprises multiple sets of the fixing module and the structured light module, and can realize multi-angle point cloud projection.

4. A mixed reality augmented display system, comprising: a structured light module, a processing module, and a mechanical arm; the structured light module comprises a camera and a projector; the structured light module is mounted on or integrated with the mechanical arm; the processing module is configured to: unify the structured light module and a target object model to a same coordinate system according to a known conversion relationship between a mechanical arm coordinate system and a model coordinate system and a current mechanical arm pose; wherein the target object model contains a to-be-projected model; the target object model is a three-dimensional patch model corresponding to a target object, and the to-be-projected model corresponds to a to-be-projected object in the target object model; input the to-be-projected model in the same coordinate system and a pose parameter of the projector into an image shooting model to generate a to-be-projected image; the image shooting model is used to shoot the to-be-projected model at the pose of the projector to generate the to-be-projected image; send the to-be-projected image to the structured light module to project the target object through the projector.

5. The mixed reality augmented display system of claim 4, wherein, The processing module is further configured to: control the mechanical arm to drive the structured light module to register to obtain a first conversion relationship between a mechanical arm coordinate system and a model coordinate system; determine a second conversion relationship between the mechanical arm coordinate system and a structured light coordinate system according to a current mechanical arm pose; unify the structured light module and the target object model to a same coordinate system according to the first conversion relationship and the second conversion relationship.

6. The mixed reality augmented display system of claim 1 or 3, wherein, The pose parameter of the projector in the same coordinate system comprises: coordinates of the projector in the same coordinate system, and a pitch angle and a roll angle of the projector.

7. The mixed reality augmented display system of claim 1 or 3, wherein, The processing module is further configured to: input intrinsic parameters of the projector, the to-be-projected model, and the pose parameter of the projector into the image shooting model to generate the to-be-projected image.

8. The mixed reality augmented display system of claim 1 or 3, wherein, The target object model is a patch model with color and / or texture.

9. The mixed reality augmented display system of claim 1 or 3, wherein, The image shooting model is a callable interface generated based on a three-dimensional model processing framework.

10. A mixed reality augmented display method, characterized by, The method comprises the following steps: acquiring a target object point cloud collected by a structured light module; wherein the structured light module comprises a camera and a projector; registering the target object point cloud with a target object model to obtain a conversion relationship between a structured light coordinate system and a model coordinate system, and unifying the structured light module and the target object model to a same coordinate system according to the conversion relationship; wherein the target object model contains a to-be-projected model; the target object model is a three-dimensional patch model corresponding to a target object, and the to-be-projected model corresponds to a to-be-projected object in the target object model; inputting the to-be-projected model and pose parameters of the projector in the same coordinate system into an image shooting model to generate a to-be-projected image; the image shooting model is used to shoot the to-be-projected model at the pose of the projector to generate the to-be-projected image; sending the to-be-projected image to the structured light module to project the to-be-projected image onto the target object by the projector.

11. The mixed-reality augmented display method of claim 10, wherein, Further comprising: performing an alarm action when the target object is detected to move; and / or, re-executing the steps of acquiring the target object point cloud collected by the structured light module to the step of sending the to-be-projected image to the structured light module to project the to-be-projected image onto the target object by the projector when the target object is detected to move.

12. A mixed reality augmented display method, characterized by, The method comprises the following steps: unifying the structured light module and the target object model to a same coordinate system according to a known conversion relationship between a mechanical arm coordinate system and a model coordinate system and a current mechanical arm pose; wherein the structured light module comprises a camera and a projector, the structured light module is installed or integrated at the end of the mechanical arm, and the target object model contains a to-be-projected model; the target object model is a three-dimensional patch model corresponding to a target object, and the to-be-projected model corresponds to a to-be-projected object in the target object model; inputting the to-be-projected model and pose parameters of the projector in the same coordinate system into an image shooting model to generate a to-be-projected image; the image shooting model is used to shoot the to-be-projected model at the pose of the projector to generate the to-be-projected image; sending the to-be-projected image to the structured light module to project the to-be-projected image onto the target object by the projector.

13. The mixed-reality augmented display method of claim 12, wherein, Further comprising: performing an alarm action when the target object is detected to move; and / or, controlling the mechanical arm to drive the structured light module to register, acquire a first conversion relationship between a mechanical arm coordinate system and a model coordinate system, and re-execute the steps of unifying the structured light module and the target object model to the same coordinate system according to the current mechanical arm pose to the step of sending the to-be-projected image to the structured light module to project the to-be-projected image onto the target object by the projector when the target object is detected to move.

14. The mixed-reality augmented display method according to claim 11 or 13, characterized by, The target object is detected to move in the following way: periodically collecting images by the camera, and determining that the target object moves when the cumulative pixel difference between two collected images is greater than a preset threshold.

Images