A medical information projection system and method

Abstract

The application provides a medical information projection system and method, which comprises a structured light module, a processing module; the structured light module comprises a camera and a projector; the processing module is in communication connection with the structured light module; each module of the medical information projection system cooperates to realize that the structured light module collects surface point cloud of a target object; the processing module generates a to-be-projected image according to model point cloud and the surface point cloud of the target object; and the projector projects the to-be-projected image to the surface of the target object. The medical information projection system of the application collects the surface point cloud of the target object through the structured light module, which is used to generate the to-be-projected image in combination with the to-be-projected point cloud in the model point cloud, and the to-be-projected image is projected to the surface of the target object through the projector of the structured light module, so that efficient and accurate projection is realized, and users can directly observe the medical information on the surface of the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to a medical information projection system and method. BACKGROUND

[0002] Currently, when a treatment plan is formulated for a patient, the surgical plan is usually discussed and planned in a three-dimensional model established based on preoperative images. The display field in the screen is limited, and there is a visual difference between the screen and the real patient, so the doctor's experience is not intuitive enough. In addition, during the operation, the surgeon is usually guided by a surgical navigation positioning system. For example, the surgical instrument is displayed in the three-dimensional model to guide the doctor to adjust the position of the surgical instrument. Under this guidance mode, the doctor needs to constantly switch the line of sight between the surgical site and the screen, which leads to incoherent surgical operation and easy fatigue.

[0003] In view of the above defects of the prior art, the present application provides a medical information projection system to solve or at least partially solve the above defects of the prior art. SUMMARY

[0004] The present application provides a medical information projection system to solve the defect that the doctor's observation of the three-dimensional model is not intuitive enough and needs to be repeatedly switched between the surgical site and the screen, which easily leads to incoherent surgical operation.

[0005] The present application provides a medical information projection system, comprising: a structured light module, a processing module;

[0006] The structured light module comprises a camera and a projector;

[0007] The processing module is in communication connection with the structured light module;

[0008] The modules of the medical information projection system cooperate to realize: the structured light module collects the surface point cloud of the target object; the processing module generates a to-be-projected image according to a model point cloud and the surface point cloud of the target object, wherein the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud contains the to-be-projected point cloud; and the projector projects the to-be-projected image to the surface of the target object.

[0009] According to the medical information projection system provided by the present application, the system further comprises a fixing device, which is a fixed connection structure or an installation structure matched with the installation position, for fixing the structured light module at a required position;

[0010] Alternatively, the system further comprises a mechanical arm in communication connection with the processing module, for moving the structured light module to a required position according to the control instruction of the processing module.

[0011] According to the medical information projection system provided by the application, the processing module comprises a memory and a processor, the processor loads a computer program stored in the memory to realize the following steps:

[0012] S11, obtaining a surface point cloud of the target object, wherein the surface point cloud of the target object is collected by the structured light module;

[0013] S12, generating a first landing point cloud corresponding to the to-be-projected point cloud on the surface of the target object according to the model point cloud and the position of the structured light module;

[0014] S13, generating a to-be-projected image according to the first landing point cloud, in combination with the surface point cloud of the target object and the collection process of the surface point cloud of the target object;

[0015] S14, sending the to-be-projected image to the projector, so that the projector projects the to-be-projected image to the surface of the target object.

[0016] According to the medical information projection system provided by the application, the to-be-projected point cloud is obtained by the following method:

[0017] Obtaining a to-be-projected region, wherein the to-be-projected region is a non-surface region of the target object;

[0018] Extracting the surface points of the to-be-projected region to obtain the to-be-projected point cloud.

[0019] According to the medical information projection system provided by the application, the S12 comprises:

[0020] S121, judging the type of the to-be-projected point cloud;

[0021] In the case that the to-be-projected point cloud is the surface point cloud of the target object, the following operation is performed:

[0022] S122, determining the to-be-projected point cloud as the first landing point cloud;

[0023] S123, unifying the model point cloud and the structured light module to the same coordinate system.

[0024] According to the medical information projection system provided by the application, the S12 comprises:

[0025] S121, judging the type of the to-be-projected point cloud;

[0026] In the case that the to-be-projected point cloud is the non-surface point cloud of the target object, the following operation is performed:

[0027] S125, unify the model point cloud and the structured light module to the same coordinate system;

[0028] S126, process the model point cloud in combination with the position of the projector in the same coordinate system to obtain a first landing point cloud corresponding to the to-be-projected point cloud on the surface of the target object.

[0029] According to the medical information projection system provided by the application, the S125 comprises:

[0030] Register the surface point cloud of the target object with the model point cloud to obtain the conversion relationship between the model point cloud coordinate system and the structured light coordinate system.

[0031] According to the conversion relationship between the model point cloud coordinate system and the structured light coordinate system, the model point cloud and the structured light module are converted to the same coordinate system.

[0032] According to the medical information projection system provided by the application, the S11 comprises: controlling a mechanical arm to drive the structured light module to collect a structured light point cloud of the target object; registering the structured light point cloud with the model point cloud to obtain a conversion relationship between a model point cloud coordinate system and a mechanical arm coordinate system; wherein the mechanical arm is installed or integrated with the structured light module.

[0033] Accordingly, the S125 comprises: according to the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system and the current projection posture of the mechanical arm, the model point cloud and the structured light module are converted to the same coordinate system.

[0034] According to the medical information projection system provided by the application, the S126 comprises:

[0035] According to the position of the projector in the same coordinate system, the projector is abstracted as a point, and each point in the to-be-projected point cloud in the same coordinate system is connected with the projector, and the intersection of each connecting line and the surface curve of the target object constitutes the first landing point cloud.

[0036] According to the medical information projection system provided by the application, the S13 comprises:

[0037] According to the first landing point cloud, a second landing point cloud corresponding to the region where the first landing point cloud is located is determined from the surface point cloud of the target object.

[0038] According to the second landing point cloud, in combination with the collection process of the surface point cloud of the target object, the to-be-projected image is generated.

[0039] According to a medical information projection system provided by the present invention, the step of determining a second landing point cloud corresponding to the region where the first landing point cloud is located from the surface point cloud of the target object based on the first landing point cloud includes:

[0040] A search tree is established for the surface point cloud of the target object. For each point in the first landing point cloud, the nearest point is searched in the search tree, and the nearest points constitute the second landing point cloud.

[0041] According to a medical information projection system provided by the present invention, the step of generating the image to be projected based on the second landing point cloud and the acquisition process of the surface point cloud of the target object includes:

[0042] Generate a blank two-dimensional image with the same resolution as the two-dimensional pattern projected during the acquisition of the surface point cloud of the target object;

[0043] For each point in the second landing point cloud, the coordinates of the corresponding two-dimensional point in the second landing point cloud are determined according to the correspondence between the pixels in the two-dimensional pattern projected during the acquisition of the surface point cloud of the target object and the generated three-dimensional points, and added to the blank two-dimensional map to finally obtain the image to be projected.

[0044] According to a medical information projection system provided by the present invention, step S13 includes:

[0045] When a point needs to be projected, the point in the image to be projected is highlighted; and / or,

[0046] When the projection requires an outline, connect the outer points in the image to be projected; and / or,

[0047] When the area to be projected is a region, morphological processing is performed on the image to be projected.

[0048] According to the present invention, a medical information projection system includes multiple structured light modules to achieve multi-angle medical information projection.

[0049] The present invention also provides a medical information projection system, comprising: a structured light module, a processing module, and a fixing device;

[0050] The structured light module includes a camera and a projector;

[0051] The processing module is communicatively connected to the structured light module;

[0052] The fixing device is used to fix the structured light module in the desired position;

[0053] The modules of the medical information projection system cooperate to realize that the structured light module collects surface point cloud of a target object; the processing module generates a to-be-projected image according to model point cloud and the surface point cloud of the target object, wherein the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud contains the to-be-projected point cloud; and the projector projects the to-be-projected image to the surface of the target object.

[0054] The application further provides a medical information projection system, comprising a structured light module, a processing module and a mechanical arm.

[0055] The structured light module comprises a camera and a projector.

[0056] The processing module is in communication connection with the structured light module.

[0057] The mechanical arm is in communication connection with the processing module, and is used for driving the structured light module to move to a required position according to a control instruction of the processing module.

[0058] The modules of the medical information projection system cooperate to realize that the structured light module collects surface point cloud of a target object; the processing module generates a to-be-projected image according to model point cloud and the surface point cloud of the target object, wherein the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud contains the to-be-projected point cloud; and the projector projects the to-be-projected image to the surface of the target object.

[0059] According to the medical information projection system provided by the application, the modules of the medical information projection system further cooperate to realize that:

[0060] The processing module calculates the projection posture of the mechanical arm according to the projection posture of the structured light module input by a user, and controls the mechanical arm to move to the projection posture.

[0061] The structured light module collects point cloud again in the projection posture.

[0062] The processing module generates a new to-be-projected image according to the point cloud collected again and the projection posture of the structured light module, and sends the new to-be-projected image to the structured light module.

[0063] The structured light module projects the new to-be-projected image to the target object through the projector.

[0064] According to the medical information projection system provided by the application, the modules of the medical information projection system further cooperate to realize that:

[0065] The projection posture of the structured light module is determined according to the adjusted posture of the mechanical arm in the following mode.

[0066] The structured light module reacquires the point cloud at the projection pose again;

[0067] The processing module generates a new to-be-projected image according to the reacquired point cloud and the projection pose of the structured light module, and sends the new to-be-projected image to the structured light module;

[0068] The structured light module projects the new to-be-projected image to the target object through the projector.

[0069] The application further provides a medical information projection method, comprising:

[0070] S11, acquiring a surface point cloud of a target object; wherein the surface point cloud of the target object is acquired by a structured light module, and the structured light module comprises a camera and a projector;

[0071] S12, generating a first landing point point cloud corresponding to a to-be-projected point cloud on a surface of a target object according to a model point cloud and a position of the structured light module; wherein the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud comprises the to-be-projected point cloud;

[0072] S13, generating a to-be-projected image according to the first landing point point cloud, in combination with the surface point cloud of the target object and a surface point cloud acquisition process of the target object;

[0073] S14, sending the to-be-projected image to the projector, so that the projector projects the to-be-projected image to the surface of the target object.

[0074] The application provides a medical information projection system and method, which have at least one of the following beneficial effects:

[0075] 1. The medical information (internal or surface) to be observed can be projected to the surface of the target object, so that the user does not need to switch the line of sight between the concerned part and the screen, and the visual field range is wider, which is convenient for intuitive research, teaching, surgical planning, surgical navigation and the like;

[0076] 2. The structured light module has the dual functions of acquiring the point cloud and projecting the medical information, the processing module processes data to generate the projection image, the whole projection system has a simple structure and low cost;

[0077] 3. The patient registration based on the structured light module has high registration accuracy, and the projection is more accurate;

[0078] 4. In the same coordinate system, the model point cloud and the target object surface point cloud collected by the structured light module are combined to generate the image to be projected, without complex calculation according to the projector parameters (external parameters, internal parameters), without detecting whether the projection landing point meets the expectation through the tracking device, without time-consuming adjustment of the projector angle and correction of the projection effect, and the projection efficiency and projection accuracy are improved. BRIEF DESCRIPTION OF DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the present application or prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0080] Figure 1 is a structural schematic diagram of a medical information projection system provided by the present application;

[0081] Figure 2 is a structural schematic diagram of a medical information projection system provided by the present application;

[0082] Figure 3 is a structural schematic diagram of a medical information projection system provided by the present application;

[0083] Figure 4 is a process schematic diagram of a medical information projection system provided by the present application to generate a first landing point cloud;

[0084] Figure 5 is a schematic diagram of the process of collecting the surface point cloud of the target object;

[0085] Figure 6 is a flow schematic diagram of a medical information projection method provided by the present application;

[0086] Figure 7 is a schematic diagram of the projection effect of a point cloud projection method provided by the present application;

[0087] Figure 8 is a schematic diagram of the projection effect of a point cloud projection method provided by the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions in the present application will be described clearly and completely in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0089] The application provides a medical information projection system and method. Figures 1-8 The application provides a medical information projection system and method.

[0090] Figure 1 Figure 1 is a structural diagram of a medical information projection system according to an embodiment of the application. As shown in the figure, the system comprises a structured light module 100, a processing module 200, and a display module 300.

[0091] The structured light module 100 comprises a camera 110 and a projector 120. The structured light module can collect three-dimensional point clouds based on the structured light principle. Structured light point cloud collection is an active point cloud collection technology, which has the advantages of non-contact, large field of view, high precision (large amount of point cloud data), and good real-time performance. In the process of point cloud collection, the structured light module 100 can project an encoded two-dimensional pattern onto a target object through the projector 120. The pattern will be deformed due to the inconsistency of the object surface height. The camera 110 captures the deformed pattern, and then decodes the pattern to obtain the correspondence between the object surface and the projection pattern points. Based on the principle of triangulation, the three-dimensional point cloud of the surface of the target object is calculated.

[0092] In the application, the structured light module 100 can be directly placed in a fixed position, for example, on a platform beside a hospital bed. The structured light module 100 can also be installed or fixed in a position, for example, suspended above a hospital bed, installed on a hospital bed, fixed and supported on the ground through a support structure, or integrated or installed on other devices, such as a cart of a surgical navigation system. The application does not limit the placement / fixed manner of the structured light module 100. It can be understood that the structured light module 100 collects visual surface point cloud data of the target object, for example, facial skin data, or tissue data in an exposed state during a craniotomy. According to the position of the structured light module 100 in space, the position of the target object in the real space can be determined, for example, the surface point cloud of the target object is corresponded to a structured light coordinate system, a hospital bed coordinate system, a surgical cart coordinate system, a mechanical arm coordinate system in which the structured light module is installed, and the like.

[0093] The model point cloud is a three-dimensional point cloud corresponding to the target object, for example, a three-dimensional point cloud generated based on medical images (magnetic resonance images, functional magnetic resonance images, CT images, phase contrast magnetic resonance angiography, etc.) of the target object, or a three-dimensional point cloud generated based on a corresponding three-dimensional design model when the target object is a physical demonstration model. The model point cloud contains the point cloud to be projected (i.e., contains medical information of interest to the doctor), such as puncture points, dissection lines, tumors, blood vessels, brain regions, and the like.

[0094] The processing module 200 obtains the surface point cloud of the target object from the structured light module 100, generates a two-dimensional image (i.e., a to-be-projected image) to be projected by the projector 120 under the perspective of the projector 120 in combination with the model point cloud, and then projects the to-be-projected image to the surface of the target object through the projector 120. The target object can be a patient, and the medical information projection can provide intuitive, accurate and efficient surgical navigation for doctors. The target object can also be a physical demonstration model and the like, which facilitates related seminars, teaching, surgical planning and the like.

[0095] The processing module 200 interacts with the structured light module 100 for data processing. The processing module 200 can be integrated with the aforementioned structured light module 100, or can be an external independent device, such as a notebook computer, a workstation, a PDA (personal digital assistant) and the like. The present application does not limit the existence form, hardware form and spatial position of the processing module 200.

[0096] The medical information projection system of the embodiment collects the surface point cloud of the target object through the structured light module, determines the relative position relationship between the target object and the structured light module, determines the position of the to-be-projected point cloud in combination with the to-be-projected point cloud in the model point cloud, generates a two-dimensional image corresponding to the to-be-projected point cloud in combination with the surface point cloud of the target object, and projects the to-be-projected image to the surface of the target object through the projector of the structured light module, thereby realizing efficient and accurate projection and facilitating users to directly observe the medical information on the surface of the target object.

[0097] Figure 7 is one of the projection effect schematic diagrams of the point cloud projection method of the present application, Figure 7 illustrates the projection effect of projecting the position of the tumor region to the head of the patient; Figure 8 is another projection effect schematic diagram of the point cloud projection method provided by the present application, Figure 8 illustrates the projection effect of projecting the tumor and its surrounding tissue to the head of the patient (different tissues can be rendered in different colors, Figure 7 , Figure 8 is a gray-scale image, and the color effect is missing.

[0098] The projector 120 can cooperate with the camera 110 to collect the point cloud of the target object, and can also project the to-be-projected point cloud to the target object. The present application conveniently and accurately generates the to-be-projected image under the perspective of the projector 120 based on the dual role of the projector 120, avoids the complex process of considering the three-dimensional shape of the to-be-projected position (the surface of the target object), the perspective of the projector, and the parameters (internal and external parameters) of the projector, and improves the processing efficiency and accuracy.

[0099] Based on the above embodiment, in one embodiment, the system further comprises a fixing device, which is a fixed connection structure or a mounting structure matched with the mounting position, for fixing the structured light module 100 at the required position.

[0100] Specifically, the fixing device can be a fixed connection structure, one end of which is connected to the structured light module, and the other end is connected or supported to the ground, wall, ceiling or other equipment position, Figure 2 is a structural diagram of a medical information projection system provided by the present application, Figure 2 which shows the state of supporting the structured light module 100 on the ground by the fixing device 300, at this time, the fixed connection structure is, for example, a tripod, a support rod, a folding bracket and the like, in addition, the structured light module 100 can also be hung at the required position by the fixed connection structure, for example, hung at the wall, ceiling and the like so that the structured light module 100 is located above the hospital bed, further, the suspension structure adopts a universal arm, which is connected with the structured light module 100 through a ball head, so that the adjustment and fixation of the position of the structured light module 100 can be realized, and the fixing device can also be a mounting structure matched with the mounting position, for example, a mounting structure matched with the mounting position on the hospital bed.

[0101] In this embodiment, the structured light module is fixed at the required position by the fixing device, which simply and low-costly constructs the medical information projection system and reduces the use cost.

[0102] Further, based on the above embodiment, in one embodiment, the system further comprises an adjusting structure for adjusting the orientation of the structured light module 100.

[0103] Specifically, after the structured light module 100 is fixed at the required position, it can not be aligned with the to-be-projected position on the target object, and the orientation of the structured light module 100 is adjusted by the adjusting structure, so that the structured light module 100 can smoothly collect the surface data of the target object and project the point cloud to the surface of the target object. The adjusting structure can include a first adjusting component and a second adjusting component, the first adjusting component is used to adjust the horizontal angle of the structured light module 100, and the second adjusting component is used to adjust the pitch angle of the structured light module.

[0104] In this embodiment, the orientation of the structured light module is conveniently adjusted by the adjusting structure, so as to project the point cloud to the target object.

[0105] Based on any of the above embodiments, in one embodiment, the system further comprises a mechanical arm, which is in communication connection with the processing module, and is used to drive the structured light module to move to the required position according to the control instruction of the processing module.

[0106] Specifically, Figure 3is a structural schematic view of a medical information projection system provided by the present application, Figure 3 The mechanical arm 400 is provided with the structured light module 100, and the structured light module 100 is installed or integrated on the mechanical arm 400, for example, installed or integrated on the end joint arm of the mechanical arm 400, and then the mechanical arm 400 drives the structured light module 100 to move to a suitable position angle according to the control instruction of the processing module 200, so as to collect the surface point cloud data of the target object at the position angle and project the point cloud to the surface of the target object.

[0107] In this embodiment, the structured light module is driven by the mechanical arm to move to the required position for point cloud collection and medical information projection, and the projection position and projection angle are more flexible. In addition, the system is compatible with the surgical navigation positioning system equipped with the mechanical arm, and the mechanical arm can be used for assisting medical information projection and assisting positioning, thereby reducing the use cost, and the medical information projection process can be combined with the patient registration of the surgical navigation positioning to a certain extent, thereby simplifying the operation process of the user.

[0108] Based on any of the above embodiments, in one embodiment, the processing module 200 comprises a memory and a processor, and the processor loads a computer program stored in the memory to realize the following functions:

[0109] S11, acquiring surface point cloud of a target object; wherein the surface point cloud of the target object is acquired by the structured light module 100;

[0110] Specifically, there are various ways to acquire the surface point cloud of the target object. For example, the system performs patient registration in advance, that is, the structured light module 100 collects the surface point cloud data of the target object, which is used to establish the corresponding relationship between the patient space and the preoperative medical image space. At this time, the processing module 200 can directly acquire the surface point cloud of the target object collected by the structured light module 100 in the patient registration process, that is, reuse the previously collected point cloud, and there is no need to repeatedly collect the point cloud (correspondingly, the structured light module 100 needs to keep the same pose as when the point cloud is collected during medical information projection). For another example, in each medical information projection process, the structured light module 100 collects the point cloud data of the target object at the current projection position, which is used to process in combination with the model point cloud to generate the to-be-projected image. The point cloud collected by the structured light module 100 can be used for registration with the model point cloud, so as to determine the conversion relationship between the structured light coordinate system and the preoperative image coordinate system.

[0111] S12, generating a first landing point point cloud corresponding to the to-be-projected point cloud on the surface of the target object according to the model point cloud and the position of the structured light module 100;

[0112] Specifically, the model point cloud and the surface point cloud of the target object are unified to the same coordinate system, and the relative position of the surface point cloud of the target object and the structured light module 100 is known, so that the to-be-projected point cloud in the model point cloud can be "observed" from the perspective of the structured light module 100 (of the projector 120), and the landing point of the to-be-projected point cloud on the surface of the target object is determined, that is, the first landing point cloud is obtained.

[0113] S13, generating a to-be-projected image according to the first landing point cloud, in combination with the surface point cloud of the target object and the acquisition process of the surface point cloud of the target object;

[0114] Specifically, the surface point cloud of the target object is acquired based on structured light, and the three-dimensional point cloud of the surface of the target object has a corresponding relationship with the two-dimensional image projected in the structured light point cloud acquisition process. According to the "projection result" of the first landing point cloud in combination with the structured light point cloud acquisition process, the corresponding "projection image" (that is, the to-be-projected image) can be inversely deduced.

[0115] S14, sending the to-be-projected image to the projector 120, so that the projector 120 projects the to-be-projected image to the surface of the target object.

[0116] Specifically, the processing module 200 sends the to-be-projected image to the projector 120, and the projector 120 does not need to change the projection parameters (projection direction, projection range, resolution and the like), but can accurately project the to-be-projected image to the surface of the target object by using the projection parameters in the structured light point cloud acquisition process, so that the medical information can be intuitively seen in the real space.

[0117] In the embodiment, the processing module loads and runs the computer program in the memory through the processor to process the surface point cloud of the target object and the model point cloud, and efficiently and accurately generates the to-be-projected image.

[0118] Based on any of the above embodiments, in an embodiment, the to-be-projected point cloud is acquired by the following method:

[0119] Acquiring a to-be-projected region; wherein the to-be-projected region is a non-surface region of the target object;

[0120] Extracting the surface points of the to-be-projected region to obtain the to-be-projected point cloud.

[0121] Specifically, a three-dimensional model is established in advance based on medical image data of a target object, such as magnetic resonance image, functional magnetic resonance image, CT image, phase-contrast magnetic resonance angiography (PC-MRA), and the like, a user delineates a region to be projected (for example, a needle insertion point) in the three-dimensional model, or automatically extracts the region to be projected (for example, extracts a blood vessel by a deep learning model, extracts a tumor by a segmentation algorithm), or the three-dimensional model itself contains the region to be projected. The processing module 200 acquires the region to be projected and extracts surface points of the region to be projected, and only the surface points of the region to be projected are processed in subsequent data processing, without processing internal points of the region to be projected, thereby reducing the amount of data processing.

[0122] It can be understood that the model point cloud is a three-dimensional point cloud extracted based on the three-dimensional model, and the timing of the process of "extracting surface points of the region to be projected in the model point cloud" can be flexibly set, for example, the process is performed in a model point cloud coordinate system, or the model point cloud and the structured light module are unified into the same coordinate system and the process is performed in the same coordinate system. In addition, the region to be projected in the processing in this embodiment is a non-surface region of the target object. For the case where the region to be projected is a surface region of the target object, the region to be projected itself is point cloud data (for example, a needle insertion point or an anatomical line), and the point cloud to be projected can be directly obtained without extracting surface points. Further, the region to be projected can be projected onto a skin surface to determine the point cloud to be projected, for example, a user takes a needle insertion point in a three-dimensional model. Due to the screen display effect, the needle insertion point taken by the user may not fall on the skin surface, and the needle insertion point needs to be projected onto the surface of the target object to obtain an accurate point cloud to be projected.

[0123] The embodiment extracts surface points of the region to be projected to obtain the point cloud to be projected, thereby reducing the amount of data processing and improving the efficiency of medical information projection.

[0124] Based on any of the above embodiments, in one embodiment, the S12 comprises:

[0125] S121, determining the type of the point cloud to be projected;

[0126] In the case where the point cloud to be projected is a surface point cloud of the target object, the following operation is performed:

[0127] S122, determining the point cloud to be projected as the first landing point cloud;

[0128] S123, unifying the model point cloud and the structured light module 100 into the same coordinate system.

[0129] Specifically, there are different data processing procedures according to different projection data types, and the processing module 200 first judges the type of the to-be-projected point cloud. For example, when the user selects a drawing type (such as an anatomical line, a needle insertion point, a tumor, a blood vessel, a brain region, etc.) when drawing the to-be-projected region, the generated to-be-projected point cloud contains a corresponding drawing type identifier, and the processing module 200 judges the type of the to-be-projected point cloud according to the drawing type identifier. For another example, the processing module 200 performs feature recognition (for example, the number of three-dimensional points contained in the point cloud, the distance between the three-dimensional points and the surface of the target object, the geometric features of the to-be-projected point cloud) on the to-be-projected point cloud or the data drawn by the user, so as to judge whether the to-be-projected point cloud is a surface point cloud or a non-surface point cloud.

[0130] If it is judged that the to-be-projected point cloud is a surface point cloud of the target object, the to-be-projected point cloud is directly determined as the first landing point cloud, and the model point cloud containing the to-be-projected point cloud is unified with the structured light module 100 in the same coordinate system, for example, in the structured light coordinate system / preoperative image coordinate system, for another example, in the bed coordinate system / surgical trolley coordinate system / coordinate system of the mechanical arm on which the structured light module 100 is installed, etc.

[0131] In the embodiment, in the case where it is judged that the to-be-projected point cloud is a surface point cloud, the to-be-projected point cloud is directly determined as the first landing point cloud, and the model point cloud is unified with the structured light module in the same coordinate system, which is convenient for processing to generate the to-be-projected image.

[0132] Based on any of the above embodiments, in one embodiment, the S12 comprises:

[0133] S121, judging the type of the to-be-projected point cloud;

[0134] In the case where the to-be-projected point cloud is a non-surface point cloud of the target object, the following operations are performed:

[0135] S125, unifying the model point cloud with the structured light module 100 in the same coordinate system;

[0136] S126, processing the model point cloud in combination with the position of the projector 120 in the same coordinate system to obtain a first landing point cloud corresponding to the to-be-projected point cloud on the surface of the target object.

[0137] Specifically, the process of judging the point cloud projection type can refer to the previous embodiment, which will not be described here. The processing module 200 unifies the model point cloud and the structured light module 100 to the same coordinate system, such as the structured light coordinate system / preoperative image coordinate system, for example, in the case of judging that the point cloud to be projected is the non-surface point cloud of the target object. The bed coordinate system / surgical trolley coordinate system / coordinate system of the mechanical arm installing the structured light module 100, and so on. Further, the point cloud to be projected in the model point cloud can be "observed" from the perspective of the structured light module 100 (of the projector 120), and the landing point of the point cloud to be projected on the surface of the target object is determined, that is, the first landing point cloud of the point cloud to be projected on the surface of the target object is obtained.

[0138] In the embodiment, in the case of judging that the point cloud to be projected is the non-surface point cloud, the model point cloud and the structured light module are unified to the same coordinate system for processing to generate the first landing point cloud of the point cloud to be projected on the surface of the target object from the perspective of the structured light module, which is convenient for generating the projection image of the element to be projected of the target object.

[0139] Based on any of the above embodiments, in one embodiment, the S125 comprises:

[0140] Registering the surface point cloud of the target object and the model point cloud to obtain the conversion relationship between the model point cloud coordinate system and the structured light coordinate system;

[0141] According to the conversion relationship between the model point cloud coordinate system and the structured light coordinate system, the model point cloud and the structured light module 100 are converted to the same coordinate system.

[0142] Specifically, the surface point cloud of the target object and the model point cloud both contain the surface information of the target object. By registration, the surfaces of the two are made to coincide as much as possible to obtain the conversion relationship between the model point cloud coordinate system and the structured light coordinate system. The position of the structured light module 100 in the structured light coordinate system is known. According to the registration relationship, the model point cloud and the structured light module 200 are converted to the same coordinate system, such as the preoperative image coordinate system / structured light coordinate system, for example, further converted to the bed coordinate system / surgical trolley coordinate system / coordinate system of the mechanical arm installing the structured light module 100, and so on.

[0143] In the embodiment, the model point cloud and the structured light module are converted to the same coordinate system by registration, which is convenient for processing and generating the first landing point cloud in the same coordinate system.

[0144] Based on any of the above embodiments, in one embodiment,

[0145] The S11 further includes: controlling the mechanical arm to drive the structured light module 100 to collect a structured light point cloud of the target object; and registering the structured light point cloud and the model point cloud to obtain a conversion relationship between a model point cloud coordinate system and a mechanical arm coordinate system; wherein the mechanical arm is mounted or integrated with the structured light module.

[0146] Correspondingly, the S125 includes: converting the model point cloud and the structured light module to the same coordinate system according to the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system and a current projection pose of the mechanical arm.

[0147] Specifically, in the embodiment, the structured light module 100 is mounted or integrated on the mechanical arm, and before projection, patient registration is further performed based on structured light: the processing module 200 controls the mechanical arm to drive the structured light module to collect a structured light point cloud at one or more poses, and registers the structured light point cloud and the model point cloud to obtain a conversion relationship between a model point cloud coordinate system and a mechanical arm coordinate system.

[0148] It can be understood that if the structured light point cloud is collected at a single pose, the structured light point cloud in the structured light coordinate system can be registered with the model point cloud to obtain the conversion relationship between the structured light coordinate system and the model point cloud coordinate system, and then the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system can be further obtained in combination with the mechanical arm pose. If the structured light point cloud is collected at a single pose, the structured light point cloud in the structured light coordinate system can also be converted to the mechanical arm coordinate according to the mechanical arm pose, and then the structured light point cloud in the mechanical arm coordinate is registered with the model point cloud to obtain the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system. If the structured light point cloud is collected at multiple poses, the structured light point cloud needs to be fused in the mechanical arm coordinate to obtain the structured light point cloud, and then the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system is obtained by registering the model point cloud.

[0149] The patient registration process described above only needs to be performed once, and when projection is performed subsequently (single-angle projection, multiple times, or multi-angle projection), the conversion relationship between the current structured light coordinate system and the mechanical arm coordinate system can be determined according to the current projection pose of the mechanical arm, and the model point cloud and the structured light module can be converted to the same coordinate system in combination with the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system, so that the structured light module can be projected at the projection pose, and the position of the structured light module 100 in the projection process does not have to be consistent with the position when the structured light point cloud is collected in the patient registration, and can be flexibly adjusted.

[0150] The structural light module can be flexibly adjusted to the required point cloud collection and point cloud projection pose by the mechanical arm driving structure light module in the embodiment, and only one patient registration can be performed, and subsequent multiple and multi-angle point cloud projection can be performed without repeated registration to determine the conversion relationship between the current structural light coordinate system and the model point cloud coordinate system, thereby improving the projection efficiency.

[0151] According to any of the above embodiments, in one embodiment, the S126 comprises:

[0152] According to the position of the projector 120 in the same coordinate system, the projector 120 is abstracted as a point, and each point in the same coordinate system is connected with the to-be-projected point cloud. The intersection of each connection and the surface curve of the target object constitutes the first landing point cloud.

[0153] Specifically, the model point cloud and the structural light module 100 are unified to the same coordinate system, and the model point cloud includes the to-be-projected point cloud inside the target object and the surface curve of the target object. Figure 4 is one of the process diagrams of the medical information projection system for generating the first landing point cloud, referring to Figure 4 The structural light module 100 includes the projector 120, and the projector 120 is abstracted as a point, for example, the optical center of the projector 120. By connecting the abstracted point of the projector with each point in the to-be-projected point cloud (the connection is the projection path), the intersection of the connection and the surface curve of the target object constitutes the first landing point cloud.

[0154] In the embodiment, the corresponding landing points of the to-be-projected point cloud on the surface of the target object under the projection angle of the projector are accurately determined according to the position of the projector and the model point cloud.

[0155] According to any of the above embodiments, in one embodiment, the S13 comprises:

[0156] According to the first landing point cloud, the second landing point cloud corresponding to the region of the first landing point cloud is determined from the surface point cloud of the target object;

[0157] According to the second landing point cloud, the to-be-projected image is generated in combination with the collection process of the surface point cloud of the target object.

[0158] Specifically, the pixel points in the first landing point cloud cannot be directly corresponding to the pixel points in the surface point cloud of the target object, and the part of the point cloud that is "coincided" with the region range of the first landing point cloud, that is, the second landing point cloud, needs to be determined from the surface point cloud of the target object. Then, in combination with the collection process of the target object point cloud, it is determined which two-dimensional pixel points in the projection image of the projector 120 can correspond to the second landing point cloud, and these two-dimensional pixel points constitute the to-be-projected two-dimensional image. Figure 5is a schematic diagram of a collection process of a surface point cloud of a target object, a projector 120 projects a two-dimensional coded image A onto the surface of the target object, and then a camera 110 captures the target object to obtain a two-dimensional image B. Based on the image A and the image B, a surface point cloud C of the target object can be generated. Exemplarily, a two-dimensional point p' in the image A and a two-dimensional point p'' in the image B correspond to a three-dimensional point p in the surface point cloud C. The second-landing-point point cloud is a part of the surface point cloud of the target object. For each point in the second-landing-point point cloud, a corresponding two-dimensional point can be determined according to the above point cloud collection process, so as to generate a to-be-projected image.

[0159] In this embodiment, the projector is abstracted as a point, and the intersection with the surface of the target object is determined as the first-landing-point point cloud by connecting the to-be-projected point cloud. The second-landing-point point cloud corresponding to the first-landing-point point cloud is determined in the (structured light) surface point cloud of the target object, the landing-point point cloud is accurately determined, the to-be-projected image is efficiently and accurately generated, and the two-dimensional image under the projection visual angle can be generated without complex calculation according to the projector parameters (external parameters and internal parameters), thereby improving the data processing efficiency.

[0160] Based on any of the above embodiments, in one embodiment, the second-landing-point point cloud corresponding to the region where the first-landing-point point cloud is located is determined from the surface point cloud of the target object according to the first-landing-point point cloud, including:

[0161] A search tree is established for the surface point cloud of the target object, and for each point in the first-landing-point point cloud, a nearest neighbor point is searched in the search tree, and each nearest neighbor point constitutes the second-landing-point point cloud.

[0162] Specifically, a search tree (for example, a KD tree, a KDB tree, a BSP tree, an octree, etc.) is established for the surface point cloud of the target object, so as to improve the search efficiency of three-dimensional points. For each point in the first-landing-point point cloud, a nearest neighbor point is searched in the search tree, and the nearest neighbor point corresponding to each point in the first-landing-point point cloud constitutes the second-landing-point point cloud.

[0163] In this embodiment, the search tree is established for the surface point cloud of the target object, and the nearest neighbor point corresponding to each point in the first-landing-point point cloud is searched in the search tree, so as to accurately and efficiently determine the landing point corresponding to the to-be-projected point cloud from the surface point cloud of the target object.

[0164] Based on any of the above embodiments, in one embodiment, the to-be-projected image is generated according to the second-landing-point point cloud in combination with the collection process of the surface point cloud of the target object, including:

[0165] A blank two-dimensional image with a resolution equal to that of the two-dimensional pattern projected in the collection process of the surface point cloud of the target object is generated.

[0166] For each point in the second landing point cloud, according to the correspondence between the pixel points in the two-dimensional pattern projected in the surface point cloud acquisition process of the target object and the generated three-dimensional points, the corresponding two-dimensional point coordinates of the point in the second landing point cloud are determined and added to the blank two-dimensional graph, and finally the to-be-projected image is obtained.

[0167] Specifically, a blank two-dimensional graph with the same resolution as the projected two-dimensional pattern is generated, which facilitates the addition of to-be-projected image pixel points. The second landing point cloud is part of the surface point cloud (collected based on structured light) of the target object. For each point in the second landing point cloud, according to the correspondence between the pixel points in the two-dimensional pattern projected in the point cloud acquisition process and the generated three-dimensional points, the coordinates of the corresponding two-dimensional point in the second landing point cloud are determined and added to the blank two-dimensional graph. The two-dimensional graph obtained after all points are added is the to-be-projected image.

[0168] The embodiment efficiently and accurately generates the to-be-projected image, without the need for complex calculations based on projector parameters (external parameters and internal parameters) to generate a two-dimensional image under the projection visual angle, thereby improving the data processing efficiency.

[0169] Based on any of the above embodiments, in one embodiment, the S13 comprises:

[0170] When a point needs to be projected, the point to be projected in the to-be-projected image is highlighted; and / or,

[0171] When a contour needs to be projected, the peripheral points in the to-be-projected image are connected; and / or,

[0172] When a region needs to be projected, morphological processing is performed on the to-be-projected image.

[0173] Specifically, if a point needs to be projected, such as a needle insertion point, a bone screw implantation point, or other discrete points, the point to be projected in the to-be-projected image is highlighted, such as displaying a cross centered on the point or enlarging the point; if a contour needs to be projected, such as a tumor contour or a brain region contour, the peripheral points in the to-be-projected image are connected to project the peripheral contour to the surface of the target object; if an entire region needs to be projected, morphological processing is performed on the to-be-projected image. Specifically, the first landing point cloud determines the second landing point cloud by searching for the nearest points. This process causes the pixel points in the second landing point cloud to correspond to the blank two-dimensional graph, which are not necessarily fine and continuous in the two-dimensional image. There may be missing pixels in the pixel point region. Image morphological processing can supplement the missing pixels in the region, further improving the point cloud projection effect.

[0174] Based on any of the above embodiments, in one embodiment, the medical information projection system comprises a plurality of structured light modules 100 to realize multi-angle point cloud projection.

[0175] Specifically, the medical information projection system can comprise a structured light module 100, a plurality of structured light modules 100 are respectively connected in communication with the processing module 200, the projection of multiple angles is realized, the projection blind area is avoided, and the point cloud projection effect is improved.

[0176] Next, another medical information projection system provided by the present application is described, and the medical information projection system described below can be mutually corresponding with the above.

[0177] Still referring to Figure 2 , the medical information projection system provided by the present application comprises a structured light module 100, a processing module 200, and a fixing device 300.

[0178] The structured light module 100 comprises a camera 110 and a projector 120.

[0179] The processing module 200 is connected in communication with the structured light module 100.

[0180] The fixing device 300 is used for fixing the structured light module 100 at a required position.

[0181] The modules of the medical information projection system cooperate to realize that the structured light module 100 collects the surface point cloud of the target object; the processing module 200 generates a to-be-projected image according to the model point cloud and the surface point cloud of the target object, wherein the model point cloud comprises a to-be-projected point cloud; and the projector 120 projects the to-be-projected image to the surface of the target object.

[0182] Specifically, the fixing device 300 fixes the structured light module 100 at a required projection position, for example, is hung above a sickbed, is installed on the sickbed, is fixed and supported on the ground through a support structure, the structured light module 100 collects the surface point cloud of the target object at this projection position (when the point cloud is projected later, the projection position is also used), the processing module 200 registers the surface point cloud of the target object with the model point cloud, determines the conversion relationship between the structured light coordinate system and the model point cloud coordinate system, unifies the model point cloud, the target object surface point cloud in the structured light coordinate system, and the structured light module to the same coordinate system, performs data processing in the same coordinate system, generates a to-be-projected image under the “viewing angle” of the projector 120, and sends the to-be-projected image to the projector 120, so that the projector 120 projects the two-dimensional image to the surface of the target object at the projection position.

[0183] The structure light module is fixed at a required position by the fixing device in the embodiment, a medical information projection system is constructed simply and at low cost, and use cost is reduced. The surface point cloud of the target object is collected by the structure light module, which can be used to determine the relative position relationship between the target object and the structure light module, and the position of the to-be-projected point cloud can be determined in combination with the to-be-projected point cloud in the model point cloud, so that a two-dimensional image corresponding to the to-be-projected point cloud can be generated, and the to-be-projected image can be accurately projected to the surface of the target object by the projector of the structure light module, efficient and accurate projection is realized, and a user can directly observe medical information on the surface of the target object.

[0184] Another medical information projection system provided by the present application is described below, and the medical information projection system described below can be mutually correspondingly referred to the medical information projection system described above.

[0185] Still referring to Figure 3 The medical information projection system provided by the present application comprises a structure light module 100, a processing module 200, and a mechanical arm 400.

[0186] The structure light module 100 comprises a camera 110 and a projector 120.

[0187] The processing module 200 is in communication connection with the structure light module 100.

[0188] The mechanical arm 400 is in communication connection with the processing module 200, and is used to drive the structure light module 100 to move to a required position according to a control instruction of the processing module 200.

[0189] The modules of the medical information projection system cooperate to realize that the structure light module 100 collects the surface point cloud of the target object; the processing module 200 generates a to-be-projected image according to a model point cloud and the surface point cloud of the target object, wherein the model point cloud comprises a to-be-projected point cloud; and the projector 120 projects the to-be-projected image to the surface of the target object.

[0190] Specifically, the structure light module 100 is installed on or integrated with the mechanical arm 400, for example, the structure light module 100 is installed on the end flange of the mechanical arm 400, or for example, the structure light module 100 is integrated on a certain joint arm of the mechanical arm 400 without occupying the end flange. The mechanical arm 400 can drive the structure light module to move to a required position according to a control instruction of the processing module 200, and collect the surface point cloud of the target object and project a two-dimensional image to the surface of the target object at the position.

[0191] The point cloud projection process in the embodiment can also be used in cooperation with a patient registration process. Specifically, the structured light module 100 carried by the mechanical arm 400 is used to collect (and fuse) structured light point clouds of the target object at one or more positions, the structured light point clouds are registered with the model point clouds, and the conversion relationship between the mechanical arm coordinate system and the model point cloud coordinate system is determined (that is, the coordinate conversion relationship between the real space and the preoperative image space is determined, and patient registration is achieved). When the point cloud projection at one or more positions is subsequently performed, the conversion relationship between the structured light coordinate system and the model point cloud coordinate system can be conveniently determined according to the posture of the mechanical arm 400, and the registration of the target object surface point cloud and the model point cloud is not required before each point cloud projection. In addition, the target object surface point cloud required for the projection process can be separately collected in the projection process, or the target object surface point cloud collected at a position in the patient registration process can be reused. It can be understood that, for the case of reusing the point cloud in the patient registration process, the target object surface point cloud collected at the position can be reused only when the point cloud projection at the position is performed. The process of generating the two-dimensional image to be projected can refer to the foregoing embodiments, and will not be described here.

[0192] The medical information projection system of the embodiment can move the structured light module to a required position for point cloud collection and projection by the mechanical arm, the projection position and projection angle can be flexibly adjusted, and the medical information projection system can also be compatible with a surgical navigation positioning system equipped with a mechanical arm. The mechanical arm can be used for assisting point cloud projection and assisting positioning, the use cost is reduced, and the point cloud projection process can also be combined with the patient registration of the surgical navigation positioning to some extent, and the operation process of the user is simplified.

[0193] Based on the foregoing embodiments, in one embodiment, the modules of the medical information projection system also cooperate to achieve:

[0194] The processing module calculates the projection posture of the mechanical arm according to the projection posture of the structured light module input by the user, and controls the mechanical arm to move to the projection posture;

[0195] The structured light module re-collects point clouds at the projection posture;

[0196] The processing module generates a new image to be projected according to the re-collected point clouds and the projection posture of the structured light module, and sends the new image to be projected to the structured light module;

[0197] The structured light module projects the new image to be projected to the target object by the projector.

[0198] Specifically, after the initial point cloud is collected and registration is completed, or after one or more projections are further performed, the projection pose can need to be adjusted, for example, the current mechanical arm pose hinders the surgical operation, for example, other content needs to be projected in a new position.

[0199] At this time, in the "active control mode of the mechanical arm", the user inputs the projection pose required by the structured light module through the input device, the processing module calculates the projection pose and motion path required by the mechanical arm to reach, and then controls the mechanical arm to move to the projection pose. The processing module obtains the position relationship between the structured light module and the point cloud to be projected in the new projection pose according to the conversion relationship between the model point cloud coordinate system obtained by registration and the mechanical arm coordinate system, and then generates a new projection image in combination with the point cloud newly collected in the projection pose, and performs projection again. The process of generating a new projection image can refer to the foregoing embodiments, which will not be described here.

[0200] In the embodiment, the structured light point cloud is newly collected and the projection image is generated and projected after the user adjusts the mechanical arm pose, which meets the demand of multi-position projection, and the registration is not needed again in the process of the second projection, which improves the projection efficiency.

[0201] Based on the foregoing embodiments, in one embodiment, the modules of the medical information projection system are further cooperated to implement:

[0202] determine the projection pose of the structured light module according to the adjusted mechanical arm pose in the following mode;

[0203] the structured light module collects point cloud again in the projection pose;

[0204] the processing module generates a new projection image to be projected according to the collected point cloud and the projection pose of the structured light module, and sends the new projection image to the structured light module;

[0205] the structured light module projects the new projection image to be projected to the target object through the projector.

[0206] Specifically, after the initial point cloud is collected and registration is completed, or after one or more projections are further performed, the projection pose can need to be adjusted, for example, the current mechanical arm pose hinders the surgical operation, for example, other content needs to be projected in a new position.

[0207] At this time, the user drags the mechanical arm to make the structured light module reach the projection pose required in the "follow-up control mode of the mechanical arm", the processing module obtains the conversion relationship between the model point cloud coordinate system and the mechanical arm coordinate system registered at this time to obtain the position relationship between the structured light module and the point cloud to be projected at the current projection pose, and then generates a new projection image in combination with the point cloud newly collected at the projection pose to perform re-projection. The process of generating a new projection image can refer to the foregoing embodiments, and will not be described here again.

[0208] In this embodiment, the structured light point cloud is newly collected and the projection image is generated and projected after the user adjusts the pose of the mechanical arm, the demand for multi-position projection is met, and the registration is not required again in the re-projection process, and the projection efficiency is improved.

[0209] Next, another medical information projection method provided by the present application is described, and the medical information projection method described below can be correspondingly referred to the medical information projection system described above.

[0210] Figure 6 is a flowchart of a medical information projection method provided by the present application, as shown in Figure 6 , the method comprises:

[0211] S11, obtaining a surface point cloud of a target object; wherein the surface point cloud of the target object is obtained by a structured light module, and the structured light module comprises a camera and a projector;

[0212] Specifically, there are various ways to obtain the surface point cloud of the target object, for example, the system performs patient registration in advance, that is, the surface point cloud data of the target object is collected by the structured light module, and is used to establish the corresponding relationship between the patient space and the preoperative medical image space, at this time, the surface point cloud of the target object collected by the structured light module 100 in the patient registration process can be directly obtained, and the point cloud does not need to be repeatedly collected. For another example, in each point cloud projection process, the point cloud data of the target object is collected by the structured light module at the current projection position, and is used to process in combination with the model point cloud to generate the projection image. The point cloud collected by the structured light module 100 can be used for registration with the model point cloud, so as to determine the conversion relationship between the structured light coordinate system and the preoperative image coordinate system.

[0213] S12, generating a first landing point point cloud corresponding to the target object surface according to the model point cloud and the position of the structured light module; wherein the model point cloud comprises the point cloud to be projected;

[0214] Specifically, the model point cloud and the surface point cloud of the target object are unified to the same coordinate system, the relative position of the surface point cloud of the target object and the structured light module is known, and therefore, the to-be-projected point cloud in the model point cloud can be "observed" from the perspective of the structured light module (of the projector) to determine the landing point of the to-be-projected point cloud on the surface of the target object, that is, to obtain the first landing point cloud.

[0215] S13, generating a to-be-projected image according to the first landing point cloud, in combination with the surface point cloud of the target object and the acquisition process of the surface point cloud of the target object;

[0216] Specifically, the surface point cloud of the target object is acquired based on structured light acquisition, and the three-dimensional point cloud of the surface of the target object has a corresponding relationship with the two-dimensional image projected in the structured light point cloud acquisition process. According to the "projection result" of the first landing point cloud in combination with the structured light point cloud acquisition process, the corresponding "projection image" (that is, the to-be-projected image) can be inversely deduced.

[0217] S14, sending the to-be-projected image to the projector so that the projector projects the to-be-projected image to the surface of the target object.

[0218] Specifically, the to-be-projected image is sent to the projector, and the projector does not need to change the projection parameters (projection direction, projection range, resolution and other parameters), and can directly project the to-be-projected image to the surface of the target object by using the projection parameters in the structured light point cloud acquisition process, so that the medical information can be directly observed in the real space.

[0219] In the embodiment, the surface point cloud of the target object is acquired, which can be used to determine the relative position relationship between the target object and the structured light module, in combination with the to-be-projected point cloud in the model point cloud, the position of the to-be-projected point cloud can be determined, and then in combination with the surface point cloud of the target object, a two-dimensional image corresponding to the to-be-projected point cloud can be generated, and the to-be-projected image can be projected to the surface of the target object by the projector of the structured light module, so that efficient and accurate projection is realized, and the user can directly observe the medical information on the surface of the target object.

[0220] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0221] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent replacements for some of the technical features therein. Such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A medical information projection system, characterized in that, include: Structured light module, processing module; The structured light module includes a camera and a projector; The processing module is communicatively connected to the structured light module; The modules of the medical information projection system work together to achieve the following: the structured light module acquires the surface point cloud of the target object; the processing module generates a projection image based on the model point cloud and the surface point cloud of the target object, wherein the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud contains the projection point cloud; the projector projects the projection image onto the surface of the target object. The processing module includes a memory and a processor. The processor loads a computer program stored in the memory to perform the following: S11. Obtain the surface point cloud of the target object; wherein the surface point cloud of the target object is acquired by the structured light module; S12. Based on the position of the model point cloud and the structured light module, generate the first landing point cloud corresponding to the point cloud to be projected on the surface of the target object; S13. Based on the first landing point cloud, combined with the surface point cloud of the target object and the correspondence between the surface point cloud of the target object and the two-dimensional image projected in the structured light point cloud acquisition process, the image to be projected is generated in reverse. S14. Send the image to be projected to the projector, so that the projector projects the image to be projected onto the surface of the target object.

2. The medical information projection system according to claim 1, characterized in that, The system also includes a fixing device, which is a fixed connection structure or an installation structure that matches the installation position, used to fix the structured light module in the required position; Alternatively, the system may also include a robotic arm, which is communicatively connected to the processing module and is used to move the structured light module to the desired position according to the control instructions of the processing module.

3. The medical information projection system according to claim 1, characterized in that, The point cloud to be projected is obtained in the following way: Obtain the region to be projected; wherein, the region to be projected is a non-surface region of the target object; Extract the surface points of the area to be projected to obtain the point cloud to be projected.

4. The medical information projection system according to claim 1, characterized in that, S12 includes: S121. Determine the type of the point cloud to be projected; If the point cloud to be projected is the surface point cloud of the target object, the following operations are performed: S122. The point cloud to be projected is determined as the first landing point cloud; S123. Unify the model point cloud and the structured light module to the same coordinate system.

5. The medical information projection system according to claim 1, characterized in that, S12 includes: S121. Determine the type of the point cloud to be projected; If the point cloud to be projected is a non-surface point cloud of the target object, the following operations are performed: S125. Unify the model point cloud and the structured light module to the same coordinate system; S126. In the same coordinate system, the model point cloud is processed in combination with the position of the projector to obtain the first landing point cloud corresponding to the point cloud to be projected on the surface of the target object.

6. The medical information projection system according to claim 5, characterized in that, The S125 includes: Register the surface point cloud of the target object with the model point cloud to obtain the transformation relationship between the model point cloud coordinate system and the structured light coordinate system; Based on the transformation relationship between the model point cloud coordinate system and the structured light coordinate system, the model point cloud and the structured light module are transformed to the same coordinate system.

7. The medical information projection system according to claim 5, characterized in that, The preceding steps, S11, include: controlling the robotic arm to drive the structured light module to acquire structured light point clouds of the target object; registering the structured light point clouds with the model point cloud to obtain the transformation relationship between the model point cloud coordinate system and the robotic arm coordinate system; wherein the robotic arm is equipped with or integrated with the structured light module; Accordingly, S125 includes: converting the model point cloud and the structured light module to the same coordinate system according to the transformation relationship between the model point cloud coordinate system and the robotic arm coordinate system and the current projection posture of the robotic arm.

8. The medical information projection system according to claim 5, characterized in that, S126 includes: Based on the position of the projector in the same coordinate system, the projector is abstracted as a point and connected to each point in the point cloud to be projected in the same coordinate system. The intersection of each line with the surface of the target object constitutes the first landing point cloud.

9. The medical information projection system according to claim 1, characterized in that, S13 includes: Based on the first point cloud of impact points, a second point cloud of impact points corresponding to the area where the first point cloud of impact points is located is determined from the surface point cloud of the target object; Based on the second landing point cloud and the acquisition process of the surface point cloud of the target object, the image to be projected is generated.

10. The medical information projection system according to claim 9, characterized in that, The step of determining a second landing point cloud corresponding to the region where the first landing point cloud is located from the surface point cloud of the target object based on the first landing point cloud includes: A search tree is established for the surface point cloud of the target object. For each point in the first landing point cloud, the nearest point is searched in the search tree, and the nearest points constitute the second landing point cloud.

11. The medical information projection system according to claim 9, characterized in that, The step of generating the image to be projected based on the second landing point cloud and the acquisition process of the surface point cloud of the target object includes: Generate a blank two-dimensional image with the same resolution as the two-dimensional pattern projected during the acquisition of the surface point cloud of the target object; For each point in the second landing point cloud, the coordinates of the corresponding two-dimensional point in the second landing point cloud are determined according to the correspondence between the pixels in the two-dimensional pattern projected during the acquisition of the surface point cloud of the target object and the generated three-dimensional points, and added to the blank two-dimensional map to finally obtain the image to be projected.

12. The medical information projection system according to claim 9, characterized in that, S13 includes: When a point needs to be projected, the point in the image to be projected is highlighted; and / or, When the projection requires an outline, connect the outer points in the image to be projected; and / or, When the area to be projected is a region, morphological processing is performed on the image to be projected.

13. The medical information projection system according to claim 1, characterized in that, The medical information projection system includes multiple structured light modules to achieve multi-angle medical information projection.

14. The medical information projection system according to claim 2, characterized in that, The various modules of the medical information projection system also work together to achieve: The processing module calculates the projection pose of the robotic arm based on the projection pose of the structured light module input by the user, and controls the robotic arm to move to the projection pose. The structured light module re-acquires point clouds at the projected pose. The processing module generates a new image to be projected based on the re-acquired point cloud and the projection pose of the structured light module, and sends it to the structured light module. The structured light module projects a new image to be projected onto the target object via the projector.

15. The medical information projection system according to claim 2, characterized in that, The various modules of the medical information projection system also work together to achieve: The projection pose of the structured light module is determined based on the robotic arm posture adjusted in follow mode; The structured light module re-acquires point clouds at the projected pose. The processing module generates a new image to be projected based on the re-acquired point cloud and the projection pose of the structured light module, and sends it to the structured light module. The structured light module projects a new image to be projected onto the target object via the projector.

16. A method for projecting medical information, characterized in that, include: S11. Acquire the surface point cloud of the target object; wherein, the surface point cloud of the target object is acquired by a structured light module, and the structured light module includes a camera and a projector; S12. Based on the model point cloud and the position of the structured light module, generate a first landing point cloud corresponding to the point cloud to be projected on the surface of the target object; wherein, the model point cloud is a three-dimensional point cloud corresponding to the target object, and the model point cloud contains the point cloud to be projected. S13. Based on the first landing point cloud, combined with the surface point cloud of the target object and the correspondence between the surface point cloud of the target object and the two-dimensional image projected in the structured light point cloud acquisition process, the image to be projected is generated in reverse. S14. Send the image to be projected to the projector, so that the projector projects the image to be projected onto the surface of the target object.

Images