Surgical assistance system based on holographic images

Abstract

The application relates to a holographic image-based surgical auxiliary system, wherein an image recognition and ranging module is used for capturing panoramic images and obtaining relative position parameter information between each system specified point; a projection mechanism in an imaging glasses module is used for mapping a human body perspective image and the relative position parameter information between each system specified point to a lens on a glasses body; a visual angle image capturing mechanism is used for capturing a visual angle picture of the glasses body; and a master control module is used for adjusting the position and size of the human body perspective image projected to the lens by the projection mechanism in real time according to the panoramic images, the visual angle picture and the relative position parameter information between each system specified point, so as to ensure that the human body perspective image presented on the lens and the corresponding human body structure can be kept coincident in real time. The holographic image-based surgical auxiliary system has the characteristics of convenient operation, good practicability and effective assistance for precise implementation of surgery.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical clinical operation, in particular to a surgery auxiliary system based on holographic image. BACKGROUND

[0002] With the continuous development of medical level, medical imaging plays an increasingly important role in the treatment process. Doctors can take pictures of the internal state of the human body with radiological equipment such as X-ray, CT, and nuclear magnetic resonance equipment to obtain imaging data to more accurately understand the lesion condition in the human body. However, in the prior art, the human body image taken is usually displayed on paper, display screen or other external imaging components. Before the operation is implemented, the doctor needs to pre-view the picture and remember the lesion position and characteristics, and then rely on memory to estimate the lesion position and size to perform the operation.

[0003] This implementation in the prior art requires very high memory of medical personnel, and errors are prone to occur during implementation. Therefore, there is an urgent need in the prior art for a device that can better assist doctors in performing operations. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a surgery auxiliary system based on holographic image, which is easy to operate, has good performance, and effectively provides real-time assistance for medical personnel.

[0005] In order to achieve the above purpose, the surgery auxiliary system based on holographic image has the following structure:

[0006] The surgery auxiliary system based on holographic image mainly comprises an image recognition and distance measurement module, an imaging glasses module and a main control module.

[0007] The image recognition and distance measurement module is used to capture panoramic images and obtain relative position parameter information between each system designated point through the panoramic images.

[0008] The imaging glasses module comprises a glasses body with transparent lenses, a projection mechanism and a view angle image capture mechanism. The projection mechanism and the view angle image capture mechanism are mounted on the glasses body. The projection mechanism is used to map the human perspective image and the relative position parameter information between each system designated point onto the lenses on the glasses body. The view angle image capture mechanism is used to capture the view angle picture of the glasses body.

[0009] The main control module is configured to adjust the position and size of the human body perspective image projected onto the lens by the projection mechanism in real time according to the relative position parameter information between the panoramic image, the perspective picture and the system specified points, so as to ensure that the human body perspective image presented on the lens and the corresponding human body structure can still be kept in real time when the relative position between the glasses main body and the human body changes.

[0010] The holographic image-based surgical assistance system described above, wherein the system specified points include specified points on the patient's body, specified points on the wearer of the glasses main body and specified points on the specified medical instrument.

[0011] The relative position parameter information between the system specified points includes relative distance parameter information and relative angle parameter information between the specified points on the patient's body, the specified points on the wearer of the glasses main body and the specified points on the specified medical instrument.

[0012] The holographic image-based surgical assistance system described above, wherein the image recognition and ranging module includes:

[0013] A camera unit is configured to capture the panoramic image and identify the position range corresponding to each of the system specified points from the panoramic image.

[0014] An infrared dot array projector unit is configured to detect the coordinate position of each of the system specified points in space.

[0015] The camera unit and the infrared dot array projector unit respectively transmit the panoramic image, the position range corresponding to each of the system specified points and the coordinate position of each of the system specified points in space to the main control module to obtain the relative position parameter information between the system specified points.

[0016] The holographic image-based surgical assistance system described above, wherein the human body perspective image is a three-dimensional reconstruction model generated according to the medical image of the patient.

[0017] The holographic image-based surgical assistance system described above, wherein the holographic image-based surgical assistance system further includes:

[0018] A control action capture camera module is configured to capture the operation control action image of the wearer of the glasses main body and transmit the operation control action image to the main control module.

[0019] The main control module adjusts the size and angle of the human body perspective image mapped onto the lens by the projection mechanism according to the operation control action image.

[0020] The computer readable storage medium mainly features that a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following operation steps:

[0021] Obtaining a panoramic image and obtaining relative position parameter information between each system specified point through the panoramic image;

[0022] Controlling a view angle image capture mechanism in the imaging glasses module to capture a view angle picture of a glasses main body in the imaging glasses module;

[0023] Controlling a projection mechanism in the imaging glasses module to map a human body perspective image and the relative position parameter information between each system specified point to a lens on the glasses main body;

[0024] Adjusting a position and a size of the human body perspective image projected by the projection mechanism to the lens in real time according to the panoramic image, the view angle picture and the relative position parameter information between each system specified point, so as to ensure that the human body perspective image presented on the lens and the corresponding human body structure can still be kept in real time coincidence when a relative position between the glasses main body and the human body changes.

[0025] The computer readable storage medium, wherein the system specified points include specified points on a patient's body, specified points on a wearer of the glasses main body and specified points on a specified medical instrument;

[0026] The relative position parameter information between each system specified point includes relative distance parameter information and relative angle parameter information between the specified points on the patient's body, the specified points on the wearer of the glasses main body and the specified points on the specified medical instrument.

[0027] The computer readable storage medium, wherein the obtaining a panoramic image and obtaining relative position parameter information between each system specified point through the panoramic image includes:

[0028] Obtaining the panoramic image and identifying a position range corresponding to each system specified point from the panoramic image;

[0029] Obtaining a coordinate position of each system specified point in space;

[0030] Obtaining relative position parameter information between each system specified point according to the panoramic image, the position range corresponding to each system specified point and the coordinate position of each system specified point in space.

[0031] The computer readable storage medium, wherein the human body perspective image is a three-dimensional reconstruction model generated according to a medical image of a patient.

[0032] The computer readable storage medium described above, wherein the real-time acquisition control action capture camera module collects the picture, acquires the operation control action image of the wearer of the glasses body, adjusts the size and angle of the human body perspective image projected on the lens according to the operation control action image.

[0033] The surgical auxiliary system based on holographic images has the following advantages:

[0034] The surgical auxiliary system based on holographic images projects the human body perspective image on the transparent lens, adjusts the position and size of the human body perspective image projected on the lens by the panoramic image, the view angle picture and the relative position parameter information between the system specified points, so that when the relative position of the glasses body and the human body changes, the human body perspective image on the lens and the corresponding human body structure can still be real-time overlapped, and the relative position parameter information between the system specified points is also mapped on the lens of the glasses body, so that the medical staff can see the overlapping state of the human body perspective image and the patient's body through the lens of the glasses body, so that the lesion position can be directly seen through the imaging glasses module without memory, and the interval distance and the angle relationship between the surgical instrument and the lesion can be confirmed through the relative position parameter information between the system specified points mapped on the lens, and since the view angle image capture mechanism is arranged, the view angle picture of the glasses body can be acquired in real time, so that when the position or posture of the wearer of the glasses body or the patient changes, the human body perspective image on the lens and the corresponding human body structure can still be real-time overlapped. The surgical auxiliary system based on holographic images is convenient to operate, has good practicability, and can effectively assist the precise implementation of surgery. BRIEF DESCRIPTION OF DRAWINGS

[0035] The concept, specific structure and technical effects of the present application will be further described below in combination with the drawings, so as to fully understand the purpose, features and effects of the present application.

[0036] Figure 1 It is a schematic diagram of the connection relationship of the structure of the surgical auxiliary system based on holographic images of the present application in an embodiment.

[0037] Figure 2 It is a schematic diagram of the relative position of the picture captured by the double-sided camera.

[0038] Figure 3 It is a positioning schematic diagram of hand key points.

[0039] Figure 4 It is a relative position relationship diagram of two points located in a horizontal plane.

[0040] Figure 5 Fig. 9 is a diagram of relative positions of two points in a vertical plane.

[0041] Figure 6 Fig. 10 is a diagram of a gesture control in a first state.

[0042] Figure 7 Fig. 11 is a diagram of a gesture control in a second state.

[0043] Figure 8 Fig. 12 is a diagram of an angle determination.

[0044] Figure 9 Fig. 13 is a diagram of a rotation instance coordinate in an embodiment.

[0045] Figure 10 Figure 11 Figure 12 Figs. 15-17 are diagrams of image processing.

[0046] Figure 13 Fig. 18 is a diagram of three-dimensional reconstruction in an embodiment. DETAILED DESCRIPTION

[0047] In order to make the technical means, creative features, purposes and effects of the application easy to understand, the application is further described below in combination with specific diagrams. However, the application is not limited to the following embodiments.

[0048] It should be understood that the structures, proportions, sizes, etc. shown in the drawings attached to the present specification are merely used to understand and read the content disclosed in the present specification by those skilled in the art, and are not used to limit the conditions that the application can be implemented, and therefore do not have substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that the application can produce, should still fall within the scope of the technical content disclosed by the application.

[0049] It should be noted that, in the present text, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between the entities or operations.

[0050] ​​In a large number of invasive surgery scenes in modern medicine, doctors need to further understand the patient's condition with the help of radiological data such as CT, magnetic resonance imaging, etc. To facilitate the formulation of intraoperative plans and the precise implementation of intraoperative operations. The present application provides a set of integrated holographic image-based surgical assistance system for doctors. The system uses holographic image technology to display the picture in front of the doctor in real time with the help of imaging glasses module equipment (such as VR, AR, MR glasses, etc.), and coincides with the patient's body. At the same time, the real-time position of the doctor and the patient, the posture and motion trajectory of the body and the corresponding special parts (including but not limited to hands, legs, head, etc.), and the real-time posture and motion trajectory of special markers (including but not limited to surgical knives, endoscopes, forceps, hemostatic forceps, etc.) are accurately captured. Then all the data and images are summarized, and the holographic image device is used to match the doctor's operation picture with the proportional three-dimensional model or two-dimensional image, so that the doctor can clearly distinguish the actual operation process. The following is a further description of the holographic image-based surgical assistance system:

[0051] In a first aspect, the holographic image-based surgical assistance system includes an image recognition and distance measurement module, an imaging glasses module, a main control module, and a control action capture camera module.

[0052] The image recognition and distance measurement module is used to capture panoramic images and obtain relative position parameter information between each system designated point through the panoramic images.

[0053] In this embodiment, the image recognition and distance measurement module includes:

[0054] The camera unit is used to shoot the panoramic images and identify the position range corresponding to each system designated point from the panoramic images.

[0055] The camera unit includes a surround camera device that can capture a 360° panorama in the operating room. The surround camera device can be composed of multiple cameras.

[0056] The infrared dot matrix projector unit is used to detect the coordinate position of each system designated point in space.

[0057] The camera unit and the infrared dot matrix projector unit respectively transmit the panoramic images, the position range corresponding to each system designated point, and the coordinate position of each system designated point in space to the main control module to obtain the relative position parameter information between each system designated point.

[0058] In implementation, the camera unit is used for panoramic shooting and identifying the relative positions of the human body parts or medical devices in the panoramic image (the corresponding anchor point markers can be set on the human body parts or medical devices to constitute the system specified points to effectively assist in identification). Since the position detection accuracy of the camera unit is generally accurate, an infrared dot array projector unit is also provided in this embodiment to perform more accurate coordinate position identification. However, since the infrared dot array projector in the infrared dot array projector unit can only detect the distance but cannot identify what the corresponding coordinates correspond to, in this embodiment, the camera unit and the infrared dot array projector unit respectively transmit the panoramic image, the position range corresponding to each system specified point and the coordinate position of each system specified point in space to the main control module. The main control module determines the relative position parameter information between the system specified points according to the panoramic image, the position range corresponding to each system specified point and the coordinate position of each system specified point in space.

[0059] The imaging glasses module includes a glasses body with transparent lenses, a projection mechanism and a perspective image capture mechanism. The projection mechanism and the perspective image capture unit are mounted on the glasses body. The projection mechanism is used to map the human body perspective image and the relative position parameter information between the system specified points to the lenses on the glasses body. The perspective image capture mechanism is used to capture the perspective view of the glasses body.

[0060] The main control module is used to adjust the position and size of the human body perspective image projected by the projection mechanism on the lenses in real time according to the panoramic image, the perspective view and the relative position parameter information between the system specified points, so as to ensure that the human body perspective image and the corresponding human body structure can be kept in real-time coincidence when the relative position of the glasses body and the human body changes.

[0061] In specific implementation, the patient's body can be marked with corresponding anchor points. The anchor point positions on the patient's body and the corresponding positions of the human body perspective image in the main control system are also matched. When the picture is aligned, only the anchor point positions need to be captured and aligned with the corresponding positions of the human body perspective image pre-stored in the system to position, so as to ensure that the human body perspective image and the corresponding human body structure can be kept in real-time coincidence. Or the body features can also be used as anchor points to assist in identification.

[0062] The control action capture camera module is used to capture the operation control action image of the glasses body wearer and transmit the operation control action image to the main control module.

[0063] The main control module adjusts the size and angle of the human body perspective image mapped to the lenses by the projection mechanism according to the operation control action image.

[0064] The system designated points include designated points on the patient's body, designated points on the wearer of the glasses, and designated points on the designated medical device.

[0065] The relative position parameter information between the designated points of each system includes the relative distance parameter information and relative angle parameter information between the designated points located on the patient's body, the designated points located on the wearer of the glasses, and the designated points on the designated medical device.

[0066] The human body perspective image is a three-dimensional reconstruction model generated based on the patient's medical images. That is, a three-dimensional reconstruction model is generated based on various two-dimensional and three-dimensional images (e.g., ultrasound images, CT images, three-dimensional models, MRI images, three-dimensional models), and then the three-dimensional reconstruction model is mapped onto a transparent lens using holographic imaging technology.

[0067] The relative position parameters between designated points of each system can be presented on the glasses through text, graphic indicators, etc. For example, when the incision direction is pointing towards the lesion, the doctor can be informed of the incision distance through numerical prompts. If the incision direction is not pointing towards the lesion, the doctor can be prompted to adjust the incision direction in time through arrows and other symbols.

[0068] To enable those skilled in the art to more intuitively understand the implementation of this holographic imaging-based surgical assistance system, a specific usage example is described below:

[0069] When the doctor (i.e., the primary wearer of the glasses) performs surgical procedures such as punctures, incisions, and fillings, the system will use the in-operative camera unit to capture 360° surround shooting to capture the real-time relationship between the doctor's hand and instruments and the patient's body. During implementation, anchor points are pre-marked on the patient's limbs, the doctor's hands, and surgical instruments to generate designated points for each system, facilitating position capture by the equipment.

[0070] During the capture process, important markers and anchor points are identified and tracked using artificial intelligence algorithms. While the infrared dot projector unit can accurately measure the actual distance to all markers and anchor points, it cannot determine which marker or anchor point the detected coordinates belong to. The camera unit, however, can identify the object captured by the infrared dot projector unit. By combining the information captured by both units, the relative positional parameters between designated points in each system can be determined.

[0071] At this time, the doctor can map the perspective image of the human body and the relative position parameter information between the designated points of each system in the lens of the glasses through the imaging glasses module (that is, a holographic image device), wherein the perspective image of the human body is in a semi-transparent state, and the perspective image of the human body and the patient's body (the body can be a complete body or a local part of the body) are kept coincident in front of the doctor. The doctor can see the specific position and depth of his operation in the three-dimensional and two-dimensional model (the perspective image of the human body) of the patient in front of him through the imaging glasses module, so that the doctor can easily judge whether the angle and intensity of the operation need to be adjusted.

[0072] In this embodiment, the perspective image of the human body is obtained according to the image obtained by the radiology department of the patient before the operation or by the ultrasound and other related technologies during the operation. The system automatically fuses the image information by using an algorithm, and the data of the patient is reconstructed and generated by three-dimensional reconstruction.

[0073] Meanwhile, the embodiment also provides a control action capture camera module for operation based on intelligent non-contact operation function. That is, the control action capture camera module is used to capture the picture of the preset position, and when a specific gesture or action is captured, the corresponding operation is performed, so that the doctor can adjust the size, position and depth of field of various images in the holographic image through gestures or eye movements without contact. Because a separate control action capture camera module is used for picture capture, false operation caused by non-operation gestures during operation is avoided. During the operation process, the doctor can also lock, rotate and set various image characteristics (for example, adjust the gray scale image display mode, automatically detect the edge of the image, sharpen the image, smooth the image, reduce the noise of the image, erode and dilate the image, etc.) of the image through related gestures or other indicators. By using the above system and corresponding functions, the doctor can not only place all the required pictures in front of himself in real time, but also can change a large number of blind operations into precise operations with direction, guidance and positioning, thereby greatly reducing the learning curve of complex operations and the probability of errors in operation.

[0074] The interaction relationship between the modules in the holographic image-based surgical assistance system in the above embodiment can be referred to Figure 1 In addition to the components mentioned above, the main control module can also be connected with the ultrasound instrument, the operating table and the shadowless lamp and other instruments in the operating room to realize comprehensive control.

[0075] The working principle of each related component of the holographic image-based surgical assistance system in the above embodiment and the related image processing principle are further described as follows:

[0076] Holographic image is a recording and reproducing technology that records and reproduces real three-dimensional images of objects using interference and diffraction principles. In the present application, the radiological imaging data (e.g. CT, nuclear magnetic resonance, etc.) in the radiology department is reconstructed into a three-dimensional model (i.e. human perspective image) through three-dimensional reconstruction. The human perspective image is projected onto the lens of the special glasses body through the holographic projection device (i.e. projection mechanism in the imaging glasses module), so that the image can be reconstructed in three-dimensional space, facilitating doctors to observe and analyze the patient's condition comprehensively.

[0077] Through the use of smart wearable devices (i.e. imaging glasses module), doctors can transmit important information and image pictures to the glasses through the wearable devices, and view or listen to information including text, numbers, pictures, three-dimensional models, videos and audio through VR, AR, MR related technologies (i.e. holographic projection technology). At the same time, the glasses body is equipped with a view angle image capture mechanism (which can be composed of a camera), which can capture the first view of the doctor through the camera on the glasses, and transmit important operation pictures and real-time treatment pictures to the main control module (i.e. central processor) for subsequent intelligent tracking, judgment and remote medical related functions.

[0078] The infrared dot array projector unit is composed of an infrared dot array projector. The infrared dot array projector accurately draws the shape profile of the object surface placed in front of the projector by projecting infrared light in a matrix form outward and receiving each infrared reflection point. At the same time, the TOF principle (TOF is the abbreviation of Time of flight, which means flight time. In a broad sense, time of flight technology can be understood as a technology that further understands the properties of ions or media by measuring the time taken by an object, particle or wave to travel a certain distance in a fixed medium (medium / distance / time are known or measurable).), which can calculate the actual distance of a point from the position of the emitter. In this embodiment, for the precise position of some important calibration objects, we can perform real-time, continuous and accurate distance measurement through this device.

[0079] The camera unit can include several high-definition cameras arranged in a 360° non-blind angle manner at the top edge position of the operating space, and a 2x2 high-definition camera matrix at the operating room shadowless lamp position. The several high-definition cameras provide the main control module (i.e. central processor) with comprehensive real-time images based on the operating room and operating table in the present application. Through these images, the control main program can coordinate different pictures to ensure that the tracking of core calibration objects (patient, surgical knife, indwelling needle, etc.) is not lost. At the same time, the operating personnel in the operating room can call the pictures collected by different cameras through the switching function.

[0080] The main control module can be composed of a computer processing system, which is the central controller of the whole system. All cameras, sensors, holographic projection devices, smart wearable devices and other related electronic equipment instruments in the holographic image-based surgical assistance system will be connected to the central processing system. The central processor needs to be responsible for running the main control program, and also needs to be responsible for processing the calibration object information in all input signals. For example, through the high-definition camera installed in the operating room, the doctor's hand movements are captured. The three-dimensional coordinates of the doctor's hand at this time are accurately calculated, and the relevant information is transmitted back to the system, and then combined with some existing three-dimensional images to simulate the real-time operation of the doctor. At the same time, the computer processing system also needs to manage different wearable devices, for example, the main doctor sees the CT three-dimensional reconstruction of the patient's lesion site, while the nurse sees the related interface for controlling the operating room shadow lamp and operating bed. The system distributes and transmits different pictures and information to different roles in the wearable device according to the settings in the program through real-time processing.

[0081] The main control module also needs to perform calibration object identification operation. The calibration object identification mainly includes two parts. One part is to track the objects through the object recognition built in the system, such as surgical knives, hemostatic forceps, and retained needles. The other part, some items that need to be captured in real time by the computer can be temporarily marked through fluorescent markers. Once the computer identifies the special color fluorescent markers, it will ask the operator through the system whether it needs to be identified and tracked. If the user selects yes, the computer will lock and track in real time in the picture. Through this selection operation, the projection mechanism can only project the distance of the specified point (such as the distance between the lesion and the surgical instrument being operated) when projecting the relative position parameter information between the system specified points, reducing the influence of other calibration objects.

[0082] The above is the hardware structure description of the holographic image-based surgical assistance system in the above embodiment. In the whole system, the main control module (central processor) pre-stores corresponding operation instructions to control the surgical assistance system to perform corresponding operations, so as to realize the linkage and control of the whole system. Through the algorithm in the present surgical assistance system, intelligent capture, control and feedback can be realized.

[0083] The algorithm function of the innovation in the invention will be described in detail below.

[0084] The camera unit can use a binocular camera or two cameras to identify the specific detection principle of the position range corresponding to each system specified point as follows:

[0085] Suppose there are two lenses, which are left lens and right lens, such as Figure 2As shown, C l For the position of the left camera, C r The left and right lenses are positioned horizontally but vertically at the same height. P represents the position of the target object (i.e., the position of any system-specified point). l P represents the position of the target object captured by the left lens at the camera baseline. r Assuming the target object captured by the right lens is positioned at the camera baseline, the lateral distance between the left and right lenses is B, the lateral distance between the left lens and the target object is X, the lateral distance between the right lens and the target object is XB, and the lateral distance between the left lens and the position of the target object captured by the left lens at the camera baseline is x. l The lateral distance from the position of the target object captured by the right lens to the position at the camera baseline is x. r The longitudinal distance from the position of the target object captured by the left lens to the position at the camera baseline is P. l The longitudinal distance from the position of the target object captured by the right lens to the position at the camera baseline is P. r The longitudinal distance from the left and right lenses to the target object is Z, and the longitudinal distance from the left and right lenses to the camera baseline is f. The positions of the left and right lenses and the camera baseline are fixed and pre-set in the main control module.

[0086] Based on the principle of similar triangles, starting from position C of the left camera... l Equation 1 can be derived as follows:

[0087]

[0088] Based on the principle of similar triangles, starting from position C of the right camera... r Equation 2 can be derived as follows:

[0089]

[0090] Based on equations 1 and 2 above, equation 3 can be derived as follows:

[0091]

[0092] By rearranging, we can obtain the following formula:

[0093]

[0094] In actual application, the above formula 3 and formula 4 can be pre-stored in the master module, and then the depth information of the target to be measured in the picture captured by the binocular camera (i.e. the longitudinal distance Z from the target to be measured to the left and right lenses) is calculated using formula 4, and then the horizontal and vertical coordinates of the target to be measured in the plane position relative to the coordinate position of the camera are determined by shifting the camera baseline position and using formula 3, and then the position of the target to be measured in space is determined.

[0095] The principle of controlling the image by the control action capture camera module is as follows:

[0096] Figure 3 A schematic diagram for positioning the key points of the hand. Figure 3 The small dots in and are the positions of the 21 key points of the hand, wherein 4, 8, 12, 16 and 20 are the positions of the fingertips, and the other positions represent the positions of the related joints. When the system is running, when the hand appears in the picture that can be captured by the control action capture camera module, the system can capture the positions of the key points of the hand.

[0097] In specific implementation, the existing hand positioning tool in the prior art can be used to achieve, such as the HandDetector algorithm in cvzone.HandTrackingModule in python (i.e. the hand key point capture algorithm in python) can be used to capture all the corresponding point positions of the hand, and through tracking these point positions, the system can master the dynamic of the doctor's hand in real time and accurately. In other embodiments, other ways can also be used to capture the hand positioning and corresponding dynamic tracking.

[0098] In implementation, an algorithm can be used to determine the positional relationship between points. For the position of any target point in three-dimensional space, the detailed spatial position information of the target point can be determined by obtaining the specific values (i.e. coordinate values) of the target point on the x-axis, y-axis and z-axis in three-dimensional space. The following will further illustrate the determination principle by combining Figure 4 and Figure 5

[0099] In calculating the positional relationship of two target points in three-dimensional space, first, we project the target point A and the target point B onto a unified plane, and from Figure 4 it can be seen that for two points in the same space, when viewed from a top-down angle, the two points can be simply understood as two points in the same plane, and formula 5 can be used to obtain the distance a between the target point A and the target point B in the horizontal plane:

[0100]

[0101] ​Wherein, x1 is the coordinate of the target point A on the X axis, y1 is the coordinate of the target point A on the Y axis, x2 is the coordinate of the target point B on the X axis, and y2 is the coordinate of the target point B on the Y axis.

[0102] After confirming the distance between the target point A and the target point B on the horizontal plane, the distance between the target point A and the target point B on the Z axis can be confirmed again. For this problem, the overhead view angle is adjusted to the front view angle, that is Figure 5 As shown in the figure, there is a height difference between the target point A and the target point B. The distance between the two points on the plane can be derived as the following formula 6: Figure 5

[0103]

[0104] In the formula, d is the straight-line distance between the target point A and the target point B; and alpha is the angle between the straight line formed by the target point A and the target point B and the X axis. Figure 5 In the formula, d is the straight-line distance between the target point A and the target point B; and alpha is the angle between the straight line formed by the target point A and the target point B and the X axis. Figure 5 In the formula, d is the straight-line distance between the target point A and the target point B; and alpha is the angle between the straight line formed by the target point A and the target point B and the X axis. Figure 5 In the formula, d is the straight-line distance between the target point A and the target point B; and alpha is the angle between the straight line formed by the target point A and the target point B and the X axis.

[0105]

[0106]

[0107] In the formula, x1 is the coordinate of the target point A on the X axis, y1 is the coordinate of the target point A on the Y axis, z1 is the coordinate of the target point A on the Z axis, x2 is the coordinate of the target point B on the X axis, y2 is the coordinate of the target point B on the Y axis, z2 is the coordinate of the target point B on the Z axis, and d is the straight-line distance between the target point A and the target point B. Figure 5 In the formula, x1 is the coordinate of the target point A on the X axis, y1 is the coordinate of the target point A on the Y axis, z1 is the coordinate of the target point A on the Z axis, x2 is the coordinate of the target point B on the X axis, y2 is the coordinate of the target point B on the Y axis, z2 is the coordinate of the target point B on the Z axis, and d is the straight-line distance between the target point A and the target point B.

[0108] In the plane shown in the figure, the angle between the straight line formed by the target point A and the target point B and the horizontal position of the ground (i.e. the angle relative to the Z axis) is theta1, wherein theta1 can be obtained by the following formulas 9, 10 and 11: Figure 6

[0109]

[0110]

[0111]

[0112] And the angle theta2 between the straight line formed by the target point A and the target point B and the X axis can be obtained by the following formula 12:​​

[0113]

[0114] By calculating the angle θ1 of the straight line formed by target point A and target point B compared to the horizontal position of the ground and the angle of the straight line formed by target point A and target point B compared to the X axis, the angle relationship between the two points in the horizontal position and the vertical position in the entire three-dimensional coordinate system can be obtained. Thus, based on the position relationship of the two points, the complete relationship of the straight line distance d of target point A and target point B in the three-dimensional space, the angle θ1 of the straight line formed by target point A and target point B compared to the vertical plane, and the angle θ2 of the straight line formed by target point A and target point B compared to the horizontal plane is preliminarily formed.

[0115] In the entire system, the positioning of the object in space is particularly critical. If the real-time human dynamics of the doctor cannot be captured, the current operation of the doctor cannot be converted into a specific imaging method in the three-dimensional image. In order to further improve the cognition of the position relationship between points, the system completes the identification of the target key point and records the related coordinates in real time through the identification in the early stage, such as the first joint of the doctor's ring finger. At the same time, this information will be transmitted to the infrared dot array sensor through the central processor, and the real distance of a specific point in the space position from the infrared dot array sensor will be automatically obtained through the ToF (Time of Flight) principle sensor.

[0116] In some use scenarios, doctors have the requirement of rotating the picture of the image returned by various devices. For example, when preparing for a lung surgery, the lung needs to be checked through an endoscope first, and the endoscope will return the camera picture in real time during the checking. However, most of the endoscopes on the market will rotate in angle when they are inserted into the patient's body. At this time, the picture seen by the doctor is not 90° vertically downward, but a random oblique angle. This angle brings certain difficulty to the doctor when observing, and in the actual operation process, if the doctor also needs to adjust the depth position of the endoscope, it will also be difficult to identify the direction of the movement. Therefore, the present application proposes a method for rotating the image by gesture to any angle. This method can adjust the rotation angle of the picture by capturing the human hand direction of the device, so that the doctor can realize the synchronous rotation of the picture by rotating the angle of the specific part of his own gesture.

[0117] A finger can be set by the preset rule of the system, and the computer calculates the pointing angle of the finger to obtain the angle of 0° (i.e. the state shown in FIG. 1A) at this time, and the picture does not rotate at all. When the finger starts to deflect to the left or right (i.e. as shown in FIG. 1B), the picture will rotate to the left or right according to the deflection angle of the finger. Figure 7 Figure 7 ​(As shown in the image), the computer then calculates the corresponding angle and, by synchronizing the angle with the angle of the screen, achieves image rotation. At this point, from... Figure 7 As can be seen, in the image, the user's index finger (i.e., the position marked with a straight line) deflected at a certain angle, and the image subsequently deflected by the same angle. During implementation, the rotation angle of the image can be marked on the screen, such as... Figure 8 The top left corner indicates the current rotation angle as 14 degrees. This rotation angle indicator allows doctors to intuitively understand whether the current image is real-time or has been adjusted, and to know the magnitude of the rotation angle for better operation. The image can follow the finger's movement continuously, but when no marker appears in the image, for example, if no finger is visible and the finger's direction of deflection cannot be detected, the image can automatically reset to zero or remain fixed at the current angle according to the design.

[0118] During the early stages of surgery, when examining the lungs with an endoscope, the image seen inside the body may appear to rotate. The holographic imaging-based surgical assistance system solution in this embodiment can, on the one hand, further predict, calibrate, and correct the surgical position, and on the other hand, enable contactless and intelligent operation of related procedures in the operating room.

[0119] The following is combined Figure 8 The control method is explained below:

[0120] pass Figure 8 The angle of the calibrated object can be clearly seen. Generally, there are three different situations.

[0121] The first type: vertically downwards, that is... Figure 8 The state shown is the movement from point A to point B;

[0122] The second method: turn to the right, that is... Figure 8 The state shown is the movement from point A to point B1;

[0123] The third type is to veer to the left, that is... Figure 9 The state shown is the movement from point A to point B2;

[0124] For the three situations mentioned above, the program needs to make a judgment first. We take point A as the benchmark position for judgment and obtain the coordinates of point A: A(x1, y1), and obtain the coordinates of point B: B(x2, y2).

[0125] By comparing the coordinates of points A and B on the X-axis, determine whether point B is located to the left or right of point A. After making this determination, calculate using trigonometric functions. Figure 10 to Figure 12the specific angle of ∠1 or ∠3. The specific value of a, b in the figure is obtained by the absolute value of the coordinate difference of point A and point B on the X axis and Y axis before calculation, wherein ∠1 can be calculated by the following formula 12:

[0126]

[0127] The calculation principle of ∠3 is the same as that of ∠1, and thus is not described here.

[0128] After the calculation is completed, the obtained value is not degree but radian, and at this time the corresponding angle is calculated by the following formula 13:

[0129]

[0130] After the angle of ∠1 is obtained, the final deflection angle ∠2 can be obtained by the following formula 14:

[0131] angle ∠2 = 90° - angle ∠1 formula 14;

[0132] The calculation principle of ∠4 is the same as that of ∠2, and thus is not described here.

[0133] After the above calculation process is completed, the next step is the processing of the image itself. In order to facilitate understanding, a processing example is described as follows:

[0134] Before processing the image, it is first necessary to clarify that the image received in the system is actually an array matrix with RGB information, and a picture is randomly selected to obtain the following data: [[68 74 62] [76 83 72] [81 89 79] ... [255 253 255] [255 253 255] [255 253 255]]

[0142] The total data is 1920x1080 groups of matrices with three arrays. The RGB matrix data represents the corresponding pixel point brightness of R (red), G (green) and B (blue) from 0 to 255. By adjusting the brightness value of each pixel point, different colors are realized, and through the final pixel point splicing, a color picture visible to the naked eye is formed. Here, it is first necessary to completely convert the coordinate system into the following matrix:

[0143] a1 a2 a3

[0144] b1 b2 b3

[0145] c1 c2 c3

[0146] If a rotation of 90° of the picture is needed in the program, the above matrix will adjust the pixel points in the corresponding positions, and the results after adjustment are as follows:

[0147] c1 b1 a1

[0148] c2 b2 a2

[0149] c3 b3 a3

[0150] Through the calculation of the matrix by the code in the program and the deflection angle of the actual calibration object calculated by the computer mentioned above, the rotation of the image can be realized. The conversion in the above case is for a rotation angle of 90°, but in the actual operation process, the rotation angle can be any number between 0° and 360°. At this time, the change of the array needs to be further optimized and adjusted in the algorithm.

[0151] The operation principle and process of the whole rotation will be described below. Figure 10 The transformation of a point is taken as an example. Point A is the original position of a pixel point in the picture. After the rotation of angle a, point A appears at the position of point B after the rotation is completed. In the whole rotation process, the displacement of any pixel point can be regarded as the movement of a point on a circle with a diameter of r.

[0152] The position of point A in the picture can be calculated by the following formula 15 and formula 16:

[0153] x1 = cos(c) r formula 15;

[0154] y1 = sin(c) r formula 16;

[0155] Wherein, x1 is the coordinate of target point A on the X axis, y1 is the coordinate of target point A on the Y axis, c is the included angle formed by the connecting line of target point A and the original point and the X axis, and r is the straight line distance from target point A to the original point.

[0156] The position of point B in the picture can be calculated by the following formula 17 and formula 18:

[0157] x2 = cos(b) r formula 17;

[0158] y2 = sin(b) r formula 18;

[0159] Wherein, x2 is the coordinate of target point B on the X axis, y1 is the coordinate of target point B on the Y axis, b is the included angle formed by the connecting line of target point B and the original point and the X axis, and r is the straight line distance from target point B to the original point.

[0160] Wherein:

[0161] ∠b = ∠a + ∠c

[0162] a is the angle between the line connecting the target point A and the origin and the line connecting the target point B and the origin;

[0163] Finally, through the calculation, the following formula 19 and the following formula 20 can be obtained:

[0164] x2 = x1 cos (a) + y1 cos (a) formula 19;

[0165] y2 = -x1 sin (a) + y1 cos (a) formula 20;

[0166] So far, the motion trajectory calculation of the position offset of any point in the entire image after rotation has been solved. When the new point coordinate is obtained, the program only needs to transfer the original point's RGB matrix information to the new point coordinate position and display, so as to successfully realize the position change of this pixel point. Then through the traversal method, the picture is traversed once, and the position of each pixel point is reset once, and finally the position after rotation can be obtained.

[0167] In the processing process, if the calibration object disappears, two different functions are also considered:

[0168] One is to save the specific value of the real-time calibration object offset angle at the moment, and if the calibration object disappears in the picture, the image saves the last calibrated angle, and the whole picture is locked according to the angle.

[0169] The other is that when the calibration object cannot be recognized in the picture, the picture will be automatically homed: the picture rotation angle is set to the original angle, and the real-time image obtained from the input source is directly output.

[0170] The above two settings can be switched and selected in the program.

[0171] In the real use scene, the tracking of a certain joint of the hand is only one of all the objects to be tracked. For other parts, the application will mainly use the combination of camera (recognition target) + infrared dot matrix sensor (precise ranging) to realize the whole multi-point precise position grabbing.

[0172] In this embodiment, the reconstruction of three-dimensional images is one of the important key technologies to achieve the scheme. The three-dimensional reconstruction used in the implementation process can be a picture generated based on the CT-related tomographic scanning technology image. Such images are usually DICOM (Digital Imaging and Communications in Medicine), which is an international standard (ISO 12052) for medical images and related information. It defines a medical image format that can be used for data exchange and meets the quality needs of clinical needs. DICOM is widely used in radiology, cardiovascular imaging, and diagnostic devices (X-ray, CT, nuclear magnetic resonance, ultrasound, etc.), and is increasingly widely used in other medical fields such as ophthalmology and dentistry. Among the tens of thousands of medical imaging devices in use, DICOM is one of the most widely used medical information standards. Currently, there are about 10 billion DICOM-compliant medical images for clinical use. In specific implementation, the method used can be a three-dimensional medical image reconstruction method based on python. The production process is as follows:

[0173] First, the pydicom package in python needs to be used to read the DICOM image. The statement used here is: pydicom.dcmread(fileName), where fileName is the file name of the image.

[0174] Generally speaking, CT images are in an ordered slice arrangement mode, and each slice (Slice) is an independent two-dimensional picture. By splicing these two-dimensional pictures, the three-dimensional image required in the present application is realized.

[0175] In the implementation process, because the interval of each slice may have uncertainty, it is necessary to first resample the entire data set to slices with the same resolution. Here we use two functions SliceThinkness and PixelSpacing to determine the slice thickness and pixel spacing. In actual use, we set the slice spacing to new_spacing = [1, 1, 1]. Finally, through the scipy package scipy.ndimage.interpolation.zoom method, all the data sets are centralized to redefine the resolution. After definition, there is no need to worry about various errors that may occur when the system processes images and three-dimensional reconstruction due to different CT slice thicknesses. It needs to be emphasized that because the original data set is resampled in resolution, the method we use scipy.ndimage.interpolation.zoom is to resample the N-dimensional image according to the new size we calculate. The specific scaling process is as follows:

[0176] (1) Calculate the interval;

[0177] (2) Calculate the adjustment multiple;

[0178] (3) Calculate the image multiple according to the new adjustment parameter;

[0179] (4) Calculate the multiple value of the original image parameter and the new image parameter;

[0180] (5) Calculate the new image interval parameter value;

[0181] (6) Finally, the image is scaled according to the calculated multiple.

[0182] Through the above processing mode, the CT scan picture with different pixel intervals can be re-adjusted in interval, so that the final processed data interval meets the preset requirements.

[0183] By performing three-dimensional reconstruction calculation on each picture, the picture with non-uniform interval and image parameters can be integrated into a picture with uniform interval and image parameters required by the computer before three-dimensional reconstruction.

[0184] Figure 11 are perspective views of three groups of reconstructed three-dimensional models, each of which shows a schematic view of three perspectives of a three-dimensional model. Among them, the three-dimensional model is formed by recombining a plurality of tomographic scan pictures obtained by CT, and the three-dimensional model is a three-dimensional image that can be observed from multiple angles. That is, each view shows a recombined three-dimensional image from a different angle. The three-dimensional image is a three-dimensional image that can be observed from multiple angles, obtained by recombining the tomographic scan pictures obtained by CT after the above scaling process and then performing image recombination by software. Figure 12 、 Figure 13 and ​ can respectively show the state of image coincidence.

[0185] After the above picture parameter adjustment is completed, the model can be reconstructed in three-dimensional space. Matplotlib (or other tools with similar functions) can be used for drawing during reconstruction. Matplotlib is a 2D drawing library in Python, which generates publication quality level graphics in various hard copy formats and cross-platform interactive environments.

[0186] There is a function Poly3DCollection in mpl_toolkits (a toolkit in Matplotlib) designed for three-dimensional reconstruction, and the marching_cubes algorithm is also needed, which is mainly used to extract isosurfaces in a three-dimensional discrete data field. In the operation process, the algorithm will process each cube in the data field one by one, separate the cubes intersecting with the isosurface, and then calculate the intersection points of the isosurface and the cube edge by interpolation. According to the position of each vertex of the cube and the isosurface, the intersection points of the isosurface and the cube edge are connected to generate a new isosurface. It should be particularly emphasized here that different thresholds will result in different results according to different image types. In the application scenario related to tomography, the radiation density measure used is hounsfield (HU), and different radiation densities correspond to different human tissues and structures. For example, the radiation density of muscle tissue is +35~+45, and the radiation density of the lung is -700~ -600. According to the tissue part or structure to be reconstructed, the parameters can be adjusted to obtain the corresponding information content.

[0187] After the above adjustment is completed, all original preprocessed image information can be obtained, and then the three-dimensional coordinate is redrawn through the Matplotlib function to obtain a reconstructed three-dimensional image. The specific processing process is as follows:

[0188] Set the grid size of the final canvas: the first grid of 1x1;

[0189] Get image dimension one data;

[0190] Get image dimension two data;

[0191] Get image dimension three data;

[0192] The effect after reconstruction can be seen in ​ The figure is a perspective image of the human body in an embodiment, and other parts of the image except the coordinates will be mapped on the glasses.

[0193] After three-dimensional reconstruction, we can real-time transfer the three-dimensional image to the doctor's wearable device or holographic projection device. Through smart wearable devices such as smart glasses and holographic projectors, the three-dimensional image can be displayed in front of the operator in real time.

[0194] The principle of learning and application of the calibration object is as follows:

[0195] In the surgical scene, the need to track and identify different objects. In this process, through the system in the built-in function with additional accessories can achieve real-time tracking of special items. For example, in the system, through the use of artificial intelligence in the early automatic learning function, has been able to complete the learning of instruments such as surgical knives. Such as using TensorFlow algorithm to learn related objects.

[0196] TensorFlow is a data flow programming-based mathematical system widely used in programming of various machine learning algorithms. In implementation, modeling learning can be performed through TensorFlow to achieve identification of specific target objects. By marking a large number of surgical knives, indwelling needles, and hemostatic forceps and other commonly used surgical instruments in the program in advance and dividing all pictures into Train (training set) and Test (test set), learning is performed using the SSD MobileNet v2320x320 model in TensorFlow. Finally, the computer can automatically identify various commonly used surgical instruments.

[0197] At the same time, the present application also sets up a scheme for another scene: if the doctor needs to use a new tool A now, the system cannot track A because there is no pre-learning and preset. At this time, the present application uses a fluorescent attachment as a special color material that the camera can capture, which can directly calibrate the object in the system preset mode to quickly identify and track the unrecorded calibration object in the system.

[0198] In the processing process, a set of color characteristic tuning values can be preset, respectively corresponding to the HUE (hue) of the color, SAT (saturation) and VALUE (color value).

[0199] The software provides a manually adjustable color filter, which realizes the identification of specific colors through visual filtering of colors. After the above filtering is completed, the operator adjusts and corrects the above values on the control interface, and observes whether the required specific color and object are filtered through the real-time visual software interface. Once the required value characteristics are determined, the corresponding values can be saved through the save function. The example here is only one implementation, and the system will analyze the image according to the above values, and is not limited thereto.

[0200] The processing process can first set the values in hasValues to the corresponding array (np.array), and filter the image through the inRange function in cv2. After filtering, we can get the specific object we just filtered. Then, use the cv2.findContours method to mark the specific object we need. In other embodiments, other methods can also be used to achieve corresponding marking.

[0201] The advantages of the holographic image-based surgical assistance system of the above embodiments are as follows:

[0202] Real-time image placement in front of the doctor's eyes using holographic projection:

[0203] In the prior art surgery, if it involves the use of ultrasound-related equipment as a guide, the doctor often needs to hold the ultrasound probe with one hand and operate various medical instruments with the other hand. At the same time, in order to be able to see the accuracy of the operation under the ultrasound guidance, the doctor needs to pay close attention to the picture of the ultrasound instrument in real time. This results in inconsistency between the doctor's hands and eyes. The ultrasound instrument is placed on both sides of the operating bed, and the doctor also finds it difficult to maintain a straight view, so many times the doctor is looking at the picture on the ultrasound device with his head tilted, while operating with his hands. This operation of the hands and eyes that cannot be synchronized greatly affects the efficiency and accuracy of the doctor's operation. Long-term tilting of the neck to look at the display of the ultrasound device also brings a lot of neck-related occupational diseases to the doctor.

[0204] This fact instance uses holographic projection technology to project the picture in real time on the smart wearable device. The doctor only needs to wear the matching smart glasses to see the externally connected picture in real time. In this way, the doctor can see his own hands operating on the patient's body part while also seeing the real-time update of the ultrasound picture. This truly achieves hand-eye synchronization, and the efficiency and accuracy will be significantly improved, and long-term tilting of the neck to operate is also avoided.

[0205] Real-time preview of 3D image data:

[0206] Before many surgeries, patients need to undergo CT or magnetic resonance imaging and other related imaging examinations. These examinations provide great protection for the doctor to develop a surgical plan. However, such imaging data needs to be saved in the computer, and the doctor often needs to repeatedly read and memorize these data before the operation. Intraoperative operation is mostly based on memory.

[0207] In this embodiment, the holographic projection technology is used to project such imaging data into the doctor's smart wearable device, such as AR glasses. The doctor can review the relevant imaging data at any time in the book, and at the same time, the doctor can perform a series of virtual operations such as rotation and zoom on the picture in front of him by using the system's gesture capture. In this way, the doctor can be more calm during the operation. Without worrying about some details that cannot be checked and confirmed in time during the operation.

[0208] Navigation guidance of virtual and real operations:

[0209] During the operation, if the doctor is performing very precise operations, the doctor often cannot judge the precise distance of the surgical instrument, such as a surgical knife, into the patient's body. Or the doctor needs to know how far the surgical knife into the patient's body is from the blood vessels or nerves.

[0210] This embodiment uses three-dimensional reconstruction technology to restore the real patient's 1:1 three-dimensional model in the computer after reconstructing CT, magnetic resonance imaging and other imaging information. At the same time, the system cooperates with infrared ranging and high-definition camera binocular ranging to accurately capture the doctor's real-time operation. In addition, through the recognition of special calibration objects such as surgical knives by the camera, the precise distance of the surgical knife into the patient's body is accurately calculated, and the relative position parameter information between the specified points of each system is mapped on the lens of the glasses main body through the identification of points, lines and surfaces in the reconstructed three-dimensional model, so that the doctor can clearly see how far he is from the important parts or lesions of the patient. Realize digital precise operation.

[0211] Using gesture operation to control the device can achieve non-contact control, which can greatly reduce pollution. In the operating environment, sterility is the most basic medical environment requirement. However, in the actual operation process, a large number of devices and instruments need to be operated and adjusted by humans: adjustment of the operating bed, adjustment of the shadow lamp position, adjustment of various parameters of the B-ultrasound instrument, adjustment of various parameters of the monitor, etc. In this process, the main surgeon often cannot personally perform such operations due to pollution issues. At this time, the assistant or nurse needs to perform related operations by gestures or verbal explanations. Such operation method is not only low in efficiency, but also poor in accuracy.

[0212] The control action capture camera module provided by the embodiment can allow doctors to use hands as virtual control calibration objects to adjust some automatic electrical equipment such as electric operating tables and electric shadowless lamps through the rotation of gestures. In this way, the efficiency can be greatly improved and the additional manpower can be saved, and the virtual control accuracy and adjustment effect of the doctors through their own gestures can be completely controlled by the main doctors. The whole process is non-contact, and the requirements of reducing contact and reducing pollution in the medical scene can still be perfectly met.

[0213] In addition, in the second aspect, the present application also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to realize the following operation steps:

[0214] Obtaining panoramic images and obtaining relative position parameter information between each system specified point through the panoramic images;

[0215] Controlling a view angle image capture mechanism in the imaging glasses module to capture a view angle picture of a glasses main body in the imaging glasses module;

[0216] Controlling a projection mechanism in the imaging glasses module to map a human body perspective image and the relative position parameter information between each system specified point to a lens on the glasses main body;

[0217] Adjusting the position and size of the human body perspective image projected by the projection mechanism on the lens in real time according to the panoramic images, the view angle picture and the relative position parameter information between each system specified point, so as to ensure that the human body perspective image presented on the lens and the corresponding human body structure can still be kept in real-time coincidence when the relative position between the glasses main body and the human body changes.

[0218] In implementation, the system specified points include specified points on the patient's body, specified points on the wearer of the glasses main body and specified points on the specified medical apparatus;

[0219] The relative position parameter information between each system specified point includes relative distance parameter information and relative angle parameter information between the specified points on the patient's body, the specified points on the wearer of the glasses main body and the specified points on the specified medical apparatus.

[0220] The obtaining panoramic images and obtaining relative position parameter information between each system specified point through the panoramic images, includes:

[0221] Obtaining the panoramic images taken and identifying the position range corresponding to each system specified point from the panoramic images;

[0222] acquire coordinate positions of each of the system specified points in space;

[0223] According to the panoramic image, the position range corresponding to each of the system specified points, and the coordinate positions of each of the system specified points in space, relative position parameter information between each of the system specified points is acquired.

[0224] The human perspective image is a three-dimensional reconstruction model generated according to a medical image of a patient.

[0225] Real-time acquisition of the picture collected by the control motion capture camera module, acquisition of the operation control motion image of the wearer of the glasses, and adjustment of the size and angle of the human perspective image mapped to the lens by the projection mechanism according to the operation control motion image.

[0226] The implementation principle in this embodiment is similar to the implementation principle of the holographic image-based surgical assistance system mentioned in the first aspect, and thus is not described here.

[0227] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in a combination of software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different ways to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0228] In the embodiments of the present application, the disclosed system, device and method can be implemented in other ways. For example, the division of units is only a logical function division, and in actual implementation, there can be another division manner. For example, multiple units or components can be combined or integrated into another system. In addition, the coupling between the units can be direct coupling or indirect coupling. In addition, the functional units in the embodiments of the present application can be integrated in one processing unit, or can be separate physical units, etc.

[0229] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the execution order, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0230] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a machine readable storage medium. Therefore, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a machine readable storage medium, and can include a plurality of instructions to make an electronic device execute all or part of the processes described in the embodiments of the present application. The storage medium can include ROM, RAM, removable disks, hard disks, magnetic disks or optical disks, and various media that can store program codes.

[0231] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative work based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the prior art according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. A holographic image-based surgical assistance system, characterized by, The system comprises an image recognition and distance measurement module, an imaging glasses module and a main control module. The image recognition and distance measurement module is used to capture panoramic images and obtain relative position parameter information between each system designated point through the panoramic images. The imaging glasses module comprises a glasses body with transparent lenses, a projection mechanism and a visual angle image capture mechanism. The main control module is used to adjust the position and size of the human body perspective image projected onto the lenses by the projection mechanism in real time according to the panoramic images, the visual angle images and the relative position parameter information between each system designated point, so as to ensure that the human body perspective image presented on the lenses and the corresponding human body structure can still be kept in real time when the relative position of the glasses body and the human body changes. The system designated points comprise designated points on the patient's body, designated points on the wearer of the glasses body and designated points on the designated medical apparatus. The relative position parameter information between each system designated point comprises relative distance parameter information and relative angle parameter information between the designated points on the patient's body, the designated points on the wearer of the glasses body and the designated points on the designated medical apparatus. The image recognition and distance measurement module comprises: A camera unit is used to capture the panoramic images and identify the position range corresponding to each system designated point from the panoramic images. An infrared dot array projector unit is used to detect the coordinate position of each system designated point in space. The camera unit and the infrared dot array projector unit respectively transmit the panoramic images, the position range corresponding to each system designated point and the coordinate position of each system designated point in space to the main control module to obtain the relative position parameter information between each system designated point. The holographic image-based surgical assistance system further comprises: A control action capture camera module is used to capture the operation control action images of the wearer of the glasses body and transmit the operation control action images to the main control module. The main control module adjusts the size and angle of the human body perspective image mapped onto the lenses by the projection mechanism according to the operation control action images. The relative position parameter information between each system designated point is presented on the glasses in the form of text and graphics. When the direction of the knife is directed to the lesion, the distance of the knife is informed by digital prompt, and when the direction of the knife is not directed to the lesion, the direction of the knife is modified in time by arrow symbol prompt.

2. The holographic image-based surgery assistance system according to claim 1, wherein, The system adjusts the rotation angle of the image returned by the device by capturing the human hand.

3. The holographic image-based surgery assistance system of claim 1, wherein, The human body perspective image is a three-dimensional reconstruction model generated according to the medical images of the patient.

4. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following operation steps: acquire panoramic images, and acquire relative position parameter information between each system specified point through the panoramic images; control the view angle image capture mechanism in the imaging glasses module to capture the view angle picture of the glasses main body in the imaging glasses module; control the projection mechanism in the imaging glasses module to map the human body perspective image and the relative position parameter information between each system specified point to the lens on the glasses main body; adjust the position and size of the human body perspective image projected to the lens by the projection mechanism in real time according to the panoramic images, the view angle picture and the relative position parameter information between each system specified point, so as to ensure that the human body perspective image presented on the lens and the corresponding human body structure can still be kept coincident in real time when the relative position between the glasses main body and the human body changes; the system specified points include specified points on the patient's body, specified points on the wearer of the glasses main body and specified points on the specified medical apparatus; the relative position parameter information between each system specified point includes relative distance parameter information and relative angle parameter information between the specified points on the patient's body, the specified points on the wearer of the glasses main body and the specified points on the specified medical apparatus; the acquisition of the panoramic images and the acquisition of the relative position parameter information between each system specified point through the panoramic images include: acquire the panoramic images taken, and identify the position range corresponding to each system specified point from the panoramic images; acquire the coordinate position of each system specified point in space; acquire the relative position parameter information between each system specified point according to the panoramic images, the position range corresponding to each system specified point and the coordinate position of each system specified point in space; real-time acquisition of the picture collected by the control action capture camera module, acquisition of the operation control action image of the wearer of the glasses main body, adjustment of the size and angle of the human body perspective image mapped to the lens by the projection mechanism according to the operation control action image; the relative position parameter information between each system specified point is presented on the glasses in the form of text and graphics indication; when the direction of the knife points to the lesion, the distance of the knife is informed by digital prompt, and when the direction of the knife does not point to the lesion, the direction of the knife is modified in time by arrow symbol prompt.

5. The computer readable storage medium of claim 4, wherein the system adjusts the rotation angle of the picture of the image returned by the device by capturing the guide of the human hand.

6. The computer-readable storage medium of claim 4, wherein, the human body perspective image is a three-dimensional reconstruction model generated according to the medical image of the patient.

Images