Interactive image analysis auxiliary method and system for deep 3D aerial imaging

Abstract

The invention provides an interactive image analysis auxiliary method and system for deep 3D aerial imaging. The method comprises the following steps: acquiring a spine medical image of any object under a preoperative standard attitude; according to the spine medical image under the preoperative standard posture, a spine three-dimensional model under the intraoperative standard posture is generated; on the basis of the spinal column three-dimensional model under the standard posture in the operation, continuous placement information of medical instruments of multiple sections of vertebrae is generated; and projecting the placement information of the medical instrument to a corresponding spine position of the object in an operation in a 3D manner so as to assist in implanting the medical instrument. According to the method and the device, the placement information of the medical breath is generated based on the preoperative spine medical image and is projected to the spine position in the operation, so that corresponding medical personnel can execute the spine operation based on the projection without additionally shooting the medical image, and the radiation-free operation in the operation is realized.

Description

Technical Field

[0001] The present application relates to the field of medical image projection technology, and more specifically, to an interactive image analysis auxiliary method and system for deep 3D aerial imaging. Background Art

[0002] Spinal surgery usually requires the placement of nails in multiple consecutive segments. This operation of nailing or implanting other medical devices requires taking multiple medical images during the operation to determine the implantation location and implantation process, resulting in excessive radiation. Summary of the invention

[0003] The problem solved by the present application is that the radiation dose during spinal surgery is too high.

[0004] To solve the above problems, the first aspect of the present application provides an interactive image analysis auxiliary method for deep 3D aerial imaging, which includes: Acquire a spinal medical image of any subject in a preoperative standard posture; Generate a three-dimensional model of the spine in a standard posture during surgery based on the spinal medical image in a standard posture before surgery; Generate placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery; The placement information of the medical device is 3D projected onto the corresponding spinal position of the subject during surgery to assist in implanting the medical device.

[0005] The second aspect of the present application provides a manufacturing system for an interactive image analysis auxiliary method for deep 3D aerial imaging, which includes: An image acquisition unit, which is used to acquire a spinal medical image of any subject in a preoperative standard posture; A model generation unit, which is used to generate a three-dimensional model of the spine in a standard posture during the operation based on the spinal medical image in a standard posture before the operation; An instrument planning unit, which is used to generate placement information of medical instruments for consecutive multiple vertebrae based on a three-dimensional model of the spine in a standard posture during surgery; A 3D projection unit is used to project the placement information of the medical device in 3D to the corresponding spinal position of the object during the operation to assist in implanting the medical device.

[0006] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory may be configured to store a program, and the processor is coupled to the memory and is used to execute the program in the memory, so as to: Acquire a spinal medical image of any subject in a preoperative standard posture; Generate a three-dimensional model of the spine in a standard posture during surgery based on the spinal medical image in a standard posture before surgery; Generate placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery; The placement information of the medical device is 3D projected onto the corresponding spinal position of the subject during surgery to assist in implanting the medical device.

[0007] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the aforementioned interactive image analysis auxiliary method for deep 3D aerial imaging.

[0008] In the present application, medical atmosphere placement information is generated based on preoperative spinal medical images and projected to the spinal position during surgery, so that the corresponding medical personnel can perform spinal surgery based on projection without the need to take additional medical images, thereby achieving radiation-free surgery during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application; Figure 2 A schematic diagram of a standard posture during an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application; Figure 3 It is an architecture diagram of a network model of an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application; Figure 4 It is an architecture diagram of a fusion module of an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application; Figure 5 It is a flowchart of 3D projection in the interactive image analysis auxiliary method of deep 3D aerial imaging according to an embodiment of the present application; Figure 6 is a schematic diagram of 3D projection in an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application; Figure 7 is an architecture diagram of an interactive image analysis auxiliary system for deep 3D aerial imaging according to an embodiment of the present application; Figure 8 is an architectural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0010] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0011] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0012] The embodiment of the present application provides the above-mentioned interactive image analysis auxiliary method for deep 3D aerial imaging, and the specific scheme of the method is as follows: Figure 1-Figure 6 As shown, the method can be performed by an interactive image analysis auxiliary system for deep 3D aerial imaging, and the interactive image analysis auxiliary system for deep 3D aerial imaging can be integrated in electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, the interactive image analysis auxiliary method for deep 3D aerial imaging includes: S101, obtaining a spinal medical image of any subject in a standard posture before surgery; In the present application, the spinal medical image may be a medical CT image (Computed Tomography, CT) containing the spine, or may be other divisible medical images such as an MRI medical image.

[0013] In this application, the standard posture before surgery is: The patient's head should be kept in the midline, arms should be placed at the sides of the body, and the torso should be lying flat on the operating table. The legs should be naturally straightened and aligned with the midline of the body, and the feet should be kept in a natural neutral position.

[0014] S102, generating a three-dimensional model of the spine in a standard posture during the operation according to the spinal medical image in a standard posture before the operation; It should be noted that the human spine has different curvature states in different postures, which will seriously affect the accuracy of spinal surgery. The conventional method is to take multiple medical images such as CT or X-rays during the operation to determine the position and angle of each vertebra in the spine in turn. However, this frequent taking of medical images will bring higher radiation exposure to patients and doctors.

[0015] In this application, by fixing the preoperative standard posture and the intraoperative standard posture, the spinal changes of the human body in different postures are reduced to spinal changes in two standard postures, greatly reducing the disorder of spinal changes; the spinal deformation in the two postures is comprehensively trained through the convolution model, so that the spinal curvature changes in the intraoperative standard posture can be accurately predicted based on the preoperative standard posture, thereby achieving accurate surgery during the operation.

[0016] S103, generating placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during the operation; S104, projecting the placement information of the medical device in 3D to the corresponding spinal position of the subject during the operation to assist in implanting the medical device.

[0017] In the present application, medical atmosphere placement information is generated based on preoperative spinal medical images and projected to the spinal position during surgery, so that the corresponding medical personnel can perform spinal surgery based on projection without the need to take additional medical images, thereby achieving radiation-free surgery during surgery.

[0018] In the present application, by mapping the spinal medical image in the preoperative standard posture to the corresponding image in the intraoperative standard posture, the curvature changes of the patient's spinal part in the two standard postures are eliminated through mapping, thereby improving the accuracy of surgical planning and assistance.

[0019] In a specific embodiment, combining Figure 2 As shown, the standard posture during the operation is: the surgical subject is in a prone position, with both upper limbs bent forward and placed on both sides of the head, and the elbows are as wide as the shoulders; there is a cushion under the chest and abdomen so that the hips are flush with the shoulders; there is a cushion under the ankles so that the toes hang naturally, and the knees are as wide as the shoulders.

[0020] like Figure 2 As shown in the figure, in the standard posture during the operation, the patient lies prone with the head tilted to one side, and a head ring is placed to protect the auricle from pressure. The upper limbs are flexed forward and placed on both sides of the head, fixed to the armrest with an arm strap, and a square pad is placed under the chest at the same level as the shoulder. An air ring is placed on the hip to protect the external genitalia from pressure, and the abdomen is left empty to maintain normal activity of the abdominal muscles and diaphragm to avoid affecting breathing. A soft pad is placed under the knees, a spleen pad is placed on the ankle to allow the toes to droop naturally, and a sponge pad is placed on the calf to fix it with a restraint belt.

[0021] In the present application, the standard posture during surgery is defined, thereby increasing the accuracy of the inference of the curvature change of the spinal column during surgery.

[0022] In a specific embodiment, the medical device includes at least one of a screw, a fusion device, a titanium rod, and a cross-connection.

[0023] In the present application, in spinal surgery, screws are mainly used for fixation of pedicles, pedicle isthmus, lamina, lateral masses, etc., as well as for support and bracing.

[0024] Fusion cage is a kind of prosthesis, which is mainly used in spinal fusion surgery to replace the damaged intervertebral disc and provide an ideal environment for the fusion of two vertebrae. In this application, the intervertebral fusion cage is a prosthesis.

[0025] In this application, titanium rods are mainly used for spinal correction in spinal surgery.

[0026] In the present application, transverse connection is the abbreviation of transverse connection device, which is mainly used to connect with the longitudinal spinal rod in the posterior spinal fixation system to form a frame structure.

[0027] In the present application, S103 generates placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery, plans the placement strategy of the medical devices based on reinforcement learning, and based on the placement strategy, directly generates the placement information of the medical devices based on the input of the placement target of the medical devices to be placed.

[0028] For example, in actual execution, four screws and a fusion device need to be placed on two adjacent vertebrae. After the placement targets of the four screws and the fusion device are input into the corresponding placement strategy, the placement strategy is executed to generate the final placement positions and placement postures of the four screws and the fusion device on the adjacent vertebrae.

[0029] In this application, S103 generates placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery, including: Acquire placement information of the medical device to be placed, wherein the placement information includes category, quantity and placement location; Selecting a placement strategy corresponding to the category, quantity and placement position based on the placement information; The placement process of the medical devices of set categories, quantities and placement positions is planned based on the selected placement strategy.

[0030] In this application, the placement information includes category, quantity and placement location, and the category is the category, quantity and placement location of medical devices. It should be noted that the determination of the category depends on the specific division range between intelligent agents in reinforcement learning; if the same model of screws is divided into different types of intelligent agents, their categories here also belong to different categories.

[0031] It should be noted that if any of the categories, quantities and placement positions are different, the placement strategies will be different. For example, for screws of the same category and quantity, the placement strategies for screws placed on the 4th vertebra and screws placed on the 5th vertebra are different.

[0032] In the present application, after selecting a placement strategy, the placement strategy is executed / calculated to obtain a planned placement process of the medical device.

[0033] In the present application, the placement process of the medical device includes the target position and implantation order of the medical device.

[0034] It should be noted that the target position of the medical device can be determined based on the placement strategy; based on the type, quantity and placement position of the medical device, the implantation order of the medical device can be determined according to a preset order.

[0035] For example, the screws are implanted in a preset order from top to bottom and from left to right, and the implantation order of the two screws is determined by the implantation positions of the screws.

[0036] In one embodiment, before planning the placement strategy of the medical device based on reinforcement learning, the method further includes: Medical device training data is obtained, wherein the medical device training data includes a preoperative 3D spinal image, position information and posture information of each screw, each fusion device, each titanium rod and each cross-link, and a placement result of each medical device; a multi-agent reinforcement learning environment is established based on the medical device training data, and the state space, action space and reward function of the agent are determined, wherein each screw, each fusion device, each titanium rod and each cross-link is an agent; a placement strategy for each agent and a collaboration strategy between agents are constructed, wherein agents of the same category but different individuals have the same strategy network structure; based on the placement strategy and collaboration strategy, the agents are trained to obtain an optimized placement strategy for each agent.

[0037] In one embodiment, the state space of the screw includes the entry point, diameter, length, transverse plane inclination and sagittal plane inclination; the reward function of the screw is determined by the offset between the screw center and the central axis of the pedicle, whether the screw breaks through the front edge of the vertebral body, the relative relationship between the screw diameter and the pedicle diameter, and the relative relationship between the screw length and the vertebral body length.

[0038] In one embodiment, the state space of the fusion device includes position, length, height, width and inclination; the reward function of the fusion device is determined by the evaluated physiological curvature of the spine after adding the fusion device and the evaluated fusion rate of the spine.

[0039] In one embodiment, the state space of the titanium rod includes position, diameter, length and curvature; and the reward function of the fusion device is determined by the stability of the titanium rod screw connection. In one embodiment, the state space of the cross-connection includes position and length; and the reward function of the fuser is determined by evaluating the stability of the evaluation results and the structural rigidity of the cross-connection.

[0040] In a specific embodiment, combining Figure 3 As shown, the S102, based on the spinal medical image in the standard posture before the operation, generates a three-dimensional model of the spine in the standard posture during the operation; including: The spinal medical image in the standard posture before surgery is input into the CNN structure and the Transformer structure respectively to obtain the corresponding first feature map and second feature map; Inputting the first feature map and the second feature map into a fusion module for fusion processing to obtain a fused first fusion map and a fused second fusion map; The first fusion image is sequentially input into the CNN structure and the global extraction module to obtain a global extraction image; The second fusion map is sequentially input into the Transformer structure and the multi-convolution module to obtain a multi-convolution feature map; After adding the global extraction map and the multi-convolution feature map, a convolution process is performed to obtain a spinal medical image in a standard posture during surgery; The three-dimensional model of the spine is generated based on the transformed image of the spine in the standard posture during the operation.

[0041] In this application, the CNN structure is used to extract multi-level local features of the input image; the Transformer structure is used to capture the long-range dependencies of different sub-blocks of the image. Through the fusion module, the outputs of the CNN structure and the Transformer structure are cross-fused and bound to each other, thereby making full use of the local context features from CNN and the global context information of SwinTransformer.

[0042] In this application, the CNN structure and the Transformer structure are conventional structures, and their specific structural details will not be repeated in this application.

[0043] In this application, a multi-convolution module refers to multiple convolutions arranged continuously.

[0044] In the present application, the convolution processing performed after the addition may be a single convolution processing or multiple consecutive convolutions.

[0045] In a specific embodiment, combining Figure 4 As shown, the first feature map and the second feature map are input into the fusion module for fusion processing to obtain the fused first fusion map and the second fusion map, including: After rearranging the second feature map, the second feature map is concatenated with the first feature map to obtain a concatenated feature map; After performing channel attention extraction and spatial attention extraction on the spliced ​​feature map in sequence, the spliced ​​feature map is added to the spliced ​​feature map to obtain an added feature map; Segmenting the added feature map into a first segmentation feature map and a second segmentation feature map; The first segmentation feature map is sequentially subjected to convolution processing, RELU processing, and convolution processing to obtain a first fusion map; After the second segmentation feature map is subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing in sequence, a second fusion map is obtained.

[0046] In the present application, rearrangement is used to adjust the dimension of the multi-dimensional data so that the second feature map has the same size as the first feature map.

[0047] In this application, CA (Channel Attention) aims to make the network pay attention to different channels of the input feature map, thereby enhancing the weights of important feature channels and weakening unimportant channels.

[0048] In this application, SA (Spatial Attention) focuses on the different positions of the input feature map in the spatial dimension, thereby guiding the network to pay more attention to important spatial positions in the image.

[0049] In this application, channel attention and spatial attention are combined. Spatial attention generates attention maps in height and width dimensions to enhance attention to specific spatial areas.

[0050] In this application, Linear usually refers to the Fully Connected Layer (FCLayer), which is one of the most basic layers in a neural network. The linear layer performs a linear transformation between the input and output to map the features of the previous layer to a higher dimension or a specific output dimension.

[0051] In this application, segmentation refers to dividing the input feature map into several parts in a certain way to perform parallel calculations or different processing operations. The segmentation in this application is segmentation on the channel dimension, dividing the input feature map into several subsets along the channel dimension, and each subset performs different convolutions or operations.

[0052] In this application, by dividing the feature maps into multiple groups, the computational complexity of each convolutional layer can be reduced, the efficiency of the network can be improved, and the expressiveness of the model can be maintained.

[0053] In this application, multi-channel feature maps from two structures are sequentially enhanced based on CA and SA on the basis of channel connection, so as to achieve complementary advantages of the two structural information and make the fusion result have both spatial local structural information and global semantic information.

[0054] In this application, the fusion results are divided into two groups by channel division, where one group of feature maps is first layer normalized (LN), then linear dimension increase, GELU activation, and linear dimension reduction are processed in sequence, and then output and sent back to the Transformer structure; the other group of feature maps are processed in sequence by spatial convolution, ReLU activation, and spatial convolution, and then output and sent back to the CNN structure. Finally, the CNN structure and the Transformer structure synchronously receive the two groups of output feature maps generated by the fusion module, and enhance the feature maps of the corresponding structures with the help of residual connections.

[0055] In a specific implementation, the processing process of the global extraction module is: Input the input feature map into a 1×1 convolution layer set in parallel to obtain a first extraction map, a second extraction map, and a third extraction map respectively; wherein the first extraction map and the third extraction map are in HW×N format, and the second extraction map is in N×HW format; Multiply the first extracted image and the second extracted image to obtain a multiplied feature image; Perform softmax processing on the multiplied feature map to obtain the multiplication coefficient; Multiplying the multiplication coefficient with the third extraction map to obtain a coefficient multiplication map; Add the coefficient multiplication map and the input feature map to obtain the globally extracted output feature map.

[0056] In the present application, the input feature map is divided into three branches for 1×1 convolution processing. The dimensions of the first branch and the third branch are adjusted to HW×N format, and the dimension of the second branch is N×HW format; then the first branch is multiplied by the third branch to obtain a feature map with a dimension of HW×HW, and a coefficient is obtained through softmax processing; the coefficient is multiplied with the third branch, and the multiplied feature map is added to the input feature map to obtain an output feature map.

[0057] In this application, the dimensions of the first branch and the second branch are reversed through 1×1 convolution, thereby realizing multiplication between feature maps; through this multiplication, long-range dependencies are captured; by setting coefficients, the dependencies are embedded in the feature maps; and on this basis, the original input is added and a residual connection is used to embed the global extraction into the model without destroying the parameters.

[0058] In this application, long-range dependencies between features are captured through global extraction, so that the model can increase the extracted global features and make up for the shortcomings of the current model.

[0059] In the present application, the response at a position is calculated as the weighted sum of the features at all positions, so that when processing the information at each position, the processing is performed on the basis of considering all the position information that can be considered.

[0060] In a specific embodiment, combining Figure 5 As shown, the S104, 3D projecting the placement information of the medical device to the corresponding spinal position of the object during the operation, includes: S401, obtaining coordinate information of a positioning frame at a specific spinal column position of the subject; the relative position of the positioning frame and the spinal column remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the subject; The positioning frame is installed or fixed on a specific spinal position, which may be the skin surface closest to a vertebra; for example, the positioning frame is a square patch with a two-dimensional positioning code, which can be displayed under the perspective of X-ray and camera.

[0061] In the present application, the relative position of the positioning frame and the spine will have a certain deviation due to the characteristics of the human body. This deviation is assumed to be non-existent in the present application (that is, the deviation has little effect on the accuracy).

[0062] In the present application, the relative position of the positioning frame and the spine remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the subject. Therefore, in the real surgical scene during the operation, as long as the position and posture of the positioning frame are determined, the position information of the spine can be confirmed.

[0063] In this step, the position information of the spine in the actual surgical scene is determined.

[0064] S402, determining a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; In this step, the intraoperative coordinate system is the coordinate system where the position information of the spine in the actual surgical scene is located; the preoperative coordinate system is the coordinate system in the preoperative medical image, wherein the placement information of medical devices for multiple vertebrae is located in the preoperative coordinate system, and the three-dimensional model of the spine in the intraoperative standard posture generated according to the spinal medical image in the preoperative standard posture is also located in the preoperative coordinate system.

[0065] In the present application, the spinal medical image in the standard posture before surgery also has a positioning frame. Based on the positioning frame in the medical image, the relative position relationship between the positioning frame and the spine in the standard posture before surgery can be determined; the relative position relationship between the positioning frame and the spine in the generated standard posture during surgery can also be determined. Generally, the relative position relationship between the two is very similar, with little difference, and is assumed to be the same.

[0066] In this step, the mapping relationship between the intraoperative coordinate system and the preoperative coordinate system can be determined through the position information of the positioning frame in the intraoperative coordinate system and the preoperative coordinate system (assuming that the actual position of the positioning frame is fixed, the change of its position information in the intraoperative coordinate system and the preoperative coordinate system is obtained by mapping the mapping relationship between the two).

[0067] S403, mapping the three-dimensional spine model and the placement information of the medical device to an intraoperative coordinate system according to the mapping relationship; In the present application, the position information of the positioning frame in the preoperative coordinate system is mapped through a mapping relationship to obtain the position information of the positioning frame in the intraoperative coordinate system (principle); similarly, the three-dimensional model of the spine and the placement information of the medical device in the preoperative coordinate system are mapped through a mapping relationship to obtain the three-dimensional model of the spine and the placement information of the medical device in the intraoperative coordinate system.

[0068] In this way, through mapping, the three-dimensional model of the spine and the placement information of the medical device are directly overlapped with the real surgical scene. In this overlapped state, the coordinates of the three-dimensional model of the spine overlap with the spine of the real surgical subject, and the placement information of the medical device also overlaps with the spine of the real surgical subject, indicating the implantation plan of the medical device at the corresponding spinal position.

[0069] S404, projecting the three-dimensional spine model and placement information of medical instruments in the intraoperative coordinate system to the corresponding spine position of the subject through a laser projection device installed in the operating room.

[0070] In this application, a laser projector with high precision and contrast can be used, which can accurately project images onto irregular surfaces, that is, the corresponding spine position of the object, to form a similar perspective effect. In order to match the projection with the curved surface of the human body, the target area (spine, back) can be three-dimensionally scanned, and the scanning data is used to capture the shape and unevenness of the human body surface; the three-dimensional scan data of the human body surface is registered with the data to be projected, and the registration technology is used to ensure that the image can adapt to the curved surface of the human body and is accurately displayed at the corresponding spine position.

[0071] In the present application, an AR device can also be used to project the three-dimensional model of the spine and the placement information of the medical device in the central coordinate system to the corresponding spinal position of the object photographed by the AR device (real surgical scene), thereby forming a perspective-like effect in the eyes of the AR user.

[0072] In the present application, by projecting the three-dimensional spine model and the placement information of the medical device to the corresponding spinal position of the object, medical staff can directly observe the planning information and smoothly perform spinal surgery.

[0073] In addition, this projection setting can ensure that there is no contact between the projection and the surgical object, thereby maintaining an overall sterile environment.

[0074] It should be noted that an additional sound module or surveillance camera can be set up to collect the voice or gestures of medical staff, and the projection can be controlled to produce corresponding changes after recognition by the large model, so as to realize contactless interactive control between the projection and the medical staff during the operation. The large model can be a BERT (Bidirectional Encoder Representations from Transformers) model or a GPT series model.

[0075] An embodiment of the present application provides an interactive image analysis assistance system for deep 3D aerial imaging, which is used to execute the interactive image analysis assistance method for deep 3D aerial imaging described in the above content of the present application. The interactive image analysis assistance system for deep 3D aerial imaging is described in detail below.

[0076] like Figure 7 As shown, the interactive image analysis auxiliary system for deep 3D aerial imaging includes: An image acquisition unit 101 is used to acquire a spinal medical image of any subject in a standard posture before surgery; A model generating unit 102, which is used to generate a three-dimensional model of the spine in a standard posture during the operation according to the spinal medical image in a standard posture before the operation; An instrument planning unit 103, which is used to generate placement information of medical instruments for multiple consecutive vertebrae based on a three-dimensional model of the spine in a standard posture during surgery; The 3D projection unit 104 is used to project the placement information of the medical device in 3D to the corresponding spinal position of the object during the operation to assist in implanting the medical device.

[0077] In one embodiment, the standard posture during the operation is: the surgical subject is in a prone position, with both upper limbs bent forward and placed on both sides of the head, and the elbows are as wide as the shoulders; there is a cushion under the chest and abdomen so that the hips are flush with the shoulders; there is a cushion under the ankles so that the toes hang naturally, and the knees are as wide as the shoulders.

[0078] In one embodiment, the medical device includes at least one of a screw, a fusion device, a titanium rod, and a cross-connection.

[0079] In one embodiment, the model generation unit 102 is further configured to: The spinal medical image in the preoperative standard posture is respectively input into the CNN structure and the Transformer structure to obtain the corresponding first feature map and the second feature map; the first feature map and the second feature map are input into the fusion module for fusion processing to obtain the fused first fusion map and the second fusion map; the first fusion map is sequentially input into the CNN structure and the global extraction module to obtain the global extraction map; the second fusion map is sequentially input into the Transformer structure and the multi-convolution module to obtain the multi-convolution feature map; after adding the global extraction map and the multi-convolution feature map, a convolution process is performed to obtain the spinal medical image in the standard posture during the operation; based on the spinal transformation image in the standard posture during the operation, the spinal three-dimensional model is generated.

[0080] In one embodiment, the model generation unit 102 is further configured to: After rearranging the second feature map, the second feature map is spliced ​​with the first feature map to obtain a spliced ​​feature map; after performing channel attention extraction and spatial attention extraction on the spliced ​​feature map in sequence, the spliced ​​feature map is added to the spliced ​​feature map to obtain an added feature map; the added feature map is segmented into a first segmentation feature map and a second segmentation feature map; the first segmentation feature map is sequentially subjected to convolution processing, RELU processing and convolution processing to obtain a first fusion map; the second segmentation feature map is sequentially subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing to obtain a second fusion map.

[0081] In one implementation, the 3D projection unit 104 is further configured to: The coordinate information of the positioning frame at the specific spinal position of the object is obtained; the relative position of the positioning frame and the spine remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the object; based on the coordinate information of the positioning frame, the mapping relationship between the intraoperative coordinate system and the preoperative coordinate system is determined; based on the mapping relationship, the three-dimensional model of the spine and the placement information of the medical device are mapped to the intraoperative coordinate system; through the laser projection equipment installed in the operating room, the three-dimensional model of the spine and the placement information of the medical device in the intraoperative coordinate system are projected to the corresponding spinal position of the object.

[0082] The interactive image analysis assistance system for deep 3D aerial imaging provided in the above-mentioned embodiments of the present application has a corresponding relationship with the interactive image analysis assistance method for deep 3D aerial imaging provided in the embodiments of the present application. Therefore, the specific content in the system has a corresponding relationship with the interactive image analysis assistance method for deep 3D aerial imaging. The specific content can be referred to the records in the interactive image analysis assistance method for deep 3D aerial imaging, and will not be repeated in this application.

[0083] The interactive image analysis assistance system for deep 3D aerial imaging provided by the above-mentioned embodiments of the present application and the interactive image analysis assistance method for deep 3D aerial imaging provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0084] The above describes the internal functions and structure of the interactive image analysis auxiliary system for deep 3D aerial imaging, such as Figure 8 As shown, in practice, the interactive image analysis auxiliary system for deep 3D aerial imaging can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0085] The memory 301 may be configured to store programs.

[0086] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0087] The memory 301 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The processor 303, coupled to the memory 301, is used to execute the program in the memory 301 for: Acquire a spinal medical image of any subject in a preoperative standard posture; Generate a three-dimensional model of the spine in a standard posture during surgery based on the spinal medical image in a standard posture before surgery; Generate placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery; The placement information of the medical device is 3D projected onto the corresponding spinal position of the subject during surgery to assist in implanting the medical device.

[0088] In one embodiment, the standard posture during the operation is: the surgical subject is in a prone position, with both upper limbs bent forward and placed on both sides of the head, and the elbows are as wide as the shoulders; there is a cushion under the chest and abdomen so that the hips are flush with the shoulders; there is a cushion under the ankles so that the toes hang naturally, and the knees are as wide as the shoulders.

[0089] In one embodiment, the medical device includes at least one of a screw, a fusion device, a titanium rod, and a cross-connection.

[0090] In one implementation, the processor 303 is further configured to: The spinal medical image in the preoperative standard posture is respectively input into the CNN structure and the Transformer structure to obtain the corresponding first feature map and the second feature map; the first feature map and the second feature map are input into the fusion module for fusion processing to obtain the fused first fusion map and the second fusion map; the first fusion map is sequentially input into the CNN structure and the global extraction module to obtain the global extraction map; the second fusion map is sequentially input into the Transformer structure and the multi-convolution module to obtain the multi-convolution feature map; after adding the global extraction map and the multi-convolution feature map, a convolution process is performed to obtain the spinal medical image in the standard posture during the operation; based on the spinal transformation image in the standard posture during the operation, the spinal three-dimensional model is generated.

[0091] In one implementation, the processor 303 is further configured to: After rearranging the second feature map, the second feature map is spliced ​​with the first feature map to obtain a spliced ​​feature map; after performing channel attention extraction and spatial attention extraction on the spliced ​​feature map in sequence, the spliced ​​feature map is added to the spliced ​​feature map to obtain an added feature map; the added feature map is segmented into a first segmentation feature map and a second segmentation feature map; the first segmentation feature map is sequentially subjected to convolution processing, RELU processing and convolution processing to obtain a first fusion map; the second segmentation feature map is sequentially subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing to obtain a second fusion map.

[0092] In one implementation, the processor 303 is further configured to: The coordinate information of the positioning frame at the specific spinal position of the object is obtained; the relative position of the positioning frame and the spine remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the object; based on the coordinate information of the positioning frame, the mapping relationship between the intraoperative coordinate system and the preoperative coordinate system is determined; based on the mapping relationship, the three-dimensional model of the spine and the placement information of the medical device are mapped to the intraoperative coordinate system; through the laser projection equipment installed in the operating room, the three-dimensional model of the spine and the placement information of the medical device in the intraoperative coordinate system are projected to the corresponding spinal position of the object.

[0093] In this application, Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.

[0094] The electronic device provided in this embodiment is based on the same inventive concept as the interactive image analysis auxiliary method for deep 3D aerial imaging provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). Memory is an example of a computer-readable medium.

[0099] The present application also provides a computer-readable storage medium corresponding to the interactive image analysis auxiliary method for deep 3D aerial imaging provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the interactive image analysis auxiliary method for deep 3D aerial imaging provided in any of the aforementioned embodiments.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the interactive image analysis auxiliary method for 3D aerial imaging provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0102] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An interactive image analysis auxiliary method for deep 3D aerial imaging, characterized in that: include: Acquire a spinal medical image of any subject in a preoperative standard posture; Generate a three-dimensional model of the spine in a standard posture during surgery based on the spinal medical image in a standard posture before surgery; Generate placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery; The placement information of the medical device is 3D projected onto the corresponding spinal position of the subject during surgery to assist in implanting the medical device.

2. The interactive image analysis auxiliary method for deep 3D aerial imaging according to claim 1, characterized in that: The standard posture during the operation is: the patient lies prone with both upper limbs bent forward and placed on both sides of the head, with elbows as wide as shoulders; a cushion is placed under the chest and abdomen so that the hips are level with the shoulders; a cushion is placed under the ankles so that the toes hang naturally, and the knees are as wide as shoulders.

3. The interactive image analysis auxiliary method for deep 3D aerial imaging according to claim 1, characterized in that: The medical device includes at least one of a screw, a fusion device, a titanium rod, and a cross-connection.

4. The interactive image analysis auxiliary method for deep 3D aerial imaging according to any one of claims 1 to 3, characterized in that: The method of generating a three-dimensional model of the spine in a standard posture during the operation according to the spinal medical image in a standard posture before the operation comprises: The spinal medical image in the standard posture before surgery is input into the CNN structure and the Transformer structure respectively to obtain the corresponding first feature map and second feature map; Inputting the first feature map and the second feature map into a fusion module for fusion processing to obtain a fused first fusion map and a fused second fusion map; The first fusion image is sequentially input into the CNN structure and the global extraction module to obtain a global extraction image; The second fusion map is sequentially input into the Transformer structure and the multi-convolution module to obtain a multi-convolution feature map; After adding the global extraction map and the multi-convolution feature map, a convolution process is performed to obtain a spinal medical image in a standard posture during surgery; The three-dimensional model of the spine is generated based on the transformed image of the spine in the standard posture during the operation.

5. The interactive image analysis auxiliary method for deep 3D aerial imaging according to claim 4, characterized in that: The first feature map and the second feature map are input into the fusion module for fusion processing to obtain the fused first fusion map and the second fusion map, including: After rearranging the second feature map, the second feature map is concatenated with the first feature map to obtain a concatenated feature map; After performing channel attention extraction and spatial attention extraction on the spliced ​​feature map in sequence, the spliced ​​feature map is added to the spliced ​​feature map to obtain an added feature map; Segmenting the added feature map into a first segmentation feature map and a second segmentation feature map; The first segmentation feature map is sequentially subjected to convolution processing, RELU processing, and convolution processing to obtain a first fusion map; After the second segmentation feature map is subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing in sequence, a second fusion map is obtained.

6. The interactive image analysis auxiliary method for deep 3D aerial imaging according to any one of claims 1 to 3, characterized in that: The 3D projection of the placement information of the medical device to the corresponding spinal position of the object during the operation includes: Acquiring coordinate information of a positioning frame at a specific spinal position of the subject; the relative position of the positioning frame and the spinal column remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the subject; Determining a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; According to the mapping relationship, the three-dimensional spine model and the placement information of the medical device are mapped to an intraoperative coordinate system; The three-dimensional model of the spine and the placement information of the medical device in the intraoperative coordinate system are projected to the corresponding spine position of the object through the laser projection equipment installed in the operating room.

7. An interactive image analysis auxiliary method system for deep 3D aerial imaging, characterized in that: include: An image acquisition unit, which is used to acquire a spinal medical image of any subject in a preoperative standard posture; A model generation unit, which is used to generate a three-dimensional model of the spine in a standard posture during the operation based on the spinal medical image in a standard posture before the operation; An instrument planning unit, which is used to generate placement information of medical instruments for consecutive multiple vertebrae based on a three-dimensional model of the spine in a standard posture during surgery; A 3D projection unit is used to project the placement information of the medical device in 3D to the corresponding spinal position of the object during the operation to assist in implanting the medical device.

8. The interactive image analysis auxiliary method system for deep 3D aerial imaging according to claim 7, characterized in that: The 3D projection unit is also used for: Acquiring coordinate information of a positioning frame at a specific spinal position of the subject; the relative position of the positioning frame and the spinal column remains unchanged in both the preoperative standard posture and the intraoperative standard posture of the subject; Determining a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; According to the mapping relationship, the three-dimensional spine model and the placement information of the medical device are mapped to an intraoperative coordinate system; The three-dimensional model of the spine and the placement information of the medical device in the intraoperative coordinate system are projected to the corresponding spine position of the object through the laser projection equipment installed in the operating room.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquire a spinal medical image of any subject in a preoperative standard posture; Generate a three-dimensional model of the spine in a standard posture during surgery based on the spinal medical image in a standard posture before surgery; Generate placement information of medical devices for multiple consecutive vertebrae based on the three-dimensional model of the spine in the standard posture during surgery; The placement information of the medical device is 3D projected onto the corresponding spinal position of the subject during surgery to assist in implanting the medical device.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the interactive image analysis auxiliary method for deep 3D aerial imaging as described in any one of claims 1-6.

Images