An interactive image analysis assistance method and system for deep 3D aerial imaging

Abstract

The application provides an interactive image analysis auxiliary method and system for deep 3D aerial imaging, which comprises the following steps: acquiring a preoperative standard posture spinal medical image of any object; generating a three-dimensional spinal model in an intraoperative standard posture according to the preoperative standard posture spinal medical image; generating placement information of a medical instrument for continuous multi-segment vertebrae based on the three-dimensional spinal model in the intraoperative standard posture; and projecting the placement information of the medical instrument to the corresponding spinal position of the object in the intraoperative 3D to assist in implanting the medical instrument. In the application, the placement information of the medical instrument is generated based on the preoperative spinal medical image, and is projected to the spinal position in the intraoperative, so that the medical personnel can perform spinal surgery based on the projection, without the need of additional medical image shooting, thereby realizing intraoperative radiation-free surgery.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image projection, in particular to a deep 3D aerial imaging interactive image analysis auxiliary method and system. BACKGROUND

[0002] In a spine surgery, multiple consecutive segments need to be implanted with screws. Such screw implantation or implantation of other medical devices requires the intraoperative shooting of multiple medical images to determine the implantation position and implantation process, resulting in excessive radiation. SUMMARY

[0003] The problem solved by the present application is the excessive radiation of intraoperative shooting in spine surgery.

[0004] To solve the above problems, the first aspect of the present application provides a deep 3D aerial imaging interactive image analysis auxiliary method, which comprises:

[0005] Obtaining a preoperative standard posture spinal medical image of any object;

[0006] Generating a three-dimensional model of the spine under the intraoperative standard posture according to the preoperative standard posture spinal medical image;

[0007] Generating placement information of a medical device for consecutive multiple vertebrae based on the three-dimensional model of the spine under the intraoperative standard posture;

[0008] 3D projecting the placement information of the medical device to the corresponding spine position of the object in the operation to assist in implanting the medical device.

[0009] The second aspect of the present application provides a manufacturing system of a deep 3D aerial imaging interactive image analysis auxiliary method, which comprises:

[0010] An image acquisition unit for acquiring a preoperative standard posture spinal medical image of any object;

[0011] A model generation unit for generating a three-dimensional model of the spine under the intraoperative standard posture according to the preoperative standard posture spinal medical image;

[0012] A device planning unit for generating placement information of a medical device for consecutive multiple vertebrae based on the three-dimensional model of the spine under the intraoperative standard posture;

[0013] A 3D projection unit for 3D projecting the placement information of the medical device to the corresponding spine position of the object in the operation to assist in implanting the medical device.

[0014] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory is configured to store a program, and the processor is coupled to the memory and used to execute the program in the memory, so as to:

[0015] Obtaining a spine medical image of an object in a preoperative standard posture;

[0016] Generating a spine three-dimensional model in an intraoperative standard posture according to the spine medical image in the preoperative standard posture;

[0017] Generating placement information of a medical instrument of a plurality of vertebrae based on the spine three-dimensional model in the intraoperative standard posture;

[0018] 3D projecting the placement information of the medical instrument to a corresponding spine position of the object in the operation to assist in implanting the medical instrument.

[0019] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the aforementioned depth 3D aerial imaging interactive image analysis auxiliary method.

[0020] In the present application, the placement information of the medical instrument is generated based on the preoperative spine medical image, and is projected to the spine position in the operation, so that the medical personnel can perform the spine surgery based on the projection, without the need of additional medical image shooting, thereby realizing the intraoperative radiation-free surgery. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of the depth 3D aerial imaging interactive image analysis auxiliary method according to the embodiment of the present application;

[0022] Figure 2 A schematic diagram of an intraoperative standard posture of the depth 3D aerial imaging interactive image analysis auxiliary method according to the embodiment of the present application;

[0023] Figure 3 An architecture diagram of a network model of the depth 3D aerial imaging interactive image analysis auxiliary method according to the embodiment of the present application;

[0024] Figure 4 An architecture diagram of a fusion module of the depth 3D aerial imaging interactive image analysis auxiliary method according to the embodiment of the present application;

[0025] Figure 5 A flowchart of 3D projection in the depth 3D aerial imaging interactive image analysis auxiliary method according to the embodiment of the present application;

[0026] Figure 63D projection diagram of an interactive image analysis auxiliary method for deep 3D aerial imaging according to an embodiment of the present application;

[0027] Figure 7 1 is an architecture diagram of an interactive image analysis assistance system for deep 3D aerial imaging according to an embodiment of the present application;

[0028] Figure 8 2 is a diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, specific embodiments of the present application are described in detail below with reference to the accompanying drawings. Although the accompanying drawings show exemplary embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0030] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which this application belongs.

[0031] The embodiment of the present application provides the above-mentioned interactive image analysis auxiliary method for deep 3D aerial imaging, the specific scheme of which is as follows: Figures 1-6 As shown, the method can be performed by an interactive image analysis auxiliary system for deep 3D aerial imaging, and the interactive image analysis auxiliary system for deep 3D aerial imaging can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, the interactive image analysis auxiliary method for deep 3D aerial imaging includes:

[0032] S101, obtaining a spinal medical image of any subject in a standard preoperative posture;

[0033] In the present application, the spinal medical image may be a medical CT image (Computed Tomography, CT) containing the spine, or may be other medical images that can be divided into different types, such as an MRI medical image.

[0034] In this application, the standard posture before surgery is:

[0035] The patient's head should be kept in the midline, arms should be placed at the sides of the body, and the torso should be straight on the operating table. The legs should be naturally straightened and aligned with the midline of the body, and the feet should be kept in a natural neutral position.

[0036] S102, generate a spine three-dimensional model in the intraoperative standard posture according to the spine medical image in the preoperative standard posture;

[0037] It should be noted that the human spine has different bending states with different postures, which seriously affects the accuracy of the spine surgery. The conventional way is to sequentially determine the position and angle of each vertebral body in the spine by taking multiple medical images such as CT or X-ray films during the operation. However, this frequent shooting of medical images will bring higher radiation effects to the patient and the doctor.

[0038] In this application, by fixing the preoperative standard posture and the intraoperative standard posture, the change of the human spine in different postures is reduced to the change of the spine in the two standard postures, greatly reducing the disorder of the change of the spine. By training the spine deformation in the two postures by the convolution model, the bending change of the spine in the intraoperative standard posture can be accurately predicted based on the preoperative standard posture, and accurate surgery in the operation is realized.

[0039] S103, generate placement information of a medical instrument of a plurality of consecutive vertebrae based on the spine three-dimensional model in the intraoperative standard posture;

[0040] S104, project the placement information of the medical instrument 3D to the corresponding spine position of the object in the operation to assist in implanting the medical instrument.

[0041] In this application, the placement information of the medical instrument is generated based on the preoperative spine medical image, and is projected to the spine position in the operation, so that the medical personnel can perform the spine surgery based on the projection, without the need for additional shooting of medical images, thereby realizing the radiation-free surgery in the operation.

[0042] In this application, the spine medical image in the preoperative standard posture is mapped to the corresponding image in the intraoperative standard posture, so that the bending change of the spine part of the patient in the two standard postures is eliminated by mapping, and the accuracy of the operation planning and assistance is improved.

[0043] In one specific embodiment, in combination with Figure 2 As shown in the figure, the intraoperative standard posture is that the surgical object is in a prone position, the upper limbs are bent forward, placed on both sides of the head, and the elbows are as wide as the shoulders; there is a soft pad under the chest and abdomen, so that the hips are flush with the shoulders; there is a soft pad on the ankle, so that the toes naturally droop, and the knees are as wide as the shoulders.

[0044] As Figure 2As shown, in the intraoperative standard posture, the patient is in a prone position, the head is tilted to one side, and the head ring is used to protect the auricle from pressure. The upper limbs are flexed forward and placed on both sides of the head, and the arm restraint is fixed on the hand rest frame. A square pad is placed under the chest to keep the shoulders level. The hip is padded with an air ring to protect the external genitalia from pressure, and the abdomen is kept empty to maintain normal activity of the abdominal muscles and diaphragm to prevent respiratory impairment. A soft pad is placed under the knees, and a spleen pad is placed under the ankles to allow the toes to naturally droop. A sponge pad is placed on the lower leg and fixed with a restraint belt.

[0045] In this application, by limiting the intraoperative standard posture, the accuracy of the speculation of the change in the curvature of the intraoperative spine is increased.

[0046] In a specific embodiment, the medical instrument includes at least one screw, a fusion cage, a titanium rod, and a cross-link.

[0047] In this application, in spinal surgery, screws are mainly used for fixation and support of pedicles, pedicle isthmuses, laminae, and lateral masses, as well as support and support.

[0048] A fusion cage is a type of prosthesis used in spinal fusion surgery to replace damaged intervertebral discs and provide an ideal environment for the fusion of two vertebrae. In this application, an intervertebral fusion cage is a prosthesis.

[0049] In this application, in spinal surgery, titanium rods are mainly used for spinal orthopedics.

[0050] In this application, a cross-link is a short name for a transverse connecting device, which is mainly used to connect with longitudinal spinal rods in posterior internal fixation systems of the spine to form a frame structure.

[0051] In this application, S103, based on the three-dimensional model of the spine in the intraoperative standard posture, generates placement information for medical instruments for continuous multiple vertebrae, plans the placement strategy of the medical instruments based on reinforcement learning, and based on the placement strategy, directly generates the placement information of the medical instruments based on the input of the placement target of the medical instruments to be placed.

[0052] For example, four screws and a fusion cage need to be placed on two adjacent vertebrae during actual execution, and after inputting the placement targets of the four screws and the fusion cage into the corresponding placement strategy, executing the placement strategy generates the final placement position and attitude of the four screws and the fusion cage on the adjacent vertebrae.

[0053] In this application, S103, based on the three-dimensional model of the spine in the intraoperative standard posture, generates placement information for medical instruments for continuous multiple vertebrae, including:

[0054] Obtain the placement information of the medical instrument to be placed, which includes the category, quantity, and placement position;

[0055] select a placement strategy corresponding to the category, quantity and placement position based on the placement information;

[0056] plan a placement process of the medical instrument of the set category, quantity and placement position based on the selected placement strategy.

[0057] In the present application, the placement information includes the category, quantity and placement position, the category is the category of the medical instrument, the quantity and the placement position. It should be noted that the judgment of the category depends on the specific division range between agents in reinforcement learning; if screws of the same model are divided into different types of agents, their categories in this case also belong to different categories.

[0058] It should be noted that any difference in the category, quantity and placement position results in a different placement strategy. For example, the placement strategies of screws of the same category and quantity placed on the fourth vertebra and the fifth vertebra are different.

[0059] In the present application, after selecting the placement strategy, executing / calculating the placement strategy can obtain the planned placement process of the medical instrument.

[0060] In the present application, the placement process of the medical instrument includes the target position of the medical instrument and the implantation sequence.

[0061] It should be noted that the target position of the medical instrument can be determined based on the placement strategy; the implantation sequence of the medical instrument can be determined according to a preset sequence based on the category, quantity and placement position of the medical instrument.

[0062] For example, when the screws are implanted, the preset sequence is from top to bottom and from left to right, and the implantation sequence of the two screws is determined by the implantation position of the screws.

[0063] In one embodiment, before the placement strategy of the medical instrument is planned based on reinforcement learning, the method further comprises:

[0064] obtaining medical instrument training data, the medical instrument training data including preoperative 3D spine image, position information and attitude information of each screw, each fusion, each titanium rod and each transverse connection, and placement result of each medical instrument; establishing a multi-agent reinforcement learning environment based on the medical instrument training data, determining state space, action space and reward function of agents, each screw, each fusion, each titanium rod and each transverse connection being an agent; constructing placement strategies of each agent and cooperation strategies between agents, the strategy network structures of agents of the same category and different individuals being the same; training the agents based on the placement strategies and cooperation strategies to obtain optimized placement strategies of each agent.

[0065] In one embodiment, the state space of the screw includes entry point, diameter, length, cross-sectional angle and sagittal angle; the reward function of the screw is determined by the offset of the screw center from the pedicle axis, whether the screw breaks through the anterior edge of the vertebral body, the relative relationship between the screw diameter and the pedicle diameter, and the relative relationship between the screw length and the vertebral body length.

[0066] In one embodiment, the state space of the fusion cage includes position, length, height, width and angle; the reward function of the fusion cage is determined by the evaluation of the physiological curvature of the spine after adding the fusion cage and the evaluation of the fusion rate of the spine.

[0067] In one embodiment, the state space of the titanium rod includes position, diameter, length and bending degree; the reward function of the fusion cage is determined by the stability of the titanium rod screw connection.

[0068] In one embodiment, the state space of the cross-link includes position and length; the reward function of the fusion cage is determined by the evaluation of the stability of the cross-link and the structural rigidity of the cross-link.

[0069] In a specific embodiment, in combination Figure 3 As shown, the S102, according to the preoperative standard posture spinal medical image, generates the intraoperative standard posture spinal three-dimensional model; including:

[0070] The preoperative standard posture spinal medical image is input into the CNN structure and the Transformer structure respectively, and the corresponding first feature map and the second feature map are obtained;

[0071] The first feature map and the second feature map are input into the fusion module for fusion processing, and the first fusion map and the second fusion map after fusion are obtained;

[0072] The first fusion map is sequentially input into the CNN structure and the global extraction module, and the global extraction map is obtained;

[0073] The second fusion map is sequentially input into the Transformer structure and the multi-convolution module, and the multi-convolution feature map is obtained;

[0074] After adding the global extraction map and the multi-convolution feature map, convolution processing is performed to obtain the intraoperative standard posture spinal medical image;

[0075] Based on the intraoperative standard posture spinal conversion image, the spinal three-dimensional model is generated.

[0076] In the present application, the CNN structure is used to extract multi-level local features of the input image, and the Transformer structure is used to capture the long-range dependency relationship of different sub-blocks of the image. The outputs of the CNN structure and the Transformer structure are cross-fused and bound to each other through a fusion module, so as to fully utilize the local context features from the CNN and the global context information from the SwinTransformer.

[0077] In the present application, the CNN structure and the Transformer structure are conventional structures, and the specific structural details thereof will not be described herein.

[0078] In the present application, the multi-convolution module is a plurality of convolutions arranged in succession.

[0079] In the present application, the convolution processing performed after the addition can be a single convolution processing or a plurality of convolutions in succession.

[0080] In one specific embodiment, the CNN structure and the Transformer structure are combined as follows: Figure 4 As shown in the figure, the first feature map and the second feature map are input into the fusion module for fusion processing to obtain a first fused feature map and a second fused feature map, which comprises:

[0081] The second feature map is rearranged and spliced with the first feature map to obtain a spliced feature map.

[0082] The spliced feature map is sequentially subjected to channel attention extraction and spatial attention extraction, and then added to the spliced feature map to obtain an added feature map.

[0083] The added feature map is divided into a first divided feature map and a second divided feature map.

[0084] The first divided feature map is sequentially subjected to convolution processing, RELU processing and convolution processing to obtain the first fused feature map.

[0085] The second divided feature map is sequentially subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing to obtain the second fused feature map.

[0086] In the present application, rearrangement is used to adjust the dimensions of multi-dimensional data, so that the size of the second feature map is the same as that of the first feature map.

[0087] In the present application, CA (Channel Attention) is used to make the network pay attention to different channels of the input feature map, so as to enhance the weight of important feature channels and weaken unimportant channels.

[0088] In this application, SA (Spatial Attention) pays attention to different positions of the input feature map in the spatial dimension, thereby guiding the network to pay more attention to important spatial positions in the image.

[0089] In this application, the channel attention and the spatial attention are combined, and the spatial attention generates an attention map in the height and width dimensions, thereby enhancing the attention to a specific spatial region.

[0090] In this application, Linear (Linear Layer) generally refers to a Fully Connected Layer (FC Layer), which is one of the most basic layers in a neural network. The linear layer performs a linear transformation between the input and the output, which is used to map the features of the previous layer to a higher dimension or a specific output dimension.

[0091] In this application, segmentation refers to dividing the input feature map into several parts in a certain way for parallel computation or different processing operations. The segmentation in this application is a segmentation in the channel dimension, which divides the input feature map along the channel dimension into several subsets, and each subset performs different convolution or operation.

[0092] In this application, by dividing the feature map into multiple groups, the computational complexity of each convolution layer can be reduced, the efficiency of the network can be improved, and the expressive ability of the model can be maintained.

[0093] In this application, the multi-channel feature maps from the two structures are sequentially subjected to CA-based global enhancement and SA-based spatial local enhancement on the basis of channel connection, realizing the complementary advantages of the two structure information, so that the fusion result has both spatial local structure information and global semantic information.

[0094] In this application, the fusion result is divided into two groups by channel division, and one group of feature maps is first subjected to layer normalization (LN), and then sequentially subjected to linear dimension lifting, GELU activation, and linear dimension reduction processing, and outputs and sends back to the Transformer structure; the other group of feature maps is sequentially subjected to spatial convolution, ReLU activation, and spatial convolution processing, and outputs and sends back to the CNN structure. Finally, the CNN structure and the Transformer structure simultaneously receive the two groups of output feature maps generated by the fusion module, and enhance the feature maps of the corresponding structure by means of residual connection.

[0095] In one specific embodiment, the processing process of the global extraction module is as follows:

[0096] The input feature map is input into the parallelly arranged 1x1 convolution layer to obtain a first extraction map, a second extraction map, and a third extraction map; wherein the first extraction map and the third extraction map are in the format of HWxN, and the second extraction map is in the format of NxHW.

[0097] multiplying the first extracted graph and the second extracted graph to obtain a multiplied feature graph;

[0098] performing softmax processing on the multiplied feature graph to obtain a multiplication coefficient;

[0099] multiplying the multiplication coefficient and the third extracted graph to obtain a coefficient multiplied graph;

[0100] adding the coefficient multiplied graph and the input feature graph to obtain an output feature graph of global extraction.

[0101] In the present application, the input feature graph is divided into three branches for 1x1 convolution processing, which adjusts the dimensions of the first branch and the third branch to be in the format of HWxN, and the dimension of the second branch to be in the format of NxHW; then the first branch and the third branch are multiplied to obtain a feature graph with a dimension of HWxHW, and a coefficient is obtained through softmax processing; the coefficient is multiplied with the third branch, and the multiplied feature graph is added to the input feature graph to obtain an output feature graph.

[0102] In the present application, the dimensions of the first branch and the second branch are reversed through 1x1 convolution, so as to realize the multiplication between the feature graphs; through the multiplication, the long-range dependency relationship is captured; through the setting of the coefficient, the dependency relationship is embedded in the feature graph; and on this basis, the original input is added and residual connection is used, so as to embed the global extraction into the model without destroying the parameters.

[0103] In the present application, the long-range dependency relationship between the features is captured through global extraction, so that the model can increase the global features extracted to make up for the defects of the current model.

[0104] In the present application, the response at a position is calculated as the weighted sum of the features at all positions, so that when processing the information of each position, all position information that can be considered is considered.

[0105] In one specific embodiment, in combination with Figure 5 As shown in the S104, the placement information 3D of the medical instrument is projected to the corresponding spine position of the subject in the operation, including:

[0106] S401, obtaining coordinate information of a positioning frame on a specific spine position of the subject; the relative position of the positioning frame and the spine remains unchanged in the preoperative standard posture and the intraoperative standard posture of the subject;

[0107] The positioning frame is installed or fixed on a specific spine position, which can be the skin surface closest to a certain vertebral body. For example, the positioning frame is a square patch with a two-dimensional positioning code, which can be displayed under the view angle of X-ray and camera.

[0108] In this application, the relative position of the positioning frame and the spine will have a certain deviation due to the characteristics of the human body. This deviation is assumed to be non-existent in this application (i.e., the deviation has little effect on accuracy).

[0109] In this application, the relative position of the positioning frame and the spine remains unchanged in the preoperative standard posture and intraoperative standard posture of the subject. Therefore, in the real surgical scene, as long as the position and posture of the positioning frame are determined, the position information of the spine can be confirmed.

[0110] In this step, the position information of the spine in the real surgical scene is determined.

[0111] S402, according to the coordinate information of the positioning frame, determine the mapping relationship between the intraoperative coordinate system and the preoperative coordinate system;

[0112] In this step, the intraoperative coordinate system is the coordinate system in which the position information of the spine in the real surgical scene is located. The preoperative coordinate system is the coordinate system in the preoperative medical image, wherein the placement information of the multi-segment vertebral medical instrument is located in the preoperative coordinate system, and the three-dimensional model of the spine in the intraoperative standard posture generated according to the spine medical image in the preoperative standard posture is also located in the preoperative coordinate system.

[0113] In this application, the spine medical image in the preoperative standard posture also has a positioning frame. Based on the positioning frame in the medical image, the relative position relationship between the positioning frame and the spine in the preoperative standard posture can be determined. The relative position relationship between the positioning frame and the generated spine in the intraoperative standard posture can also be determined. Generally, the relative position relationship of the two is very similar, and the difference is very small, which is assumed to be the same.

[0114] In this step, the mapping relationship between the intraoperative coordinate system and the preoperative coordinate system can be determined through the position information of the positioning frame in the intraoperative coordinate system and the preoperative coordinate system (assuming that the actual position of the positioning frame is fixed and unchanged, the change of the position information of the positioning frame in the intraoperative coordinate system and the preoperative coordinate system is obtained by the mapping relationship between the two).

[0115] S403, according to the mapping relationship, map the three-dimensional model of the spine and the placement information of the medical instrument to the intraoperative coordinate system;

[0116] In the present application, by mapping relationship, the position information of the positioning frame under the preoperative coordinate system is mapped, that is, the position information of the positioning frame under the intraoperative coordinate system is obtained (principle); similarly, by mapping relationship, the placement information of the spinal three-dimensional model and the medical instrument under the preoperative coordinate system is mapped, that is, the placement information of the spinal three-dimensional model and the medical instrument under the intraoperative coordinate system is obtained.

[0117] In this way, by mapping, that is, the placement information of the spinal three-dimensional model and the medical instrument is directly coincided with the real operation scene, in this coincidence state, the coordinates of the spinal three-dimensional model are coincided with the spine of the real operation object, and the placement information of the medical instrument is also coincided with the spine of the real operation object, indicating the implantation planning of the medical instrument at the corresponding spinal position.

[0118] S404, by the laser projection device installed in the operating room, the placement information of the spinal three-dimensional model and the medical instrument under the intraoperative coordinate system is projected to the corresponding spinal position of the object.

[0119] In the present application, a laser projector with high precision and contrast can be used, which can accurately project images to irregular surfaces, that is, the corresponding spinal position of the object, forming a similar perspective effect. In order to match the projection with the curved surface of the human body, the target area (spine, back) can be scanned three-dimensionally, and the scanning data is used to capture the shape and unevenness of the human body surface; the three-dimensional scanning data of the human body surface is registered with the data to be projected, and through the registration technology, it is ensured that the image can adapt to the curved surface of the human body and be accurately displayed at the corresponding spinal position.

[0120] In the present application, AR equipment can also be used to project the placement information of the spinal three-dimensional model and the medical instrument under the intraoperative coordinate system to the corresponding spinal position of the object photographed by the AR equipment (real operation scene), so as to form a similar perspective effect in the eyes of the AR user.

[0121] In the present application, by projecting the placement information of the spinal three-dimensional model and the medical instrument to the corresponding spinal position of the object, medical staff can directly observe the planning information and successfully perform spinal surgery.

[0122] In addition, such projection setting can make the projection contactless with the operation object, so as to maintain the overall sterile environment.

[0123] It should be noted that a sound module or a monitoring camera can be additionally arranged to collect the voice or gesture action of the medical staff, and the corresponding change is generated by controlling the projection after large model recognition, so as to realize the non-contact interaction control of the projection in the operation and the medical staff. The large model can be a BERT (Bidirectional Encoder Representations from Transformers) model or a GPT series model.

[0124] The embodiment of the present application provides an interactive image analysis auxiliary system for deep 3D aerial imaging, which is used for executing the interactive image analysis auxiliary method for deep 3D aerial imaging described in the above content of the present application. The interactive image analysis auxiliary system for deep 3D aerial imaging is described in detail as follows.

[0125] As shown in Figure 7 The interactive image analysis auxiliary system for deep 3D aerial imaging comprises:

[0126] An image acquisition unit 101 is configured to acquire a spine medical image of any object in a preoperative standard posture;

[0127] A model generation unit 102 is configured to generate a spine three-dimensional model in an intraoperative standard posture according to the spine medical image in the preoperative standard posture;

[0128] An instrument planning unit 103 is configured to generate placement information of a medical instrument of consecutive vertebrae based on the spine three-dimensional model in the intraoperative standard posture;

[0129] A 3D projection unit 104 is configured to project the placement information of the medical instrument to a corresponding spine position of the object in the operation in a 3D manner, so as to assist in implanting the medical instrument.

[0130] In an embodiment, the intraoperative standard posture is that the surgical object is in a prone position, both upper limbs are flexed forward and placed on both sides of the head, and the elbow and the shoulder are the same width; a soft pad is placed under the chest and abdomen, so that the hip is flush with the shoulder; a soft pad is placed on the ankle, so that the toes naturally droop, and both knees are the same width as the shoulder.

[0131] In an embodiment, the medical instrument comprises at least one of a screw, a fusion cage, a titanium rod and a transverse connection.

[0132] In an embodiment, the model generation unit 102 is further configured to:

[0133] The preoperative standard posture spinal medical image is input into the CNN structure and the Transformer structure respectively to obtain corresponding first feature maps and second feature maps; the first feature maps and the second feature maps are input into a fusion module for fusion processing to obtain first fusion maps and second fusion maps after fusion; the first fusion maps are sequentially input into the CNN structure and a global extraction module to obtain a global extraction map; the second fusion maps are sequentially input into the Transformer structure and a multi-convolution module to obtain a multi-convolution feature map; after the global extraction map and the multi-convolution feature map are added, convolution processing is performed to obtain the intraoperative standard posture spinal medical image; and based on the intraoperative standard posture spinal conversion image, the spinal three-dimensional model is generated.

[0134] In an implementation manner, the model generation unit 102 is further configured to:

[0135] After the second feature maps are rearranged, the second feature maps are spliced with the first feature maps to obtain spliced feature maps; after the spliced feature maps are sequentially subjected to channel attention extraction and spatial attention extraction, the spliced feature maps are added to the spliced feature maps to obtain added feature maps; the added feature maps are divided into first divided feature maps and second divided feature maps; the first divided feature maps are sequentially subjected to convolution processing, RELU processing and convolution processing to obtain first fusion maps; and the second divided feature maps are sequentially subjected to rearrangement processing, LN processing, linear processing, GELU processing and linear processing to obtain second fusion maps.

[0136] In an implementation manner, the 3D projection unit 104 is further configured to:

[0137] Coordinate information of a positioning frame at a specific spinal position of the object is acquired; the relative position of the positioning frame and the spine remains unchanged in the preoperative standard posture and the intraoperative standard posture of the object; a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system is determined according to the coordinate information of the positioning frame; the spinal three-dimensional model and the placement information of the medical instrument are mapped to the intraoperative coordinate system according to the mapping relationship; and the spinal three-dimensional model and the placement information of the medical instrument in the intraoperative coordinate system are projected to the corresponding spinal position of the object by a laser projection device installed in an operating room.

[0138] The above embodiments of the present application provide a depth 3D aerial imaging interactive image analysis auxiliary system, which has a corresponding relationship with the depth 3D aerial imaging interactive image analysis auxiliary method provided by the embodiments of the present application, and the specific contents in the system have a corresponding relationship with the depth 3D aerial imaging interactive image analysis auxiliary method. The specific contents can be referred to the records in the depth 3D aerial imaging interactive image analysis auxiliary method, and will not be described here.

[0139] The above embodiments of the present application provide a depth 3D aerial imaging interactive image analysis auxiliary system and the depth 3D aerial imaging interactive image analysis auxiliary method provided by the embodiments of the present application have the same inventive concept and the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0140] The internal functions and structures of the depth 3D aerial imaging interactive image analysis auxiliary system are described above, such as Figure 8 As shown in the figure, in practice, the depth 3D aerial imaging interactive image analysis auxiliary system can be implemented as an electronic device, including a memory 301 and a processor 303.

[0141] The memory 301 can be configured to store programs.

[0142] In addition, the memory 301 can also be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0143] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The processor 303 is coupled to the memory 301 for executing programs in the memory 301 for:

[0144] Obtaining a spine medical image of a subject in a preoperative standard posture;

[0145] Generating a spine three-dimensional model in an intraoperative standard posture according to the spine medical image in the preoperative standard posture;

[0146] Generating placement information of a medical instrument for a plurality of consecutive vertebrae based on the spine three-dimensional model in the intraoperative standard posture;

[0147] 3D projecting the placement information of the medical instrument to a corresponding spine position of the subject in the operation to assist in implanting the medical instrument.

[0148] In an embodiment, the intraoperative standard posture is that the surgical subject is in a prone position, both upper limbs are flexed forward and placed on both sides of the head, and the elbow and shoulder are the same width; there is a soft pad under the chest and abdomen, so that the hips are flush with the shoulders; there is a soft pad on the ankle, so that the toes naturally droop, and the knees are the same width as the shoulders.

[0149] In an embodiment, the medical instrument includes at least one of a screw, a fusion cage, a titanium rod, and a crosslink.

[0150] In an embodiment, the processor 303 is further configured to:

[0151] input the spine medical image in the preoperative standard posture into the CNN structure and the Transformer structure respectively to obtain corresponding first feature maps and second feature maps; input the first feature maps and the second feature maps into a fusion module for fusion processing to obtain first fusion maps and second fusion maps after fusion; input the first fusion maps into the CNN structure and the global extraction module in sequence to obtain a global extraction map; input the second fusion maps into the Transformer structure and the multi-convolution module in sequence to obtain a multi-convolution feature map; add the global extraction map and the multi-convolution feature map, and then perform convolution processing to obtain the spine medical image in the intraoperative standard posture; and generate the spine three-dimensional model based on the spine medical image in the intraoperative standard posture.

[0152] In an embodiment, the processor 303 is further configured to:

[0153] rearrange the second feature maps and splice them with the first feature maps to obtain spliced feature maps; perform channel attention extraction and spatial attention extraction on the spliced feature maps in sequence, and then add the spliced feature maps to obtain added feature maps; divide the added feature maps into first divided feature maps and second divided feature maps; perform convolution processing, RELU processing, and convolution processing on the first divided feature maps in sequence to obtain the first fusion maps; perform rearrangement processing, LN processing, linear processing, GELU processing, and linear processing on the second divided feature maps in sequence to obtain the second fusion maps.

[0154] In an embodiment, the processor 303 is further configured to:

[0155] obtain coordinate information of a positioning frame at a specific spine position of the object; the relative position of the positioning frame to the spine remains unchanged in the preoperative standard posture and the intraoperative standard posture of the object; determine a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; map the spine three-dimensional model and the placement information of the medical instrument to the intraoperative coordinate system according to the mapping relationship; and project the spine three-dimensional model and the placement information of the medical instrument in the intraoperative coordinate system to the corresponding spine position of the object through a laser projection device installed in an operating room.

[0156] In the present application, Figure 8 only some components are shown schematically, and it does not mean that the electronic device only includes Figure 8 the components shown.

[0157] The electronic device provided by the embodiment has the same beneficial effects as the method adopted, run or implemented by the application program stored in the electronic device.

[0158] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. In addition, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 The flow or the plurality of flows and / or blocks Figure 1 The device that implements the functions specified in the flow or the plurality of flows and / or blocks. These computer program instructions can also be stored in a computer-readable memory that can cause the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flowcharts and / or block diagrams. Figure 1 The flow or the plurality of flows and / or blocks Figure 1 The device that implements the functions specified in the flow or the plurality of flows and / or blocks.

[0160] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 The flow or the plurality of flows and / or blocks Figure 1 The device that implements the functions specified in the flow or the plurality of flows and / or blocks.

[0161] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. The memory can include non-persistent memory in the form of various computer-readable media, random access memory (RAM), and / or non-volatile memory such as read only memory (ROM) or Flash memory. Memory is an example of computer-readable media.

[0162] The application also provides a computer-readable storage medium corresponding to the interactive image analysis assisting method of deep 3D aerial imaging provided by the foregoing embodiments, and a computer program (i.e., a program product) is stored on the computer-readable storage medium. When the computer program is executed by a processor, the interactive image analysis assisting method of deep 3D aerial imaging provided by any of the foregoing embodiments is executed.

[0163] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for storing information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0164] The computer-readable storage medium provided by the foregoing embodiments of the application has the same beneficial effects as the method adopted, executed or implemented by the application program stored therein, based on the same inventive concept as the interactive image analysis assisting method of deep 3D aerial imaging provided by the embodiments of the application.

[0165] It should be noted that in the specification provided herein, a large number of specific details are explained. However, it can be understood that the embodiments of the application can be practiced without these specific details. In some examples, well-known structures and techniques are not shown in detail in order not to obscure the understanding of the present specification.

[0166] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0167] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. An interactive image analysis assistance method for deep 3D aerial imaging, characterized in that, The method comprises: acquiring a preoperative standard posture spinal medical image of any object; generating a three-dimensional model of the spine in an intraoperative standard posture according to the preoperative standard posture spinal medical image; generating placement information of a medical instrument of a continuous multi-segment vertebra based on the three-dimensional model of the spine in the intraoperative standard posture; 3D projecting the placement information of the medical instrument to the corresponding spinal position of the object in the operation to assist in implanting the medical instrument; The method of generating a three-dimensional model of the spine in an intraoperative standard posture according to a preoperative standard posture spinal medical image comprises: inputting the preoperative standard posture spinal medical image into a CNN structure and a Transformer structure respectively to obtain corresponding first feature maps and second feature maps; inputting the first feature maps and the second feature maps into a fusion module for fusion processing to obtain first fused maps and second fused maps after fusion; inputting the first fused maps into the CNN structure and a global extraction module in sequence to obtain a global extraction map; inputting the second fused maps into the Transformer structure and a multi-convolution module in sequence to obtain multi-convolution feature maps; adding the global extraction map and the multi-convolution feature maps and then performing convolution processing to obtain the intraoperative standard posture spinal medical image; generating the three-dimensional model of the spine based on the intraoperative standard posture spinal conversion image; The method of inputting the first feature maps and the second feature maps into a fusion module for fusion processing to obtain first fused maps and second fused maps after fusion comprises: rearranging the second feature maps and then splicing them with the first feature maps to obtain spliced feature maps; performing channel attention extraction and spatial attention extraction on the spliced feature maps in sequence and then adding them with the spliced feature maps to obtain added feature maps; dividing the added feature maps into first divided feature maps and second divided feature maps; performing convolution processing, RELU processing and convolution processing on the first divided feature maps in sequence to obtain the first fused maps; performing rearrangement processing, LN processing, linear processing, GELU processing and linear processing on the second divided feature maps in sequence to obtain the second fused maps.

2. The interactive image analysis aided method of deep 3D aerial imaging according to claim 1, characterized in that, The intraoperative standard posture is that the surgical object is in a prone position, both upper limbs are flexed forward and placed on both sides of the head, and the elbows and shoulders are the same width; there is a soft pad under the chest and abdomen so that the hips are flush with the shoulders; there is a soft pad on the ankle so that the toes naturally droop, and both knees are the same width as the shoulders.

3. The interactive image analysis aid method of deep 3D aerial imaging according to claim 1, characterized in that, The medical instrument comprises at least one of a screw, a fusion cage, a titanium rod and a transverse connection.

4. The interactive image analysis assisted method of deep 3D aerial imaging according to any one of claims 1-3, characterized in that, The method of 3D projecting the placement information of the medical instrument to the corresponding spinal position of the object in the operation comprises: acquiring coordinate information of a positioning frame on a specific spinal position of the object; the relative position of the positioning frame to the spine remains unchanged in the preoperative standard posture and the intraoperative standard posture of the object; determining a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; mapping the three-dimensional model of the spine and the placement information of the medical instrument to the intraoperative coordinate system according to the mapping relationship; projecting the three-dimensional model of the spine and the placement information of the medical instrument in the intraoperative coordinate system to the corresponding spinal position of the object by a laser projection device installed in the operating room.

5. An interactive image analysis assistance method system for deep 3D aerial imaging, characterized in that, The method comprises: An image acquisition unit configured to acquire a spine medical image of any subject in a preoperative standard posture; A model generation unit configured to generate a spine three-dimensional model in an intraoperative standard posture according to the spine medical image in the preoperative standard posture; An instrument planning unit configured to generate placement information of a medical instrument for consecutive vertebrae based on the spine three-dimensional model in the intraoperative standard posture; A 3D projection unit configured to 3D project the placement information of the medical instrument to a corresponding spine position of the subject in the intraoperative standard posture to assist implantation of the medical instrument; The model generation unit is further configured to: input the spine medical image in the preoperative standard posture into a CNN structure and a Transformer structure respectively to obtain corresponding first feature maps and second feature maps; input the first feature maps and the second feature maps into a fusion module for fusion processing to obtain first fused maps and second fused maps after fusion; input the first fused maps into the CNN structure and a global extraction module in sequence to obtain a global extraction map; input the second fused maps into the Transformer structure and a multi-convolution module in sequence to obtain multi-convolution feature maps; add the global extraction map and the multi-convolution feature maps, and then perform convolution processing to obtain a spine medical image in the intraoperative standard posture; and generate the spine three-dimensional model based on the spine medical image in the intraoperative standard posture; The inputting of the first feature maps and the second feature maps into the fusion module for fusion processing to obtain the first fused maps and the second fused maps after fusion includes: rearranging the second feature maps and splicing the rearranged second feature maps with the first feature maps to obtain spliced feature maps; performing channel attention extraction and spatial attention extraction on the spliced feature maps in sequence, adding the spliced feature maps and the added feature maps, dividing the added feature maps into first divided feature maps and second divided feature maps, performing convolution processing, RELU processing and convolution processing on the first divided feature maps in sequence to obtain the first fused maps, and performing rearrangement processing, LN processing, linear processing, GELU processing and linear processing on the second divided feature maps in sequence to obtain the second fused maps.

6. The interactive image analysis aid method system of deep 3D aerial imaging according to claim 5, characterized in that, The 3D projection unit is further configured to: acquire coordinate information of a positioning frame on a specific spine position of the subject; the relative position of the positioning frame to the spine remains unchanged in the preoperative standard posture and the intraoperative standard posture of the subject; determine a mapping relationship between an intraoperative coordinate system and a preoperative coordinate system according to the coordinate information of the positioning frame; map the spine three-dimensional model and the placement information of the medical instrument to the intraoperative coordinate system according to the mapping relationship; and project the spine three-dimensional model and the placement information of the medical instrument in the intraoperative coordinate system to the corresponding spine position of the subject by using a laser projection device installed in an operating room.

7. An electronic device, comprising: comprise: a memory and a processor; the memory is configured to store a program; the processor is coupled to the memory and configured to execute the program to: acquire a spine medical image of any subject in a preoperative standard posture; generate a spine three-dimensional model in an intraoperative standard posture according to the spine medical image in the preoperative standard posture; and Based on the three-dimensional model of the spine in the intraoperative standard posture, the placement information of the medical instrument of the continuous multi-segment vertebra is generated; The placement information of the medical instrument is 3D projected to the corresponding spine position of the object in the operation to assist in implanting the medical instrument; The three-dimensional model of the spine in the intraoperative standard posture is generated according to the spine medical image in the preoperative standard posture; comprising: The spine medical image in the preoperative standard posture is input into the CNN structure and the Transformer structure respectively to obtain the corresponding first feature map and the second feature map; The first feature map and the second feature map are input into the fusion module for fusion processing to obtain the first fusion map and the second fusion map after fusion; The first fusion map is sequentially input into the CNN structure and the global extraction module to obtain the global extraction map; The second fusion map is sequentially input into the Transformer structure and the multi-convolution module to obtain the multi-convolution feature map; After adding the global extraction map and the multi-convolution feature map, convolution processing is performed to obtain the spine medical image in the intraoperative standard posture; Based on the spine conversion image in the intraoperative standard posture, the three-dimensional model of the spine is generated; The first feature map and the second feature map are input into the fusion module for fusion processing to obtain the first fusion map and the second fusion map after fusion, comprising: After rearranging the second feature map, it is spliced with the first feature map to obtain a spliced feature map; After sequentially performing channel attention extraction and spatial attention extraction on the spliced feature map, the spliced feature map is added to obtain an added feature map; The added feature map is divided into a first divided feature map and a second divided feature map; The first divided feature map is sequentially subjected to convolution processing, RELU processing and convolution processing to obtain the first fusion map; After sequentially performing rearrangement processing, LN processing, linear processing, GELU processing and linear processing on the second divided feature map, the second fusion map is obtained.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the interactive image analysis assisted method of deep 3D aerial imaging according to any one of claims 1-4.

Images