Surgery navigation positioning method and device based on structured light image

Through the surgical navigation and positioning method based on structured light images, real-time semantic segmentation is used for real-time semantic segmentation using deep learning models, the existing surgical navigation system's inaccurate navigation and positioning in spinal surgery is solved, and high accuracy is achieved , low-cost real-time navigation, suitable for surgical robots with automatic resection function.

CN118644644BActive Publication Date: 2024-11-22PEKING UNIVERSITY THIRD HOSPITAL (THE THIRD CLINICAL MEDICAL SCHOOL OF PEKING UNIVERSITY)
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411095106.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-22
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

The existing surgical navigation system has problems such as insufficient navigation positioning, poor real-time, high cost and low scalability in spinal surgery, which affects the accuracy and safety of the surgery.

Method used

Using the surgical navigation and positioning method based on structured light images, by acquiring three-dimensional reconstruction images and real-time structured light images, semantic segmentation is used for deep learning models, real-time registration of surgical sites is achieved, reducing dependence on expensive equipment, and improving the accuracy of registration Sex and real-time.

Benefits of technology

Real-time high-precision positioning and navigation during the operation process, reducing the difficulty of surgical operations, reducing manual intervention, and suitable for surgical robots with automatic resection function, reducing overall costs, and improving the safety and automation of the operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644644B_ABST
    Figure CN118644644B_ABST
Patent Text Reader

Abstract

The present application provides a method, device, electronic device and computer-readable storage medium for surgical navigation positioning based on structured light images, the method comprising: obtaining a reference point cloud of a three-dimensional reconstructed image of a surgical site; obtaining a point cloud to be registered of a real-time structured light image; using a pre-trained structured light point cloud segmentation model to semantically segment the point cloud to be registered, and obtaining an effective point cloud of an effective registration area in the surgical site; registering the effective point cloud with the corresponding point cloud in the reference point cloud, thereby achieving registration of the point cloud to be registered with the reference point cloud, and obtaining a real-time registration result; and performing surgical navigation positioning based on the real-time registration result. In this way, a simple instrument can be used to accurately capture the anatomical structure of the surgical site in real time, achieve real-time and high-precision intraoperative positioning navigation, and reduce the difficulty of surgical operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of surgical navigation and positioning technology, and in particular, relates to a surgical navigation and positioning method, device, electronic device and computer-readable storage medium based on structured light images. Background Art

[0002] In recent years, with the rapid development of medical technology, medical imaging technology, computer algorithms and optical precision positioning technology have made significant progress. In this context, surgical navigation systems, as a high-precision auxiliary tool, have gradually emerged in spinal surgery. By providing real-time three-dimensional images of the surgical area, surgical navigation systems help doctors locate the lesion more accurately, thereby improving the accuracy and safety of surgery.

[0003] However, although surgical navigation systems have brought many conveniences to spinal surgery, existing systems still have certain limitations. At present, the vast majority of surgical navigation systems on the market mainly use the method of intraoperative image registration: using landmarks fixed on the bone surface and performing point cloud registration through a laser positioning detector. Although this method is widely used, it has the following disadvantages:

[0004] First, due to the complex structure of the spine, the intervertebral joints between the vertebrae have micro-movements during the operation, resulting in the inability of the operation point and the bony landmarks to move rigidly and synchronously, which leads to registration errors, which may gradually increase as the operation progresses and are difficult to detect. In addition, as the operation progresses, the bony landmarks may also move due to the operation, further increasing the registration error, resulting in inaccurate navigation positioning of the surgical system.

[0005] Second, the surgical navigation system based on the corresponding point registration of bony landmarks cannot achieve real-time tracking. The model registered by this method comes from the intraoperative (preoperative) CT scan and is registered with the tracker position. However, there is a time interval of several minutes between the preoperative (intraoperative) CT scan and the operation. Once the first displacement error occurs, the surgical navigation system cannot be calibrated and corrected in time due to its non-real-time characteristics.

[0006] Third, although the surgical navigation system based on the corresponding point registration of bony landmarks can achieve high-precision registration, it needs to be equipped with intraoperative CT equipment or O-arm intraoperative scanning equipment that costs up to tens of millions. Not only is the equipment expensive and complicated to operate, but it is also easily affected by factors such as light and temperature in the operating room, resulting in a decrease in registration accuracy. Therefore, it has high requirements on environmental conditions.

[0007] Fourth, the surgical navigation system based on the corresponding point registration of bony landmarks has little room for future expansion and application. Most medical research and development teams are currently developing surgical robots with automatic cutting functions, which have extremely high requirements for registration accuracy, real-time performance, and error correction capabilities. Preliminary experiments have found that the current surgical navigation system cannot fully meet safety requirements and requires manual error correction and manual supervision to improve safety.

[0008] In summary, the problems of existing surgical navigation systems mainly include inaccurate navigation positioning, poor real-time performance, extremely high cost, and low scalability. These problems not only affect the accuracy of the surgical navigation system, but may also have a negative impact on the surgical effect. Therefore, how to improve the accuracy of the surgical navigation system has become a top priority. Summary of the invention

[0009] In view of this, the present application provides a surgical navigation positioning method, device, electronic device and computer-readable storage medium based on structured light images, which are used to solve the technical problems raised by the background technology.

[0010] According to the first aspect of the present application, a surgical navigation and positioning method based on structured light images is provided, comprising: obtaining a reference point cloud of a three-dimensional reconstructed image of a part to be operated on, wherein the three-dimensional reconstructed image of the part to be operated on is reconstructed based on a computer tomography image of the part to be operated on; obtaining a point cloud to be registered of a real-time structured light image, wherein the real-time structured light image is obtained by real-time scanning of a surgical part exposed by a surgical incision using a structured light camera; performing semantic segmentation on the point cloud to be registered using a pre-trained structured light point cloud segmentation model to obtain a valid point cloud of a valid registration area in the surgical part, wherein the structured light point cloud segmentation model is used to identify a valid registration area and an invalid registration area in the point cloud to be registered; registering the valid point cloud with the corresponding point cloud in the reference point cloud to achieve registration of the point cloud to be registered with the reference point cloud, and obtaining a real-time registration result; and performing surgical navigation and positioning based on the real-time registration result.

[0011] According to a second aspect of the present application, a surgical navigation and positioning device based on structured light images is provided, comprising: a reference point cloud acquisition module, used to acquire a reference point cloud of a three-dimensional reconstructed image of a part to be operated on, wherein the three-dimensional reconstructed image of the part to be operated on is obtained by reconstructing a computer tomography image of the part to be operated on; a point cloud acquisition module to be registered, used to acquire a point cloud to be registered of a real-time structured light image, wherein the real-time structured light image is obtained by real-time scanning of a surgical part exposed by a surgical wound using a structured light camera; an effective point cloud acquisition module, used to perform semantic segmentation on the point cloud to be registered using a pre-trained structured light point cloud segmentation model, and obtain an effective point cloud of an effective registration area in the surgical part, wherein the structured light point cloud segmentation model is used to identify an effective registration area and an invalid registration area in the point cloud to be registered; a real-time registration result acquisition module, used to register the effective point cloud with a corresponding point cloud in the reference point cloud, thereby achieving registration of the point cloud to be registered with the reference point cloud, and obtaining a real-time registration result; and a surgical navigation and positioning module, used to perform surgical navigation and positioning based on the real-time registration result.

[0012] According to a third aspect of the present application, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the steps of any one of the above-mentioned surgical navigation and positioning methods based on structured light images are implemented.

[0013] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of any one of the above-mentioned surgical navigation and positioning methods based on structured light images are implemented.

[0014] In summary, the surgical navigation positioning method, device, electronic device and computer-readable storage medium based on structured light images provided in the embodiments of the present application have at least the following beneficial effects: by real-time scanning of the surgical site to obtain the point cloud to be registered of the real-time structured light image, the latest point cloud to be registered can be obtained in real time to ensure the real-time performance during the operation. And by accurately identifying the effective point cloud of the effective registration area in the point cloud to be registered through the structured light point cloud segmentation model, the accuracy of the registration can be further improved. By aligning the effective point cloud with the corresponding point cloud in the reference point cloud, the real-time registration result of the point cloud to be registered and the reference point cloud can be quickly obtained, and surgical navigation positioning can be performed based on the real-time registration result. In this way, the anatomical structure of the surgical site during the operation can be accurately captured in real time through a simple instrument, so as to achieve real-time and high-precision intraoperative positioning and navigation, and reduce the difficulty of surgical operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram showing the structure of a surgical navigation and positioning system provided by an embodiment of the present application is shown;

[0017] Figure 2 A schematic flow chart of a surgical navigation and positioning method based on structured light images provided in an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram showing a real-time structured light image with an effective registration area marked provided by an embodiment of the present application;

[0019] Figure 4 A schematic diagram of the structure of a structured light point cloud segmentation model provided in an embodiment of the present application is shown;

[0020] Figure 5 A schematic diagram showing a rough registration result of a point cloud to be registered and a reference point cloud provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of the structure of a surgical navigation and positioning device based on structured light images provided in an embodiment of the present application is shown;

[0022] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to make the above and other features and advantages of the present application more clear, the present application is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0024] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that specific details need not be adopted to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid blurring the present application.

[0025] There is a pedicle fixation surgical navigation system based on structured light scanning, which may include a structured light scanner, a dynamic reference and surgical instruments. The structured light scanner is used to collect structured light three-dimensional images of the affected bone. By installing visible light markers on the dynamic reference and surgical instruments, their initial homogeneous coordinates in the structured light image coordinate system are determined, thereby achieving surgical navigation. Although it is a surgical navigation system based on structured light scanning, it can reduce equipment costs and achieve real-time tracking of dynamic references and surgical instruments through structured light scanners and visible light markers. However, it still does not meet the requirements of the medical research and development team in terms of positioning accuracy and real-time performance, and requires manual error correction.

[0026] Based on this, the applicant proposes a technical solution that combines deep learning technology with a 3D structured light camera to perform continuous dynamic scanning of the anatomical structure of the surgical site during surgery to achieve real-time tracking and registration, thereby achieving efficient real-time intraoperative positioning and navigation. Among them, the deep learning model can automatically learn and identify features in the point cloud data, thereby accurately segmenting the spinal structure, greatly reducing human intervention and errors, and has high accuracy. And the structured light point cloud segmentation model constructed by multiple residual attention layers and feature fusion layers can capture the complex features of the point cloud data, thereby improving the accuracy of the segmentation. In addition, the constructed structured light point cloud segmentation model has structural advantages: it adopts a deep learning model structure design, and by combining the residual network and the attention mechanism, it can maintain stable performance when processing large amounts of structured light point cloud data.

[0027] On the one hand, an embodiment of the present application provides a surgical navigation and positioning system. Figure 1 A schematic diagram of the structure of a surgical navigation and positioning system provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the surgical navigation positioning system 100 may include a positioning detector 110, a surgical navigation positioning device 120, and a display device 130. The positioning detector 110 may be a device for three-dimensional imaging based on structured light technology. For example, a four-dimensional (3D) structured light camera. It should be noted that when a four-dimensional 3D structured light camera is used as a positioning detector, the four-dimensional 3D structured light camera needs to be dynamically calibrated.

[0028] The positioning detector 110 establishes communication connections with the surgical navigation positioning device 120 and the display device 130 respectively, and is used to perform real-time scanning at a preset scanning frequency for the surgical site exposed by the incision of the surgical site to obtain a point cloud to be registered of the real-time structured light image. The positioning detector 110 is also used to transmit the point cloud to be registered to the surgical navigation positioning device 120 and to transmit the point cloud to be registered to the display device 130.

[0029] The surgical navigation positioning device 120 can perform any of the steps of the surgical navigation positioning method based on structured light images provided in the embodiments of the present application. In this way, the effective registration area in the structured light point cloud data can be identified by the structured light point cloud segmentation model constructed based on deep learning technology, thereby achieving high-precision real-time tracking and registration between the computed tomography (CT) image and the structured light image, thereby improving the real-time and accuracy of surgical positioning navigation.

[0030] In some embodiments, the surgical navigation positioning device 120 sends the real-time registration result to the display device 130. The display device 130 can be used to visualize the real-time registration result. In addition, the display device 130 can also be used to display at least one of the real-time structured light image, the CT image, and the three-dimensional reconstructed image of the surgical site.

[0031] It should be noted that the surgical navigation positioning device 120 and the display device 130 can be separately arranged or integrated together. In the embodiment of the present application, there is no limitation on the arrangement of the surgical navigation positioning device 120 and the display device 130.

[0032] The surgical navigation and positioning system provided in the embodiment of the present application effectively overcomes the shortcomings of the existing surgical navigation system registration method. First, the surgical navigation and positioning system provided in the embodiment of the present application adopts the registration based on structured light image, and can directly detect the micro-movement between vertebrae by real-time acquisition of effective point cloud and registration with reference point cloud, which greatly reduces the registration error caused by the movement of bony landmarks. And any slight deviation can be discovered and corrected in time.

[0033] Secondly, the surgical navigation and positioning system provided in the embodiment of the present application supports real-time tracking, which significantly improves the accuracy of the surgery. By combining real-time structured light images and structured light point cloud segmentation models, the imaging data during surgery can be obtained instantly, and the registration information can be updated and calibrated in a timely manner, effectively overcoming the problem of time interval between preoperative (intraoperative) CT scanning and actual operation in traditional methods. In this way, not only the accuracy of registration is improved, but also the safety of surgery is enhanced.

[0034] Thirdly, compared with the traditional surgical navigation system based on the corresponding point registration of bony landmarks, the surgical navigation positioning system provided by the embodiment of the present application uses more economical and efficient equipment, and does not require expensive and complex intraoperative scanning equipment, reducing the dependence on expensive equipment and reducing the overall cost. At the same time, the improved solution has lower requirements on environmental conditions, and only needs to ensure that the surgical field and the structured light camera are not blocked and are not easily affected by factors such as light and temperature in the operating room, thereby maintaining a high-precision registration effect.

[0035] Finally, the surgical navigation and positioning system provided by the embodiment of the present application has great potential for future expansion and application, and is particularly suitable for surgical robots with automatic recognition and cutting functions developed by the medical team in the early stage, meeting their requirements for high precision, real-time performance, and error correction capabilities, without the need for frequent manual intervention and correction, and improving the automation and safety of surgery. In the future, this solution is expected to play a greater role in the field of intelligent surgical robots and promote the development of surgical navigation technology.

[0036] In summary, the surgical navigation and positioning system provided in the embodiments of the present application has significant advantages in terms of registration accuracy, real-time performance, cost-effectiveness, and future expansion applications, and can provide more reliable and efficient technical support for modern surgical intelligent equipment.

[0037] Another aspect of the present application is to provide a surgical navigation positioning method based on structured light images, which can be applied to Figure 1 The surgical navigation positioning device 120 is shown.

[0038] It should be noted that the surgical navigation positioning method based on structured light images provided in the embodiments of the present application can be applied to various surgical operations. This application takes the application scenario of spinal surgery navigation task as an example to introduce the surgical navigation positioning method in detail. The "coordinates" involved in the embodiments of the present application refer to the coordinates of the world coordinate system.

[0039] Figure 2 A schematic diagram of a surgical navigation positioning method based on structured light images provided in an embodiment of the present application is shown. Figure 2 As shown, the surgical navigation positioning method based on structured light images may include the following steps.

[0040] S21, obtaining a reference point cloud of a three-dimensional reconstructed image of the surgical site.

[0041] The CT image involved in the embodiment of the present application may be a series of two-dimensional tomographic images obtained by scanning the surgical site using CT technology. The three-dimensional reconstructed image of the surgical site is an image including three-dimensional information of the surgical site, which can be reconstructed based on the CT image of the surgical site. Specifically, the CT image information is converted into three-dimensional point cloud data (i.e., reference point cloud) by using a three-dimensional reconstruction algorithm to form a three-dimensional reconstructed image of the surgical site. It should be noted that the three-dimensional reconstructed image is an anatomical structure image of the surgical site.

[0042] In one embodiment of the present application, the surgical navigation positioning device 120 can obtain a CT image of the surgical site from an external device and reconstruct a three-dimensional reconstruction image of the surgical site. In another embodiment of the present application, the surgical navigation positioning device 120 can directly obtain a reference point cloud of the three-dimensional reconstruction image of the surgical site from an external device.

[0043] The "reference point cloud" involved in the embodiment of the present application may include information such as the three-dimensional coordinates, color, and label of each point, wherein the label is used to mark the bone region to which each point belongs.

[0044] S22, obtaining a point cloud to be registered of the real-time structured light image.

[0045] The "real-time structured light image" involved in this application is obtained by scanning the surgical site exposed by the surgical wound in real time using a structured light camera. In other words, the real-time structured light image can reflect the real-time status of the surgical site. In one embodiment of the present application, the structured light camera can scan the surgical site exposed by the surgical wound in real time at a preset high frequency. The preset high frequency is not less than 15 frames per second (FPS). It should be noted that due to movement or deformation during the operation, it is necessary to continuously acquire new structured light images of the surgical site at a preset high frequency to ensure the real-time and accuracy of the registration results. The preset high frequency can be set according to the surgical requirements and is not limited in the embodiment of the present application.

[0046] The point cloud to be registered may include information such as the three-dimensional coordinates, color, and reflection intensity of each point.

[0047] The surgical site may refer to the part of the exoskeleton exposed under the surgical wound. The scope of the surgical site may be the same as the scope of the site to be operated on, or may not exceed a certain range, for example, not exceed 3 cm.

[0048] S23, using a pre-trained structured light point cloud segmentation model to perform semantic segmentation on the point cloud to be registered, to obtain a valid point cloud of a valid registration area in the surgical site.

[0049] The "structured light point cloud segmentation model" involved in this application can be constructed based on deep neural network technology to identify valid registration areas and invalid registration areas in the point cloud to be registered. The valid registration area can refer to a simple bone surface area, that is, a bone surface area that does not contain skin and flesh tissue. The invalid registration area can refer to a non-simple bone surface area, that is, a bone surface area covered with skin and flesh tissue. The valid point cloud can be a collection of data for each point in the valid registration area.

[0050] Figure 3 A schematic diagram of a real-time structured light image with a marked effective registration area provided by an embodiment of the present application is shown. Figure 3 As shown in the figure, the white area is the effective registration area.

[0051] The pre-trained structured light point cloud segmentation model may refer to a structured light point cloud segmentation model that has been optimized through learning with a large number of labeled structured light sample images, and can accurately distinguish different areas in the structured light point cloud data. The pre-trained structured light point cloud segmentation model can meet the model evaluation indicators. Among them, the model evaluation indicators may include but are not limited to three indicators: average intersection-over-union ratio, average class accuracy, and overall point accuracy.

[0052] In an embodiment of the present application, the point cloud to be registered is input into a pre-trained structured light point cloud segmentation model, and the pre-trained structured light point cloud segmentation model performs semantic segmentation on the point cloud to be registered, thereby identifying the valid registration area and the invalid registration area in the surgical site, and outputting the valid point cloud data of the valid registration area and the invalid point cloud data of the invalid registration area.

[0053] It should be noted that the valid point cloud has additional regional label information compared to the invalid point cloud data.

[0054] S24, by registering the valid point cloud with the corresponding point cloud in the reference point cloud, the registration of the point cloud to be registered and the reference point cloud is achieved, and a real-time registration result is obtained.

[0055] The "corresponding point cloud" involved in the embodiments of the present application may refer to a point cloud in the reference point cloud that has the same features as the valid point cloud. In other words, for each point in the valid point cloud, a corresponding point with the most similar features is found in the reference point cloud. The corresponding point cloud is a collection of these corresponding points.

[0056] In one embodiment of the present application, the real-time registration result may include the real-time position and orientation of each point in the point cloud to be registered in the reference point cloud. The real-time registration result may also include a three-dimensional model of the surgical site. The three-dimensional model of the surgical site is part or all of the three-dimensional model of the surgical site.

[0057] In one embodiment of the present application, for each valid point in the valid point cloud, a corresponding point with the most similar features is found in the reference point cloud, and each valid point in the point cloud to be registered is aligned with the corresponding point in the reference point cloud, so that the registration of the point cloud to be registered and the reference point cloud can be achieved, and real-time registration results of the point cloud to be registered and the reference point cloud can be obtained.

[0058] It should be noted that the “alignment” involved in the embodiments of the present application may refer to the same posture and direction of two groups of point clouds.

[0059] S25, performing surgical navigation positioning based on the real-time registration results.

[0060] Specifically, based on the real-time registration results, the actual position and direction of the surgical site in three-dimensional space can be determined, so as to plan the dynamic navigation channel of the surgery based on the relative position and direction between the surgical site and the surgical operation point.

[0061] In the above embodiment, the point cloud to be registered of the real-time structured light image is obtained by scanning the surgical site in real time. In this way, the latest point cloud to be registered can be obtained in real time to ensure the real-time performance during the operation. And the accuracy of the registration can be further improved by accurately identifying the effective point cloud of the effective registration area in the point cloud to be registered through the structured light point cloud segmentation model. By aligning the effective point cloud with the corresponding point cloud in the reference point cloud, the real-time registration result of the point cloud to be registered and the reference point cloud can be quickly obtained, and surgical navigation positioning can be performed based on the real-time registration result. In this way, the anatomical structure of the surgical site during the operation can be accurately captured in real time through a simple instrument, so as to achieve real-time and high-precision intraoperative positioning and navigation, and reduce the difficulty of surgical operation.

[0062] In some embodiments, S25, performing surgical navigation positioning based on the real-time registration result may include: acquiring the real-time position of the surgical operation point; and determining the dynamic navigation channel of the surgery based on the real-time position of the surgical operation point and the real-time registration result.

[0063] The "surgical operation point" involved in the embodiments of the present application can be considered as the location of the surgical instrument. The surgical operation point can be obtained by a high-precision sensor. The dynamic navigation channel can be a virtual path used to guide the user or surgical robot to operate along a predetermined path to ensure the safety and accuracy of the operation.

[0064] In an embodiment of the present application, the real-time position of the surgical operation point can be mapped to the three-dimensional model in the real-time registration result, so as to track the current position of the surgical operation point in the surgical site, and then decide on the dynamic navigation channel of the operation.

[0065] In the above embodiment, real-time tracking of the position of the surgical operation point can provide accurate data support for surgical navigation positioning. By combining the real-time position of the surgical operation point with the real-time alignment result, a favorable dynamic navigation channel can be decided to provide precise guidance and guarantee for the surgical operation, thereby achieving efficient, accurate and safe surgical navigation positioning.

[0066] The applicant found that structured light point cloud and natural language are completely different data types, that is, structured light point cloud is an unstructured and disordered data set, and the classification and recognition of structured light point cloud using general semantic segmentation models is not accurate enough. Therefore, the inventor proposed a structured light point cloud segmentation model based on the Transformer learning framework. The structured light point cloud segmentation model introduces a residual attention layer, so that the structured light point cloud segmentation model has inherent permutation invariance and can cover the problem of feature map redundancy, which is more suitable for processing the classification task of unstructured, disordered and irregular point clouds.

[0067] In some embodiments, Figure 4A schematic diagram of the structure of a structured light point cloud segmentation model provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the structured light point cloud segmentation model 40 may include an input embedding layer 41, multiple residual attention layers 42, a first feature fusion layer 43, a fused convolution layer 44, a global pooling layer 45, a second feature fusion layer 46, a third feature fusion layer 47, multiple linear dimensionality reduction layers 48, and a classification result output layer 49. Among them, the input embedding layer 41, multiple residual attention layers 42, the first feature fusion layer 43 and the fused convolution layer 44 constitute an encoder, and the global pooling layer 45, the second feature fusion layer 46, the third feature fusion layer 47, multiple linear dimensionality reduction layers 48 and the classification result output layer 49 constitute a segmentation network.

[0068] The input embedding layer 41 is used to semantically encode each point in the point cloud to be registered to obtain point-by-point semantic features.

[0069] In one embodiment, the input embedding layer 41 can be constructed by cascading one-dimensional convolutional linear layers to implement semantic encoding based on analog point embedding. Through semantic encoding, the original features of each point in the point cloud to be registered can be mapped to a new, higher-dimensional feature space. The new feature space can capture complex relationships that are not easily perceived in the original features. For example, the point cloud to be registered , where d represents the original feature of each point, N represents the number of points in the point cloud, and R represents a set of real numbers. Optionally, d is 4. The original features may include three-dimensional coordinates and colors. The three-dimensional coordinates may include X-axis coordinates, Y-axis coordinates, and Z-axis coordinates. The original features may be represented as .

[0070] For example, the input scale of the input embedding layer 41 can be expressed as (B, 3, 1024), and the output scale can be expressed as (B, 64, 1024), where B is the batch size, 1024 is the number of points, and 3 represents the 3D coordinates of the point cloud.

[0071] In one embodiment, the linear dimension reduction layer can be expressed as LBR, including a linear fully connected layer (Linear), a batch normalization layer (Batch Norm), and a linear activation layer (ReLU).

[0072] The multiple residual attention layers 43 include multiple attention modules connected by a residual network. The multiple residual attention layers 43 are used to obtain multiple attention features of the point cloud to be registered at different scales based on point-by-point semantic features.

[0073] The "attention module" involved in the embodiment of the present application is built based on the attention mechanism. The operation process of the attention mechanism is as follows: First, each input element is represented by three vectors: Query, Key, and Value. These vectors are obtained by different linear transformations of the input elements (that is, multiplying by the weight matrix); then, the score of each element is obtained by calculating the dot product of the Query and all the Keys. This score reflects the degree of association between the current element and other elements, and a higher score means that the two elements are more related. Then, these scores are converted into probability values ​​through the softmax function. This probability value is the attention weight, which reflects the degree of attention of the model to each element. Finally, use these attention weights to weighted sum all the Values ​​to get the final output.

[0074] like Figure 4 As shown, in the multiple residual attention layers 43, the multiple attention modules are connected using a four-layer residual network. In this way, the multiple residual attention layers 43 can obtain multiple attention features at different scales through different attention modules.

[0075] In one embodiment of the present application, the i-th attention module can be represented by , the output can be expressed as . Where i can represent the serial number of the attention module. For example Represents the first attention module. The output of the first attention module , the output of the i+1th attention module .

[0076] The first feature fusion layer 43 is used to splice multiple attention features and output spliced ​​features.

[0077] In this way, by splicing the attention features at different scales together, the redundancy of feature maps can be eliminated as much as possible and the noise can be suppressed, which will improve the performance of the model and facilitate the output of downstream tasks.

[0078] The fusion convolution layer 44 is used to fuse the spliced ​​features to obtain the point-by-point features of the point cloud to be registered. .

[0079] In some embodiments, the fused convolution layer 44 may include a one-dimensional linear convolution layer LBR, that is, a linear fully connected layer, a batch normalization layer, and a linear activation layer. The "feature fusion" involved in the embodiment of the present application can be to extract and integrate key information in the spliced ​​features through methods such as convolution operations, normalization, and nonlinear activation, thereby generating a more compact and more representative point-by-point feature representation.

[0080] Point-by-point features It is a feature extracted separately for each point in the point cloud to be registered, which can reflect the local details and shape information in the point cloud to be registered.

[0081] For example, the dimension of the concatenated features output by the first feature fusion layer 43 may be expressed as C+256, and the dimension of the point-by-point features output by the fusion convolution layer 44 may be 1024.

[0082] The global pooling layer 45 includes a maximum pooling layer and an average pooling layer, which is used to , obtain the maximum pooling features and average pooling features of the point cloud to be registered.

[0083] In one embodiment of the present application, the maximum pooling layer can be used to perform a maximum pooling operation on the point-by-point features of the point cloud to be registered, thereby obtaining a maximum pooling feature. The maximum pooling feature can be the most significant feature of the point cloud to be registered.

[0084] The average pooling layer can be used to perform an average pooling operation on the point-by-point features of the point cloud to be registered, thereby obtaining an average pooling feature. The average pooling feature can be the overall trend feature of the point cloud to be registered.

[0085] The second feature fusion layer 46 is used to concatenate and fuse the maximum pooling feature and the average pooling feature to obtain a global fusion feature. .

[0086] Global fusion features It can reflect the global structure of the entire point cloud and help understand the overall semantics and contextual information of the point cloud to be registered. In this way, by splicing and fusing the two types of features, a more comprehensive feature representation can be obtained.

[0087] The third feature fusion layer 47 is used to concatenate and fuse the global fusion features with the point-by-point features to obtain the final features.

[0088] The "final features" involved in the embodiments of the present application may include local details, global structures, and contextual information of the point cloud to be registered, and may be used for downstream tasks such as point cloud classification and segmentation. In the third feature fusion layer 47, the global fusion features and the point-by-point features are concatenated together to form a new, richer feature representation, namely the final features, which enables the model to simultaneously utilize the detailed local structure and overall contextual information in the point cloud.

[0089] Multiple linear dimensionality reduction layers 48 are used to reduce the dimensionality of the final features.

[0090] In the embodiment of the present application, the multiple linear dimensionality reduction layers 48 can be composed of multiple one-dimensional linear convolution layers LBR, which are used to perform multiple dimensionality reduction on the final features. The one-dimensional linear convolution layer LBR can be configured to have only a convolution kernel size of 1x1, which can be used to perform linear transformation.

[0091] For example, if the dimension of the final feature is 1024, after the first dimension reduction is performed using the first one-dimensional linear convolution layer LBR in the multiple linear dimension reduction layers 48, the dimension of the final feature is 512, and after the second dimension reduction is performed using the second one-dimensional linear convolution layer LBR in the multiple linear dimension reduction layers 48, the dimension of the final feature is 256. In this way, the model can gradually remove redundant information and retain the most important features through the method of gradual dimension reduction.

[0092] The classification result output layer 49 is used to map the final features after dimensionality reduction and output the classification result of each point in the point cloud to be registered.

[0093] In an embodiment of the present application, the classification result output layer 49 can be constructed by a linear fully connected layer. That is, in the classification result output layer 49, the linear fully connected layer is used to map the features of each point in the final features after dimensionality reduction to obtain different types of classification scores for each point, and then obtain the classification result of each point.

[0094] In the classification result output layer 49, the classification result of each point may exist in the form of a label. Different label values ​​are set for different classification results.

[0095] In the above embodiment, the structured light point cloud segmentation model constructed by multiple residual attention layers has inherent permutation invariance, so that point cloud data can be processed more accurately. In addition, the model can cover the problem of feature map redundancy, making it more suitable for processing unstructured, disordered, irregular point cloud semantic segmentation tasks.

[0096] In some embodiments, Figure 4 As shown, the plurality of linear dimensionality reduction layers 48 may include a first linear dimensionality reduction layer, a second linear dimensionality reduction layer, and a dropout layer.

[0097] The first linear dimensionality reduction layer and the second linear dimensionality reduction layer are both composed of a one-dimensional linear convolution layer LBR. The inactivation layer is set between the first linear dimensionality reduction layer and the second linear dimensionality reduction layer, and is used to discard a preset number of neurons in the first linear dimensionality reduction layer with a preset probability to prevent overfitting and improve the generalization ability of the model.

[0098] In one embodiment of the present application, the deactivation layer is used to discard a preset number of neurons in the first linear dimensionality reduction layer with a preset probability, which can be performed as follows: in the deactivation layer, the outputs of the preset number of neurons in the first linear dimensionality reduction layer are set to 0 according to a preset probability (for example, 0.5). In this way, the discarded neurons will not have any effect on the output of the next layer. And because the discarding process is random, a different subset of neurons will be discarded each time, so that the model can adapt to different subsets of neurons during training and avoid over-reliance on any specific combination of neurons.

[0099] In some embodiments, the pre-trained structured light point cloud segmentation model may be trained using a moving training sample set collected by a structured light camera in response to the movement of the scanned object.

[0100] The "mobile acquisition" involved in the embodiments of the present application refers to the structured light camera moving and scanning within a preset range on the camera coordinate system. The scanned object refers to the object captured by the structured light camera. The preset range may mean that the displacement of each axis of the camera coordinate system is within 5 cm and / or the rotation angle is within ±30°. In other words, the structured light camera moves during the acquisition of the training sample set. The mobile training sample set may include structured light images of the same scanned object at different angles.

[0101] In this way, the model trained by the above-mentioned mobile training sample set will not affect the classification result of the point cloud even if the surgical site is not fixed during the operation, so there is no need to increase the intraoperative voxel rigidity through other auxiliary means.

[0102] In some embodiments, the classification result may include a valid registration area and an invalid registration area. Using a pre-trained structured light point cloud segmentation model to semantically segment the point cloud to be registered, and obtaining a valid point cloud of the effective registration area in the surgical site, includes: using a pre-trained structured light point cloud segmentation model to semantically segment the point cloud to be registered, and obtaining a valid registration area and an invalid registration area in the point cloud to be registered; obtaining a valid point cloud of the effective registration area in the surgical site.

[0103] In one embodiment of the present application, a pre-trained structured light point cloud segmentation model is used to infer the point cloud to be registered (i.e., forward propagation), and a classification label can be assigned to each point in the point cloud to be registered. Different classification labels represent different classification results. Specifically, points belonging to the pure bone surface are assigned a classification label representing a valid registration area, and points in the non-pure bone surface area are assigned a classification label representing an invalid registration area.

[0104] Moreover, the model can divide the point cloud into different regions according to the label of each point, and obtain the valid point cloud of the valid registration area according to the divided regions.

[0105] In the above embodiment, a valid point cloud is obtained by using a pre-trained structured light point cloud segmentation model so that the point cloud data that contributes to the registration process can be used for registration later, thereby improving the accuracy and efficiency of the registration.

[0106] In some embodiments, S24, by aligning the effective point cloud with the corresponding point cloud in the reference point cloud, the point cloud to be registered and the reference point cloud are registered to obtain a real-time registration result, which may include: using a point cloud coarse registration algorithm to coarsely align the effective point cloud with the corresponding point cloud in the reference point cloud to obtain a coarse registration result between the point cloud to be registered and the reference point cloud; using a point cloud fine registration algorithm to optimize the coarse registration result to obtain a real-time registration result between the point cloud to be registered and the reference point cloud.

[0107] In one embodiment of the present application, a point cloud coarse registration algorithm can be used to quickly estimate the approximate transformation (such as translation and rotation) between the effective point cloud and the corresponding point cloud in the reference point cloud. Specifically, in the point cloud coarse registration algorithm, the effective point cloud and the corresponding point cloud are aligned through the positional relationship between the effective point cloud and the corresponding point cloud in the reference point cloud, and the initial change matrix of the point cloud to be registered and the reference point cloud is estimated, and then the point cloud to be registered and the reference point cloud are coarsely registered according to the initial change matrix, thereby obtaining a coarse registration result of the point cloud to be registered and the reference point cloud.

[0108] Figure 5 A schematic diagram showing a rough registration result of a point cloud to be registered and a reference point cloud provided in an embodiment of the present application is shown, Figure 5 As shown in the figure, the overlapping parts of the point cloud to be registered and the reference point cloud are represented by green, and the non-overlapping parts are represented by red. 1s marked with different colors (green 1 is a feature point in the reference point cloud, and red 1 is a feature point in the point cloud to be registered) represent the positions of a feature point in the point cloud to be registered and the reference point cloud, respectively. 2s marked with different colors (green 2 is a feature point in the reference point cloud, and red 2 is a feature point in the point cloud to be registered) represent the positions of another feature point in the point cloud to be registered and the reference point cloud, respectively. It can be seen that the point cloud to be registered and the reference point cloud do not completely overlap and are not completely aligned.

[0109] In one embodiment of the present application, in the point cloud fine registration algorithm, the coarse registration result of the point cloud to be registered and the reference point cloud is used as the initial value, and the nearest point pair is found in an iterative manner, and a new transformation matrix is ​​calculated based on these point pairs, and then the transformation matrix is ​​applied to update the position of the point cloud until the preset stop condition is met (such as reaching the maximum number of iterations, the error is less than a certain threshold, etc.). In this way, through multiple iterations, the alignment result of the point cloud to be registered and the reference point cloud can be gradually optimized, and the real-time registration result of the point cloud to be registered and the reference point cloud can be obtained.

[0110] In the above embodiment, the coarse registration algorithm can quickly provide a rough alignment for the two point clouds, and the fine registration algorithm makes fine adjustments based on this to achieve a more accurate registration. In this way, the amount of calculation is greatly reduced through a staged processing method, the efficiency of the registration is improved, and the accuracy of the two sets of point clouds can be ensured.

[0111] Another aspect of the present application is to provide a surgical navigation positioning device based on structured light images. Figure 6 A schematic diagram of the structure of a surgical navigation positioning device based on structured light images provided in an embodiment of the present application is shown. Figure 6 As shown, the surgical navigation and positioning device based on structured light images may include the following modules.

[0112] The reference point cloud acquisition module 61 is used to acquire a reference point cloud of a three-dimensional reconstructed image of the surgical site, wherein the three-dimensional reconstructed image of the surgical site is reconstructed based on a computer tomography image of the surgical site.

[0113] The to-be-registered point cloud acquisition module 62 is used to acquire the to-be-registered point cloud of the real-time structured light image. The real-time structured light image is obtained by scanning the surgical site exposed by the surgical wound in real time using a structured light camera.

[0114] The effective point cloud acquisition module 63 is used to perform semantic segmentation on the point cloud to be registered using a pre-trained structured light point cloud segmentation model to obtain an effective point cloud of an effective registration area in the surgical site. The structured light point cloud segmentation model is used to identify effective registration areas and invalid registration areas in the point cloud to be registered.

[0115] The real-time registration result obtaining module 64 is used to realize the registration of the point cloud to be registered with the reference point cloud by registering the effective point cloud with the corresponding point cloud in the reference point cloud, and obtain the real-time registration result.

[0116] The surgical navigation positioning module 65 is used to perform surgical navigation positioning based on the real-time registration result.

[0117] In some embodiments, the surgical navigation positioning module 65 may include the following sub-modules.

[0118] A real-time position acquisition submodule is used to acquire the real-time position of the surgical operation point;

[0119] The channel determination submodule is used to determine the dynamic navigation channel of the surgery based on the real-time position of the surgery operation point and the real-time registration result.

[0120] In some embodiments, the structured light point cloud segmentation model includes: an input embedding layer, multiple residual attention layers, a first feature fusion layer, a fused convolution layer, a global pooling layer, a second feature fusion layer, a third feature fusion layer, multiple linear dimensionality reduction layers, and a classification result output layer;

[0121] The input embedding layer is used to perform semantic encoding on each point in the point cloud to be registered to obtain point-by-point semantic features;

[0122] The multiple residual attention layers include multiple attention modules connected by a residual network, and are used to obtain multiple attention features of the point cloud to be registered at different scales based on the point-by-point semantic features;

[0123] A first feature fusion layer concatenates the multiple attention features and outputs a concatenated feature;

[0124] The fusion convolution layer is used to perform feature fusion on the splicing features to obtain point-by-point features of the point cloud to be registered;

[0125] The global pooling layer includes a maximum pooling layer and an average pooling layer, which are used to obtain the maximum pooling feature and the average pooling feature of the point cloud to be registered based on the point-by-point features;

[0126] The second feature fusion layer is used to concatenate and fuse the maximum pooling feature and the average pooling feature to obtain a global fusion feature;

[0127] The third feature fusion layer is used to concatenate and fuse the global fusion feature with the point-by-point feature to obtain a final feature;

[0128] A plurality of linear dimensionality reduction layers, for reducing the dimensionality of the final features;

[0129] The classification result output layer is used to map the final features after dimensionality reduction and output the classification result of each point in the point cloud to be registered.

[0130] In some embodiments, the multiple linear dimensionality reduction layers include a first linear dimensionality reduction layer, a second linear dimensionality reduction layer and an inactivation layer, wherein the inactivation layer is disposed between the first linear dimensionality reduction layer and the second linear dimensionality reduction layer, and is used to discard a preset number of neurons in the two linear dimensionality reduction layers with a preset probability.

[0131] In some embodiments, the input embedding layer includes a cascaded one-dimensional linear convolution layer, and the linear dimensionality reduction layer includes a linear fully connected layer, a batch normalization layer, and a linear activation layer.

[0132] In some embodiments, the effective point cloud acquisition module 63 is specifically used to perform semantic segmentation on the point cloud to be registered using a pre-trained structured light point cloud segmentation model to obtain effective registration areas and invalid registration areas in the point cloud to be registered; and extract the effective point cloud of the effective registration area in the surgical site.

[0133] In some embodiments, the real-time registration result obtaining module 64 is specifically used to use a point cloud coarse registration algorithm to coarsely align the valid point cloud with the corresponding point cloud in the reference point cloud to obtain a coarse registration result between the point cloud to be registered and the reference point cloud; and use a point cloud fine registration algorithm to optimize the coarse registration result to obtain a real-time registration result between the point cloud to be registered and the reference point cloud.

[0134] It should be understood that the specific features, operations and details described hereinabove with respect to the method of the present application may also be similarly applied to the device and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the device or system of the present application.

[0135] It should be understood that each module / unit of the device of the present application can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the electronic device or independent of the processor in the form of hardware or firmware, or can be stored in the memory of the electronic device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0136] According to another aspect of the embodiments of the present application, an electronic device is provided. Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. Figure 7 As shown, the electronic device 70 includes a processor 71 and a memory 72 storing computer program instructions. When the processor 71 executes the computer program instructions, the above steps of surgical navigation positioning based on structured light images are implemented. The electronic device can be generally a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities.

[0137] In one embodiment, the electronic device 70 may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device may be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 70 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, a computer program, etc. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 70 may be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of surgical navigation positioning based on structured light images provided in any of the above embodiments are performed.

[0138] The present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the steps of the surgical navigation and positioning method based on structured light images provided in any of the above embodiments are implemented.

[0139] Those skilled in the art will appreciate that the method steps of the present application can be completed by instructing related hardware such as electronic devices or processors through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, which causes the steps of the present application to be executed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0140] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A surgical navigation positioning method based on structured light images, characterized in that: include: Acquire a reference point cloud of a three-dimensional reconstructed image of the part to be operated on, wherein the three-dimensional reconstructed image of the part to be operated on is reconstructed based on a computer tomography image of the part to be operated on; Acquire a point cloud to be registered of a real-time structured light image, wherein the real-time structured light image is obtained by scanning a surgical site exposed by a surgical wound in real time using a structured light camera; Using a pre-trained structured light point cloud segmentation model to perform semantic segmentation on the point cloud to be registered, and obtain a valid point cloud of a valid registration area in the surgical site, wherein the structured light point cloud segmentation model is used to identify a valid registration area and an invalid registration area in the point cloud to be registered; The structured light point cloud segmentation model includes: an input embedding layer, multiple residual attention layers, a first feature fusion layer, a fused convolution layer, a global pooling layer, a second feature fusion layer, a third feature fusion layer, multiple linear dimensionality reduction layers and a classification result output layer; the input embedding layer is used to semantically encode each point in the point cloud to be registered to obtain point-by-point semantic features; the multiple residual attention layers include multiple attention modules connected by residual networks, which are used to obtain multiple attention features of the point cloud to be registered at different scales based on the point-by-point semantic features; the first feature fusion layer splices the multiple attention features and outputs the spliced ​​features; the fused convolution layer is used to splice the spliced ​​features The features are combined to obtain the point-by-point features of the point cloud to be registered; the global pooling layer includes a maximum pooling layer and an average pooling layer, which are used to obtain the maximum pooling features and the average pooling features of the point cloud to be registered based on the point-by-point features; the second feature fusion layer is used to splice and fuse the maximum pooling features and the average pooling features to obtain the global fusion features; the third feature fusion layer is used to splice and fuse the global fusion features with the point-by-point features to obtain the final features; multiple linear dimensionality reduction layers are used to reduce the dimensionality of the final features; the classification result output layer is used to map the final features after dimensionality reduction and output the classification results of each point in the point cloud to be registered; By registering the valid point cloud with the corresponding point cloud in the reference point cloud, the registration of the point cloud to be registered and the reference point cloud is achieved, and a real-time registration result is obtained; Surgical navigation positioning is performed based on the real-time registration result.

2. The method according to claim 1, characterized in that The performing surgical navigation positioning based on the real-time registration result includes: Obtain the real-time location of the surgical operation point; A dynamic navigation channel for the surgery is determined based on the real-time position of the surgery operation point and the real-time registration result.

3. The method according to claim 2, characterized in that The multiple linear dimensionality reduction layers include a first linear dimensionality reduction layer, a second linear dimensionality reduction layer and an inactivation layer, The deactivation layer is arranged between the first linear dimensionality reduction layer and the second linear dimensionality reduction layer, and is used to discard a preset number of neurons in the two linear dimensionality reduction layers with a preset probability.

4. The method according to claim 1 or 3, characterized in that: The input embedding layer includes a cascaded one-dimensional linear convolution layer, and the linear dimension reduction layer includes a linear fully connected layer, a batch normalization layer and a linear activation layer.

5. The method according to claim 1 or 3, characterized in that: The method of using a pre-trained structured light point cloud segmentation model to semantically segment the point cloud to be registered to obtain a valid point cloud of a valid registration area in the surgical site includes: Using a pre-trained structured light point cloud segmentation model to perform semantic segmentation on the point cloud to be registered, and obtain a valid registration area and an invalid registration area in the point cloud to be registered; A valid point cloud of a valid registration area in the surgical site is extracted.

6. The method according to claim 1, characterized in that By registering the valid point cloud with the corresponding point cloud in the reference point cloud, the registration of the point cloud to be registered and the reference point cloud is achieved, and a real-time registration result is obtained, including: Using a point cloud coarse registration algorithm, the effective point cloud is coarsely registered with a corresponding point cloud in the reference point cloud to obtain a coarse registration result of the point cloud to be registered with the reference point cloud; The rough registration result is optimized by using a point cloud fine registration algorithm to obtain a real-time registration result of the point cloud to be registered and the reference point cloud.

7. A surgical navigation and positioning device based on structured light images, characterized in that: include: A reference point cloud acquisition module, used to acquire a reference point cloud of a three-dimensional reconstructed image of the surgical site, wherein the three-dimensional reconstructed image of the surgical site is reconstructed based on a computer tomography image of the surgical site; A module for acquiring a point cloud to be registered, used for acquiring a point cloud to be registered of a real-time structured light image, wherein the real-time structured light image is obtained by scanning the surgical site exposed by the surgical wound in real time using a structured light camera; An effective point cloud acquisition module is used to perform semantic segmentation on the point cloud to be registered using a pre-trained structured light point cloud segmentation model to obtain an effective point cloud of an effective registration area in the surgical site, wherein the structured light point cloud segmentation model is used to identify an effective registration area and an invalid registration area in the point cloud to be registered; The structured light point cloud segmentation model includes: an input embedding layer, multiple residual attention layers, a first feature fusion layer, a fused convolution layer, a global pooling layer, a second feature fusion layer, a third feature fusion layer, multiple linear dimensionality reduction layers and a classification result output layer; the input embedding layer is used to semantically encode each point in the point cloud to be registered to obtain point-by-point semantic features; the multiple residual attention layers include multiple attention modules connected by residual networks, which are used to obtain multiple attention features of the point cloud to be registered at different scales based on the point-by-point semantic features; the first feature fusion layer splices the multiple attention features and outputs the spliced ​​features; the fused convolution layer is used to splice the spliced ​​features The features are combined to obtain the point-by-point features of the point cloud to be registered; the global pooling layer includes a maximum pooling layer and an average pooling layer, which are used to obtain the maximum pooling features and the average pooling features of the point cloud to be registered based on the point-by-point features; the second feature fusion layer is used to splice and fuse the maximum pooling features and the average pooling features to obtain the global fusion features; the third feature fusion layer is used to splice and fuse the global fusion features with the point-by-point features to obtain the final features; multiple linear dimensionality reduction layers are used to reduce the dimensionality of the final features; the classification result output layer is used to map the final features after dimensionality reduction and output the classification results of each point in the point cloud to be registered; A real-time registration result obtaining module is used to realize the registration of the point cloud to be registered with the reference point cloud by registering the effective point cloud with the corresponding point cloud in the reference point cloud, and obtain a real-time registration result; A surgical navigation positioning module is used to perform surgical navigation positioning based on the real-time registration result.

8. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the steps of the surgical navigation and positioning method based on structured light images as described in any one of claims 1-6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the steps of the surgical navigation and positioning method based on structured light images as described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Operation navigation system adopting structural light image registering and registering signal collection method

    CN109464196A

  • Multi-modal MR image brain tumor segmentation method based on deep learning and multi-guidance

    CN112365496A

  • Surgical navigation point cloud segmentation and registration method based on structured light

    CN114792326A

  • Orthopedic robot positioning method, system, equipment and medium

    CN115211966A

  • Orthopedic puncture channel display method based on 3D structured light point cloud scanning

    CN115590617A