Surgical instrument registration method and system based on augmented reality technology

Abstract

The invention provides a surgical instrument registration method and system based on an augmented reality technology, and relates to the technical field of intelligent medical systems. The method comprises the steps that based on a plurality of sensor modules, target RGB images and target space depth information of an operation area are collected from different visual angles; generating a surgical area three-dimensional model according to the target RGB image and the target space depth information; determining a target instrument type corresponding to a to-be-registered target surgical instrument in the surgical area according to the instrument identification model; loading a target standard virtual component matched with the target equipment type from a virtual component database; and dynamically superposing the target standard virtual component to the three-dimensional model of the surgical area according to the real-time tracking data of the device. Therefore, the system can still accurately and stably complete registration of the surgical instruments in a complex and changeable surgical environment, and provides a solid technical guarantee for modern informatization surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of smart medical systems, and in particular to a surgical equipment registration method and system based on augmented reality technology. Background Art

[0002] In the current smart medical system, the registration of surgical instruments is an essential step before conducting smart surgery, and the precise positioning and registration of instruments are the key prerequisites to ensure that modern information-based surgeries, such as automated surgery or remote-assisted surgery, can proceed smoothly.

[0003] At present, surgical equipment registration methods mainly include optical tracking systems and electromagnetic tracking systems. Optical tracking systems usually use infrared cameras and reflective markers to determine the position and posture of surgical instruments. However, optical systems are sensitive to ambient light, and strong light or shadows in the operating room will interfere with the normal operation of the system, resulting in reduced registration accuracy. On the other hand, electromagnetic tracking systems use electromagnetic fields to track the position and posture of surgical instruments. They have a certain penetration ability and can overcome the occlusion problem to a certain extent. However, electromagnetic systems are easily interfered by metal objects. At the same time, many equipment and tools in the operating room contain metal components, which will also affect the accuracy of surgical device tracking results.

[0004] In addition, optical tracking systems require multiple cameras and computing equipment, which not only takes up a lot of space, but is also difficult to carry and install, making them unsuitable for use in emergency situations. Electromagnetic systems have lower accuracy and stability than optical systems, and are prone to position drift and errors, especially in complex surgical environments, such as those during earthquakes, fires, or mass casualty events.

[0005] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0006] The embodiments of the present invention provide a surgical instrument registration method and system based on augmented reality technology, which are used to at least solve at least one problem in the above-mentioned related technologies.

[0007] In a first aspect, an embodiment of the present invention provides a surgical equipment registration method based on augmented reality technology, which is applied to AR glasses. The method includes: based on multiple sensor modules, respectively collecting target RGB images and target spatial depth information of a surgical area from different perspectives; each of the sensor modules has associated orientation information and perspective information; the sensor module includes a camera, a depth sensor, a laser rangefinder, and an inertial sensor; generating a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information; inputting the target RGB image and the target spatial depth information into an equipment recognition model, so that the equipment recognition model determines the target equipment type corresponding to the target surgical equipment to be registered in the surgical area; loading a target standard virtual component matching the target equipment type from a virtual component database; the virtual component database pre-stores multiple standard virtual components, each of which has a corresponding equipment type; obtaining real-time equipment tracking data for the target surgical equipment, and dynamically superimposing the target standard virtual component to the three-dimensional model of the surgical area according to the device real-time tracking data; the real-time equipment tracking data includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment, and the real-time posture of the surgical equipment.

[0008] In a second aspect, an embodiment of the present invention provides a surgical equipment registration system based on augmented reality technology, the system comprising: a data acquisition unit, for acquiring a target RGB image and target spatial depth information of a surgical area from different perspectives based on a plurality of sensor modules; each of the sensor modules has associated orientation information and perspective information; the sensor module comprises a camera, a depth sensor, a laser rangefinder and an inertial sensor; a three-dimensional model generation unit, for generating a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information; an equipment type recognition unit, for inputting the target RGB image and the target spatial depth information into an equipment recognition model, so that the equipment type recognition unit can be used to generate a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information. The recognition model determines the target equipment type corresponding to the target surgical equipment to be registered in the surgical area; the virtual component matching unit is used to load the target standard virtual component matching the target equipment type from the virtual component database; the virtual component database pre-stores multiple standard virtual components, each of which has a corresponding equipment type; the virtual component overlay unit is used to obtain the real-time tracking data of the equipment for the target surgical equipment, and dynamically overlay the target standard virtual component to the three-dimensional model of the surgical area according to the real-time tracking data of the device; the real-time tracking data of the equipment includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment and the real-time posture of the surgical equipment.

[0009] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method.

[0010] In a fourth aspect, an embodiment of the present invention provides a storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of the above-mentioned method of the present invention.

[0011] In a fifth aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product comprises a computer program stored on a storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the above method.

[0012] The invention provides a surgical equipment registration method, system, electronic device and non-transitory computer-readable storage medium based on augmented reality technology, thereby optimizing the reliability and convenience of surgical equipment registration operations in modern information-based surgeries based on AR (augmented reality) technology or other remote image remote control, and can produce at least the following technical effects:

[0013] (1) By collecting the target RGB image and target spatial depth information of the surgical area from different perspectives through multiple sensor modules, a high-precision three-dimensional model of the surgical area can be generated. Based on the fusion of multi-sensor data, the error that may occur in a single sensor can be effectively reduced, thereby improving the accuracy of surgical equipment registration. In addition, the combination of laser rangefinder and depth sensor can accurately measure the spatial position of surgical equipment, avoiding the problem of sensitivity of the optical system to ambient light and overcoming the weakness of the electromagnetic system to metal interference, thereby achieving more stable and accurate positioning of surgical equipment.

[0014] (2) By inputting the target RGB image and spatial depth information into the equipment recognition model, the type of surgical equipment can be accurately identified, and the matching target standard virtual components can be loaded from the virtual component database. Based on the robustness of the deep learning model, it can maintain stable equipment recognition and registration performance in complex surgical environments. In addition, by capturing real-time equipment tracking data, including the real-time posture of AR glasses, the real-time position of AR glasses, the real-time position of surgical equipment, and the real-time posture of surgical equipment, the stability of equipment registration in a dynamic surgical environment is ensured, position drift and error are avoided, and the reliability of the system registration results is improved.

[0015] (3) In this technical solution, the virtual components of surgical instruments are dynamically superimposed on the three-dimensional model of the surgical area through augmented reality (AR) technology, realizing real-time registration and tracking of the instruments. The surgical team can intuitively observe and operate the surgical instruments, which helps to improve the accuracy of modern information-based surgical operations. The use of AR glasses makes the system highly portable and flexible, suitable for rapid deployment and use in emergency situations, easy to carry and install, and can maintain good registration performance in various complex environments. It is particularly suitable for emergency scenarios such as earthquakes, fires or mass casualties.

[0016] This technical solution provides a new surgical equipment registration solution by integrating the depth information and RGB images of the surgical area, which can avoid the sensitivity of the optical system to strong light or shadows, and overcome the interference of the electromagnetic system to metal objects, ensuring the stability and reliability of the system in different surgical environments. As a result, the system can still accurately and stably complete the registration of surgical instruments in complex and changeable surgical environments, providing a solid technical guarantee for modern information-based surgery, such as automated surgery or remote-assisted surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart showing an example of a surgical equipment registration method based on augmented reality technology according to an embodiment of the present invention is shown;

[0019] Figure 2 A flowchart showing an example of a surgical equipment registration method based on augmented reality technology according to an embodiment of the present invention is shown;

[0020] Figure 3 An operation flow chart of an example of collecting target space depth information according to an embodiment of the present invention is shown;

[0021] Figure 4 An operation flow chart of an example of generating a three-dimensional model of a surgical area according to a target RGB image and target spatial depth information according to an embodiment of the present invention is shown;

[0022] Figure 5 A schematic diagram showing a structural connection of an example of a device identification model according to an embodiment of the present invention is shown;

[0023] Figure 6An operation flow chart showing an example of dynamically superimposing a target standard virtual component onto a three-dimensional model of a surgical area according to an embodiment of the present invention;

[0024] Figure 7 A structural block diagram of an example of a surgical equipment registration system based on augmented reality technology according to an embodiment of the present invention is shown;

[0025] Figure 8 It is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0028] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0029] Figure 1 A flowchart of an example of a surgical instrument registration method based on augmented reality technology according to an embodiment of the present invention is shown.

[0030] It should be noted that in modern information-based surgery, in order to achieve remote control of surgical equipment or intelligent assisted path planning, the surgical equipment needs to be registered first. That is, during the operation, the virtual surgical equipment model and the actual physical surgical equipment's position and posture in space are accurately aligned and matched to ensure that the virtual and real-world equipment correspond to each other in the same coordinate system, making it easier for doctors to perform surgery in an augmented reality environment.

[0031] Regarding the executor of the method of the embodiment of the present invention, it can be any controller or processor with computing or processing capabilities, which can be integrated or partially integrated in AR glasses to significantly improve the accuracy, reliability and adaptability of surgical equipment registration by applying multi-view sensor data fusion and AR technology.

[0032] In some examples, it may be integrated into the configuration server through software, hardware, or a combination of software and hardware, which should not be limited here.

[0033] The following will take the surgical equipment registration platform as an exemplary implementation subject to introduce the details of the technical solution involved in the present invention. However, it should be understood that one or more steps involved in the following process can be implemented by one or more controllers or software installed and deployed in the client or server.

[0034] like Figure 1 As shown, in step S110, based on multiple sensor modules, the target RGB image and target spatial depth information of the surgical area are collected from different perspectives.

[0035] Here, each sensor module has associated orientation information and viewing angle information, and the sensor module includes a camera, a depth sensor, a laser rangefinder, and an inertial sensor.

[0036] In some embodiments, multiple sensor modules are integrated into the AR glasses, which are installed at different positions and have associated orientation information and viewing angle information to collect target RGB images and target spatial depth information of the surgical area from different viewing angles, respectively, thereby ensuring that comprehensive surgical area data can be collected from multiple angles. It should be noted that a dual depth sensing mechanism is set in the sensor module, and the posture and motion data of the sensor module are collected through inertial sensors to ensure the accuracy and synchronization of the sensor data collected. In addition, a laser rangefinder is used for high-precision distance measurement to supplement the data of the depth sensor to improve the accuracy of the depth information, thereby greatly ensuring the reliability of the surgical equipment registration results in spatial information.

[0037] In step S120, a three-dimensional model of the surgical area is generated according to the target RGB image and the target spatial depth information.

[0038] In some embodiments, the collected target RGB image and target spatial depth information can be used to generate a three-dimensional model of the surgical area using a three-dimensional reconstruction algorithm. More specifically, the three-dimensional reconstruction algorithm combines multi-view data to generate a high-precision three-dimensional model of the surgical area through point cloud fusion and surface reconstruction technology. In addition, in the process of generating the three-dimensional model, depth information is used to perform data correction to reduce noise and errors and improve the accuracy of the model. As a result, the generated three-dimensional model of the surgical area can accurately reflect the actual situation at the surgical site, provide scene environment information of surgical instruments, provide a reliable basis for subsequent identification and tracking of surgical instruments, improve the accuracy and safety of surgical operations, and reduce surgical risks caused by environmental errors.

[0039] In step S130 , the target RGB image and the target spatial depth information are input into an equipment recognition model, so that the equipment recognition model determines the target equipment type corresponding to the target surgical equipment to be registered in the surgical area.

[0040] Here, the equipment recognition model can adopt various non-restrictive deep learning models, such as CNN or Transformer, etc. In some embodiments, the equipment recognition model is based on a deep learning algorithm and is trained with a large amount of surgical equipment images and depth data, so that it can efficiently recognize different types of surgical equipment. More specifically, the equipment recognition model can accurately identify the target surgical equipment to be registered in the surgical area and output the corresponding equipment type through steps such as feature extraction, pattern matching and classification.

[0041] In step S140, a target standard virtual component matching the target equipment type is loaded from a virtual component database. The virtual component database pre-stores a plurality of standard virtual components, each of which has a corresponding equipment type.

[0042] Here, the virtual component database can be maintained synchronously by the AR glasses and the surgical equipment registration platform, and pre-stores standard virtual components corresponding to multiple equipment types. For example, the standard virtual components pre-stored in the virtual component database include screws, steel plates and drills of different types and specifications, ensuring that they are completely consistent with the size, shape and function of the actual surgical equipment, thereby loading standard virtual components that match the identified screws, steel plates and drills from the virtual component database. In addition, each standard virtual component has detailed equipment type and specification information. Therefore, compared with real-time image modeling, the use of standardized virtual components ensures the consistency and accuracy of surgical equipment registration, and by directly calling the pre-stored standard virtual components, the loading process is fast and efficient, reducing waiting time.

[0043] In step S150, the real-time tracking data of the target surgical instrument is obtained, and the target standard virtual component is dynamically superimposed on the three-dimensional model of the surgical area according to the real-time tracking data of the device.

[0044] Here, the real-time tracking data of the equipment includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment, and the real-time posture of the surgical equipment. Thus, by acquiring the posture and position data of the AR glasses and surgical equipment in real time, the accuracy of the dynamic superposition is ensured, and the superposition deviation caused by the movement of the surgical equipment or AR glasses (especially the AR glasses in the wearing state) is avoided. Furthermore, computer vision and sensor fusion technology can also be used to update the position and posture of the virtual components in the three-dimensional model of the surgical area in real time, ensuring that the dynamic superposition of the virtual components is synchronized with the actual operation of the surgical equipment, and providing real-time visual feedback. Thus, real-time dynamic superposition ensures the precise alignment of the virtual components with the actual equipment during the operation, enhances the surgical team's intuitive understanding of the surgical area and equipment, reduces the risk of misoperation, and improves the accuracy and safety of surgical operations.

[0045] Through the embodiment of the present invention, multiple sensor modules are used to collect the target RGB image and target spatial depth information of the surgical area from different perspectives, and the multi-perspective data collection ensures that the image and depth information of the surgical area are more comprehensive and accurate. The orientation and perspective information of the sensor module helps to generate an accurate three-dimensional model of the surgical area. In addition, the AR glasses integrate multiple sensor modules, are compact, easy to carry and install, and are suitable for use in emergency situations. Compared with the optical tracking system that requires multiple cameras and a large number of computing devices, the embodiment of the present invention is more convenient and suitable for rapid deployment in complex situations such as earthquakes, fires or mass casualties. In addition, the equipment recognition model is used to automatically identify the type of target surgical equipment in the surgical area, and the matching standard virtual components are loaded from the virtual component database, which improves work efficiency and ensures the standardization and consistency of the surgical equipment registration process. In addition, by acquiring the posture and position data of the AR glasses and surgical equipment in real time, the virtual components are dynamically superimposed on the three-dimensional model of the surgical area, realizing real-time tracking and accurate registration of surgical equipment, so that any equipment movement during the operation can be accurately captured, ensuring the accuracy and safety of the surgical operation. Therefore, through multi-sensor fusion, augmented reality technology, automated identification and standardized matching, high-precision, stable and real-time positioning and registration of surgical instruments are achieved, improving the safety and efficiency of smart surgery.

[0046] It should be noted that, through the above operations such as step S110 to step S150, the automatic registration of surgical instruments in the surgical area using AR glasses is realized. In some examples of the embodiments of the present invention, AR glasses can also provide a manual calibration function to adjust or confirm the registration details by receiving human-computer interaction instructions.

[0047] Figure 2 A flowchart of an example of a surgical instrument registration method based on augmented reality technology according to an embodiment of the present invention is shown.

[0048] In step S210, an automated registration operation of surgical equipment is performed based on the AR glasses.

[0049] For more specific operation details of step S210, reference may be made to the above description of the operations in conjunction with steps S110 to S150, which will not be repeated here.

[0050] In step S220, it is detected whether there is a human-computer interaction instruction. The human-computer interaction instruction includes any one of the following: a voice interaction instruction, a control interaction instruction, and a gesture interaction instruction.

[0051] In some embodiments, AR glasses can have a built-in voice recognition module or touchpad to monitor interactive instructions from surgeons in real time. More specifically, the voice recognition module recognizes the doctor's voice commands, such as the doctor's command "calibrate screws" or "confirm position"; the touchpad on the AR glasses inputs specific commands, such as sliding to adjust the position of the equipment; the camera and depth sensor recognize the doctor's gestures, such as sensing finger pointing or palm waving to perform specific operations. Thus, the flexibility and adaptability of the system are improved through a variety of human-computer interaction methods, allowing doctors to choose the most convenient interaction method according to actual conditions, ensuring that the system can quickly respond to the doctor's registration assistance operations.

[0052] In step S231, when it is detected that the first human-computer interaction instruction satisfies the first preset interaction condition, calibration information corresponding to the first human-computer interaction instruction is parsed, and the position or posture of the target standard virtual component is adjusted according to the calibration information.

[0053] In some embodiments, when it is detected that the first human-computer interaction instruction satisfies the first preset interaction condition (for example, the voice command "calibrate the screw"), the platform system parses the calibration information corresponding to the instruction and extracts key information in the instruction, such as specific parameters for adjusting the direction, angle or position. Then, according to the parsed calibration information, the position or posture of the target standard virtual component is dynamically adjusted. For example, the system can rotate the virtual screw to a certain angle or move it to a specific position according to the doctor's gesture or voice command. Thus, through the parsing and real-time adjustment of the human-computer interaction instruction, the alignment of the virtual component with the actual surgical equipment is made more precise, so that the doctor can flexibly calibrate the deviation of the equipment position or posture during the automated registration process according to the actual equipment position, thereby improving the accuracy and flexibility of the surgical operation.

[0054] In step S233, when it is detected that the second human-machine interaction instruction satisfies the second preset interaction condition, it is confirmed that the registration of the target surgical instrument is completed.

[0055] In some embodiments, once the confirmation instruction (e.g., the voice command "confirm position") is effective, the system locks the position and posture of the current virtual component and updates the relevant data in the three-dimensional model of the surgical area to complete the equipment registration process. Thus, by adding a registration confirmation mechanism, it is possible to ensure that the positioning and posture of the surgical equipment are accurately recorded after the doctor's confirmation, preventing accidental adjustment or misoperation, and providing additional safety protection.

[0056] Figure 3 An operational flow chart of an example of collecting target spatial depth information according to an embodiment of the present invention is shown.

[0057] like Figure 3 As shown, in step S310, based on the depth sensor and the laser rangefinder in the sensor module, the first depth distribution information and the second depth distribution information of the surgical area are respectively collected.

[0058] Specifically, the depth data D of the surgical area is obtained by a depth sensor. s and the depth data D obtained by the laser rangefinder l The depth map D provided by the depth sensor s is a two-dimensional matrix, each element D s (x, y) represents the depth value at the pixel (x, y). The depth data D provided by the laser rangefinder l is a sparse set of depth points, each of which contains position and depth information (x i ,y i ,d i ), where x i ,y i is the pixel coordinate in the image, d iis the depth value of the point.

[0059] In step S320, the first depth distribution information and the second depth distribution information are fused to determine corresponding fused depth distribution information.

[0060] Here, in order to enhance the accuracy of the depth sensor data, we need to fuse the depth data of the laser rangefinder with the depth sensor data. Specifically, by weighted averaging, the laser rangefinder data D l Fusion to the depth sensor data D s In order to improve the accuracy of the overall depth map, the fused depth map is denoted as D f , the specific fusion formula is as follows:

[0061] D f (x,y)=w s (x,y)·D s (x,y)+w l (x,y)·D l (x,y) , formula (1)

[0062] Where D f (x, y) represents the depth value of the fused depth distribution information at (x, y), D s (x, y) represents the depth value of the depth sensor at (x, y); D l (x, y) represents the depth value of the laser rangefinder at (x, y). If the laser rangefinder has no measurement value at (x, y), then D l (x,y)=0;w s (x,y) represents the weight of the depth sensor data, w l (x,y) represents the weight of the laser rangefinder data;

[0063] For laser rangefinder data, if there is no measurement value at (x,y), then w l (x,y)=0, otherwise initialize weight w l (x, y) = 1. Considering that the accuracy of laser rangefinder data is usually higher than that of depth sensor data, the weights can be adjusted to better reflect this, so the weights are adjusted using the following formula:

[0064] w l (x,y)=k×w l (x,y) , formula (2)

[0065] Wherein, k represents the weight enhancement coefficient of the laser rangefinder, and k>1, for example, k=3.

[0066] In order to ensure the rationality of the fused depth value, the weights need to be normalized:

[0067]

[0068]

[0069] In step S330, bilateral filtering is performed on the fused depth distribution information to obtain corresponding target space depth information.

[0070] It should be noted that the fused depth map D f There may still be some noise and discontinuities, and the quality of the depth map can be improved through further optimization processing.

[0071] More specifically, a bilateral filtering method is used to smooth the depth map while retaining edge details. Bilateral filtering is an image filtering technique that considers both spatial and intensity domain information, and can smooth noise while retaining edge details in the image.

[0072]

[0073] Where D f′ (x, y) represents the depth value of the filtered depth map at (x, y), Ω represents the neighborhood range centered at (x, y); W p represents the normalization coefficient to ensure that the sum of weights is 1; σ d Represents the standard deviation of the spatial domain, which is used to control the degree of smoothness within the spatial range; σ r Indicates the standard deviation of the intensity domain, which is used to control the degree of smoothing within the depth value range.

[0074] In an embodiment of the present invention, by combining the data of the depth sensor and the laser rangefinder and adopting a weighted average fusion method, the accuracy of depth measurement can be significantly improved. The depth sensor provides comprehensive depth information, while the data accuracy of the laser rangefinder is relatively high. Through weighted fusion, the advantages of both can be effectively utilized to improve the overall accuracy of the depth map, which helps to improve the precise positioning and registration of surgical instruments. In addition, since there may be various interference factors (such as equipment noise, ambient light changes, etc.) in the surgical environment, and surgical instruments usually have complex geometric shapes and fine structures, and clear edge details help to accurately identify and locate the equipment, bilateral filtering can be used to retain the edge details in the depth map during the smoothing process using the information in the intensity domain, while maintaining the integrity of the depth information, removing these noises, and improving the stability and reliability of the registration of surgical instruments. Therefore, bilateral filtering simultaneously performs weighted averaging in the spatial domain and the intensity domain, so that the depth map maintains the clarity of the edges while smoothing, and enhances the detail performance of the depth map, especially at the edges and complex structure areas of the equipment, which can provide higher resolution and more delicate depth information.

[0075] Figure 4 The flowchart shows an example of generating a three-dimensional model of a surgical area based on a target RGB image and target spatial depth information according to an embodiment of the present invention.

[0076] like Figure 4 As shown, in step S410, the target RGB image is spatially aligned and matched with the target spatial depth information.

[0077] In some embodiments, after the image correction operation, an alignment correction operation is performed.

[0078] The image distortion correction formula is as follows:

[0079] x corrected =x·(1+k1r 2 +k2r 4 +k3r 6 ), Formula (6)

[0080] y corrected =y·(1+k1r 2 +k2r 4 +k3r 6 ), Formula (7)

[0081] In the formula, x, y represent the pixel coordinates in the original image, x corrected ,y corrected represents the corrected pixel coordinates, k1, k2, k3 represent the lens distortion coefficients, and r represents the distance from the pixel to the optical center of the camera.

[0082] The alignment correction formula is as follows:

[0083]

[0084]

[0085] Where u, v represent the pixel coordinates of the aligned image plane, X, Y, Z represent the three-dimensional point coordinates in the camera coordinate system, and Z is the depth value obtained from the depth map; f x ,f y represents the focal length of the camera, c x ,c y Represents the coordinates of the camera optical center (principal point).

[0086] By spatially aligning the RGB image and the depth map, we ensure that the two are precisely aligned in the same coordinate system. The combination of distortion correction and pixel coordinate transformation greatly reduces the deviation between image and depth information, ensuring data consistency and accuracy.

[0087] In step S420, three-dimensional point cloud data of the surgical area is generated according to the aligned and matched target RGB image and target spatial depth information.

[0088] More specifically, each pixel is traversed and the corresponding three-dimensional point is generated according to the depth value to form a point cloud data set. The point cloud is divided into a voxel grid, and the points in each voxel are replaced by its centroid to reduce the number of point clouds. Statistical methods are used to remove outliers and enhance the overall quality of the point cloud. Therefore, the generated point cloud is processed by voxel filtering and statistical outlier removal methods to eliminate noise and outliers, further improving the overall quality and credibility of the point cloud.

[0089] In step S430, a three-dimensional model is reconstructed based on the three-dimensional point cloud data to generate a three-dimensional model of the surgical area.

[0090] In some implementations, point cloud registration is performed based on the orientation information and viewing angle information corresponding to each sensor module. For example, the posture information provided by the IMU sensor can be used to perform preliminary point cloud registration.

[0091] More specifically, based on the orientation information and viewing angle information corresponding to each sensor module, point cloud registration is performed on the three-dimensional point cloud data of the surgical area.

[0092] P aligned =R IMU ·P+t IMU , Formula (10)

[0093] Where P aligned represents the point cloud after registration, P represents the initial point cloud, R IMURepresents the rotation matrix of the sensor module, t IMU Represents the translation vector of the sensor module.

[0094] Therefore, the attitude information provided by the IMU sensor is used for initial registration to improve the registration accuracy of the point cloud data and ensure that the point cloud data from different perspectives can be accurately aligned.

[0095] In step S440, the point cloud data is converted into a continuous surface model using the Poisson equation, and the details of the continuous surface model are enhanced based on a Marching Cubes algorithm to generate a triangular mesh.

[0096] Here, the Poisson equation is used to convert the point cloud data into a continuous surface model. By calculating the point cloud gradient field and solving the Poisson equation, a high-precision continuous surface model is generated. Poisson reconstruction can effectively process noisy point cloud data and generate a smooth and detailed surface model. In addition, the continuous surface model is converted into a highly detailed triangular mesh by combining Poisson reconstruction and the Marching Cubes algorithm. The Marching Cubes algorithm achieves detail enhancement and triangular mesh generation through voxel division and isosurface construction, ensuring high resolution and precision of the three-dimensional model.

[0097] In step S450, the triangular mesh is simplified by using the QEM (Quadric Error Metrics) method, and the normal vector of the mesh is smoothed by using the Laplacian smoothing algorithm.

[0098] Therefore, the QEM method is used to simplify the mesh, effectively reducing the number of triangular facets and improving the processing efficiency of the 3D model. At the same time, the Laplacian smoothing algorithm is used to smooth the mesh normal vectors, eliminate sharp edges and noise, and improve the visual effect and rendering quality of the model.

[0099] In step S460, the color values ​​in the target RGB image are mapped to the vertex colors of the three-dimensional model, and the mapped colors are subjected to illumination correction to ensure visual consistency under different illumination conditions.

[0100] More specifically, the color mapping formula and lighting correction formula are as follows:

[0101] C(u,v)=I(u,v), Formula (11)

[0102]

[0103] In the formula, C(u,v) represents the color value of the point on the 3D model, I(u,v) represents the color value on the RGB image, C′(u,v) represents the corrected color value, and I current Indicates the current light intensity; I target Indicates the target light intensity, which is preset to the ideal light value in the operating room.

[0104] Here, the color values ​​in the RGB image are mapped to the vertex colors of the 3D model to ensure the accuracy and consistency of the model surface color. The color mapping formula is used to accurately implement the texture mapping of the 3D model and enhance the visual effect. In addition, real-time illumination measurement and dynamic correction methods are used to perform illumination correction according to changes in ambient light intensity. By real-time measurement of the current light intensity I current , and the target light intensity I target Compare and dynamically adjust color values ​​to ensure visual consistency and realism in different lighting conditions.

[0105] Through the embodiments of the present invention, through high-quality point cloud data generation and three-dimensional model reconstruction, the system can adapt to different surgical environments and complex operating scenarios. Even in the case of changes in ambient light and noise interference, the system can still provide a high-precision and high-reliability three-dimensional model. As a result, a high-precision three-dimensional model of the surgical area can be quickly generated to provide real-time support for the registration and positioning of surgical equipment. In emergency situations (such as earthquakes, fires, etc.), the system can respond quickly and provide accurate spatial references to improve the safety and efficiency of surgery. On the other hand, through high-precision three-dimensional model reconstruction and realistic texture mapping, intuitive three-dimensional visual effects are provided. Doctors can view the three-dimensional model of the surgical area in real time through AR glasses, which enhances the intuitiveness and controllability of the operation, allowing the system to quickly identify and locate surgical equipment and achieve an efficient equipment registration process.

[0106] Figure 5 A schematic diagram of structural connections of an example of a device identification model according to an embodiment of the present invention is shown.

[0107] like Figure 5 As shown, the equipment recognition model 500 adopts a hybrid deep learning model based on fusion of multimodal features and self-supervised learning, which includes an input layer 510, a feature extraction module 520, an image segmentation module 530, a multimodal feature fusion module 540, a global average pooling layer 550 and an output layer 560.

[0108] More specifically, the input layer 510 is used to receive the target RGB image and the target spatial depth information to determine the corresponding RGB-D data. RGB-D , Where H is the image height, W is the image width, and 4 represents 3 RGB channels and 1 depth channel.

[0109] By fusing RGB and depth information, the model can take full advantage of the advantages of both data sources and significantly improve the recognition accuracy of surgical instruments. RGB images provide rich color and texture information, while depth maps provide spatial structure information of objects. The combination of the two can more comprehensively describe the characteristics of surgical instruments.

[0110] The feature extraction module 520 uses a multi-scale convolution layer and extracts features in parallel through convolution kernels of different sizes, and then fuses these features to obtain a multi-scale feature map.

[0111] Exemplarily, the multi-scale convolution formula is:

[0112] F MS =σ([W1*I RGB-D +b1]⊕[W2*I RGB-D +b2]⊕[W3*I RGB-D +b3]) , formula (13)

[0113] In the formula, F MS Represents the output feature map of the multi-scale convolutional layer, I RGB-D represents the input feature map, W i represents the i-th convolution kernel, k i is the convolution kernel size, C i is the number of output channels of the i-th convolution kernel, b i represents the bias of the i-th convolution kernel, * represents the convolution operation, ⊕ represents the feature map concatenation operation, and σ represents the activation function (such as ReLU).

[0114] By using convolution kernels of different sizes to extract features in parallel, the characteristic information of surgical instruments at different scales can be captured, thereby improving the model's ability to recognize instrument details.

[0115] The image segmentation module 530 uses U-Net to perform high-precision segmentation through an encoder-decoder structure to obtain a corresponding segmentation feature map.

[0116] It should be noted that U-Net is a convolutional neural network for image segmentation, which performs high-precision segmentation through an encoder-decoder structure. In the encoder part, multiple convolutional layers and pooling layers are used to gradually reduce the spatial dimensions of the feature map and extract high-level features. In the decoder part, upsampling layers and convolutional layers are used to gradually restore the spatial dimensions of the feature map, while combining the features of the encoder for fine segmentation.

[0117] Therefore, the U-Net network structure achieves accurate segmentation of surgical instruments through the encoder-decoder method. The segmented feature map can accurately locate and identify the contours and boundaries of surgical instruments, ensuring the accuracy of feature extraction.

[0118] The multimodal feature fusion module 540 includes a feature fusion module 541 and a self-supervised learning module 542. The feature fusion module 541 is used to fuse the segmentation feature map and the multi-scale feature map to determine the corresponding initial fusion feature map. The self-supervised learning module 542 is used to enhance the initial fusion feature map through a self-supervised learning mechanism to determine the target fusion feature map.

[0119] Here, in the multimodal feature fusion module, the feature map segmented by U-Net is fused with the feature map of the multi-scale convolutional layer, and the feature representation is enhanced by combining the self-supervised learning mechanism.

[0120] More specifically, the feature fusion formula and the self-supervised learning mechanism formula are:

[0121] F fused =F MS ⊕F U-Net , Formula (14)

[0122] F self =σ(W self ·flatten(F fused )+b self ), Formula (15)

[0123] In the formula, F fused represents the initial fusion feature map, F U-Net Represents the segmentation feature map output by U-Net, F MS Represents the multi-scale feature map output by the multi-scale convolutional layer; F self represents the target fusion feature map, σ represents the activation function of the self-supervised learning module; W self and b self They represent the weight matrix and bias term of the self-supervised learning module respectively.

[0124] Therefore, through the self-supervised learning mechanism, the model can be pre-trained on unlabeled data and learn richer and more general feature representations, further improving the efficiency and accuracy of feature extraction.

[0125] The global average pooling layer 550 is used to perform global average pooling on the target fused feature map to determine the corresponding global pooling feature vector.

[0126] The fused features are pooled through the global average pooling layer and converted into classifier input. More specifically, the global average pooling formula is as follows:

[0127]

[0128] In the formula, F GAP represents the global pooling feature vector, F self [i, j] represents the value of the target fusion feature map at position (i, j), and H and W represent the height and width of the target fusion feature map, respectively.

[0129] Here, the traditional fully connected layer is omitted, and the feature map is directly converted into the classifier input through the global average pooling layer, which significantly reduces the number of model parameters and reduces the computational complexity. At the same time, global average pooling can effectively prevent overfitting and improve the generalization ability of the model.

[0130] The output layer 560 is used to predict the surgical instrument category probability distribution by using the softmax function:

[0131]

[0132] In the formula, represents the predicted probability distribution of surgical equipment categories, C is the number of categories, and W out and b out Represent the weight matrix and bias term of the output layer respectively.

[0133] Through the embodiment of the present invention, based on the hybrid deep learning model design of multimodal feature fusion and self-supervised learning, the surgical equipment recognition model can achieve the advantages of high precision, high efficiency, strong robustness and convenient deployment in this technical scenario. In addition, due to the optimized design of the above model structure, the reasoning process is more efficient, which is suitable for the needs of real-time recognition of surgical equipment and can quickly and accurately identify and locate surgical equipment in an actual surgical environment.

[0134] In some examples of the embodiments of the present invention, in order to make the equipment recognition model have higher accuracy and robustness in the surgical equipment recognition task, a comprehensive multi-task loss function is formed by combining classification loss, self-supervised learning loss and feature contrast loss.

[0135] Here, classification loss is used to measure the classification accuracy of the model in the equipment recognition task, and the cross-entropy loss function (Cross-Entropy Loss) is used. Self-supervised learning loss is used to enhance feature representation. By training self-supervised tasks on unlabeled data, the loss of image reconstruction tasks, i.e., image reconstruction loss function, can be used. Feature contrast loss is used to enhance the distinguishability of feature representation, so that similar samples are closer in the feature space and dissimilar samples are farther apart. The loss function in the contrastive learning framework (Contrastive Learning) can be used. Furthermore, in order to balance the impact of each loss, a weight coefficient is introduced to fuse each loss function to form a comprehensive multi-task loss function.

[0136] More specifically, the loss function of the equipment recognition model is for:

[0137]

[0138]

[0139]

[0140]

[0141] In the formula, represents the classification loss, using the cross entropy loss function; N represents the total number of samples in the data sample set, C represents the total number of surgical equipment categories, and y i,c represents the true label of the i-th sample, Indicates the predicted probability that the i-th sample belongs to the c-th class; represents the self-supervised learning loss, using the image reconstruction loss function; X i represents the original image corresponding to the i-th sample, Indicates that for X i the reconstructed image; represents feature contrast loss, using contrast learning loss function; z i represents the feature representation of sample i, represents the positive sample feature representation of sample i, z j represents the feature representation of sample j, and sim() represents the similarity function.

[0142] Through the design of the above loss function provided by the embodiment of the present invention, the cross entropy loss is used to ensure the high accuracy of the model in the task of surgical equipment classification, so that the model can accurately distinguish different categories of surgical equipment. As a result, the improvement of classification accuracy helps to reduce the potential risks caused by equipment recognition errors during surgery and ensure the smooth progress of surgery. The self-supervised learning loss is trained through unlabeled data, and the model can learn richer and more general feature representations and improve the generalization ability of unknown scenes. As a result, the self-supervised learning mechanism utilizes a large amount of unlabeled data, reduces the dependence on large-scale labeled data, and improves the adaptability of the model in a variable surgical environment. The feature contrast loss makes the distance between similar samples in the feature space closer and the distance between dissimilar samples farther through the contrast learning framework, thereby enhancing the distinguishability of feature representation. As a result, enhancing feature distinguishability can effectively improve the recognition ability of the model in complex surgical scenes, such as the robustness under conditions of illumination changes, partial occlusion, etc. Furthermore, the comprehensive loss function combines the three losses of classification, self-supervised learning, and contrastive learning. Through multi-task learning, the model can learn multiple feature representations at the same time. Collaborative optimization can make the model perform well in many aspects, improve the comprehensive ability of the model, and ensure that it can perform well in a variety of surgical scenarios. In particular, through the introduction of self-supervised learning and contrastive learning, the model can learn more common features in multiple tasks, reduce the risk of overfitting of a single task, and enhance the generalization ability of the model, so that the model can still perform well in unseen data and scenarios, and is suitable for a wide range of practical applications. In this way, the stability and reliability of the model in different surgical environments and conditions are improved, ensuring the performance of the model in actual surgical operations.

[0143] Figure 6 The flowchart shows an example of dynamically superimposing a target standard virtual component onto a three-dimensional model of a surgical area according to an embodiment of the present invention.

[0144] like Figure 6 As shown, by using an unscented Kalman filter to smooth the real-time tracking data of surgical instruments and AR glasses, and dynamically updating the position and posture of the target standard virtual component based on the smoothed data, the accuracy and real-time performance of surgical instrument recognition and virtual component superposition can be significantly improved.

[0145] More specifically, in step S610, an unscented Kalman filter is used to smooth the device real-time tracking data.

[0146] In some embodiments, sensors and AR glasses in the surgical area are used to obtain real-time updated data of the surgical environment, including the position and posture of surgical equipment, and changes in the surgical environment. Then, based on the real-time updated data, the dynamic adjustment parameters of the target standard virtual components are calculated to ensure that the virtual model is always aligned with the actual equipment. An unscented Kalman filter (UKF) is used for smoothing to reduce noise and errors.

[0147] In some embodiments, an unscented Kalman filter (UKF) is used to smooth the real-time tracking data of surgical instruments and AR glasses, and the formula is as follows:

[0148] t adjusted (t) = UKF(t instrument (t),t AR (t)), Formula (22)

[0149] R adjusted (t) = UKF(R instrument (t),R AR (t)), Formula (23)

[0150] In the formula, t instrument (t) represents the position vector of the surgical instrument at time t, and t adjusted (t) represents the real-time position after smoothing; R instrument (t) represents the posture rotation matrix of the surgical instrument at time t, and R adjusted (t) represents the real-time attitude after smoothing; UKF() represents the unscented Kalman filter function; t AR (t) and R AR (t) represent the position vector and attitude rotation matrix of the AR glasses at time t respectively.

[0151] It should be noted that the specific steps of the unscented Kalman filter function include state prediction, covariance prediction, Kalman gain calculation, state update and covariance update, and can be referred to the records in the current relevant technology, which will not be expanded here.

[0152] The real-time tracking data of surgical equipment and AR glasses is smoothed by using the unscented Kalman filter to reduce noise and errors, ensuring high accuracy and stability of position and posture data. The data processed by UKF is more accurate and reliable, the alignment of virtual components with actual surgical equipment is more precise, the deviation between virtual and actual equipment is reduced, and accurate operation during surgery is guaranteed.

[0153] In step S620 , the real-time position and real-time posture of the target standard virtual component are updated according to the smoothed real-time tracking data of the device.

[0154] More specifically, based on the smoothed data, the real-time position and posture of the target standard virtual component are updated as follows:

[0155] t updated (t) = t initial +t adjusted (t)+△t(t), formula (24)

[0156]

[0157] The adaptive adjustment of position and attitude is calculated through feedback control, and the formula is as follows:

[0158] △t(t)=K p (t instrument (t)-t updated (t-1))+K d (v instrument (t)-v updated (t-1)), Formula (26)

[0159] △q(t)=Slerp(q instrument (t),q updated (t-1),K q ), Formula (27)

[0160] In the formula, t updated (t) represents the updated real-time position of the target standard virtual component at time t, t initial represents the initial position of the target standard virtual component, t adjusted (t) represents the real-time position at time t after smoothing, △t(t) represents the adaptive adjustment amount of the position at time t determined by the proportional differential feedback system; q updated (t) represents the updated real-time attitude quaternion of the target virtual component at time t, q initial Represents the initial attitude quaternion of the target virtual component, q adjusted (t) represents the real-time attitude quaternion at time t after smoothing, △q(t) represents the attitude adaptive adjustment amount at time t, Represents quaternion multiplication; v instrument (t) represents the speed of the surgical instrument at time t, v updated (t-1) represents the updated virtual component speed at time t-1, K p Represents the position proportional gain, K d Represents the position differential gain, K q represents the attitude adjustment gain, q instrument (t) represents the quaternion of the surgical instrument’s posture at time t, and q updated(t-1) represents the virtual component posture quaternion after being updated at time t-1.

[0161] The posture adjustment of virtual components is achieved through quaternion interpolation (Slerp) to ensure the smoothness and accuracy of posture transformation. Quaternion interpolation can handle complex rotation transformations and avoid the singularity problem caused by Euler angles, so that virtual components can maintain smooth and stable posture adjustment in a rapidly changing surgical environment, improving the visualization of surgical equipment. In addition, through the adaptive adjustment algorithm, the position and posture of virtual components are dynamically adjusted according to real-time tracking data to ensure the synchronization of virtual models with actual surgical equipment. The adaptive adjustment algorithm can respond to changes in surgical equipment in real time, provide fast and accurate adjustments, and enable virtual components to closely follow the movement and rotation of actual equipment, improving the dynamic response capability and adaptability of the system.

[0162] Through the embodiment of the present invention, the noise and error in the real-time tracking data are significantly reduced through UKF and adaptive adjustment algorithms, ensuring high-precision alignment and real-time update of virtual components. As a result, the alignment error between virtual and actual equipment is reduced, the reliability and stability of the system are improved, and it is ensured that the virtual components are always highly consistent with the actual surgical equipment during the operation, reducing the risk of the operation.

[0163] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0164] Figure 7 A structural block diagram of an example of a surgical equipment registration system based on augmented reality technology according to an embodiment of the present invention is shown.

[0165] like Figure 7 As shown, the surgical equipment registration system 700 based on augmented reality technology includes a data acquisition unit 710 , a three-dimensional model generation unit 720 , an equipment type identification unit 730 , a virtual component matching unit 740 and a virtual component superposition unit 750 .

[0166] The data acquisition unit 710 is used to collect the target RGB image and target spatial depth information of the surgical area from different perspectives based on multiple sensor modules; each of the sensor modules has associated orientation information and perspective information; the sensor modules include a camera, a depth sensor, a laser rangefinder and an inertial sensor.

[0167] The three-dimensional model generating unit 720 is used to generate a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information.

[0168] The equipment type identification unit 730 is used to input the target RGB image and the target spatial depth information into an equipment identification model, so as to determine the target equipment type corresponding to the target surgical equipment to be registered in the surgical area through the equipment identification model.

[0169] The virtual component matching unit 740 is used to load a target standard virtual component that matches the target equipment type from a virtual component database; the virtual component database pre-stores a plurality of standard virtual components, each of which has a corresponding equipment type.

[0170] The virtual component overlay unit 750 is used to obtain real-time equipment tracking data for the target surgical equipment, and dynamically overlay the target standard virtual component onto the three-dimensional model of the surgical area according to the real-time tracking data of the device; the real-time equipment tracking data includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment, and the real-time posture of the surgical equipment.

[0171] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the above-mentioned surgical equipment registration method based on augmented reality technology of the present invention.

[0172] In some embodiments, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned surgical equipment registration method based on augmented reality technology.

[0173] In some embodiments, an embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a surgical equipment registration method based on augmented reality technology.

[0174] Figure 8 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for executing a surgical instrument registration method based on augmented reality technology provided by another embodiment of the present invention. Figure 8 As shown, the device includes:

[0175] One or more processors 810 and memory 820, Figure 8 A processor 810 is taken as an example.

[0176] The device for executing the surgical instrument registration method based on augmented reality technology may further include: an input device 830 and an output device 840 .

[0177] The processor 810, the memory 820, the input device 830 and the output device 840 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0178] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the surgical equipment registration method based on augmented reality technology in the embodiment of the present invention. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, the surgical equipment registration method based on augmented reality technology in the above method embodiment is implemented.

[0179] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include a memory remotely arranged relative to the processor 810, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] The input device 830 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 840 may include a display device such as a display screen.

[0181] The one or more modules are stored in the memory 820 , and when executed by the one or more processors 810 , perform the surgical instrument registration method based on augmented reality technology in any of the above method embodiments.

[0182] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0183] The electronic devices of the embodiments of the present invention exist in various forms, including but not limited to:

[0184] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0185] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0186] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0187] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0188] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical equipment registration method based on augmented reality technology, applied to AR glasses, characterized in that: The method comprises: Based on multiple sensor modules, target RGB images and target spatial depth information of the surgical area are collected from different perspectives respectively; each of the sensor modules has associated orientation information and perspective information respectively; the sensor modules include a camera, a depth sensor, a laser rangefinder and an inertial sensor; Generate a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information; Inputting the target RGB image and the target spatial depth information into an equipment recognition model, so as to determine the target equipment type corresponding to the target surgical equipment to be registered in the surgical area by the equipment recognition model; Loading a target standard virtual component matching the target equipment type from a virtual component database; the virtual component database pre-stores a plurality of standard virtual components, each of which has a corresponding equipment type; Real-time equipment tracking data for the target surgical equipment is obtained, and the target standard virtual component is dynamically superimposed on the three-dimensional model of the surgical area according to the real-time tracking data of the device; the real-time equipment tracking data includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment, and the real-time posture of the surgical equipment.

2. The method according to claim 1, wherein: After dynamically superimposing the target standard virtual component onto the three-dimensional model of the surgical area according to the real-time tracking data of the device, the method further includes: Detecting whether there is a human-computer interaction instruction; the human-computer interaction instruction includes any one of the following: a voice interaction instruction, a control interaction instruction, and a gesture interaction instruction; When it is detected that the first human-computer interaction instruction satisfies the first preset interaction condition, parsing calibration information corresponding to the first human-computer interaction instruction, and adjusting the position or posture of the target standard virtual component according to the calibration information; and When it is detected that the second human-machine interaction instruction satisfies the second preset interaction condition, it is confirmed that the registration of the target surgical instrument is completed.

3. The method according to claim 1, wherein: The method is based on multiple sensor modules to collect target RGB images and target spatial depth information of the surgical area from different perspectives, including: Based on the depth sensor and the laser rangefinder in the sensor module, first depth distribution information and second depth distribution information of the surgical area are respectively collected; The first depth distribution information and the second depth distribution information are fused to determine corresponding fused depth distribution information: D f (x,y)=w s (x,y)·D s (x,y)+w l (x,y)·D l (x,y) Where D f (x, y) represents the depth value of the fused depth distribution information at (x, y), D s (x, y) represents the depth value of the depth sensor at (x, y); D l (x, y) represents the depth value of the laser rangefinder at (x, y). If the laser rangefinder has no measurement value at (x, y), then D l (x,y)=0;w s (x,y) represents the weight of the depth sensor data, w l (x,y) represents the weight of the laser rangefinder data; For laser rangefinder data, if there is no measurement value at (x,y), then w l (x,y)=0, otherwise initialize weight w l (x,y)=1, and the weight is adjusted by the following formula: w l (x,y)=k×w l (x,y) In the formula, k represents the weight enhancement coefficient of the laser rangefinder, and k>1; In order to ensure the rationality of the fused depth value, the weights need to be normalized: A bilateral filtering process is performed on the fused depth distribution information to obtain the corresponding target space depth information: Where D f′ (x, y) represents the depth value of the filtered depth map at (x, y), Ω represents the neighborhood range centered at (x, y); W p represents the normalization coefficient to ensure that the sum of weights is 1; σ d Represents the standard deviation of the spatial domain, which is used to control the degree of smoothness within the spatial range; σ r Indicates the standard deviation of the intensity domain, which is used to control the degree of smoothing within the depth value range.

4. The method according to claim 1, wherein: Generating a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information includes: Performing spatial alignment and matching of the target RGB image and the target spatial depth information; Generate three-dimensional point cloud data of the surgical area based on the aligned and matched target RGB image and target spatial depth information; Reconstructing a three-dimensional model according to the three-dimensional point cloud data to generate a three-dimensional model of the surgical area specifically includes: Based on the orientation information and viewing angle information corresponding to each sensor module, the three-dimensional point cloud data of the surgical area is aligned: P aligned =R IMU ·P+t IMU Where P aligned represents the point cloud after registration, P represents the initial point cloud, R IMU Represents the rotation matrix of the sensor module, t IMU Represents the translation vector of the sensor module; The point cloud data is converted into a continuous surface model using the Poisson equation, and the details of the continuous surface model are enhanced based on the marching cube algorithm to generate a triangular mesh; The QEM method is used to simplify the triangular mesh, and the Laplace smoothing algorithm is used to smooth the normal vector of the mesh. Map the color values ​​in the target RGB image to the vertex colors of the 3D model and perform lighting correction on the mapped colors to ensure visual consistency under different lighting conditions: C(u,v)=I(u,v) In the formula, C(u,v) represents the color value of the point on the 3D model, I(u,v) represents the color value on the RGB image, C′(u,v) represents the corrected color value, and I current Indicates the current light intensity; I target Indicates the target light intensity, which is preset to the ideal light value in the operating room.

5. The method according to claim 1, wherein: The equipment recognition model adopts a hybrid deep learning model based on fusion of multimodal features and self-supervised learning, which includes an input layer, a feature extraction module, an image segmentation module, a multimodal feature fusion module, a global average pooling layer and an output layer; The input layer is used to receive the target RGB image and the target spatial depth information to determine corresponding RGB-D data; The feature extraction module uses a multi-scale convolution layer and extracts features in parallel through convolution kernels of different sizes, and then fuses these features to obtain a multi-scale feature map; The image segmentation module adopts U-Net to perform high-precision segmentation through an encoder-decoder structure to obtain a corresponding segmentation feature map; The multimodal feature fusion module includes a feature fusion module and a self-supervised learning module; the feature fusion module is used to fuse the segmentation feature map and the multi-scale feature map to determine the corresponding initial fusion feature map; the self-supervised learning module is used to enhance the initial fusion feature map through a self-supervised learning mechanism to determine a target fusion feature map: F self =σ(W self ·flatten(F fused )+b self ) In the formula, F fused represents the initial fusion feature map, F U-Net Represents the segmentation feature map output by U-Net, F MS Represents the multi-scale feature map output by the multi-scale convolutional layer; F self represents the target fusion feature map, σ represents the activation function of the self-supervised learning module; W self and b self Represent the weight matrix and bias term of the self-supervised learning module respectively; The global average pooling layer is used to perform global average pooling on the target fusion feature map to determine a corresponding global pooling feature vector; In the formula, F GAP represents the global pooling feature vector, F self [i, j] represents the value of the target fusion feature map at position (i, j), and H and W represent the height and width of the target fusion feature map, respectively. The output layer is used to predict the surgical instrument category probability distribution by using the softmax function: In the formula, represents the predicted probability distribution of surgical equipment categories, C is the number of categories, and W out and b out Represent the weight matrix and bias term of the output layer respectively.

6. The method according to claim 5, wherein: The loss function of the equipment recognition model for: In the formula, represents the classification loss, using the cross entropy loss function; N represents the total number of samples in the data sample set, C represents the total number of surgical equipment categories, and y i,c represents the true label of the i-th sample, Indicates the predicted probability that the i-th sample belongs to the c-th class; represents the self-supervised learning loss, using the image reconstruction loss function; X i represents the original image corresponding to the i-th sample, Indicates that for X i the reconstructed image; represents feature contrast loss, using contrast learning loss function; z i represents the feature representation of sample i, represents the positive sample feature representation of sample i, z j represents the feature representation of sample j, and sim() represents the similarity function.

7. The method according to claim 1, wherein: The method of dynamically superimposing the target standard virtual component onto the three-dimensional model of the surgical area according to the real-time tracking data of the device includes: The real-time tracking data of the device is smoothed using an unscented Kalman filter: t adjusted (t)=UKF(t instrument (t),t AR (t)) R adjusted (t)=UKF(R instrument (t),R AR (t)) Where, t instrument (t) represents the position vector of the surgical instrument at time t, and t adjusted (t) represents the real-time position after smoothing; R instrument (t) represents the posture rotation matrix of the surgical instrument at time t, and R adjusted (t) represents the real-time attitude after smoothing; UKF() represents the unscented Kalman filter function; t AR (t) and R AR (t) respectively represent the position vector and attitude rotation matrix of the AR glasses at time t; According to the smoothed device real-time tracking data, the real-time position and real-time posture of the target standard virtual component are updated: t updated (t)=t initial +t adjusted (t)+△t(t) △t(t)=K p (t instrument (t)-t updated (t-1))+K d (v instrument (t)-v updated (t-1)) △q(t)=Slerp(q instrument (t),q updated (t-1),K q ) Where, t updated (t) represents the updated real-time position of the target standard virtual component at time t, t initial represents the initial position of the target standard virtual component, t adjusted (t) represents the real-time position at time t after smoothing, △t(t) represents the adaptive adjustment amount of the position at time t determined by the proportional differential feedback system; q updated (t) represents the updated real-time attitude quaternion of the target virtual component at time t, q initial Represents the initial attitude quaternion of the target virtual component, q adjusted (t) represents the real-time attitude quaternion at time t after smoothing, △q(t) represents the attitude adaptive adjustment amount at time t, Represents quaternion multiplication; v instrument (t) represents the speed of the surgical instrument at time t, v updated (t-1) represents the updated virtual component speed at time t-1, K p Represents the position proportional gain, K d Represents the position differential gain, K q represents the attitude adjustment gain, q instrument (t) represents the quaternion of the surgical instrument’s posture at time t, and q updated (t-1) represents the virtual component posture quaternion after being updated at time t-1.

8. A surgical equipment registration system based on augmented reality technology, characterized in that: The system comprises: A data acquisition unit, used to acquire a target RGB image and target spatial depth information of a surgical area from different viewing angles based on a plurality of sensor modules; each of the sensor modules has associated orientation information and viewing angle information; the sensor modules include a camera, a depth sensor, a laser rangefinder and an inertial sensor; A three-dimensional model generating unit, used for generating a three-dimensional model of the surgical area according to the target RGB image and the target spatial depth information; an equipment type recognition unit, configured to input the target RGB image and the target spatial depth information into an equipment recognition model, so as to determine the target equipment type corresponding to the target surgical equipment to be registered in the surgical area by the equipment recognition model; A virtual component matching unit, used for loading a target standard virtual component matching the target equipment type from a virtual component database; the virtual component database pre-stores a plurality of standard virtual components, each of which has a corresponding equipment type; A virtual component overlay unit is used to obtain real-time equipment tracking data for the target surgical equipment, and dynamically overlay the target standard virtual component onto the three-dimensional model of the surgical area according to the real-time tracking data of the device; the real-time equipment tracking data includes the real-time posture of the AR glasses, the real-time position of the AR glasses, the real-time position of the surgical equipment, and the real-time posture of the surgical equipment.

Images