Surgical assistance system, method, electronic device and computer storage medium

By collecting and fusing the reference and auxiliary images of the surgical site in real time in the surgical assistance system, the position and contour of the target object are enhanced, and the problem of non-rigid and decent shape of the surgical surface is solved, and the position deviation of the target object is achieved during the operation Real-time accurate positioning and efficient operation.

CN118236174BActive Publication Date: 2024-11-15FOURTH MILITARY MEDICAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410296468.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-15
Publication Date
2024-11-15
Estimated Expiration
2044-03-15

AI Technical Summary

Technical Problem

During the operation, since the surgical surface is non-rigid and decent, changes in human tissue and patient position cause the target object to be offset, and the prior art is difficult to achieve real-time accurate positioning, which increases the difficulty and duration of the operation.

Method used

A surgical assist system is designed, including a reference image detector, an auxiliary image detector and a data processing unit, which collects and fuses the reference image and auxiliary image of the surgical site in real time, and enhances the display of the target object in the reference image through the data processing unit. and the outline of the target, generate the target fusion image.

Benefits of technology

Real-time accurate positioning and display of target objects is achieved, surgical efficiency is improved, the accuracy of target objects is ensured, and the operation time is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118236174B_ABST
    Figure CN118236174B_ABST
Patent Text Reader

Abstract

The present disclosure provides a surgical assistance system, method, electronic device and computer storage medium, which belongs to the field of medical equipment technology. The system includes: a reference image detector, an auxiliary image detector, a data processing unit and a display unit; the reference image detector is configured to collect a reference image of the surgical site in real time during surgery; the auxiliary image detector is configured to collect an auxiliary image of the surgical site in real time during surgery; the data processing unit is configured to enhance the display of the target object in the reference image according to the position and contour of the target object in the auxiliary image to obtain a target fused image; the display unit is configured to display the target fused image. The target object can be enhanced and displayed in real time during surgery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of medical equipment, and in particular to a surgery assisting system, method, electronic equipment and computer storage medium. Background Art

[0002] With the development of medical imaging technology, various imaging technologies are used before and during surgery to assist doctors in performing surgery. Static images are usually collected before surgery to assist doctors in roughly determining the location of the target object (such as a tumor). During surgery, imaging technologies such as endoscopes can be used to determine the specific location of the target object.

[0003] However, since the surgical surface is a non-rigid surface, human tissue, patient position, and surgical anatomical surface will change during the operation, and the position of the target object will also change. The doctor needs to constantly determine the position of the target object, which increases the difficulty and duration of the operation. Summary of the invention

[0004] The present disclosure provides a surgery assisting system, method, electronic device and computer storage medium, which can enhance the display of target objects in real time during surgery.

[0005] The technical solution of the present disclosure is achieved as follows:

[0006] In a first aspect, the present disclosure provides a surgical assistance system, the system comprising: a reference image detector, an auxiliary image detector, a data processing unit, and a display unit;

[0007] a reference image detector configured to acquire a reference image of a surgical site during surgery in real time;

[0008] an auxiliary image detector configured to acquire auxiliary images of a surgical site during surgery in real time;

[0009] A data processing unit is configured to enhance and display the target object in the reference image according to the position and outline of the target object in the auxiliary image to obtain a target fused image;

[0010] The display unit is configured to display the target fused image.

[0011] In a second aspect, the present disclosure provides a surgical assistance method, which includes: acquiring a baseline image of a surgical site during surgery in real time; acquiring an auxiliary image of the surgical site during surgery in real time; enhancing the display of the target object in the baseline image based on the position and outline of the target object in the auxiliary image to obtain a target fusion image; and displaying the target fusion image.

[0012] In a third aspect, the present disclosure provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the surgical assistance method as described in the second aspect.

[0013] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a program or instruction, which, when executed by a processor, implements the surgical assistance as described in the second aspect. Steps of the method.

[0014] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product includes a computer program or instructions, and when the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the surgical assistance method as described in the second aspect.

[0015] In a sixth aspect, the present disclosure provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the surgical assistance method as described in the second aspect.

[0016] The present disclosure provides a surgical assistance system, which includes: a reference image detector, an auxiliary image detector, a data processing unit, and a display unit; the reference image detector is configured to collect a reference image of the surgical site during surgery in real time; the auxiliary image detector is configured to collect an auxiliary image of the surgical site during surgery in real time; the data processing unit is configured to enhance the display of the target object in the reference image according to the position and outline of the target object in the auxiliary image to obtain a target fusion image; the display unit is configured to display the target fusion image. During the operation, since the surgical surface is a non-rigid surface type, the position of the target object may be offset due to changes in human tissue, patient position, surgical anatomical surface, etc., resulting in an increase in the time it takes for the doctor to locate the target object. In this solution, the displayed target fusion image is fused and updated in real time during the operation, thereby ensuring the accurate display of the position of the target object, and the enhanced display of the outline of the target object can also facilitate the doctor's operation and improve the efficiency of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A structural block diagram of a surgical assistance system provided by the present disclosure;

[0018] Figure 2 This is one of the structural schematic diagrams of a surgical assistance system provided by the present disclosure;

[0019] Figure 3 This is a second structural schematic diagram of a surgical assistance system provided by the present invention;

[0020] Figure 4 A schematic diagram of coordinate conversion between a reference image and an ultrasonic image provided by the present disclosure;

[0021] Figure 5 A schematic diagram of a process for determining a target tumor using ultrasound data provided by the present disclosure;

[0022] Figure 6 A schematic diagram of the enhanced display of a target tumor in a reference image provided by the present disclosure;

[0023] Figure 7 A schematic diagram of displaying distance information in a target fused image provided by the present disclosure;

[0024] Figure 8 This is a third structural schematic diagram of a surgical assistance system provided by the present disclosure;

[0025] Fig. 9 A schematic diagram of coordinate conversion between a reference image and a thermal imaging image provided by the present disclosure;

[0026] Fig.10 A schematic diagram of the enhanced display of a target blood vessel in a reference image provided by the present disclosure;

[0027] Fig.11 A schematic diagram of a process for determining a target blood vessel in a thermal imaging image by a target detection model provided by the present disclosure;

[0028] Fig.12 This is a fourth structural schematic diagram of a surgical assistance system provided by the present disclosure;

[0029] Fig.13 A schematic diagram of the comparison of fused images provided by the present disclosure;

[0030] Fig.14 This is a fifth structural schematic diagram of a surgical assistance system provided by the present disclosure;

[0031] Fig.15 A schematic diagram of the interaction between the remote device and the surgical assistance system provided by the present disclosure;

[0032] Fig.16 A schematic diagram of annotated images provided for the present disclosure;

[0033] Fig.17 This is a sixth structural schematic diagram of a surgical assistance system provided by the present disclosure;

[0034] Fig.18 A schematic diagram of a surgical assistance method provided by the present invention;

[0035] Fig.19 A schematic diagram of the hardware structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the present disclosure to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of the present application.

[0037] Among medical imaging technologies, those commonly used before surgery include computed tomography (CT), magnetic resonance imaging (MRI), and the like.

[0038] Among them, CT imaging generates cross-sectional images of the inside of the body by combining X-ray imaging technology and computer analysis. The basic principle of CT imaging is: there is an X-ray source and an X-ray detector opposite it inside the CT scanner. When the X-ray source rotates around the patient, it emits thin beams of X-rays from multiple angles through the body. These X-rays are captured by the detector after penetrating the body tissue. Different types of tissue (such as bone, muscle, fat and air) absorb X-rays to different degrees, so the intensity of X-rays received by the detector will also be different. The X-ray signals captured by the detector are then converted into electrical signals and then into digital data for computer processing. The computer uses an algorithm called tomographic reconstruction to process these data obtained from multiple angles. By analyzing the penetration and absorption of X-rays at different angles, the computer can calculate the density of each small volume unit (called voxel) in the scanning area. The density of these voxels is then converted into different grayscales to generate images. Finally, these reconstructed cross-sectional images are displayed on the computer screen for doctors to analyze and interpret.

[0039] The principle of MRI imaging is: MRI is a medical imaging technology that uses strong magnetic fields and radio waves to obtain detailed images of the internal structure of the human body. Specifically, during the MRI scan, the patient is located in a strong magnetic field. This magnetic field aligns the spin directions of hydrogen nuclei (mainly hydrogen atoms in water molecules and fat) inside the body. Once the hydrogen nuclei are aligned, the scanner emits a series of radio wave pulses of a specific frequency, which temporarily disrupt the alignment of the hydrogen nuclei. When the radio wave pulses stop, the hydrogen nuclei will realign to their original state and release energy in the process. This energy is detected by the receiver of the MRI machine in the form of radio signals. Different types of tissues (such as fat and water) will realign hydrogen nuclei at different rates, so the energy released will also be different. The detected signals are sent to a computer, which processes these signals through complex mathematical and physical algorithms and converts them into images. The brightness of each point depends on the rate and amount of hydrogen nuclei in the tissue at that point to realign and release energy, which reflects the nature and state of different tissues. The resulting image can show different body structures, including brain tissue, muscles, joints, internal organs, etc.

[0040] Therefore, CT or MRI images are convenient for surgical planning and provide a clear representation of the surface and contours of organs, but they are inaccurate in locating tumors and key blood vessels inside organs. In addition, since the imaging devices of CT and MRI images are large, they cannot be integrated into auxiliary systems for real-time imaging during surgery.

[0041] The techniques commonly used during surgery include endoscopic imaging, red green blue (RGB) imaging, thermal imaging technology, and ultrasonic imaging (multi-purpose phased array ultrasonic imaging).

[0042] Among them, endoscopic imaging is a medical imaging technology that uses endoscopic equipment to allow doctors to observe and operate internal organs of the human body without major surgery. An endoscope is a slender tubular instrument, usually equipped with a light source and a camera, which can transmit images of internal organs back to an external display for doctors to view. Specifically, one end of the endoscopic device is equipped with a light source, usually an LED light or other high-intensity light source, to ensure that the internal organs are fully illuminated for easy imaging. The front end of the endoscope is usually equipped with one or more miniature cameras or optical lens systems to capture images of internal organs. The captured image is transmitted to an external display via a fiber optic bundle or cable inside the endoscope. In traditional endoscopes, optical fibers are used to transmit images from one end of the endoscope to the other; in more modern digital endoscopes, images are transmitted in electronic form through a thin cable. The transmitted image is displayed on an external display, usually in real time, so that doctors can directly see the condition of the internal organs and perform diagnosis or surgical operations accordingly.

[0043] The principle of RGB imaging is a technology that combines red, green, and blue light to produce images with a wide range of colors. This method is based on the working principle of the human visual system. There are three types of color-sensing cells in the human eye, which are most sensitive to red, green, and blue light. By adjusting the intensity of these three colors of light, almost all colors in the visible spectrum can be simulated. Specifically, in digital imaging devices (such as digital cameras or scanners), when light passes through the lens and shines on the sensor, the tiny photosensitive elements on the sensor (called pixels) capture the intensity of the light. Each pixel can only capture one color of light (red, green, or blue), which is achieved by placing color filters on the pixels. The most common arrangement is the Bayer filter array. In the Bayer filter array, each pixel is assigned to be red, green, or blue sensitive, with twice the number of green pixels as red or blue to better imitate the human eye's higher sensitivity to green light. When light hits the sensor, each pixel only records the light intensity information of its corresponding color. Since each pixel can only capture a single color, image processing algorithms (such as interpolation algorithms) are required to reconstruct the complete color information. This process is usually carried out in the camera's image processor, which calculates the color values ​​of neighboring pixels to estimate the complete RGB value for each pixel location. After the above processing, each pixel will have a complete set of RGB values, representing the color of that pixel location. These values ​​can be mixed in different proportions to produce a wide range of colors. The final RGB image can be displayed on an electronic display. RGB imaging allows digital images to show complex and rich colors in a way similar to how the human eye perceives.

[0044] Phased array ultrasound imaging is a technique that uses multiple ultrasound transmitters and receivers (i.e., arrays) for imaging. The principle of array ultrasound probe imaging is that the array ultrasound probe contains multiple small piezoelectric crystal elements that can generate ultrasound waves under the action of voltage. By precisely controlling the emission time of each of these elements, the ultrasound waves can be emitted in a specific shape and direction. This method is called beamforming. The probe can adjust the direction and focus of the ultrasound beam by changing the phase relationship (i.e., the relative emission time) between different elements. This allows different areas inside the body to be scanned without physically moving the probe, and the imaging quality at different depths can also be optimized by adjusting the focal depth. When ultrasound waves encounter tissues of different densities and elasticities (such as organ interfaces), echoes are generated. These echoes are detected by the same piezoelectric elements on the probe and converted into electrical signals. The detected electrical signals are sent to a computer for processing. By analyzing the time delay and intensity of the echoes, the computer can determine the depth of the echo source and the reflection intensity, which is used to generate an image. By synthesizing the echoes from different directions and depths, the computer can reconstruct a two-dimensional or three-dimensional image representing the structure in the body. In this process, each pixel of the image represents the reflection characteristics of the tissue at the corresponding location. By rapidly and continuously changing the direction and focus of the beam, an array ultrasound probe is able to provide real-time dynamic images.

[0045] Thermal imaging is based on the principle of infrared radiation. The human body and other organisms generate heat due to their normal physiological activities, and this heat is radiated from the surface of the skin in the form of infrared radiation. Thermal imaging detectors can capture this infrared radiation and convert it into images. Specifically, different parts of the human body emit different intensities of infrared radiation due to slight differences in temperature (such as caused by blood flow, inflammation or tumors). Thermal imaging detectors contain one or more infrared sensors that are very sensitive to infrared radiation within a specific wavelength range. When infrared radiation is incident on these sensors, the sensors generate electrical signals based on the intensity of the received radiation. These electrical signals are then converted into digital data for further processing. In this step, the detector usually amplifies, filters and digitizes the signal to improve the image quality. The digitized data is used to construct an image. The signal of each sensor element corresponds to a pixel on the image. The color or grayscale value of the pixel is determined by the intensity of the received infrared radiation and reflects the temperature of the measured area. In order to make the thermal image more useful, temperature calibration is usually required, which means matching the color or grayscale value on the image with the actual temperature. In this way, doctors can intuitively identify changes in body temperature based on changes in color. By analyzing the temperature distribution patterns on the thermogram, doctors can identify unusual hot or cold spots, which may indicate inflammation, blood circulation problems, tumors, or other medical conditions.

[0046] Therefore, endoscopic images and RGB images act as the doctor's "eyes" during surgery and have the characteristics of visual intuition. Endoscopic images and RGB images are closest to the actual images seen by the naked eye. Thermal imaging images can achieve real-time imaging and are very obvious for blood vessel visualization. Ultrasonic images can achieve real-time imaging and are very obvious for tumor visualization. However, thermal imaging and ultrasonic imaging have poor imaging quality, low signal-to-noise ratio, and are difficult to interpret.

[0047] Before and during surgery, various imaging technologies can be combined to assist doctors in locating the target object of their interest. However, static images collected before surgery can usually only help doctors roughly determine the location of the target object. However, the human body has a complex tissue structure, and even experienced doctors find it difficult to accurately determine the location of the target object during surgery based on static images. In addition, because the target object is usually hidden deep inside the tissues and organs, doctors cannot quickly determine the location of the target object based on their experience and real-time imaging.

[0048] In view of the above problems, the usual solution is to use image fusion technology, combining the advantages of various imaging technologies, fusing preoperative images and intraoperative images, and using fused images to more intuitively display the structure, position, surface information, internal blood vessels, tumors and other information of various organ tissues.

[0049] Since the preoperative CT / MRI images are static images, and the surgical surface is a non-rigid surface, during the operation, the human body tissue, patient position, and surgical anatomical surface will change, and the position of the target object that the doctor is concerned about will also change, but the position of the target object in the preoperative image remains unchanged. Therefore, when the preoperative image and the intraoperative image are fused, the target object positioning is not accurate enough, and the target object displayed in the fused image has a deviation. However, the operation requires high precision, and the existence of this deviation may cause the doctor to make an incorrect judgment and increase the operation time.

[0050] Therefore, the present application aims to provide a surgical assistance system that can accurately locate and display a target object in real time. Figure 1 A surgical assistance system is shown in the present disclosure. Figure 1 As shown, the surgical auxiliary system includes: a reference image detector 101, an auxiliary image detector 102, a data processing unit 103, and a display unit 104. The reference image detector 101 is configured to acquire a reference image of the surgical site during surgery in real time. The auxiliary image detector 102 is configured to acquire an auxiliary image of the surgical site during surgery in real time. The data processing unit 103 is configured to enhance the display of the target object in the reference image according to the position and contour of the target object in the auxiliary image to obtain a target fused image. The display unit 104 is configured to display the target fused image.

[0051] The image collected by the reference image detector 101 has the feature that it can be viewed directly by the human eye, that is, the difference between the collected image of the surgical site and the actual picture of the surgical site is small, such as the color of the reference image is different from the actual picture seen by the naked eye, and the reference image is the image after the actual picture is magnified a certain multiple, etc. Optionally, the reference image detector 101 can be any detector capable of collecting images that can be viewed directly by the naked eye, such as an endoscope detector, an RGB image detector, and an exoscopic detector.

[0052] The image collected by the auxiliary image detector 102 is better for displaying the target object, that is, although the auxiliary image is not intuitive to the human eye, it can highlight the target object. Optionally, the auxiliary image detector 102 can be at least one of an ultrasonic detector and a thermal imaging detector.

[0053] The target object can be any object that the doctor is concerned about, such as a specific tissue structure, tumor, blood vessel, etc.

[0054] Therefore, during the operation, since the surgical surface is a non-rigid surface, the position of the target object may be offset. In this solution, the target fusion image is updated in real time during the operation, ensuring the accurate display of the position of the target object and highlighting the outline of the target object to facilitate the doctor's accurate positioning and improve the efficiency of the operation.

[0055] In some embodiments, Figure 2 The surgical assistance system including an ultrasonic probe is shown. The auxiliary image probe 102 includes an ultrasonic probe 1021 ; accordingly, the target object includes a target tumor; and the auxiliary image includes an ultrasonic image.

[0056] The ultrasonic detector may preferably be a phased array ultrasonic probe, which includes multiple ultrasonic transmitters and receivers. The phased array can usually be divided into linear, matrix, annular and fan-shaped according to the array form. Phased array probes have a variety of different array arrangements, and their types can be divided into: one-dimensional linear array, two-dimensional matrix, annular array, fan-shaped array, concave array, convex array, dual linear array, etc. according to the arrangement of array elements. Different array arrangements will produce different sound field characteristics, so that the phased array can be used for detection under different working conditions. According to actual needs, a phased array ultrasonic probe with a suitable array arrangement can be selected.

[0057] The fusion of ultrasonic images with reference images (such as RGB images) can adopt augmented reality (AR) technology. AR technology is a technology that combines the real world with virtual information based on computer real-time computing and multi-sensor fusion.

[0058] Specifically, in order to fuse the reference image and the ultrasonic image, the ultrasonic image needs to be converted to the same coordinate system as the reference image. In the surgical assistance system provided by the present application, the position of the reference image detector 101 is fixed and unchanged, and the ultrasonic detector 1021 has a fixed storage position when not in use, such as Figure 3 As shown, the dotted line 301 is an optional storage position of the ultrasonic probe 1021. When the ultrasonic probe 1021 is removed and starts to work, the positioning sensor (such as two gyroscope sensors) carried by the ultrasonic probe 1021 will perform real-time inertial measurement, and the data processing unit 103 can sense the current posture of the probe and the position relative to the storage position in real time.

[0059] For example, Figure 4 As shown, the coordinate system of the reference image detector 101 is used as the reference coordinate system, and the first pixel point in the upper left corner of the image collected by the reference image detector 101 is used as the coordinate origin. The coordinates corresponding to the first pixel in the upper left corner of the image collected by the ultrasonic detector 1021 at the initial position 301 (i.e., the storage position) in the reference coordinate system are determined. During the operation, the ultrasonic detector 1021 moves from position 301 to position 402. The two gyroscope sensors can determine the posture of the ultrasonic detector 1021 and the position relative to the initial position in real time. The data processing unit 103 converts the ultrasonic image collected by the ultrasonic detector 1021 into an ultrasonic image in the reference coordinate system according to the posture of the ultrasonic detector 1021 and the position relative to the initial position and the coordinates corresponding to the first pixel in the upper left corner at the initial position, so that the ultrasonic image and the reference image can be fused.

[0060] The auxiliary image detector 102 is specifically a two-dimensional matrix phased array ultrasound probe. The process of determining the target tumor through the collected two-dimensional phased array ultrasound data is as follows: Figure 5As shown. Collecting ultrasound data 501 specifically includes: using a two-dimensional phased array ultrasonic sensor, adjusting the time delay of transmission and reception and scanning in different directions to obtain a real-time two-dimensional ultrasonic data set. Signal processing 502 specifically includes: using beam imaging technology (enhancing the target's echo signal by coherently superimposing the signal of each receiving element) to process the collected two-dimensional ultrasonic data set, and demodulating and filtering the signal after beam imaging to emphasize the target tumor and reduce noise; by analyzing the time delay and intensity of the processed ultrasonic data, the depth and reflection intensity of the ultrasonic data source are determined, and this information is used to generate an image. Image analysis 503 specifically includes using image analysis technology to detect the target tumor in the image according to different densities; then using a segmentation algorithm to segment the image into different tissue regions, focusing on the target tumor. Positioning and contour extraction 504 specifically includes: by analyzing the information of the target tumor in the image, according to a preset response threshold, removing background noise and irrelevant features in the image, thereby screening out important feature responses; then according to Determine the maximum response area in the image, which is the approximate location of the target tumor; where R(θ) represents the maximum response in the θ direction, M represents the number of pixel rows, N represents the number of pixel columns, and I(i,j) represents the pixel value at pixel position i, j in the image; in order to further accurately determine the location and contour of the target tumor, after determining the approximate location of the target tumor, use a specific gradient operator to determine the gradient intensity (indicating the amplitude of the pixel intensity change at the point) and gradient direction (indicating the direction of the strongest change) of each pixel point in the image, and only retain the pixel point with the local maximum gradient value in the gradient direction on the gradient intensity map. Other non-maximum points are set to 0 to obtain the refined edge, and then the double threshold (high threshold and low threshold) is used to determine the real edge and potential edge. Pixels above the high threshold are considered strong edges, pixels below the low threshold are excluded, and pixels between the two are considered weak edges. Whether they are real edges depends on whether they are connected to strong edges. Finally, starting from the strong edge points, the weak edge points are connected to the strong edge points using edge tracking technology, so as to ensure that only when the weak edge points are connected to the strong edge points, they are included in the final edge image, and the contour obtained by the final connection is the accurate contour information of the target tumor. Real-time update and display 505 specifically includes: when the ultrasonic scanning is in progress, the image is updated in real time to reflect the new ultrasonic data (such as frame-by-frame image processing); the real-time updated image is fused with the reference image and displayed on the display unit 104, so that the doctor can observe the target tumor position and contour in real time.

[0061] For example, Figure 6As shown, it is a schematic diagram of the enhanced display of the target tumor in the reference image. The image indicated by the number 60 is the RGB image of the surgical site collected by the RGB detector, the image indicated by the number 61 is the ultrasonic image of the surgical site collected by the ultrasonic detector, and the image indicated by the number 62 is the enhanced display of the target tumor in the reference image. Among them, the arrow 611 is the target tumor. In this way, during the operation, it can assist the doctor to locate the position of the target tumor more accurately and improve the efficiency of the operation.

[0062] Optionally, the display of the target fusion image can be real-time fusion and continuous display; it can also be displayed when needed according to the control instruction, such as based on the control instruction indicating the highlighting of the target tumor, the target fusion image including the target tumor is controlled to stop displaying after a fixed time, and only the reference image is displayed; or the target fusion image including the target tumor is controlled to stop displaying after being displayed for a first preset number of times (such as flashing 5 times at 2s intervals) at a first preset interval, and only the reference image is displayed. In this way, the doctor can flexibly choose whether to display according to actual needs, which allows the doctor to clearly identify the tumor location without interfering with the surgical process.

[0063] In order to further assist doctors in performing tumor cutting surgery, the distance information between the medical device and the tumor can be displayed in real time. Therefore, in some embodiments of the present disclosure, the data processing unit 103 is further configured to: determine the distance information between the target tumor and the surgical instrument based on the ultrasonic image; the display unit 104 is also configured to display the distance information.

[0064] Phased array ultrasonic positioning is very accurate. Taking intracranial glioma resection surgery as an example, the phased array ultrasonic detector needs to be inserted into the gap of the patient's cerebral cortex in advance, 5cm±1cm away from the edge of the glioma, and a real-time array scan of the surgical site is performed. Due to the different densities of the tumor and the surgical instrument, which are inconsistent with the reflection characteristics of the ultrasound, the distance between the surgical instrument and the edge of the tumor can be accurately determined, and the positioning accuracy can reach about 0.5 mm. It can provide doctors with accurate distance information to assist doctors in surgery.

[0065] In this embodiment, the distance between the target tumor and the surgical instrument is determined by: based on the imaging capability and distance measurement principle of phased array ultrasound, the propagation distance of each beam is determined by the propagation time of each beam, and then the average of the propagation distances of each beam is determined as the relative distance between the surgical instrument and the edge of the tumor (the basic principle is to calculate the distance using the propagation speed of the sound wave and the time difference of the echo). The specific formula is: N represents the number of beams, d irepresents the propagation distance of the ith beam, which is equal to half the product of the speed of sound and the echo duration, and D represents the relative distance between the surgical instrument and the target tumor. In practice, factors such as the beam angle, scanning plane, and propagation speed may need to be considered. The specific calculation method depends on the performance and configuration of the ultrasonic imaging device.

[0066] For example, Figure 7 As shown, reference numeral 701 indicates a surgical instrument displayed in the ultrasonic image, reference numeral 702 indicates a target tumor displayed in the ultrasonic image, and reference numeral 703 indicates that the relative distance between the surgical instrument and the target tumor 702 is 2.58 mm in the target fusion image.

[0067] Optionally, the distance information is outputted by voice broadcast.

[0068] Specifically, the output mode of the distance information can be continuously updated and output, or it can be output when needed according to the control instruction, such as controlling the distance information to stop outputting after a fixed time based on the control instruction indicating the output of the distance information; or controlling the distance information to stop outputting after a second preset number of times (flashing and displaying 5 times at 2s intervals, and then stopping the display of the distance information) at a second preset interval. In this way, the doctor can be reminded and the display of too much information that interferes with the operation can be avoided.

[0069] Optionally, when the distance between the surgical instrument and the target tumor is less than the distance threshold corresponding to the target tumor type, an alarm message is output. The alarm message is used to prompt the doctor that the distance between the surgical instrument and the target tumor is less than the distance threshold. Each type of tumor corresponds to a distance threshold. The distance thresholds corresponding to different types of tumors may be the same or different. The output mode of the alarm message includes at least one of the following: voice, text, alarm pattern, and alarm indicator light.

[0070] During surgery, cutting off a large blood vessel may cause excessive bleeding, which affects the surgery and increases the pain of the patient. Therefore, blood vessels are also the focus of doctors during surgery. Therefore, in some embodiments of the present disclosure, Figure 8 The surgical assistance system including an ultrasonic detector is shown. The auxiliary image detector 102 also includes: a thermal imaging detector 1022; accordingly, the target object also includes: a target blood vessel; and the auxiliary image also includes: a thermal imaging image.

[0071] During the operation, the surgical surface is constantly covered with fresh blood, which seriously affects the operation. However, since the temperature of human tissue is mostly provided by arterial blood, the temperature of arterial blood is greater than the temperature of venous blood, which is greater than the temperature of human tissue. This temperature difference is usually between 0.5°C. The present disclosure uses a high-precision thermal imaging sensor to achieve high-precision temperature measurement, which is used to distinguish human tissue, blood vessels and blood that has already flowed out, and determine the distribution and direction of larger arteries and veins near the surgical surface.

[0072] Specifically, the reference image and the thermal imaging image are fused, such as Fig. 9 As shown, taking the reference image as an RGB image as an example, the positions and relative positions of the RGB detector 1011 and the thermal imaging detector 1022 are fixed, and the distance between the two is d. The process of image fusion is: camera calibration, which is the process of obtaining the camera intrinsic parameter matrix and distortion parameters. Through calibration, the pixel coordinates can be converted to camera coordinates, and the coordinate system conversion can be performed; calibration usually uses objects with known structures such as chessboards, captures images at multiple angles, and uses camera calibration tools to obtain the camera intrinsic parameter matrix and distortion parameters. Coordinate system conversion, coordinate system conversion is to convert the coordinates of each pixel point in the image of the thermal imaging image detector into the image coordinate system of the RGB image detector, considering the distance between the two cameras. This step ensures that the coordinate systems of the two images are consistent; that is, for each pixel point (X B , Y B ), and mapped to the coordinates (X A , Y A ), the coordinate mapping formula is: X A =X B +2d,Y A and Y B Image fusion: Image fusion is to fuse the thermal image after coordinate transformation to the RGB image, and copy the pixel value of each point on the thermal image to the corresponding position on the RGB image. In order to avoid the fused image affecting the actual size of the RGB image, transparency control or weighted fusion is required. The transparency of the thermal image can be adjusted to make it a superposition of the RGB image without affecting the size of the object in the RGB image.

[0073] Optionally, the target blood vessel can be displayed by continuously updating and displaying the target fusion image including the target blood vessel, or can be displayed when needed according to a control instruction, such as controlling the target fusion image including the target blood vessel to stop displaying after a fixed time based on a control instruction indicating the display of the target blood vessel, and only displaying the reference image; or controlling the target fusion image including the target blood vessel to stop displaying after a third preset number of times (flickering 5 times at 2s intervals) at a third preset interval. In this way, the doctor can be reminded of the location of the blood vessel and avoid interfering with the surgery.

[0074] It should be noted that, in order to distinguish the target blood vessel from the target tumor, the color of the target tumor and the color of the target blood vessel displayed in the target fusion image may be different, such as the target tumor image is black and the target blood vessel is red.

[0075] For example, Fig.10 The figure shows a schematic diagram of the enhanced display of the target blood vessel in the reference image. The image indicated by reference numeral 10 is an RGB image of the surgical site acquired by the RGB detector, and the image indicated by reference numeral 11 is a thermal imaging image of the surgical site acquired by the thermal imaging detector, wherein the highlighted white line in the figure is the target blood vessel; the image indicated by reference numeral 12 is an enhanced display of the target blood vessel in the reference image, wherein the highlighted white line in the figure is the target blood vessel. In this way, during the operation, it can assist the doctor to more accurately determine the location of the target blood vessel and avoid mistakenly cutting a larger blood vessel, which leads to increased bleeding.

[0076] Since the blood that has just flowed out may affect the positioning of the position and contour of the blood vessel, in order to more accurately locate the position of the target blood vessel, in some embodiments of the present disclosure, a target detection model is preloaded in the data processing unit 103; before enhancing the display of the target object in the reference image based on the position and contour of the target object in the auxiliary image to obtain the target fused image, the data processing unit 103 is also configured to: determine the position and contour of the blood vessel in the auxiliary image through the target detection model.

[0077] Specifically, the process of determining the target blood vessel in the thermal imaging image by the target detection model is as follows: Fig.11As shown. Data preparation 1101 is specifically used to collect thermal imaging images during surgery and label each image accordingly. The label is used to identify the location and contour of the blood vessels in the image. Data preprocessing 1102 is used to preprocess the thermal imaging images, including operations such as adjusting the size of the image, graying, normalizing, and denoising to ensure the data quality and consistency of the input model. Model construction 1103 is used to construct an initial target detection model, which can be any model suitable for detecting a target in an image; data set partitioning 1104 is used to divide the data set into a training set and a test set to ensure that the model is evaluated on unseen data. Model training 1105 is used to train the initial target detection model using the training set to obtain a target detection model. During the training period, the model updates the weights through back propagation and the optimizer, and gradually learns to extract image features to correctly determine the location and contour of the blood vessels. Model evaluation 1106 is used to evaluate the performance of the model using the test set, and to check indicators such as accuracy, precision, and recall rate to ensure that the model performs well on unseen data. The model application 1107 is used to determine the position and contour of the target blood vessel in the thermal imaging image through the target detection model based on the thermal imaging image.

[0078] Among them, the specific process of determining the position and contour of the target blood vessel in the thermal imaging image through the target detection model is as follows: extracting features in the thermal imaging image through a convolution operation, such as to distinguish between outflowing blood and blood vessels. Since the outflowing blood and blood vessels have large differences in shape and temperature, the extracted features include at least shape features and temperature features; performing a maximum pooling operation on the obtained feature map to extract the most significant features and obtain a target feature map. This operation can not only improve the network's anti-noise ability, but also reduce the amount of calculation; predicting the position and size of the target blood vessel bounding box based on the target feature map, and determining possible candidate areas; performing a more detailed analysis on each candidate area to determine whether it contains the target blood vessel; determining the contour of the target blood vessel based on the determined candidate area including the target blood vessel, and outputting the position and contour of the target blood vessel and a confidence score indicating the degree of confidence in its prediction.

[0079] In this way, by introducing the target detection model in the data processing unit, the target blood vessel can be located more accurately and the positioning accuracy can be improved.

[0080] The display during the operation is usually near-eye display, but when the operation is long, the near-eye display will cause visual fatigue to the doctor and affect the operation. Fig.12 As shown, the display unit is a light field display screen 1041; the light field display screen 1041 is specifically configured to display an enlarged virtual image 1042 of the target fused image at a preset position.

[0081] Among them, light field display technology can produce a three-dimensional virtual image in front of the user's eyes by precisely controlling the direction and position of light. In near-eye display applications, such as head-mounted display devices or augmented reality glasses, light field technology can be used to generate an enlarged virtual image. The light field display screen uses an optical lens to pull the image of the ordinary display screen away, presenting a long-distance enlarged virtual image to the doctor. In this way, compared with traditional near-eye display devices, it can avoid doctors' visual fatigue and fogging of near-eye devices, effectively ensuring the continuity of the operation and reducing the labor intensity of doctors during the operation.

[0082] Conventional AR display information will be displayed at the top layer of the fused image, which does not conform to the near-far relationship of the actual physical world image. Therefore, in some embodiments of the present disclosure, the data processing unit 103 is specifically configured to enhance the display of the target object in the reference image according to the position and outline of the target object in the auxiliary image, and the upper and lower positional relationship between the target object and the surgical instrument, to obtain a target fused image including the upper and lower positional relationship between the target object and the surgical instrument.

[0083] When the auxiliary image is superimposed on the base image, the target object is displayed on the upper layer of the base image, such as Fig.13 As shown, Fig.13 The figure a shows conventional AR fusion. The blood vessels in the area indicated by the reference numeral 1300 are displayed on the upper layer of the surgical instrument. However, in reality, the surgical instrument should be on the upper layer of the blood vessels. In the present disclosure, when the auxiliary image is superimposed on the reference image, it is fused and displayed according to the upper and lower position relationship between the surgical instrument and the target object, which complies with the perspective rule, such as Fig.13 In the target fusion image shown in b, since the blood vessels are located in the lower layer of the surgical instrument, in the target fusion image, part of the blood vessels in the area indicated by the reference numeral 1301 are blocked by the surgical instrument in the upper layer. That is, the digital information superimposed in the digital display screen conforms to the perspective rule, and the target fusion image includes the depth information of the upper and lower position relationship between the surgical instrument and the target object, and the display effect is more consistent with the actual surgical situation.

[0084] The surgical assistance system provided by the present disclosure can be used to assist doctors in performing surgery, and can also be used as a teaching device to assist interns in performing surgery. Therefore, in some embodiments of the present disclosure, for example Fig.14 As shown, the surgical assistance system also includes: a data communication unit 105; the data communication unit 105 is configured to connect to a remote device; transmit the content displayed by the display unit 104 to the remote device; receive the annotated image with annotations returned by the remote device; the display unit 104 is also configured to display the annotated image.

[0085] Specifically, Fig.15As shown, the remote device establishes a network connection with the surgical assistance system, and the intern and the instructor can make video and voice calls through the network connection. Through the network connection, the image displayed by the display unit can also be transmitted to the remote device in real time, such as the RGB image or fusion image displayed by the display unit. The instructor can annotate the received image and return the annotated image to the display unit for display. Fig.16 As shown, the dotted line box 1601 is the blood vessel marked by the instructor, and the dotted line box 1602 is the tumor marked by the instructor. In this way, for inexperienced interns, the location of the target object can be quickly determined based on the instructor's markings.

[0086] During the operation, in order to facilitate the control of various parts of the surgical assistance system, in some embodiments of the present application, such as Fig.17 As shown, the surgical assistance system further includes: an instruction receiving unit 106; the instruction receiving unit is configured to receive a control instruction from a user; and a data processing unit 103, configured to implement corresponding control according to the instructions in the control instruction.

[0087] The instruction receiving unit 106 may be a microphone for receiving voice control instructions, or a camera for receiving gesture control instructions, etc. Since doctors usually have both hands occupied during surgery, the instruction receiving unit is preferably a microphone.

[0088] Specifically, taking the control command as a voice control command as an example, the surgical assistance system disclosed in the present invention can at least implement the following control commands. When the system is powered on, the system can be turned on and off by voice; the display of the display unit can be controlled by voice, such as displaying a fused image, displaying a reference image, or displaying the distance between the surgical instrument and the target object; the reference image detector and the thermal imaging image detector can be simultaneously zoomed in or out by voice; the display of the light field display screen can be controlled; the display mode of the distance information can be controlled, such as text display, voice broadcast, etc. In this way, the doctor can control the system through commands, simplifying the operation of the system during the operation.

[0089] Alternatively, if Fig.17 As shown, the surgical auxiliary system may further include a surgical lamp 107. The surgical lamp 107 is configured to provide an illumination source during the surgical procedure. Preferably, the surgical lamp 107 is a cold light illumination source.

[0090] The present disclosure also provides a surgical assistance method, such as Fig.18 As shown, the method includes the following steps 1801 to 1804.

[0091] In step 1801, a reference image of the surgical site during surgery is acquired in real time.

[0092] In step 1802, auxiliary images of the surgical site during surgery are acquired in real time.

[0093] In step 1803, the target object is enhanced and displayed in the reference image according to the position and outline of the target object in the auxiliary image to obtain a target fused image.

[0094] In step 1804, the target fused image is displayed.

[0095] In some embodiments of the present disclosure, the target object includes: a target tumor; and the auxiliary image includes: an ultrasonic image.

[0096] In some embodiments of the present disclosure, after collecting auxiliary images of the surgical site during surgery in real time, the distance information between the target tumor and the surgical instrument is determined based on the ultrasonic image; and the distance information is displayed.

[0097] In some embodiments of the present disclosure, the target object further includes: a target blood vessel; and the auxiliary image further includes: a thermal imaging image.

[0098] In some embodiments of the present disclosure, before enhancing the display of the target object in the reference image based on the position and outline of the target object in the auxiliary image to obtain the target fused image, the position and outline of the blood vessel in the auxiliary image are determined by a target detection model.

[0099] In some embodiments of the present disclosure, displaying the target fused image includes: displaying an enlarged virtual image of the target fused image at a preset position.

[0100] In some embodiments of the present disclosure, based on the position and outline of the target object in the auxiliary image, the target object is enhanced and displayed in the reference image to obtain a target fusion image, including: based on the position and outline of the target object in the auxiliary image and the upper and lower positional relationship between the target object and the surgical instrument, the target object is enhanced and displayed in the reference image to obtain a target fusion image including the upper and lower positional relationship between the target object and the surgical instrument.

[0101] In some embodiments of the present disclosure, the method further includes: connecting to a remote device; transmitting the content displayed by the display unit to the remote device; receiving an annotated image with annotations returned by the remote device; and displaying the annotated image.

[0102] In some embodiments of the present disclosure, the method further includes: receiving a control instruction input by a user; and implementing corresponding control based on instructions in the control instruction.

[0103] It should be noted that the relevant description and technical effects of the embodiment of the surgical assistance method can refer to the relevant description of the above-mentioned surgical assistance system, and can achieve the same technical effects. In order to avoid repetition, they will not be repeated here.

[0104] Please refer to Fig.19, which shows a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure. In some examples, the electronic device may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The electronic device has a communication function and can access a wired network or a wireless network. The electronic device may generally refer to one of a plurality of terminals, and those skilled in the art may know that the number of the above terminals may be more or less. It can be understood that the electronic device undertakes the calculation and processing work of the technical solution of the present disclosure, and the present disclosure does not limit this.

[0105] like Fig.19 As shown, the electronic device in the present disclosure may include one or more of the following components: a processor 1910 and a memory 1920 .

[0106] Optionally, the processor 1910 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1920, and calling data stored in the memory 1920. Optionally, the processor 1910 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 1910, but may be implemented by a separate chip.

[0107] The memory 1920 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 1920 includes a non-transitory computer-readable storage medium. The memory 1920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0108] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the electronic device also includes a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, a sensor (such as an acceleration sensor, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module and other components, which will not be described in detail here.

[0109] The present disclosure also provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is used to be executed by a processor to implement the surgical assistance method described in the above embodiments.

[0110] The present disclosure also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes to implement the surgical assistance methods described in the above-mentioned embodiments.

[0111] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned surgical assistance method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0112] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0113] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, servers and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0117] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by a general or special-purpose computer.

[0118] It should be noted that the technical solutions described in the present disclosure can be combined arbitrarily without conflict.

[0119] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A surgical assistance system, characterized in that: The system comprises: a reference image detector, an auxiliary image detector, a data processing unit, and a display unit. The data processing unit is preloaded with a target detection model. The auxiliary image detector comprises: a thermal imaging detector; The reference image detector is configured to acquire a reference image of the surgical site during surgery in real time; The auxiliary image detector is configured to collect auxiliary images of the surgical site during surgery in real time, and the auxiliary images include: thermal imaging images; The data processing unit is configured to determine the position and contour of the target blood vessel in the thermal imaging image according to the shape characteristics and the temperature characteristics through the target detection model, and to enhance and display the target object in the reference image according to the position and contour of the target object in the auxiliary image to obtain a target fused image, wherein the target object includes: a target blood vessel; The display unit is configured to display a target fused image including the target blood vessel.

2. The system according to claim 1, characterized in that The auxiliary image detector further includes: an ultrasonic detector; correspondingly, the target object further includes: a target tumor; and the auxiliary image further includes: an ultrasonic image.

3. The system according to claim 2, characterized in that The data processing unit is further configured to: Determining distance information between the target tumor and a surgical instrument according to the ultrasonic image; The display unit is further configured to display the distance information.

4. The system according to any one of claims 1 to 3, characterized in that: The display unit is a light field display screen; The light field display screen is specifically configured to display an enlarged virtual image of the target fused image at a preset position.

5. The system according to any one of claims 1 to 3, characterized in that: The data processing unit is specifically configured to enhance the display of the target object in the reference image based on the position and outline of the target object in the auxiliary image, and the upper and lower positional relationship between the target object and the surgical instrument, so as to obtain a target fusion image including the upper and lower positional relationship between the target object and the surgical instrument.

6. The system according to any one of claims 1 to 3, characterized in that: The system further comprises: a data communication unit; The data communication unit is configured to connect to a remote device; transmitting the content displayed by the display unit to the remote device; Receive the annotated image returned by the remote device; The display unit is further configured to display the annotated image.

7. The system according to any one of claims 1 to 3, characterized in that: The system further comprises: an instruction receiving unit; The instruction receiving unit is configured to receive a control instruction input by a user; The data processing unit is further configured to implement corresponding control based on the instructions in the control instruction.

8. A surgical assistance method, characterized in that: The method comprises: Real-time acquisition of intraoperative reference images of the surgical site; Collecting auxiliary images of the surgical site in real time, wherein the auxiliary images include: thermal imaging images; Determining the position and contour of the blood vessel in the thermal imaging image according to the shape characteristics and temperature characteristics through the target detection model; According to the position and outline of the target object in the auxiliary image, the target object is enhanced and displayed in the reference image to obtain a target fused image, wherein the target object includes: a target blood vessel; A target fused image including the target blood vessel is displayed.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the surgical assistance method as described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the surgery assisting method as claimed in claim 8.