Image assistance method and system, electronic equipment, storage medium and program product

Abstract

The invention provides an image assisting method and system, an electronic device, a storage medium and a program product, and belongs to the technical field of intelligent medicines.The method comprises the steps that an operation image is obtained based on a camera device on a hot steam ablation operation catheter; based on the parameters of the camera equipment, generating image coordinates and superposing the image coordinates on the surgical image; based on the image coordinates, determining a sheath tube moving distance of the hot steam ablation operation; under the condition that the movement distance of the sheathing canal reaches the target distance, a surgical operation prompt is initiated. According to the method, the camera device on the hot steam ablation operation catheter is used for obtaining the operation image, additional endoscope equipment is not needed, the sheath tube moving distance of the hot steam ablation operation is determined through the image coordinates overlapped on the operation image, and under the condition that the sheath tube moving distance reaches the target distance, the operation prompt is initiated; the defects that judgment of a doctor on the moving distance of the sheathing canal completely depends on visual estimation and operation hand feeling, standard operation is difficult to achieve, and the uniformity of the operation effect is affected are overcome.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and in particular to an image-assisted method, system, electronic device, storage medium, and program product. Background Technology

[0002] Steam ablation is an important treatment for benign prostatic hyperplasia (BPH). It involves injecting high-temperature steam into the diseased prostate tissue, causing coagulative necrosis and thus achieving therapeutic results. Currently, steam ablation relies on endoscopic equipment to acquire and transmit images of the surgical field. Surgeons simultaneously monitor both the ablation equipment parameters and the endoscopic images. The judgment of the sheath movement distance depends entirely on visual estimation and tactile feedback, making standardized operation difficult and affecting the uniformity of surgical outcomes. Summary of the Invention

[0003] This invention provides an image-assisted method, system, electronic device, storage medium, and program product, which solves the problem of difficulty in achieving standardized operation and affecting the uniformity of surgical results.

[0004] This invention provides an image-assisted method, comprising: Surgical images are acquired using a camera device mounted on a surgical catheter for thermal steam ablation. Based on the parameters of the camera device, image coordinates are generated and superimposed on the surgical image; Based on the image coordinates, the sheath movement distance of the thermal steam ablation procedure is determined; Once the sheath has traveled the target distance, a surgical procedure prompt will be initiated.

[0005] As one embodiment, generating image coordinates based on the parameters of the camera device, and superimposing the image coordinates onto the surgical image, includes: Based on the optical and calibration parameters of the camera device, the distance parameter corresponding to each pixel of the surgical image is determined; Based on the distance parameter, rectangular coordinates are generated, and the rectangular coordinates are used as the image coordinates; The origin of the image coordinates is superimposed on the center of the surgical image, and the coordinate lines and scale information of the image coordinates are superimposed on the surgical image to achieve the superposition of the image coordinates on the surgical image.

[0006] As one embodiment, superimposing the coordinate lines and scale information of the image coordinates onto the surgical image includes: Call the graphics drawing object to overlay the coordinate lines and scale information of the image coordinates onto the surgical image.

[0007] As one embodiment, determining the sheath movement distance during the thermal steam ablation procedure based on the image coordinates includes: The initial position of the sheath on the image coordinates and the real-time position of the sheath on the image coordinates are obtained to determine the sheath movement distance of the thermal steam ablation procedure.

[0008] The present invention also provides an image-assisted system, comprising: A camera device, mounted on the hot steam ablation surgical catheter, is used to acquire surgical images; The host device is used to generate image coordinates based on the parameters of the camera device and superimpose the image coordinates onto the surgical image; determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates; and initiate a surgical operation prompt when the sheath movement distance reaches the target distance.

[0009] As one embodiment, the host device is used to generate image coordinates based on parameters of the camera device and superimpose the image coordinates onto the surgical image; and to determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates, including: The host device is configured to: acquire the surgical image from the camera device and store it in the original image buffer of the shared buffer based on a first thread; acquire the surgical image from the original image buffer based on a second thread, generate image coordinates based on the parameters of the camera device, superimpose the image coordinates onto the surgical image to obtain a processed surgical image, and store the processed surgical image in the processed image buffer of the shared buffer; acquire the processed surgical image from the processed image buffer based on a third thread and display the processed surgical image; and acquire the processed surgical image from the second thread based on a fourth thread, and determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates. The first thread has a higher priority than the second thread, and the second thread has a higher priority than the third thread and the fourth thread.

[0010] As an example, the host device is also configured to execute an exception handling process when an exception occurs during the operation of the first thread, the second thread, the third thread, and the fourth thread. The exception handling process includes initiating an exception alarm and recording exception information.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image-assisted method as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image-assisted method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image-assisted method as described above.

[0014] This invention provides an image-assisted method, system, electronic device, storage medium, and program product. It acquires surgical images using a camera device on a steam ablation surgical catheter; generates image coordinates based on parameters of the camera device and superimposes these coordinates onto the surgical image; determines the sheath movement distance for the steam ablation procedure based on the image coordinates; and initiates a surgical operation prompt when the sheath movement distance reaches a target distance. This invention utilizes a camera device on a steam ablation surgical catheter to acquire surgical images, eliminating the need for additional endoscopic equipment. The surgeon only needs to focus on the ablation device used in the steam ablation procedure, reducing the number of devices the surgeon needs to monitor and allowing for greater concentration. Furthermore, by determining the sheath movement distance using the superimposed image coordinates on the surgical image and initiating a surgical operation prompt when the sheath movement distance reaches the target distance, it overcomes the shortcomings of surgeons relying entirely on visual estimation and tactile feedback to judge the sheath movement distance, which makes standardized operation difficult and affects the uniformity of surgical outcomes. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the image-assisted method provided by the present invention.

[0017] Figure 2 This is a schematic diagram of the image-assisted system provided by the present invention.

[0018] Figure 3 The functional architecture diagram of the image assistance system provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the multi-threaded processing flow provided by the present invention.

[0020] Figure 5 This is a flowchart illustrating the image assistance method implemented by the image assistance system provided by the present invention.

[0021] Figure 6 This is a schematic diagram comparing the puncture spacing error of the present invention and the traditional method.

[0022] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] In prostate thermal steam ablation surgery, precisely controlling the advance and retreat distance of the sheath is crucial to ensuring treatment effectiveness and safety. Current traditional surgical methods have the following drawbacks: Lack of quantitative reference system: Doctors mainly rely on visual observation of the surgical field and operation feel to judge the distance of sheath movement. There is no intuitive digital coordinate reference. The prostate tissue is soft and there are no obvious anatomical landmarks during the movement of the sheath, making it difficult to achieve the standardized requirements of precise puncture. High equipment dependence and poor compatibility: Traditional surgery requires the use of a prostate thermal steam ablation main device in conjunction with a third-party large endoscope. The device is bulky (the endoscope main unit usually weighs >5kg) and has complex connection lines (including light source lines, data lines, etc.), which increases the difficulty of operating room layout. There are differences in communication protocols between different brands of equipment, which may cause image delay or signal interruption problems, affecting the smoothness of the surgery. The operation is highly subjective and inconsistent: there are significant differences in the criteria for judging the distance of sheath movement among doctors with different years of experience; Insufficient precision and high risk: Without quantitative coordinate guidance, it is difficult to accurately control the subtle movements of sensitive areas such as the apex of the prostate and the periurethral region. Errors in puncture depth may cause the steam ablation range to deviate from the target area, increasing the risk of complications such as urethral injury and postoperative urinary incontinence. Unstable image quality: Third-party endoscopes are susceptible to problems such as optical path contamination and light source attenuation, which can easily cause glare and blurring in the images, further increasing the difficulty of distance judgment.

[0025] This invention provides an image-assisted method, system, electronic device, storage medium, and program product. It integrates a camera device onto a steam ablation surgical catheter, eliminating the need for a third-party endoscope. This solves the problems of high device dependence, poor compatibility, and unstable image quality. Furthermore, this invention determines the sheath movement distance during steam ablation surgery by overlaying image coordinates onto the surgical image. When the sheath movement distance reaches the target distance, a surgical operation prompt is initiated, providing the doctor with an intuitive and quantitative reference for the sheath movement distance. This facilitates the standardization of surgical operations, avoids affecting the uniformity of surgical outcomes, and solves the problems of lack of a quantitative reference system, strong operational subjectivity and poor consistency, and insufficient accuracy and high risk. The invention will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 This is a flowchart illustrating the image-assisted method provided by the present invention, as shown below. Figure 1 As shown, the present invention provides an image-assisted method, the method comprising the following steps: Step S100: Acquire surgical images using a camera device on the hot steam ablation surgical catheter; Step S200: Based on the parameters of the camera device, generate image coordinates and superimpose the image coordinates onto the surgical image; Step S300: Based on the image coordinates, determine the sheath movement distance of the thermal steam ablation procedure; Step S400: When the sheath has moved a certain distance to the target distance, a surgical operation prompt is initiated.

[0027] The hot steam ablation surgical catheter is a thin, flexible tube that precisely delivers high-temperature steam (hot steam) into the body to the diseased tissue (usually tumors or proliferative tissue), causing coagulative necrosis of the diseased tissue, thereby achieving the therapeutic goal.

[0028] The sheath is usually made of materials with good biocompatibility and mechanical strength, such as polytetrafluoroethylene and polyurethane, to ensure its flexibility, lubricity and tissue compatibility. Its inner diameter is matched with the hot steam ablation catheter to provide a direct channel for the catheter to reach the lesion.

[0029] Optionally, in step S100, the camera device is integrated into the catheter and moves with the catheter. When the catheter is used for hot steam ablation surgery, the camera device acquires images of the surgical field in real time. It should be noted that saline solution flows out from the tip of the catheter to rinse the lens surface of the camera device to ensure that the camera device captures clear images.

[0030] Furthermore, the imaging device uses a small-sized, high-definition camera to avoid increasing the weight of the catheter and adversely affecting the doctor's operation. Specifically, the imaging device can use a medical-grade miniature camera module, including a CMOS sensor, with dimensions less than 1.5mm x 1.5mm x 2.5mm, integrated into a protective cover near the tip of the catheter. The protective cover is used to fix and protect the camera module. The protective cover is made of medical-grade fused silica material to prevent the camera module from being damaged by hot steam. It should be noted that the protective cover is transparent, allowing the camera module to capture images through it. In addition, saline solution flows out from the tip of the catheter to rinse the surface of the protective cover, ensuring clear images captured by the imaging device.

[0031] Optionally, in step S200, the image coordinates can be used as a size measurement standard for human tissue in the surgical image, providing a positional reference for the doctor. Furthermore, after the doctor completes one high-temperature steam injection, the image coordinates are superimposed on the center of the surgical image. The position of the image coordinates remains unchanged until the next high-temperature steam injection, thus providing the doctor with a stable positional reference.

[0032] Optionally, in step S300, the designated location of the sheath (e.g., the tip of the sheath) in each surgical image is repeatedly identified and its position on the image coordinates is determined, and the sheath movement distance is determined by the position difference.

[0033] Optionally, in step S400, the target distance is used to characterize the distance difference between adjacent high-temperature steam injection positions. When the sheath travels a distance equal to the target distance, it means that a high-temperature steam injection needs to be performed. The surgical operation prompt is used to prompt the doctor to perform the surgical operation.

[0034] Furthermore, the target distance can be determined based on the volume of the lesion tissue, or it can be customized based on experience; this invention does not limit this.

[0035] Furthermore, surgical procedure prompts can alert doctors to perform steam therapy through methods such as flashing graphic coordinates, sound cues, or display markers. The prompting method can be configured according to the doctor's preferences to ensure effective reminders without interfering with the surgical procedure. Flashing graphic coordinates can manifest as a bright, colored outline of the coordinate axis flashing; sound cues emit a crisp, short confirmation tone; and display markers can be text-based. Doctors can pre-select the dominant sensory cues (visual or auditory) in the settings, adjust the flashing frequency and tone, and configure different prompting schemes for different surgical stages (such as localization, ablation, and evaluation) to ensure the prompting method aligns with their individual thinking and work style.

[0036] Understandably, this invention utilizes a camera device on the surgical catheter of a thermal steam ablation procedure to acquire surgical images, eliminating the need for additional endoscopic equipment. Doctors only need to focus on the ablation device used in the thermal steam ablation procedure, reducing the number of devices they need to monitor and allowing them to concentrate their attention. Furthermore, the sheath movement distance in the thermal steam ablation procedure is determined by the image coordinates superimposed on the surgical image. When the sheath movement distance reaches the target distance, a surgical operation prompt is initiated, solving the problem that doctors' judgment of the sheath movement distance relies entirely on visual estimation and tactile feedback, making it difficult to achieve standardized operation and affecting the uniformity of surgical results.

[0037] As one embodiment, generating image coordinates based on the parameters of the camera device, and superimposing the image coordinates onto the surgical image, includes: Based on the optical and calibration parameters of the camera device, the distance parameter corresponding to each pixel of the surgical image is determined; Based on the distance parameter, rectangular coordinates are generated, and the rectangular coordinates are used as the image coordinates; The origin of the image coordinates is superimposed on the center of the surgical image, and the coordinate lines and scale information of the image coordinates are superimposed on the surgical image to achieve the superposition of the image coordinates on the surgical image.

[0038] Optionally, based on the optical parameters and calibration data of the camera equipment, the actual millimeter length corresponding to each pixel is calculated. This requires first determining the camera equipment's intrinsic parameter matrix and scene depth information. The intrinsic parameter matrix is ​​obtained through camera equipment calibration, including the focal length (…). , (in pixels) and principal point ( , Scene depth ( Z (Unit: millimeters) refers to the object distance, that is, the distance from human tissue to the optical center of the camera equipment.

[0039] The specific calculation formula is: the actual size (mm / pixel) corresponding to each pixel equals the depth. Z Divide by focal length (for x (direction) or (for y (Direction). That is, and This formula originates from the principle of similar triangles in perspective projection models, where the focal length... = f / d x , f Physical focal length d xThe physical size of a pixel (mm / pixel).

[0040] The calibration parameters provide precise intrinsic parameters and distortion coefficients to ensure calculation accuracy. In practical applications, image distortion needs to be corrected first, and then the depth Z is combined for calculation to map the pixel coordinates to the actual world size.

[0041] It is important to note that the camera device in this invention is integrated within a protective shield near the tip of the catheter. This shield is made of medical-grade fused silica. During image distortion correction, the influence of the shield on image distortion must be considered to improve the accuracy of the focal length parameters. In the composite optical system consisting of the camera and the medical-grade fused silica protective shield, the calibration of the focal length parameters requires iterative optimization based on a modified ray tracing model. The parameter optimization process for the camera device is described in detail below.

[0042] Model parameters for constructing a ray tracing model The expression is as follows: ; in, Indicates that the camera equipment is in x Equivalent focal length in direction, Indicates that the camera equipment is in y Equivalent focal length in direction, , Indicates the coordinates of the principal point. , This represents the radial distortion coefficient of the lens. , Let R represent the tangential distortion coefficient of the lens, R represent the extrinsic rotation matrix from the world coordinate system to the camera coordinate system, and t represent the translation vector from the world coordinate system to the camera coordinate system. This indicates the refractive index of fused silica. d Indicates the physical thickness of the protective shield. and This indicates the angle of inclination of the surface normal of the protective cover relative to the optical axis of the camera.

[0043] The conversion formula between the world coordinate system and the camera coordinate system is as follows: ; in, Represents the world point. This represents the coordinates of the world point in the camera coordinate system.

[0044] Assume the front surface (outer side) of the protective shield is located in the camera coordinate system. Z = Z glass The plane, pointing from the camera optical center O to point The direction of the light is .

[0045] The coordinates of the intersection point between the ray and the front surface of the protective shield can be obtained by solving the equations of the line and the plane: in, Indicates the intersection point. This indicates the coordinates of the intersection point in the camera coordinate system.

[0046] Light enters the protective shield from the air, undergoing the first refraction, at the angle of incidence. ,in, The angle of refraction is calculated based on Snell's law, where the outer normal to the protective shield surface is given by the following formula: ; in, Indicates the angle of refraction.

[0047] The direction of light propagation inside the protective shield after the first refraction The propagation distance inside the protective shield can be calculated using vector formulas. d After (thickness), the surface point is reached. The calculation formula is as follows: .

[0048] At point A second refraction occurs at the incident angle of θ. angle of departure satisfy According to the angle of departure The direction of the outgoing ray v′ can be calculated, and the backward extension of the outgoing ray intersects the normalized imaging plane of the camera (assuming no protective shield). Z The intersection of the planes (where 1 = 1) This refers to the ideal normalized coordinates after refraction by the protective shield.

[0049] Since the refraction calculation has partially altered the light path, the original distortion model center (principal point) remains the reference point. However, distortion is an inherent property of the lens. Therefore, the distortion model is applied to the normalized coordinates as follows: ; in, These are the distorted, normalized coordinates.

[0050] The distorted normalized coordinates are projected onto the pixel plane using the following formula: ; in, For the final calculated pixel coordinates .

[0051] The optimization objective of the camera's focal length parameter is to minimize the reprojection error considering refraction. The expression for the objective function is as follows: ; in, It is the first i The first calibration image j The observed pixel coordinates of each corner point It uses the aforementioned projection chain that includes refraction, based on the current parameters. θ The calculated projection pixel coordinates.

[0052] Initialize the model parameters of the ray tracing model ,in, , , , , , , , These are the initial values ​​obtained using the standard Zhang Zhengyou method without considering the protective shield. Usually fixed, d It can be obtained through measurement and can be used as a fixed value or an initial value. and Initialized to 0, R and t are calculated from each calibration plot and the initial intrinsic parameters.

[0053] During the iterative search for the objective function, the error is calculated. E For parameter vectors θ The partial derivatives of each optimizable parameter are used to determine the Jacobian matrix. J Jacobian matrix J This is used to characterize how each parameter affects the total error. The partial derivative calculation process applies the chain rule and the Jacobian matrix. J Automatic differentiation is used for analysis.

[0054] Construct and solve the normal equations , obtain parameter increment Update parameters According to the error E Whether to decrease to adjust the damping factor λ, and determine whether convergence has occurred.

[0055] In the normal equation, The approximation of the Hessian matrix in the least squares problem approximates the curvature information of the error function E(θ). I Let e ​​represent the identity matrix and e represent the residual vector. This represents the gradient vector.

[0056] The conditions for ending the iterative solution include one of the following: the parameter increment is small enough, the error decrease is small enough, the gradient is small enough, and the maximum number of iterations is reached.

[0057] Optionally, the present invention generates a rectangular coordinate system in the center of the real-time image and marks it with a scale in units of 0.5mm to achieve accurate conversion between pixel coordinates and actual human tissue dimensions.

[0058] Furthermore, the coordinate lines can be displayed with high-contrast colors to ensure clear visibility against different tissue backgrounds; the length of the scale marks is automatically adjusted as the image is scaled to ensure accurate reading at any magnification.

[0059] Optionally, superimposing the coordinate lines and scale information of the image coordinates onto the surgical image includes: Call the graphics drawing object to overlay the coordinate lines and scale information of the image coordinates onto the surgical image.

[0060] Taking QPainter as an example, QPainter is a powerful tool for 2D drawing in Qt. By using QPainter to overlay the coordinate lines and scale information of the image coordinates onto the surgical image, the efficiency of coordinate overlay can be improved.

[0061] It is understood that this invention uses digital coordinates to quantify the distance the sheath moves, providing a precise positional reference. When the sheath moves to the target distance, it automatically alerts the user, ensuring precise control of the puncture interval and significantly improving the standardization of the surgery.

[0062] As one embodiment, determining the sheath movement distance during the thermal steam ablation procedure based on the image coordinates includes: The initial position of the sheath on the image coordinates and the real-time position of the sheath on the image coordinates are obtained to determine the sheath movement distance of the thermal steam ablation procedure.

[0063] Optionally, the distance calculation uses a cumulative counting method. When the cumulative movement distance reaches the target distance (taking 5mm ± 0.3mm as an example), the doctor is reminded to perform hot steam therapy through methods such as flashing coordinate scales, sound prompts, or interface markers. It is understood that this invention provides a scheme for determining the sheath movement distance. By setting the sheath movement distance, the doctor can perform the puncture operation based on coordinate information and prompts to complete the hot steam therapy.

[0064] The image assistance system provided by the present invention is described below. The image assistance system described below can be referred to in correspondence with the image assistance method described above.

[0065] Figure 2 This is a schematic diagram of the image-assisted system provided by the present invention, as shown below. Figure 2 As shown, the present invention also provides an image-assisted system, comprising: Camera device 10 is mounted on the hot steam ablation surgical catheter to acquire surgical images; The host device 20 is used to generate image coordinates based on the parameters of the camera device and superimpose the image coordinates onto the surgical image; determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates; and initiate a surgical operation prompt when the sheath movement distance reaches the target distance.

[0066] Optionally, the camera device uses a small-sized high-definition camera to avoid increasing the weight of the catheter and adversely affecting the doctor's operation. Surgical images are transmitted to the host device via a data interface. Specifically, the camera device can use a medical-grade miniature camera module, including a CMOS sensor, with dimensions less than 1.5mm x 1.5mm x 2.5mm, integrated into a protective cover near the tip of the catheter. The protective cover is used to fix and protect the camera module. It is made of medical-grade fused silica to prevent damage to the camera module from hot steam. It should be noted that the protective cover is transparent, allowing the camera module to capture images through it.

[0067] Optionally, the image-assisted system also includes a human-computer interaction device, which is used to display processed surgical images and operating interface, receive user configuration parameters, and can also be used to display surgical operation prompts.

[0068] Figure 3 The functional architecture diagram of the image assistance system provided by the present invention is as follows: Figure 3 As shown, the image assistance system provided by the present invention includes a user interaction layer, a core function layer, a data processing layer, a hardware interface layer, and a support layer. The user interaction layer is implemented based on a human-computer interaction device, the core function layer, the data processing layer, and the support layer are implemented based on a host device, and the hardware interface layer is implemented based on a data interface.

[0069] The user interaction layer is used to display the processed surgical images and operation interface, and to receive user configuration parameters. Core functional layer: Used to implement core functions such as digital caliper coordinate generation, image overlay, and distance calculation; Data processing layer: processes raw surgical images, including acquisition, preprocessing, and transmission; Hardware interface layer: Communicates with the camera device integrated on the duct to enable device control and calibration; Support layer: Provides basic services such as multi-threaded management, memory sharing, and configuration management.

[0070] Furthermore, the main interface of the human-computer interaction system centers on the real-time image, occupying over 70% of the screen area to ensure a clear surgical field of view. The sidebar displays key information, including movement distance, cumulative count, and system status. The main interface supports image zooming, freezing, and screenshot functions, facilitating doctors' observation of details and recording of key steps. The coordinate display style is configurable, including color, thickness, and scale density, to meet individual needs. The operation interface is simple and intuitive, with key functions accessible via shortcut keys, reducing operational steps.

[0071] Understandably, this invention integrates the imaging device directly into the catheter tip, eliminating the need for third-party image observation equipment. Surgery can be completed with just one host unit, simplifying equipment configuration, reducing operating room space requirements and equipment connection complexity, and lowering equipment procurement and maintenance costs. The dedicated hardware interface design avoids compatibility issues between different devices, improving system stability and reliability, and reducing the risk of equipment malfunction during surgery. Precise control of the operating interval ensures uniform distribution of hot steam therapy, improving the consistency of treatment effects and reducing postoperative complications and recurrence rates.

[0072] As one embodiment, the host device is used to generate image coordinates based on parameters of the camera device and superimpose the image coordinates onto the surgical image; and to determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates, including: The host device is configured to: acquire the surgical image from the camera device and store it in the original image buffer of the shared buffer based on a first thread; acquire the surgical image from the original image buffer based on a second thread, generate image coordinates based on the parameters of the camera device, superimpose the image coordinates onto the surgical image to obtain a processed surgical image, and store the processed surgical image in the processed image buffer of the shared buffer; acquire the processed surgical image from the processed image buffer based on a third thread and display the processed surgical image; and acquire the processed surgical image from the second thread based on a fourth thread, and determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates. The first thread has a higher priority than the second thread, and the second thread has a higher priority than the third thread and the fourth thread.

[0073] Figure 4 This is a schematic diagram of the multi-threaded processing flow provided by the present invention, such as... Figure 4 As shown, this invention employs a multi-threaded design, which improves overall performance through parallel processing and ensures real-time requirements.

[0074] The first thread can be understood as the acquisition thread, which acquires surgical images from the camera in real time at a frame rate of 30-60fps and stores them in the raw image buffer. The first thread is set to the highest priority to ensure continuous acquisition of image data and avoid data loss.

[0075] The second thread can be understood as the processing thread: it acquires the surgical image from the original image buffer, performs preprocessing and coordinate overlay, and stores the processed surgical image into the processed image buffer. The second thread uses a medium-to-high priority to ensure timely image processing while avoiding excessive resource consumption that could affect the acquisition thread.

[0076] The third thread can be understood as the display thread: it retrieves the processed surgical image from the processed image buffer, updates the display of the human-computer interaction interface, and ensures that the doctor can observe the surgical field of view with coordinates in real time. The third thread adopts a medium priority to ensure smooth refresh of the human-computer interaction interface without affecting the core processing flow.

[0077] The fourth thread can be understood as a computation thread: it receives the processed surgical images from the processing thread, analyzes the sheath position and calculates the sheath movement distance, and feeds the results back to the processing thread to trigger surgical operation prompts. The fourth thread uses medium priority and works in parallel with the processing thread to improve overall processing efficiency.

[0078] Optionally, this invention is developed using a Qt5.14.2 C++ environment and combined with a professional image processing library to implement efficient image algorithms, ensuring cross-platform compatibility and system stability.

[0079] The first thread uses a dedicated driver to communicate with the image device, supporting the Universal Serial Bus Video Class (UVC) standard protocol to ensure stable acquisition of image data.

[0080] The second thread preprocesses the surgical images, including using a hybrid noise reduction algorithm to denoise the surgical images, effectively removing noise from the surgical environment while preserving tissue details; using an adaptive contrast enhancement algorithm to enhance the surgical images, improving the distinction between different tissue types and making the anatomical structures clearer; and performing automatic white balance adjustment on the surgical images to adapt to possible changes in lighting during the operation and maintain image color consistency.

[0081] The second thread is also used to implement automatic calibration functions, using standard models for calibration to ensure coordinate accuracy. It is also used to receive manual fine-tuning parameters input by doctors through the human-computer interaction interface to adapt to the anatomical characteristics of different patients. It is also used to implement dynamic calibration, such as automatically performing drift correction every 10 minutes to compensate for possible system errors. It is also used to save and recall calibration data to support parameter management for different catheter models.

[0082] Furthermore, each thread exchanges data through a shared buffer and uses a semaphore mechanism for synchronization, ensuring data consistency and processing efficiency. The multi-threaded architecture provided by this invention enables the system to perform complex image processing and analysis functions while ensuring real-time performance.

[0083] It is understood that this invention employs a multi-threaded architecture and optimized image processing algorithms to ensure that the overall system latency is less than 15ms and the image refresh rate reaches 60fps, meeting the real-time requirements of surgery and providing doctors with a smooth operating experience.

[0084] As an example, the host device is also configured to execute an exception handling process when an exception occurs during the operation of the first thread, the second thread, the third thread, and the fourth thread. The exception handling process includes initiating an exception alarm and recording exception information.

[0085] Figure 5 This is a flowchart illustrating the image assistance method implemented by the image assistance system provided by the present invention, as shown below. Figure 5 As shown, before the operation, the catheter of the integrated imaging device is connected to the host device, and the system is started; after the system is powered on, it performs a self-test to check the status of the hardware devices and the integrity of the software modules; it loads the user configuration parameters and calibration data; it initializes the camera and completes the parameter settings of the image sensor to ensure that the image quality meets the requirements of the surgery. The doctor adjusts the system parameters according to the patient's condition, including the coordinate display style and prompting method.

[0086] The system enters normal operating mode, with the acquisition thread acquiring real-time images at a frame rate of 60fps. Noise reduction, contrast enhancement, and white balance adjustment are performed on the surgical images to improve image clarity. Digital caliper coordinates are added to the pre-processed images. The coordinate-added images are displayed in real-time on the human-computer interaction interface to track changes in the sheath position, triggering a prompt when the sheath moves 5mm. The doctor performs the puncture operation based on the coordinate information and prompts, completing the hot steam therapy. During operation, the system monitors the status of each stage in real time. If abnormalities such as image acquisition failure, processing errors, or hardware malfunctions occur, the system immediately enters the abnormal handling process, issuing alarms through sound and visual means and recording fault information to ensure surgical safety.

[0087] After the surgery is completed, the doctor issues a termination command, the system stops image acquisition and processing, saves the image data and operation logs during the surgery, releases system resources, and completes the shutdown process.

[0088] As one embodiment, the host device is also used for: Based on the optical and calibration parameters of the camera device, the distance parameter corresponding to each pixel of the surgical image is determined; Based on the distance parameter, rectangular coordinates are generated, and the rectangular coordinates are used as the image coordinates; The origin of the image coordinates is superimposed on the center of the surgical image, and the coordinate lines and scale information of the image coordinates are superimposed on the surgical image to achieve the superposition of the image coordinates on the surgical image.

[0089] Optionally, based on the optical parameters and calibration data of the camera equipment, the actual millimeter length corresponding to each pixel is calculated. This requires first determining the camera equipment's intrinsic parameter matrix and scene depth information. The intrinsic parameter matrix is ​​obtained through camera equipment calibration, including the focal length (…). , (in pixels) and principal point ( , Scene depth ( Z (Unit: millimeters) refers to the object distance, that is, the distance from human tissue to the optical center of the camera equipment.

[0090] The specific calculation formula is: the actual size (mm / pixel) corresponding to each pixel equals the depth. Z Divide by focal length (for x (direction) or (for y (Direction). That is, and This formula originates from the principle of similar triangles in perspective projection models, where the focal length... = f / d x , f Physical focal length d x The physical size of a pixel (mm / pixel).

[0091] The calibration parameters provide precise intrinsic parameters and distortion coefficients to ensure calculation accuracy. In practical applications, image distortion needs to be corrected first, and then the depth Z is combined for calculation to map the pixel coordinates to the actual world size.

[0092] It is important to note that the camera device in this invention is integrated within a protective shield near the tip of the catheter. This shield is made of medical-grade fused silica. During image distortion correction, the influence of the shield on image distortion must be considered to improve the accuracy of the focal length parameters. In the composite optical system consisting of the camera and the medical-grade fused silica protective shield, the calibration of the focal length parameters requires iterative optimization based on a modified ray tracing model. The parameter optimization process for the camera device is described in detail below.

[0093] Model parameters for constructing a ray tracing model The expression is as follows: ; in, Indicates that the camera equipment is in x Equivalent focal length in direction, Indicates that the camera equipment is in y Equivalent focal length in direction, , Indicates the coordinates of the principal point. , This represents the radial distortion coefficient of the lens. , Let R represent the tangential distortion coefficient of the lens, R represent the extrinsic rotation matrix from the world coordinate system to the camera coordinate system, and t represent the translation vector from the world coordinate system to the camera coordinate system. This indicates the refractive index of fused silica. d Indicates the physical thickness of the protective shield. and This indicates the angle of inclination of the surface normal of the protective cover relative to the optical axis of the camera.

[0094] The conversion formula between the world coordinate system and the camera coordinate system is as follows: ; in, Represents the world point. This represents the coordinates of the world point in the camera coordinate system.

[0095] Assume the front surface (outer side) of the protective shield is located in the camera coordinate system. Z = Z glass The plane, pointing from the camera optical center O to point The direction of the light is .

[0096] The coordinates of the intersection point between the ray and the front surface of the protective shield can be obtained by solving the equations of the line and the plane: in, Indicates the intersection point. This indicates the coordinates of the intersection point in the camera coordinate system.

[0097] Light enters the protective shield from the air, undergoing the first refraction, at the angle of incidence. ,in, The angle of refraction is calculated based on Snell's law, where the outer normal to the protective shield surface is given by the following formula: ; in, Indicates the angle of refraction.

[0098] The direction of light propagation inside the protective shield after the first refraction The propagation distance inside the protective shield can be calculated using vector formulas. d After (thickness), the surface point is reached. The calculation formula is as follows: .

[0099] At point A second refraction occurs at the incident angle of θ. angle of departure satisfy According to the angle of departure The direction of the outgoing ray v′ can be calculated, and the backward extension of the outgoing ray intersects the normalized imaging plane of the camera (assuming no protective shield). Z The intersection of the planes (where 1 = 1) This refers to the ideal normalized coordinates after refraction by the protective shield.

[0100] Since the refraction calculation has partially altered the light path, the original distortion model center (principal point) remains the reference point. However, distortion is an inherent property of the lens. Therefore, the distortion model is applied to the normalized coordinates as follows: ; in, These are the distorted, normalized coordinates.

[0101] The distorted normalized coordinates are projected onto the pixel plane using the following formula: ; in, For the final calculated pixel coordinates .

[0102] The optimization objective of the camera's focal length parameter is to minimize the reprojection error considering refraction. The expression for the objective function is as follows: ; in, It is the first i The first calibration image j The observed pixel coordinates of each corner point It uses the aforementioned projection chain that includes refraction, based on the current parameters. θ The calculated projection pixel coordinates.

[0103] Initialize the model parameters of the ray tracing model ,in, , , , , , , , These are the initial values ​​obtained using the standard Zhang Zhengyou method without considering the protective shield. Usually fixed, d It can be obtained through measurement and can be used as a fixed value or an initial value. and Initialized to 0, R and t are calculated from each calibration plot and the initial intrinsic parameters.

[0104] During the iterative search for the objective function, the error is calculated. E For parameter vectors θ The partial derivatives of each optimizable parameter are used to determine the Jacobian matrix. J Jacobian matrix J This is used to characterize how each parameter affects the total error. The partial derivative calculation process applies the chain rule and the Jacobian matrix. J Automatic differentiation is used for analysis.

[0105] Construct and solve the normal equations , obtain parameter increment Update parameters According to the error E Whether to decrease to adjust the damping factor λ, and determine whether convergence has occurred.

[0106] In the normal equation, The approximation of the Hessian matrix in the least squares problem approximates the curvature information of the error function E(θ). I Let e ​​represent the identity matrix and e represent the residual vector. This represents the gradient vector.

[0107] The conditions for ending the iterative solution include one of the following: the parameter increment is small enough, the error decrease is small enough, the gradient is small enough, and the maximum number of iterations is reached.

[0108] As one embodiment, the host device is also used for: Call the graphics drawing object to overlay the coordinate lines and scale information of the image coordinates onto the surgical image.

[0109] As one embodiment, the host device is also used for: The initial position of the sheath on the image coordinates and the real-time position of the sheath on the image coordinates are obtained to determine the sheath movement distance of the thermal steam ablation procedure.

[0110] Optionally, the distance calculation adopts a cumulative counting method. When the cumulative movement distance reaches the target distance (taking 5mm±0.3mm as an example), the doctor is reminded to perform hot steam therapy through methods such as flashing coordinate scale, sound prompts, or interface markers.

[0111] The technical effects of the present invention will now be described in detail with reference to the accompanying drawings.

[0112] Figure 6 This is a schematic diagram comparing the puncture spacing error of the present invention and the traditional method, as shown in the figure. Figure 6 As shown, red represents the error data of the traditional method, and blue represents the error data of the present invention. It can be seen that when using the traditional method for puncture, the average error is 1.82mm, and the proportion of errors ≤0.5mm is only 32%; while after using the present invention, the average error is reduced to 0.21mm, and the proportion of errors ≤0.5mm reaches 99%. The accuracy improvement is significant, with the average error reduced by 88.5% and the proportion of high-precision punctures increased by 209%, effectively ensuring precise control of the puncture interval every 5mm.

[0113] This invention significantly improves the accuracy, efficiency, and standardization of prostate thermal steam ablation surgery by integrating image acquisition device design, digital coordinate quantization display, and real-time image processing technologies, while reducing surgical risks and equipment costs.

[0114] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the image-assisted method, the method including: Surgical images are acquired using a camera device mounted on a surgical catheter for thermal steam ablation. Based on the parameters of the camera device, image coordinates are generated and superimposed on the surgical image; Based on the image coordinates, the sheath movement distance of the thermal steam ablation procedure is determined; Once the sheath has traveled the target distance, a surgical procedure prompt will be initiated.

[0115] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the image-assisted methods provided by the above methods, the methods comprising: Surgical images are acquired using a camera device mounted on a surgical catheter for thermal steam ablation. Based on the parameters of the camera device, image coordinates are generated and superimposed on the surgical image; Based on the image coordinates, the sheath movement distance of the thermal steam ablation procedure is determined; Once the sheath has traveled the target distance, a surgical procedure prompt will be initiated.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image-assisted methods provided by the methods described above, the methods comprising: Surgical images are acquired using a camera device mounted on a surgical catheter for thermal steam ablation. Based on the parameters of the camera device, image coordinates are generated and superimposed on the surgical image; Based on the image coordinates, the sheath movement distance of the thermal steam ablation procedure is determined; Once the sheath has traveled the target distance, a surgical procedure prompt will be initiated.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image-assisted method, characterized in that, include: Surgical images are acquired using a camera device mounted on a surgical catheter for thermal steam ablation. Based on the parameters of the camera device, image coordinates are generated and superimposed on the surgical image; Based on the image coordinates, the sheath movement distance of the thermal steam ablation procedure is determined; Once the sheath has traveled the target distance, a surgical procedure prompt will be initiated.

2. The image-assisted method of claim 1, wherein, The step of generating image coordinates based on the parameters of the camera device, and superimposing the image coordinates onto the surgical image, includes: Based on the optical and calibration parameters of the camera device, the distance parameter corresponding to each pixel of the surgical image is determined; Based on the distance parameter, rectangular coordinates are generated, and the rectangular coordinates are used as the image coordinates; The origin of the image coordinates is superimposed on the center of the surgical image, and the coordinate lines and scale information of the image coordinates are superimposed on the surgical image to achieve the superposition of the image coordinates on the surgical image.

3. The image-assisted method of claim 2, wherein, The step of superimposing the coordinate lines and scale information of the image coordinates onto the surgical image includes: Call the graphics drawing object to overlay the coordinate lines and scale information of the image coordinates onto the surgical image.

4. The image-assisted method according to any one of claims 1 to 3, characterized in that, Determining the sheath movement distance during the steam ablation procedure based on the image coordinates includes: The initial position of the sheath on the image coordinates and the real-time position of the sheath on the image coordinates are obtained to determine the sheath movement distance of the thermal steam ablation procedure.

5. An image-assisted system, characterized in that include: A camera device, mounted on the hot steam ablation surgical catheter, is used to acquire surgical images; The host device is used to generate image coordinates based on the parameters of the camera device and superimpose the image coordinates onto the surgical image; determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates; and initiate a surgical operation prompt when the sheath movement distance reaches the target distance.

6. The image-assisted system of claim 5, wherein, The host device is used to generate image coordinates based on the parameters of the camera device and superimpose the image coordinates onto the surgical image; Based on the image coordinates, the sheath movement distance during the steam ablation procedure is determined, including: The host device is configured to: acquire the surgical image from the camera device and store it in the original image buffer of the shared buffer based on a first thread; acquire the surgical image from the original image buffer based on a second thread, generate image coordinates based on the parameters of the camera device, superimpose the image coordinates onto the surgical image to obtain a processed surgical image, and store the processed surgical image in the processed image buffer of the shared buffer; acquire the processed surgical image from the processed image buffer based on a third thread and display the processed surgical image; and acquire the processed surgical image from the second thread based on a fourth thread, and determine the sheath movement distance of the thermal steam ablation surgery based on the image coordinates. The first thread has a higher priority than the second thread, and the second thread has a higher priority than the third thread and the fourth thread.

7. The image-assisted system of claim 6, wherein, The host device is also used to execute an exception handling process when an exception occurs during the operation of the first thread, the second thread, the third thread and the fourth thread. The exception handling process includes initiating an exception alarm and recording exception information.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image-assisted method as described in any one of claims 1 to 4. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the image-assisted method as described in any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image-assisted method as described in any one of claims 1 to 4.

Images