Intraoperative risk assessment system based on multi-modal operation image fusion

Abstract

The invention relates to the technical field of medical image processing, in particular to an intraoperative risk assessment system based on multi-modal operation image fusion. The method comprises the following steps: acquiring a multi-modal operation grayscale image; determining difference images of the same type; acquiring a change area on the difference image, and performing image projection to obtain a modal change sequence; determining an abnormal coefficient according to the consistency performance of the modal change sequence; performing dynamic time warping matching on the modal change sequence, and determining a midpoint offset coefficient according to the position difference between a central point and a point matched with corresponding warping processing in the sequence; and in combination with the abnormal coefficient and the midpoint offset coefficient, determining an operation influence degree, and in combination with the operation influence degrees of all types of operation grayscale images in the sampling period, realizing risk assessment. According to the method, operation data of different modes are fused, operation scenes which are complex in operation and fine are compressed into simple multi-mode sequences for processing, the quantitative evaluation capability of risks in the operation is enhanced, and the risk evaluation accuracy and timeliness are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an intraoperative risk assessment system based on multimodal surgical image fusion. Background Art

[0002] The intraoperative risk assessment system based on multimodal surgical image fusion enhances the visualization of surgery by integrating different types of medical data such as CT, MRI, ultrasound and endoscopic images, helping surgeons make real-time assessments and decisions during surgery.

[0003] Related technologies utilize artificial intelligence and machine learning algorithms to intelligently analyze fused imaging data, identify potential surgical risks, and monitor the patient's condition in real time, providing prompt notifications to the surgical team. However, surgical procedures are complex and delicate, and artificial intelligence and machine learning algorithms, when faced with complex scenarios and multimodal data, are unable to make quick decisions or accurately analyze risks, making quantitative assessment of intraoperative risks inefficient. Summary of the Invention

[0004] In order to solve the technical problem that artificial intelligence and machine learning algorithms in related technologies cannot make quick decisions and accurately implement risk analysis, resulting in insufficient efficiency in quantitative assessment of intraoperative risks, the present invention provides an intraoperative risk assessment system based on multimodal surgical image fusion. The technical solutions adopted are as follows: The present invention proposes an intraoperative risk assessment system based on multimodal surgical image fusion, which includes: The acquisition module is used to acquire multimodal surgical grayscale images at different sampling times within the same sampling period; and to determine the difference image at the pixel point between two adjacent surgical grayscale images of the same type in the temporal sequence; The anomaly analysis module is used to obtain the change area on the difference image, use the central axis parallel to the long side of the minimum circumscribed rectangle of the change area as the projection axis, perform image projection, and obtain the modal change sequence, where the element value of the modal change sequence is determined by the grayscale value of the pixel point in the direction perpendicular to the projection axis during the projection process; based on the consistency performance of the modal change sequence in modal change sequences of the same type, the anomaly coefficient of each modal change sequence is determined; The offset analysis module is used to perform dynamic time warping matching on the modal change sequences of two adjacent surgical grayscale images of the same type in time sequence, and determine the midpoint offset coefficient based on the position difference between the center point of each modal change sequence and the corresponding regularized matching point in the sequence; The evaluation module is used to combine the abnormal coefficient and the midpoint offset coefficient to determine the operation impact of two adjacent surgical grayscale images, and to combine the operation impact of all types of surgical grayscale images within the sampling period to achieve risk assessment.

[0005] Furthermore, determining the difference image at pixel points of two temporally adjacent surgical grayscale images of the same type includes: The absolute value of the grayscale value difference between pixels at the same position on the image is used as the grayscale value of the pixel at the same position on the difference image to obtain the difference image.

[0006] Furthermore, the method for obtaining the modal change sequence includes: Determine the line connecting each pixel point in the change area with the endpoint of the projection axis, and use the tangent value of the angle between the line and the projection axis as the grayscale weight of the corresponding pixel point; Calculate the product of the grayscale value and grayscale weight of each pixel in the change area as the modal coefficient; The sum of the modal coefficients of all pixel points in the change area on the straight line perpendicular to the projection axis is used as the element value; Arrange the element values in the direction of the projection axis to obtain the modal change sequence.

[0007] Furthermore, based on the consistency of the modal change sequence among the modal change sequences of the same type, the abnormal coefficient of each modal change sequence is determined, including: Determine the element anomaly index based on the difference characteristics of the element values with the same sequence number between any modal change sequence and the same type of modal change sequence; Determine the length anomaly index based on the length of the modal change sequence; The product of the element anomaly index and the length anomaly index is calculated and normalized as the anomaly coefficient.

[0008] Furthermore, based on the difference characteristics of the element values of the same sequence number in any modal change sequence and the same type of modal change sequence, the element abnormality index is determined, including: Calculate the mean value of the elements with the same sequence number in the same type of modal change sequence as the standard element with the corresponding sequence number; The absolute value of the difference between the value of any sequence element in any modal change sequence and the standard element with the same sequence number is taken as the standard difference of the corresponding sequence number; Calculate the mean of the standard deviation of all serial numbers in the modal change sequence as the element abnormality indicator.

[0009] Furthermore, according to the length of the modal change sequence, a length anomaly index is determined, including: Calculate the mean length of the same type of modal change sequence as the standard length; The absolute value of the difference between the length of any modal change sequence and the standard length is used as the length anomaly indicator.

[0010] Furthermore, according to the position difference between the center point of each modal change sequence and the corresponding regularization matching point in the sequence, the midpoint shift coefficient is determined, including: Calculate the absolute value of the sequence number difference between the center point and the matching point in the adjacent modal change sequence to obtain the position difference index; The mean of all position difference indices corresponding to the center point of the modal change sequence is normalized and used as the midpoint shift coefficient.

[0011] Furthermore, the abnormal coefficient and the midpoint shift coefficient are combined to determine the operation influence of two adjacent surgical grayscale images, including: The product of the anomaly coefficient and the midpoint shift coefficient is calculated and normalized to obtain the operational impact.

[0012] Furthermore, the risk assessment is performed by combining the operational impact of all types of surgical grayscale images within the sampling period, including: The impact of all operations in each type of surgical grayscale image is averaged and normalized to serve as the risk indicator of the corresponding type; The average of all types of risk indicators is used as the risk coefficient; and risk assessment is performed based on the risk coefficient.

[0013] Furthermore, when the risk coefficient is greater than a preset coefficient threshold, an early warning reminder is issued.

[0014] The present invention has the following beneficial effects: In an embodiment of the present invention, multimodal surgical grayscale images are acquired at different sampling moments within the same sampling period; a difference image of two adjacent surgical grayscale images of the same type at pixel points in a temporal sequence is determined, and then, based on image projection, it is changed into a one-dimensional modal change sequence. By performing analysis through the one-dimensional modal change sequence, the complexity of two-dimensional image analysis can be effectively avoided, analysis efficiency can be improved, and analysis time can be reduced, thereby adapting to more precise and time-valuable surgical scenarios; the abnormal coefficient of each modal change sequence is determined by the overall difference in element values in the modal change sequence, and the abnormal coefficient represents the abnormal effect of the image change; then, dynamic time regularization matching of adjacent surgical grayscale images is performed to determine the midpoint offset coefficient, which represents the temporal offset effect. By analyzing the abnormal coefficient and the midpoint offset coefficient, dynamic change analysis between different surgical grayscale images is achieved, making the operation impact more accurate and reliable. In summary, the embodiments of the present invention integrate surgical data of different modalities, compressing complex and delicate surgical scenes into simple multimodal sequence processing, improving information processing efficiency, timely realizing multimodal change analysis of tissue deformation, enhancing the efficiency of quantitative assessment of intraoperative risks, and improving the timeliness of risk assessment while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a structural diagram of an intraoperative risk assessment system based on multimodal surgical image fusion provided by one embodiment of the present invention; Figure 2 A schematic diagram of a multimodal surgical grayscale image provided by one embodiment of the present invention; Figure 3 A schematic diagram of a projection axis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of an intraoperative risk assessment system based on multimodal surgical image fusion proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] The following describes in detail a specific solution of an intraoperative risk assessment system based on multimodal surgical image fusion provided by the present invention with reference to the accompanying drawings.

[0020] See also Figure 1 , which shows a structural diagram of an intraoperative risk assessment system based on multimodal surgical image fusion provided by an embodiment of the present invention, including: an acquisition module 101, an abnormality analysis module 102, a deviation analysis module 103 and an assessment module 104.

[0021] The acquisition module 101 is used to acquire multimodal surgical grayscale images at different sampling times within the same sampling period; and determine the difference image at the pixel point between two adjacent surgical grayscale images of the same type in time sequence.

[0022] The intraoperative risk assessment system, based on multimodal surgical image fusion, enhances surgical visualization by integrating different types of medical data, including CT, MRI, ultrasound, and endoscopic images, helping surgeons make real-time assessments and decisions during surgery. The system utilizes artificial intelligence and machine learning algorithms to intelligently analyze imaging data, identify potential surgical risks, and monitor patient status in real time, providing timely alerts to the surgical team.

[0023] Multimodal technology provides important information for surgical strategy formulation and intraoperative identification of responsible blood vessels in a more intuitive and accurate manner, and provides timely warnings for the impact of abnormal operations, thereby reducing surgical risks.

[0024] The effectiveness of intraoperative risk assessment depends primarily on the effectiveness of multimodal surgical image fusion, which is more about the relevance and real-time performance of multimodal information integration under the actual intraoperative tissue capture capability. The acquisition conditions of different surgical images are inconsistent, and the analysis results of different surgical images need to be reasonably correlated. Furthermore, the overall fusion analysis process needs to be highly real-time to facilitate timely early warning and feedback. Artificial intelligence and machine learning algorithms in related technologies often require more computing resources when faced with complex and delicate surgical scenarios, making it difficult to make quick decisions and resulting in poor efficiency.

[0025] Based on the above scenarios and problems, this application combines different types of surgical images for analysis, thereby achieving more accurate and timely abnormality analysis and improving risk assessment efficiency.

[0026] In an embodiment of the present invention, the multimodal surgical images may specifically include CT, MRI, ultrasound, and endoscopic images, etc., and may be adjusted according to actual needs.

[0027] Specifically: 1. Deploy DSA, intraoperative ultrasound, and sliding CT equipment, and connect them to the central processing terminal via gigabit fiber.

[0028] 2. Enable the hardware synchronization interface (such as the Linux timestamp synchronization module of Syntalos software) to ensure that the clock error of each device is less than 1ms; fix the infrared optical marker ball on the operating table as the common spatial reference system for multimodal imaging.

[0029] 3. Preoperative CT / MRI data are imported into the navigation system and aligned with the optical tracking system (error ≤ 0.5 mm).

[0030] 4. Adopt a master-slave synchronization architecture: the DSA's 30Hz video stream is used as the master clock, and ultrasound and CT align the acquisition timing through hardware trigger signals.

[0031] 5. Timestamp recording: Timestamp recording of each operation and location change for subsequent analysis.

[0032] 6. Match the collected multimodal images according to the timestamps to obtain all multimodal surgical images.

[0033] After the corresponding surgical image is acquired, image preprocessing and image grayscale processing are performed to obtain a surgical grayscale image. The image preprocessing and image grayscale processing are well known to those skilled in the art and will not be further limited or elaborated. Figure 2 , Figure 2 A schematic diagram of a multimodal surgical grayscale image provided by one embodiment of the present invention.

[0034] In the embodiment of the present invention, after determining the surgical grayscale image, the changes in the same type of surgical grayscale images at different times can be analyzed to analyze the impact of the operation. Among them, difference analysis is an effective method for analyzing image changes in time sequence.

[0035] Furthermore, in some embodiments of the present invention, the difference image at the pixel point of two adjacent surgical grayscale images of the same type in time sequence is determined, including: taking the absolute value of the grayscale value difference between the pixel points at the same position on the image as the grayscale value of the pixel point at the same position on the difference image to obtain the difference image.

[0036] The two adjacent surgical grayscale images of the same type can specifically be CT images taken at two adjacent sampling moments in time. Specifically, by calculating the grayscale value difference of a pixel at the same image location on the CT images taken at two adjacent sampling moments, if a change occurs, a grayscale change will occur between the two adjacent frames of the image, and this grayscale change can be reflected in the difference image. The specific acquisition of the difference image is well known to those skilled in the art and will not be further described.

[0037] The anomaly analysis module 102 is used to obtain the change area on the difference image, use the central axis parallel to the long side of the minimum circumscribed rectangle of the change area as the projection axis, perform image projection, and obtain a modal change sequence, wherein the element value of the modal change sequence is determined by the grayscale value of the pixel point in the direction of the vertical projection axis during the projection process; and determine the anomaly coefficient of each modal change sequence based on the consistency performance of the modal change sequence in the modal change sequence of the same type.

[0038] The effectiveness of intraoperative risk assessment mainly depends on the effect of multimodal surgical image fusion. Its effect is more about the ability of the multimodal information integration results to capture the tissue information of the actual intraoperative process. In other words, it is necessary to consider the accuracy of the multimodal imaging information on the respective tendencies of the tissue structure, and to reflect the detail of capturing the real information when there are tissue structure changes during the operation.

[0039] Since different modal types of multimodal surgical imaging have their own emphasis, such as real-time performance and accuracy, the specific manifestations are: the real-time perspective capability of digital silhouette angiography reaches 15-30 frames per second, which can dynamically track catheter movement and blood flow changes, and provide a "real-time roadmap" during interventional surgery, such as guiding the correction of the puncture needle trajectory during TIPS surgery; the high-frequency probe when acquiring ultrasound images can reach 60Hz, which can display tissue deformation in real time.

[0040] Therefore, the sampling times of different types of surgical grayscale images are inconsistent. It is necessary to analyze different types of surgical grayscale images separately within the same sampling period, obtain difference images, and perform change analysis based on the difference images to achieve abnormality analysis.

[0041] Among them, the changed area can be specifically the area composed of pixels in the difference image whose grayscale values are not 0, or a corresponding pixel grayscale threshold can be set, for example 5, and the area composed of pixels in the difference image whose grayscale values are greater than 5 is regarded as the changed area, and there is no restriction on this.

[0042] After determining the change region, in order to facilitate modal analysis of the change region and improve processing efficiency, it is necessary to process the two-dimensional change region into a one-dimensional sequence form.

[0043] In the embodiment of the present invention, the central axis parallel to the longest side of the minimum circumscribed rectangle of the change area is used as the projection axis. Figure 3 , Figure 3 A schematic diagram of a projection axis provided by an embodiment of the present invention.

[0044] Furthermore, in some embodiments of the present invention, the method for obtaining the modal change sequence includes: determining the line connecting each pixel point in the change area and the end point of the projection axis, and taking the tangent value of the angle between the line and the projection axis as the grayscale weight of the corresponding pixel point; calculating the product of the grayscale value and the grayscale weight of each pixel point in the change area as the modal coefficient; taking the sum of the modal coefficients of all pixel points in the change area on the straight line perpendicular to the projection axis as the element value; arranging the element values in the direction of the projection axis to obtain the modal change sequence.

[0045] Among them, the grayscale weight represents the influence weight of the pixel itself, which is the tangent value of the angle between the pixel and the projection axis. The modal coefficient can be determined by multiplying the grayscale weight and the grayscale value. Then, the sum of the modal coefficients of the pixels in all change areas on the straight line perpendicular to the projection axis is integrated as the element value; the element values are arranged in the direction of the projection axis to obtain the modal change sequence.

[0046] Furthermore, in some embodiments of the present invention, the anomaly coefficient of each modal change sequence is determined based on the consistency performance of the modal change sequence in the same type of modal change sequences, including: determining the element anomaly index based on the difference characteristics of the element values with the same sequence number between any modal change sequence and the same type of modal change sequence; determining the length anomaly index based on the length of the modal change sequence; calculating the product value of the element anomaly index and the length anomaly index, and normalizing it as the anomaly coefficient.

[0047] Among them, the same sequence number indicates the same sequence position in the modal change sequence. The elements in the sequence are numbered in order, 1, 2, 3..., thereby determining the sequence numbers of different sequence positions. The same sequence number can specifically mean that it is the third element or the fifth element in the same type of modal change sequence. The corresponding element value difference can represent the difference characteristics between sequences.

[0048] In an embodiment of the present invention, an element abnormality index is determined based on the difference characteristics of the element values with the same sequence number between any modal change sequence and the same type of modal change sequence, including: calculating the mean value of the element values with the same sequence number in the same type of modal change sequence as the standard element of the corresponding sequence number; taking the absolute value of the difference between the element value of any sequence number in any modal change sequence and the standard element with the same sequence number as the standard difference of the corresponding sequence number; and calculating the mean value of the standard difference under all sequence numbers of the modal change sequence as the element abnormality index.

[0049] Since the number of different types of modal change sequences is different, in order to characterize the overall situation, the mean value of the elements with the same sequence number in the same type of modal change sequence is used as the standard element of the corresponding sequence number, that is, the standard element represents the overall level under the corresponding sequence number, and the absolute value of the difference between the value of any sequence number element in any modal change sequence and the standard element with the same sequence number is the standard difference. The larger the value of the standard difference, the greater the difference effect between the element value under the corresponding sequence number and the overall situation, that is, the more abnormal it is.

[0050] Therefore, in the embodiment of the present invention, the standard deviation mean of all sequence numbers of the modal change sequence is calculated as the element abnormality indicator.

[0051] The larger the value of the element anomaly index is, the greater the difference between the modal change sequence and all other modal change sequences of the same type is, and therefore, the more abnormal the position is.

[0052] Digital subtraction angiography tends to partially visualize blood vessels, which are widely distributed and sparsely distributed. Direct image display and differentiation are not effective. However, based on the length change of the modal change sequence, abnormality analysis can be achieved more clearly and intuitively.

[0053] For different modal change sequences, the lengths of different types of modal change sequences are different. This is due to the differences in the data formats collected. The lengths of modal change sequences of the same type should be consistent. Therefore, in an embodiment of the present invention, the mean length of modal change sequences of the same type is calculated as the standard length; the absolute value of the difference between the length of any modal change sequence and the standard length is used as the length anomaly indicator.

[0054] Through length anomalies and element value anomalies, we can accurately and efficiently analyze the abnormal effects of modal change sequences at different times, calculate the product value of the element anomaly index and the length anomaly index, and normalize it as the anomaly coefficient. The anomaly coefficient represents the consistency difference between the corresponding modal change sequence itself and the whole, and the consistency difference represents the abnormal characteristics.

[0055] The offset analysis module 103 is used to perform dynamic time warping matching on the modal change sequences of two adjacent surgical grayscale images of the same type in time sequence, and determine the midpoint offset coefficient based on the position difference between the center point of each modal change sequence and the corresponding regularized matching point in the sequence.

[0056] Because intraoperative manipulation can affect the image quality of certain modalities, primarily due to factors such as physical intervention, tissue deformation, and equipment interference, for example, when surgical instruments pull or resect tissue, the spatial position of organs / lesions shifts, leading to anatomical misalignment between preoperative and real-time intraoperative images. This can be seen, for example, in brain tissue drift caused by cerebrospinal fluid loss during brain tumor surgery. Multimodal fusion systems that rely on spatial consistency (such as MRI+CT) need to further consider the ability to capture information about the effects of intraoperative manipulation on the changing representation of multimodal images, considering the structure of the modal change sequence affected by intraoperative manipulation.

[0057] For the modal change sequence of the corresponding modality type, its information capture capability when affected by intraoperative operations is actually the transition between image clarity and blur. The change of tissue structure information within the image field of view can be expressed as the corresponding structural information change of the modal change sequence. The change of the structural information can be effectively analyzed through the midpoint offset of the modal change sequence.

[0058] First, time series matching is performed based on the dynamic time warping algorithm, and the modal change sequences of two adjacent surgical grayscale images of the same type in time series are matched. The dynamic time warping algorithm can effectively deal with the "stretching and compression" effects on elements, and this "stretching and compression" effect is consistent with the displacement scenario of the spatial position of the organ / lesion. Therefore, effective matching can be achieved based on the dynamic time warping algorithm. The dynamic time warping algorithm is an algorithm well known to those skilled in the art and is not limited to this.

[0059] It should be noted that since the dynamic time warping algorithm matching is not a one-to-one match, but may produce a one-to-many or many-to-one situation, in the one-to-many situation, the matched sequence number is the average of all corresponding sequence numbers.

[0060] Furthermore, in some embodiments of the present invention, the midpoint offset coefficient is determined based on the position difference between the center point of each modal change sequence and the corresponding regularized matching point in the sequence, including: calculating the absolute value of the serial number difference between the center point and the matching point in the adjacent modal change sequence to obtain a position difference index; and normalizing the mean of all position difference indices corresponding to the center point of the modal change sequence as the midpoint offset coefficient.

[0061] In an embodiment of the present invention, the midpoint offset corresponds to the actual offset effect of the adjacent modal change sequence. The larger the value of the midpoint offset coefficient, the greater the change trend, thereby producing a larger grayscale change, causing a midpoint offset in the dynamic time warping process.

[0062] It can be understood that each modal change sequence may be adjacent to two other modal change sequences in time sequence, corresponding to two matching situations, and position difference indicators are obtained respectively. Therefore, in the embodiment of the present invention, the mean of all position difference indicators corresponding to the center point of the modal change sequence is normalized and used as the midpoint offset coefficient.

[0063] The acquisition of the midpoint offset coefficient can realize the analysis of the change amplitude during the operation through the offset effect. The larger the change amplitude, the more obvious the midpoint offset.

[0064] The evaluation module 104 is used to determine the operation impact of two adjacent surgical grayscale images by combining the abnormal coefficient and the midpoint shift coefficient, and to implement risk evaluation by combining the operation impact of all types of surgical grayscale images within the sampling period.

[0065] The above-mentioned abnormal coefficient and midpoint offset coefficient are combined to realize the operation impact analysis. The operation impact of two adjacent surgical grayscale images is determined by combining the abnormal coefficient and the midpoint offset coefficient, including: calculating the product of the abnormal coefficient and the midpoint offset coefficient, and normalizing it as the operation impact.

[0066] In this embodiment of the present invention, the anomaly coefficient represents the difference in consistency between the modality change sequence itself and all similar modality change sequences. A greater consistency difference indicates a more pronounced change between two adjacent surgical grayscale images and a greater degree of influence from the procedure. The midpoint shift coefficient, on the other hand, represents the temporal shift effect of a two-dimensional image on one dimension. A more pronounced shift indicates a greater magnitude of overall image change, and therefore a greater degree of influence from the procedure. Combined analysis is performed to determine the product of the anomaly coefficient and the midpoint shift coefficient, which is then normalized to represent the degree of manipulation influence.

[0067] The sampling period is then analyzed, and the average of the impact of all surgical grayscale images within the sampling period is calculated as the risk coefficient. Risk assessment is performed based on the risk coefficient. If the risk coefficient exceeds a preset threshold, an early warning alert is issued.

[0068] Since different types of surgical grayscale images are acquired at different frequencies during the sampling period, in an embodiment of the present invention, all operation impacts of each type of surgical grayscale image are first averaged and normalized to serve as the type risk index of the corresponding type. Then, the mean of the type risk indexes of all types is used as the risk coefficient.

[0069] On the other hand, in other embodiments, considering that the effects of different types of surgical grayscale images may actually be different, for example, the effect of DSA images is better than that of CT images, the type risk index of DSA images can be given a higher weight to achieve weighted risk coefficient analysis, and there is no restriction on this.

[0070] Among them, the preset coefficient threshold is the threshold value of the risk coefficient. In an embodiment of the present invention, the preset coefficient threshold can be specifically 0.8, for example. That is, when the risk coefficient is greater than 0.8, an early warning reminder is issued to relevant surgical personnel to facilitate timely investigation.

[0071] In an embodiment of the present invention, multimodal surgical grayscale images are acquired at different sampling moments within the same sampling period; a difference image of two adjacent surgical grayscale images of the same type at pixel points in a temporal sequence is determined, and then, based on image projection, it is changed into a one-dimensional modal change sequence. By performing analysis through the one-dimensional modal change sequence, the complexity of two-dimensional image analysis can be effectively avoided, analysis efficiency can be improved, and analysis time can be reduced, thereby adapting to more precise and time-valuable surgical scenarios; the abnormal coefficient of each modal change sequence is determined by the overall difference in element values in the modal change sequence, and the abnormal coefficient represents the abnormal effect of the image change; then, dynamic time regularization matching of adjacent surgical grayscale images is performed to determine the midpoint offset coefficient, which represents the temporal offset effect. By analyzing the abnormal coefficient and the midpoint offset coefficient, dynamic change analysis between different surgical grayscale images is achieved, making the operation impact more accurate and reliable. In summary, the embodiments of the present invention integrate surgical data of different modalities, compressing complex and delicate surgical scenes into simple multimodal sequence processing, improving information processing efficiency, timely realizing multimodal change analysis of tissue deformation, enhancing the efficiency of quantitative assessment of intraoperative risks, and improving the timeliness of risk assessment while ensuring accuracy.

[0072] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. An intraoperative risk assessment system based on multimodal surgical image fusion, characterized in that: The system comprises: The acquisition module is used to acquire multimodal surgical grayscale images at different sampling times within the same sampling period; and to determine the difference image at the pixel point between two adjacent surgical grayscale images of the same type in the temporal sequence; The anomaly analysis module is used to obtain the change area on the difference image, use the central axis parallel to the long side of the minimum circumscribed rectangle of the change area as the projection axis, perform image projection, and obtain the modal change sequence, where the element value of the modal change sequence is determined by the grayscale value of the pixel point in the direction perpendicular to the projection axis during the projection process; based on the consistency performance of the modal change sequence in modal change sequences of the same type, the anomaly coefficient of each modal change sequence is determined; The offset analysis module is used to perform dynamic time warping matching on the modal change sequences of two adjacent surgical grayscale images of the same type in time sequence, and determine the midpoint offset coefficient based on the position difference between the center point of each modal change sequence and the corresponding regularized matching point in the sequence; The evaluation module is used to combine the abnormal coefficient and the midpoint offset coefficient to determine the operation impact of two adjacent surgical grayscale images, and to combine the operation impact of all types of surgical grayscale images within the sampling period to achieve risk assessment.

2. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: The step of determining a difference image at pixel points between two adjacent surgical grayscale images of the same type in a temporal sequence includes: The absolute value of the grayscale value difference between pixels at the same position on the image is used as the grayscale value of the pixel at the same position on the difference image to obtain the difference image.

3. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: The method for obtaining the modal change sequence includes: Determine the line connecting each pixel point in the change area with the endpoint of the projection axis, and use the tangent value of the angle between the line and the projection axis as the grayscale weight of the corresponding pixel point; Calculate the product of the grayscale value and grayscale weight of each pixel in the change area as the modal coefficient; The sum of the modal coefficients of all pixel points in the change area on the straight line perpendicular to the projection axis is used as the element value; Arrange the element values in the direction of the projection axis to obtain the modal change sequence.

4. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: According to the consistency performance of the modal change sequence in the modal change sequence of the same type, the abnormal coefficient of each modal change sequence is determined, including: Determine the element anomaly index based on the difference characteristics of the element values with the same sequence number between any modal change sequence and the same type of modal change sequence; Determine the length anomaly index based on the length of the modal change sequence; The product of the element anomaly index and the length anomaly index is calculated and normalized as the anomaly coefficient.

5. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 4, characterized in that: Based on the difference characteristics of the element values of the same sequence number in any modal change sequence and the same type of modal change sequence, the element abnormality index is determined, including: Calculate the mean value of the elements with the same sequence number in the same type of modal change sequence as the standard element with the corresponding sequence number; The absolute value of the difference between the value of any sequence element in any modal change sequence and the standard element with the same sequence number is taken as the standard difference of the corresponding sequence number; Calculate the mean of the standard deviation of all serial numbers in the modal change sequence as the element abnormality indicator.

6. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 4, characterized in that: According to the length of the modal change sequence, determine the length anomaly indicators, including: Calculate the mean length of the same type of modal change sequence as the standard length; The absolute value of the difference between the length of any modal change sequence and the standard length is used as the length anomaly indicator.

7. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: The midpoint offset coefficient is determined based on the position difference between the center point of each modal change sequence and the corresponding regularization matching point in the sequence, including: Calculate the absolute value of the sequence number difference between the center point and the matching point in the adjacent modal change sequence to obtain the position difference index; The mean of all position difference indices corresponding to the center point of the modal change sequence is normalized and used as the midpoint shift coefficient.

8. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: Combining the anomaly coefficient and the midpoint shift coefficient, the operation influence of two adjacent surgical grayscale images is determined, including: The product of the anomaly coefficient and the midpoint shift coefficient is calculated and normalized to obtain the operational impact.

9. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 1, characterized in that: Risk assessment is achieved by combining the operational impact of all types of surgical grayscale images within the sampling period, including: The impact of all operations in each type of surgical grayscale image is averaged and normalized to serve as the risk indicator of the corresponding type; The average of all types of risk indicators is used as the risk coefficient; and risk assessment is performed based on the risk coefficient.

10. The intraoperative risk assessment system based on multimodal surgical image fusion according to claim 9, characterized in that: When the risk factor is greater than a preset coefficient threshold, an early warning reminder is issued.

Images