A see-through image enhancement method, system, device and computer storage medium

Abstract

The application discloses a perspective image enhancement method, system, device and computer storage medium, relates to the technical field of surgical robots, and obtains a current perspective image; target structure model and auxiliary observation information of the current perspective image are obtained; a projection registration relationship of the target structure model to the current perspective image is established, and an enhanced candidate layer set is generated; registration confidence data is generated according to consistency information of the auxiliary observation information and a residual error of the projection registration relationship; registration conflict scores are generated according to conflict information of the auxiliary observation information and the registration confidence data; and the enhanced candidate layer set is subjected to hierarchical display gating according to the registration confidence data and the registration conflict scores. In the application, different display strategies can be adopted for different layers according to different values of the registration confidence data and the registration conflict scores, the registration quality can be converted into a control variable that can directly drive the display intensity of the layer, and misleading enhancement of the layer is avoided.

Description

Technical Field

[0001] This application relates to the field of surgical robot technology, and more specifically, to a method, system, device, and computer storage medium for enhancing fluoroscopic images. Background Technology

[0002] In procedures such as bronchoscopy and percutaneous puncture, X-ray fluoroscopy is an important visualization tool for instrument positioning. Fluoroscopy images can show the trajectory of instruments in the body in real time. However, the difference in X-ray attenuation between soft tissues and lesions is small, resulting in low contrast of lesions, airways, and some risk structures in fluoroscopy images, making them difficult to identify directly. To solve this problem, a preoperative CT (Computed Tomography) / CBCT (Cone Beam Computed Tomography) model can be registered in two dimensions with intraoperative fluoroscopy images and then fixedly superimposed. Specifically, the projection transformation is obtained based on the C-arm geometric parameters and a small number of landmarks, and then the lesion outline or planned path is overlaid on the fluoroscopy image.

[0003] However, this scheme equates a single registration with the reliable display of all layers. When local registration shifts, the precise boundaries of lesions or blood vessels may still be displayed with high intensity, resulting in misleading enhancement.

[0004] In conclusion, how to accurately enhance perspective images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a perspective image enhancement method, which can, to some extent, solve the technical problem of how to accurately enhance perspective images. This application also provides a perspective image enhancement system, an electronic device, and a computer-readable storage medium.

[0006] To achieve the above objectives, this application provides the following technical solution: A perspective image enhancement method, comprising: Get the current perspective image; Obtain auxiliary observation information of the target structure model and the current perspective image; Establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set; Based on the consistency information of the auxiliary observation information and the residual of the projection registration relationship, registration reliability data is generated; Based on the conflict information of the auxiliary observation information and the registration confidence data, a registration conflict score is generated; Based on the registration confidence data and the registration conflict score, the enhanced candidate layer set is subjected to hierarchical display gating; The auxiliary observation information includes instrument pose and / or endoscopic semantics; based on the consistency information of the auxiliary observation information and the residuals of the projection registration relationship, registration reliability data is generated, including: The consistency information between the device pose and the device image in the current perspective image is determined and normalized to obtain the normalized device consistency error. The consistency information between the endoscopic semantics and the target structural model is determined and normalized to obtain the normalized endoscopic semantics inconsistency rate. Determine the temporal variation of the current perspective image in a relatively stable state and normalize it to obtain the normalized temporal variation. The residuals of the projection registration relationship are normalized to obtain the normalized projection residuals; The negative evidence term is obtained by multiplying the normalized projection residual, the normalized instrument consistency error, the normalized endoscope semantic inconsistency rate, and the normalized time series change by their respective negative weights and then summing them. The positive evidence item is obtained by multiplying the quality score and reasonableness score of the current perspective image by their respective positive weights and then summing them. A reference confidence level is generated based on the sum of the baseline bias and the negative evidence items and the positive evidence items; The reference confidence level is mapped to a predetermined bounded range to obtain the registration confidence level data.

[0007] Preferably, a registration conflict score is generated based on the conflict information of the auxiliary observation information and the registration confidence data, including: The normalized projection residual is checked for whether it exceeds the limit, and a corresponding first indication value is generated; The normalized instrument consistency error is checked for whether it exceeds the limit, and a corresponding second indication value is generated. The normalized endoscopic semantic inconsistency rate is checked for whether it exceeds the limit, and a corresponding third indicator value is generated. Determine the distance from the end effector of the instrument to each target risk structure in the instrument pose and normalize it to obtain the normalized risk distance; Based on the normalized risk distance, a risk indication value is generated; Based on the registration confidence data, a registration indication value is generated; The first indication value, the second indication value, the third indication value, the risk indication value, and the registration indication value are weighted and truncated to obtain a registration conflict score.

[0008] Preferably, based on the registration confidence data and the registration conflict score, a hierarchical display gating system is implemented for the enhanced candidate layer set, including: For candidate layers in the enhanced candidate layer set, determine the source credibility index, structural display importance index, and risk sensitivity index of the candidate layers; A fourth index value is generated based on the product of the risk sensitivity index and the registration conflict score; A fifth indicator value is generated based on the difference between the set value and the fourth indicator value; A display intensity base value is generated based on the product of the registration confidence data, the source confidence index, the structure display importance index, and the fifth index value; The display intensity base value is truncated to a preset range to obtain the target display intensity value of the candidate layer; The display intensity of the candidate layers is controlled according to the target display intensity value.

[0009] Preferably, based on the registration confidence data and the registration conflict score, a hierarchical display gating system is implemented for the enhanced candidate layer set, including: If the registration confidence data is greater than or equal to a set high confidence threshold and the registration conflict score is less than a conflict trigger threshold, then the set of enhanced candidate layers is fully displayed. In response to the registration confidence data being greater than or equal to a set confidence threshold and less than a set high confidence threshold, the enhanced candidate layer set is then displayed in a faded manner. In response to the registration confidence data being greater than or equal to a set low confidence threshold and less than a set medium confidence threshold, the enhanced candidate layer set is displayed in a coarser manner. If the registration confidence data is less than a set low confidence threshold and the registration conflict score is greater than or equal to the conflict triggering threshold, then the enhanced candidate layer set is hidden.

[0010] Preferably, after performing hierarchical display gating on the enhanced candidate layer set, the method further includes: After the enhanced candidate layer set is displayed in a faded, coarse, or hidden manner, if the registration confidence data is greater than or equal to the set confidence threshold and the registration conflict score is less than the conflict trigger threshold within N consecutive periods, then the enhanced candidate layer set is restored layer by layer. The order of layer-by-layer restoration is as follows: instrument real image and text prompt layer, instrument axis layer, airway path layer, lesion rough area layer, lesion precise boundary and risk structure boundary layer.

[0011] Preferably, auxiliary observation information for acquiring the target structural model and the current perspective image includes: Use the frame timestamp of the current perspective image as the master clock; Select the target structural model and auxiliary observation information of the current perspective image with the smallest time difference from the master clock and not exceeding the preset synchronization window.

[0012] A perspective image enhancement system, comprising: The image acquisition module is used to acquire the current perspective image; The data acquisition module is used to acquire auxiliary observation information of the target structure model and the current perspective image; The layer generation module is used to establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set; The credibility assessment module is used to generate registration credibility data based on the consistency information of the auxiliary observation information and the residual of the projection registration relationship; The conflict assessment module is used to generate a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data; The gated display module is used to perform hierarchical display gating on the enhanced candidate layer set based on the registration confidence data and the registration conflict score; The auxiliary observation information includes instrument pose and / or endoscopic semantics; The credibility assessment module is used to: determine and normalize the consistency information between the instrument pose and the instrument image in the current fluoroscopic image to obtain a normalized instrument consistency error; determine and normalize the consistency information between the endoscope semantics and the target structural model to obtain a normalized endoscope semantic inconsistency rate; determine and normalize the temporal variation of the relatively stable state of the current fluoroscopic image to obtain a normalized temporal variation; normalize the residual of the projection registration relationship to obtain a normalized projection residual; multiply the normalized projection residual, the normalized instrument consistency error, the normalized endoscope semantic inconsistency rate, and the normalized temporal variation by their respective negative weights and sum them to obtain a negative evidence item; multiply the quality score and reasonableness score of the current fluoroscopic image by their respective positive weights and sum them to obtain a positive evidence item; generate a reference credibility based on the reference bias and the sum of the negative and positive evidence items; and map the reference credibility to a predetermined bounded range to obtain the registration credibility data.

[0013] An electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of any of the perspective image enhancement methods described above.

[0014] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the perspective image enhancement methods described above.

[0015] This application provides a perspective image enhancement method, which involves: acquiring a current perspective image; acquiring auxiliary observation information of a target structural model and the current perspective image; establishing a projection registration relationship between the target structural model and the current perspective image to generate a set of enhancement candidate layers; generating registration confidence data based on the consistency information of the auxiliary observation information and the residuals of the projection registration relationship; generating a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data; and performing layered display gating on the set of enhancement candidate layers based on the registration confidence data and the registration conflict score. In this application, after establishing the projection registration relationship, it is not directly used to enhance the display. Instead, auxiliary observation information is further acquired, and the consistency of this auxiliary observation information is used to generate registration reliability data, thereby enabling the perception of errors in the registration relationship. Subsequently, the conflict information of the auxiliary observation information and the registration reliability data are considered simultaneously to generate a registration conflict score, which can sensitively identify high-risk misleading scenarios. Finally, since the reliability data reflects whether the registration relationship is reliable, and the registration conflict score reflects whether there are multi-source contradictions and the risk level of the current scenario, if different display strategies are adopted for different layers according to different values ​​of the registration reliability data and the registration conflict score, the registration quality can be transformed from a background evaluation indicator into a control variable that can directly drive the display intensity of the layer, avoiding misleading enhancement of the layer when the registration is uncertain and improving the enhancement accuracy of the perspective image. Furthermore, during the generation of registration credibility data, increased geometric, instrumental, endoscopic, and temporal errors all imply a less reliable current projection relationship, thus entering as negative terms; improved image quality and plausibility enhance the credibility of the current results, thus entering as positive terms; normalization followed by fusion ensures that observations from different units are comparable in dimensionality, and finally, mapping prevents output from exceeding limits, thereby improving the accuracy of registration credibility data. This registration credibility data can then be used to further improve the enhancement accuracy of perspective images. The perspective image enhancement system, electronic device, and computer-readable storage medium provided in this application also solve the corresponding technical problems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1A flowchart of a perspective image enhancement method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a perspective image enhancement system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 4 This is another structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Please see Figure 1 , Figure 1 A flowchart of a perspective image enhancement method provided in an embodiment of this application.

[0020] This application provides a perspective image enhancement method that can be applied to computers and the like, and may include the following steps: Step S101: Obtain the current perspective image.

[0021] In practical applications, the current fluoroscopic image of the object can be acquired first, and the object can be flexibly determined as needed. The current fluoroscopic image can be an image frame acquired in real time by the intraoperative X-ray fluoroscopy equipment. For example, the fluoroscopic image stream can be acquired from the C-arm X-ray machine through a video capture card or network interface. Each frame of the image contains a pixel grayscale value matrix and corresponding imaging geometric parameters such as C-arm angle, source-detector distance, and pixel pitch.

[0022] Step S102: Obtain auxiliary observation information of the target structure model and the current perspective image.

[0023] In practical applications, it is also necessary to obtain the target structural model of the fluoroscopic object, as well as the auxiliary observation information of the current fluoroscopic image. The auxiliary observation information refers to dynamic observation data that can be used to verify the reliability of the registration relationship, in addition to the fluoroscopic image and the preoperative model. This data may include instrument pose and / or endoscopic semantics, etc.

[0024] The target structural model refers to the model obtained before operating on the fluoroscopic object. While fluoroscopic images can display instruments in real time, the contrast of lesions, airways, and some soft tissue risk structures is low. However, the target structural model is affected by the object's position, breathing, C-arm angle, and deformation caused by the instrument. Combining the target structural model with fluoroscopic images can provide more accurate guidance for instrument operation and path planning. The target structural model can be structured data obtained after segmenting and modeling a preoperative CT / CBCT model. It can include lesion masks, airway centerlines, blood vessel / pleural surfaces, and planned paths. For example, after reading preoperative CT / CBCT volume data, performing voxel standardization, coordinate orientation unification, and resolution resampling, the airway tree, lesions, blood vessels, pleura, bony structures, and planned paths are segmented to generate a 3D mesh, centerline, risk structure distance field, and structural importance labels.

[0025] In an exemplary embodiment, the device pose can be obtained by detecting a perspective image or by electromagnetic tracking, optical tracking, or a robot encoder. It can include the three-dimensional position coordinates of the device end in the tracking coordinate system, the device axial direction vector, a timestamp, and a tracking quality index. The tracking quality index is used to reflect the stability of the tracking signal. When the device moves out of the effective working range of the tracker or is subjected to electromagnetic interference, the tracking quality index decreases.

[0026] In an exemplary embodiment, endoscopic semantics can be obtained through endoscopic image processing; for example, a camera is set at the front end of the endoscope to acquire real-time video images of the airway during endoscopic procedures, and semantic recognition is performed on each frame of the endoscopic image. For example, the airway cavity region can be identified through a deep learning segmentation network, the location and type of airway bifurcation can be identified through a bifurcation detection algorithm, and the direction of the lumen center and the direction of travel can be estimated through optical flow or depth estimation methods. The recognition results may include bifurcation category labels, pixel coordinates of the lumen center in the image, visible direction vector, and recognition confidence, etc.

[0027] It is understandable that if the time sources of the target structural model, auxiliary observation information, and perspective image are not consistent, it will lead to inconsistencies between the layer position and the current perspective frame, resulting in misleading enhancement. To avoid this, during the acquisition of auxiliary observation information for the target structural model and the current perspective image, the frame timestamp of the current perspective image can be used as the master clock; the target structural model and auxiliary observation information with the smallest time difference from the master clock and not exceeding a preset synchronization window are selected. For example, the synchronization window can be set to 50ms to 150ms. Data exceeding the synchronization window will not be included in the current cycle fusion, and a source missing flag will be recorded. That is, the data sources, coordinate systems, time attributes, and update frequencies of the target structural model, perspective image, instrument pose, and endoscopic semantics are all different. If they are directly used for fusion calculation in the same cycle without synchronization, layer misalignment across frames and coordinates will inevitably occur. Therefore, using the timestamp of the current fluoroscopic frame as the master clock, a periodic state packet is established for each fluoroscopic frame cycle. Input buffer queues for each data source are maintained. When a new fluoroscopic frame is received, its timestamp t0 is recorded as the reference time for the current cycle. Then, the instrument pose buffer queue, endoscope semantic buffer queue, and all associated input sources are traversed. For each data source, the observation with the smallest absolute difference between its timestamp and t0 is selected. If the absolute value of this minimum time difference does not exceed a preset synchronization window, the observation is included in the periodic state packet for the current cycle, and its source identifier, timestamp, and quality flag are recorded in the state packet. If no observation with a time difference of t0 within the synchronization window exists in any of the buffer queues, the source is marked as missing in the current cycle. For sources marked as missing, data is not falsified, and expired data is not used as the current state. Instead, a source missing flag is set in the periodic state packet for subsequent missing source processing when calculating registration reliability.

[0028] Step S103: Establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set.

[0029] In practical applications, preoperative CT / CBCT models are three-dimensional volumetric data acquired before surgery, with the CT coordinate system as their spatial reference. Fluorescent images, on the other hand, are two-dimensional projections obtained under C-arm imaging geometry, with the fluoroscopic pixel coordinate system as their spatial reference. Therefore, it is necessary to map the three-dimensional structural points in the CT coordinate system to the two-dimensional pixel coordinates in the fluoroscopic pixel coordinate system. This requires establishing a projection registration relationship between the target structural model and the current fluoroscopic image, generating a set of enhancement candidate layers. The projection registration relationship refers to the geometric relationship that maps the coordinates of the target structural model to the current fluoroscopic pixel coordinates. The enhancement candidate layer set can include multiple candidate layers such as lesion contour layers, airway pathway layers, instrument axis layers, and risk structure layers. Specifically, this involves mapping the three-dimensional points in the target structural model to the pixel coordinates of the current fluoroscopic image. Assuming the homogeneous coordinates of a point in the target structural model in the CT coordinate system are... The homogeneous transformation matrix from the current periodic CT coordinate system to the C-arm coordinate system is: The current equivalent projection matrix of the C-arm is The process of projecting this point onto the perspective pixel coordinates can be represented as follows: , This allows the dynamic registration transformation to be chained with the current C-arm projection geometry, enabling the same structure to be updated to the current perspective frame as body position, respiration, and C-arm angle change; among which, This is the index of the current perspective cycle, a dimensionless integer used to bind data and states within the same cycle. It can be generated by the perspective frame trigger, and t≥1. The homogeneous coordinates of the preoperative structural point are represented by a four-dimensional column vector, which represents the projected lesion, path, or risk structural point. It can be obtained by sampling from the target structural model. The unit is the position component mm, and the last part is dimensionless. It is a four-dimensional column vector. express The homogeneous transformation from the CT coordinate system to the C-arm coordinate system within the cycle is used to transform the preoperative structure to the current C-arm geometry. It can be a 4×4 matrix, updated by registration optimization. The rotation is dimensionless, and the translation dimension can be mm. express The current C-arm equivalent projection matrix within the period is used to project the three-dimensional points of the C-arm onto the detector plane. It can be a 3×4 matrix, obtained from the C-arm intrinsic and extrinsic parameters and detector calibration. Its dimension can be mapped to homogeneous pixel coordinates. express Homogeneous perspective pixel coordinates within a period, used to preserve the projection scale term in the perspective plane, can be a three-dimensional column vector, obtained by matrix multiplication, with the dimension of homogeneous pixel quantity; , , express The three components of the homogeneous coordinates within the period are used to complete homogeneous normalization. ,Depend on The decomposition shows that the dimensions of the first two terms can change with the projection scale, while the dimension of the third term can be the scale. express The final perspective pixel coordinates of the period are used to generate the drawing position of the enhanced candidate layer. The value is taken from the effective pixel range of the detector and is obtained by normalizing the aligned sub-coordinates. The dimension can be pixel.

[0030] Understandable The change will alter the structure's position in the three-dimensional space of the C-arm. The change will alter its two-dimensional projection position on the detector; only when both are periodic... Only then will the candidate layer match the current perspective frame. When the scale factor... When the point is close to zero or the projection point is outside the detector's effective range, the point is marked as undisplayable instead of being forcibly cropped into the image.

[0031] Step S104: Generate registration reliability data based on the consistency information of auxiliary observation information and the residuals of projection registration relationship.

[0032] In practical applications, registration reliability data needs to be generated based on the consistency information of auxiliary observation information and the residuals of the projection registration relationship. Since the auxiliary observation information originates from independent data channels, its consistency with the projection registration relationship reflects the reliability of the current registration relationship. When a deviation occurs in the registration relationship, this deviation manifests as an inconsistency between the auxiliary observation information and the projection registration relationship. Therefore, by detecting consistency, the reliability of the registration relationship can be quantified. In this way, unreliable registration states can be autonomously identified, providing a decision-making basis for subsequent display control and preventing the direct output of erroneous augmentation information from the source. Registration reliability data is a quantitative indicator that comprehensively reflects the reliability of the current periodic projection registration relationship. Registration reliability data can include registration reliability values, registration reliability levels, etc.

[0033] In an exemplary embodiment, during the process of generating registration reliability data based on the consistency information of auxiliary observation information and the residual of projection registration relationship, the consistency information between the instrument pose and the instrument image in the current fluoroscopic image can be determined and normalized to obtain the normalized instrument consistency error. For example, the spatial deviation between the fluoroscopic visible instrument position and the instrument pose can be divided by a preset instrument consistency threshold to obtain the normalized instrument consistency error. The consistency information between the endoscopic semantics and the target structure model can be determined and normalized to obtain the normalized endoscopic semantic inconsistency rate. For example, the mismatch ratio between the endoscopic visual semantics and the preoperative airway model can be divided by a preset semantic threshold to obtain the normalized semantic inconsistency rate. The relatively stable state of the current fluoroscopic image can be determined. The time-series variation is normalized to obtain the normalized time-series variation; the residual of the projection registration relationship is normalized to obtain the normalized projection residual, for example, by dividing the pixel deviation of the projection marker by a preset projection error threshold; the normalized projection residual, normalized instrument consistency error, normalized endoscope semantic inconsistency rate, and normalized time-series variation are multiplied by their respective negative weights and summed to obtain the negative evidence term; the quality score and reasonableness score of the current fluoroscopic image are multiplied by their respective positive weights and summed to obtain the positive evidence term; a reference confidence level is generated based on the baseline bias and the sum of the negative and positive evidence terms; the reference confidence level is mapped to a predetermined bounded range to obtain the registration confidence level data.

[0034] Specifically, the process of generating registration reliability data can be represented as follows: .in, Indicates the current period t The registration confidence level is used to determine the display level that the enhancement result is allowed to reach. The value range is (0,1) and is dimensionless. Represents the Logistic mapping function. , is used to map comprehensive evidence to bounded confidence level, with a value range of (0,1) and dimensionless; It represents the reference bias, which can be set by calibration data or expert initial values. It is used to represent the reference confidence level under error-free conditions. It is a real number and dimensionless. Represents the normalized projection residual, derived from the projection marker points. p The pixel error is divided by the projection error threshold to reflect the geometric consistency between the preoperative structure and the fluoroscopic image. The value range is [0,1] and is dimensionless. This represents the normalized instrument consistency error, which can be determined by the position of the fluoroscopic instrument. i The value is obtained by normalizing the deviation from the tracking prediction position and is used to detect the conflict between the tracking coordinates and the perspective coordinates. The value range is [0,1] and it is dimensionless. The normalized endoscopic semantic inconsistency rate can be represented by the number of mismatched bifurcations / directions. eDivide by the effective semantic number to obtain the value, which is used to provide evidence of intracavitary anatomical consistency. The value range is [0,1] and is dimensionless. This represents the normalized time-series variation, which can be derived from the current transformation. s The change in the relatively stable state is normalized and used to suppress erroneous submissions caused by single-cycle mutations. The value range is [0,1] and is dimensionless. A perspective image quality score can be expressed by contrast, sharpness, and occlusion rate. X The value obtained is used to reflect the availability of the evidence in this period, and its range is [0,1], which is dimensionless; Indicates operation A The rationality score can be calculated based on the location of the device within the airway / safety passage and the risk distance. It is used to constrain the registration results to conform to the interventional anatomy relationship. The value range is [0,1] and is dimensionless. This represents the projection residual weight, which can be calibrated by validation cases or preset, and is used to adjust the negative contribution of projection geometric errors to confidence. Dimensionless; This represents the weight of the device consistency error, which can be calibrated by validation cases or preset, and is used to adjust for the negative contribution of device imaging and tracking prediction conflicts. Dimensionless; This represents the semantic error weight of endoscopy, which can be calibrated or preset by validation cases and used to adjust for the negative contribution of airway bifurcation or inconsistency in direction. Dimensionless; The weights representing temporal variations can be calibrated by validation cases or preset, and are used to adjust for the negative contribution of registration state mutations. Dimensionless; The weighting of fluoroscopic image quality can be calibrated or preset by validation cases and used to adjust the positive contribution of fluoroscopic evidence quality. Dimensionless; The weighting for anatomical rationality can be determined by validated cases or pre-defined, and is used to adjust the positive contribution of the rationality of safe passages and risk distances. , dimensionless.

[0035] Understandably, during the generation of registration confidence data, increased geometric, instrument, endoscopy, and temporal errors all imply that the current projection relationship is less reliable, and therefore are entered as negative terms. Improved image quality and anatomical rationality can enhance the confidence evidence of the current results, and therefore are entered as positive terms. Normalization followed by fusion ensures that observations from different units are comparable in terms of scale, and the Logistic function prevents the output from going out of bounds. Furthermore, when a source is missing, the corresponding error is set to the upper limit or its validity flag is set to unavailable, and the baseline confidence is lowered. Zero error is not used to replace missing observations. For example, when the semantic source of endoscopy is missing, the weight originally allocated to the semantic consistency term of endoscopy is proportionally added to the projection residual term and the instrument consistency term, thereby increasing the reliance on available evidence and ensuring that the confidence calculation is not raised due to the absence of a certain source.

[0036] Step S105: Generate a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data.

[0037] In practical applications, registration reliability data alone can reflect whether registration is accurate, but it cannot reflect the severity of the consequences of inaccurate registration. When there are conflicts between auxiliary observation information, it indicates that there are contradictions between different data sources. At this time, the registration relationship is in an uncertain state. In this uncertain state, if the display is further enhanced, the enhanced information may mislead the operator. For example, the instrument tracking system indicates that the instrument tip is located in a certain airway branch, and the instrument image is also visible in the fluoroscopic image in this area, but the endoscopic semantics identifies a completely different airway bifurcation sequence, indicating that there is a significant deviation in the tracking coordinates or preoperative model in the current area. In this case, although the global residual of projection registration may still be within an acceptable range, the conflict between local key sources has already constituted a high-risk state. To avoid this situation, conflict information and reliability data need to be included in the scoring system to identify high-risk states in a timely manner when registration is uncertain and multi-source observations are contradictory. This provides triggering conditions for subsequent display gating, enabling the risk of being misled to be perceived and responded to in a timely manner. In other words, it is also necessary to generate a registration conflict score based on the conflict information of auxiliary observation information and the registration confidence data. The registration conflict score is used to comprehensively reflect the degree of conflict between multi-source observations and the proximity of instruments to risk structures. Unlike the registration confidence data, the conflict score focuses on detecting situations where there are serious conflicts in some areas even if the overall confidence is acceptable.

[0038] In an exemplary embodiment, during the process of generating a registration conflict score based on conflict information from auxiliary observation information and registration reliability data, the following steps can be taken: Detecting whether the normalized projection residual exceeds the limit, generating a corresponding first indicator value; Detecting whether the normalized instrument consistency error exceeds the limit, generating a corresponding second indicator value; Detecting whether the normalized endoscope semantic inconsistency rate exceeds the limit, generating a corresponding third indicator value; Determining and normalizing the distance from the instrument tip to each target risk structure in the instrument pose, obtaining a normalized risk distance; The target risk structure can be a high-risk structure, such as at least one of blood vessels, pleura, and non-penetrating tissues; Generating a risk indicator value based on the normalized risk distance; Generating a registration indicator value based on the registration reliability data; Performing weighted calculations and truncating the first indicator value, second indicator value, third indicator value, risk indicator value, and registration indicator value to obtain the registration conflict score.

[0039] Specifically, the process of generating registration conflict scores can be represented as follows: .in, Indicates the current period t The registration conflict score is used to trigger conflict classification and high-risk layer suppression. Its value range is [0,1] and it is dimensionless. This represents a conditional indicator function, which takes the value 1 if the condition is true and 0 otherwise. It is used to convert critical error exceeding the limit into discrete conflict evidence and is dimensionless. , , The three types of normalized errors represent the projection, instrument, and endoscope conflicts, respectively, and their values ​​range from [0,1] and are dimensionless. This represents the minimum normalized risk distance, which can be obtained by dividing the distance from the device to each risk structure by its respective safety distance and taking the minimum value. A value less than 1 indicates that the device has entered at least one safety distance boundary. Dimensionless; The positive part function, max(0,x), is used to inflate the penalty only when the risk distance is insufficient. Its unit is π / 2. x Same, non-negative; This indicates the projection over-limit weight, which can be verified by risk or preset, and is used to adjust the conflict contribution of projection over-limit. Dimensionless; This indicates the weight of the device exceeding its limits, which can be determined through risk verification or pre-setting, and is used to adjust the contribution of device conflicts. Dimensionless; This indicates the weighting of endoscopy exceeding limits, which can be determined through risk verification or pre-setting, and used to adjust the contribution of semantic conflicts. Dimensionless; This represents the risk distance weight, which can be verified or preset by risk to make it more conservative when approaching risk structures. Dimensionless; This indicates a low-confidence weight, which can be verified through risk assessment or preset, and is used to convert globally low-confidence evidence into conflicting evidence. Dimensionless; This represents the upper and lower limit cutoff function, taking 0 for values ​​less than 0 and 1 for values ​​greater than 1, and is dimensionless. Here, we can let... to make Keep it within the scope of interpretability.

[0040] Understandably, in the process of generating registration conflict scores, both discrete out-of-limit evidence and continuous risk evidence can be retained simultaneously. The first three terms ensure that a significant conflict from any key source is not averaged out by other high-quality sources; the risk distance term strengthens the handling of the same registration error when it is near blood vessels or the pleura; and the low-confidence term covers the situation where multiple small errors accumulate together. The possibility of error superposition and its consequences can be incorporated into the image display gating. Furthermore, a conflict persistence term can be added to the registration conflict score generation process. For example, the proportion of registration conflict scores exceeding the conflict trigger threshold in the most recent M periods can be generated as the conflict persistence term. Weighting and truncating the first indicator value, second indicator value, third indicator value, risk indicator value, registration indicator value, and conflict persistence term yields the registration conflict score. In this way, a brief increase in error caused by single-frame noise will not immediately trigger a high-risk conflict, but the accumulation of errors over multiple periods will gradually increase the conflict score, eventually triggering high-risk handling. This avoids false triggering caused by single-frame noise and ensures that continuous anomalies can be identified in a timely manner.

[0041] Step S106: Based on the registration confidence data and registration conflict score, perform hierarchical display gating on the enhanced candidate layer set.

[0042] In practical applications, layered display gating is the process of determining the display action for each layer in the enhancement candidate layer set based on registration confidence data and registration conflict scores. Display actions can include full display, faded display, coarsened display, and hiding. Furthermore, after all enhancement candidate layers have been synthesized in the background buffer, they can be switched to the foreground for display in an atomic commit manner.

[0043] In an exemplary embodiment, during the process of gating the hierarchical display of the enhanced candidate layer set based on registration confidence data and registration conflict scores, the gating occurs in response to the registration confidence data being greater than or equal to a set high confidence threshold. And the registration conflict score is less than the conflict triggering threshold. If the registration confidence data is greater than or equal to the set confidence threshold, then the complete set of enhanced candidate layers will be displayed; If the confidence level is less than the set high confidence threshold, the enhanced candidate layer set will be dimmed; if the registration confidence level is greater than or equal to the set low confidence threshold, the enhanced candidate layer set will be dimmed. If the registration confidence score is less than the set confidence threshold, the enhanced candidate layer set will be displayed in a coarser format; if the registration confidence score is less than the set low confidence threshold and the registration conflict score is greater than or equal to the conflict triggering threshold, the enhanced candidate layer set will be hidden. Dimensionless, with a value range of 0.75–0.90; Dimensionless, with a value range of 0.55–0.75, and ; Dimensionless, with a value range of 0.30–0.55, and ; Dimensionless, with a value range of 0.50–0.80, and At that time, a conflict state is entered.

[0044] In addition, corresponding gating actions can be performed when at least one of the following anomalies is detected: for example, when the fluoroscopic frame, C-arm geometry, or target structure model is missing, an input anomaly is identified, no new enhancement overlay is generated, and only the original fluoroscopic image is displayed; when any key dynamic source, such as the target structure model or auxiliary observation information, exceeds the synchronization window, a synchronization anomaly is identified, and the relevant layers are faded or hidden; when the projection residual or instrument consistency error exceeds the limit, a registration anomaly is identified, the previous stable low-risk layer is frozen, and the precise outline of the high-risk layer is hidden; when the registration conflict score reaches the conflict threshold, or when the low confidence level is reached and the instrument is close to the risk structure, a high-risk conflict is identified, and the precise boundary of the lesion, the blood vessel and pleura layers can be hidden, and a review prompt can be displayed.

[0045] In an exemplary embodiment, during the process of hierarchical display gating of the enhanced candidate layer set, the following can be further defined for each candidate layer in the enhanced candidate layer set: source credibility index, structural display importance index, and risk sensitivity index; a fourth index value is generated based on the product of the risk sensitivity index and the registration conflict score; a fifth index value is generated based on the difference between the set value and the fourth index value; a display intensity base value is generated based on the product of the registration credibility data, source credibility index, structural display importance index, and the fifth index value; the display intensity base value is truncated to a preset range to obtain the target display intensity value of the candidate layer; and the display intensity of the candidate layer is controlled according to the target display intensity value, for example, the candidate layer is fully displayed when the target display intensity value is greater than the full display intensity threshold, and the candidate layer is hidden when the target display intensity value is less than the hiding threshold, etc. The full display intensity threshold can be in the range of 0.70–0.85, and the hiding threshold can be in the range of 0.10–0.30, etc.

[0046] Specifically, the process of generating the intensity value can be represented as follows: .in, Indicates that layer l is in the periodt The display intensity is used to map to display transparency and display action. It is dimensionless and its value ranges from [0,1]. l This represents the structural layer category index, which can be generated from classifications such as lesions, airways, instruments, blood vessels, and pleura. It is used to support different gating parameters for different structures and is a dimensionless positive integer. Indicates the reliability of the layer source, which can be generated based on the source status and time freshness of the layer. For example, if a source is marked as expired or low quality, its source reliability is reduced accordingly. This is used to prevent a single low-quality source from contaminating the corresponding layer. It is dimensionless and its value range is [0,1]. The importance of the structure is indicated by the procedure and operation stage. For example, in the airway positioning stage of bronchoscopy navigation, the importance of the airway path layer can be set to the highest, while in the operation stage near the lesion, the importance of the lesion center layer is increased. It is used to distinguish the display priority of instruments, targets and auxiliary structures. It is dimensionless and can take values ​​in the range of [0,1]. This indicates the layer's risk sensitivity, which can be preset based on the consequences of errors. High-risk structures show faster attenuation during conflicts. It is dimensionless and can range from [0,1]. Layer priority can be set according to structural importance and risk sensitivity. For example, for instrument layers containing instrument ends, axes, extensions, etc., a priority setting can be configured. Set to high For low settings, the default display method is highlighted lines / endpoint markers; for airway path layers that include the current path, next bifurcation direction, etc., you can set... Set to medium to high For the middle layer, the default display mode is set to semi-transparent path lines; for lesion layers containing lesion centers, outlines, target areas, etc., the following settings can be configured: Set to high For high-resolution images, the default display mode is set to outline / target circle; for risk structure layers containing blood vessels, pleura, restricted areas, etc., the default display mode can be set to [high resolution]. Set to high The highest setting is set to default display mode as warning color border / distance indicator; for text label layers containing credibility, distance, review prompts, etc., the following settings can be configured: Set to high Set to low, and the default display mode is text / icon.

[0047] Understandably, during the generation of display intensity, global registration confidence limits the overall displayability of the preoperative model, source confidence limits individual layers, structural importance determines priority under normal conditions, and risk sensitivity determines the decay rate when conflicts occur. For precise lesion boundaries, blood vessels, and the pleura, larger values ​​can be set. For instrument image and text verification prompts, do not use the preoperative projection intensity control or set it to a low value. Therefore, the same conflict will not simply shut down all information, but will prioritize suppressing the layers most likely to mislead operations. It can be mapped to full display, fade, coarse, or hide according to the threshold; when a high-risk conflict is reached, the high-risk layer is directly hidden, and it is not subject to the relaxation of continuous value results.

[0048] In an exemplary embodiment, if the registration performance of the perspective image is improved, the candidate layers can be restored. This involves gating the enhanced candidate layer set with layered display, then weakening, coarsening, or hiding the enhanced candidate layer set. In response to a situation where, within N consecutive cycles, the registration confidence data is greater than or equal to a set confidence threshold and the registration conflict score is less than a conflict trigger threshold, the enhanced candidate layer set is restored layer by layer. N is a positive integer, and can be 3 to 10, etc. The order of layer-by-layer restoration is: device real image and text prompt layer, device axis layer, airway path layer, lesion rough area layer, lesion precise boundary layer, and risk structure boundary layer. Furthermore, during the restoration process, if any step triggers a registration conflict again, it immediately reverts to the previous safe state, forming a fast-decreasing, slow-rising hysteresis restoration mechanism to avoid frequent layer flickering at threshold critical states.

[0049] In an exemplary embodiment, key states of the current cycle can also be recorded, including the registration transformation matrix, registration confidence data, registration conflict score, display intensity of each layer, and display actions of each layer, forming a status log for postoperative review and parameter optimization. The projection registration relationship of the current cycle is written to a stable state only when the registration confidence data is greater than or equal to a set high confidence threshold and the registration conflict score is less than a conflict trigger threshold, serving as the basis for registration initialization in the next cycle; otherwise, the previous stable state is retained to prevent single-cycle erroneous registration from contaminating the initialization baseline of subsequent cycles.

[0050] In an exemplary embodiment, to ensure the security of the enhancement results, projection geometry constraints, instrument consistency constraints, endoscopy semantic consistency constraints, anatomical path continuity constraints, and risk structure distance constraints can be introduced during the joint optimization and display output stages. It can also ensure that the actual fluoroscopic image of the instrument is never covered or replaced, and the enhancement layer is only used as an auxiliary display and superimposed on the original fluoroscopic image.

[0051] In the projection geometry constraints, the median of the reprojection error of each marker point in the projection registration relationship can be taken as a statistical measure of the projection geometry error. When this median error exceeds the preset projection error threshold, it indicates that the current registration transformation has a significant geometric deviation from the perspective observation. At this time, high-risk layers such as lesion outlines and risk structure boundaries should not be displayed in a strong prompt manner, but only in a low transparency or dashed line manner.

[0052] In the instrument consistency constraint, the spatial deviation between the instrument position detected in the perspective image and the instrument position predicted by the tracking system, as well as the angular deviation between the two in their axial directions, can be calculated separately. When the spatial deviation exceeds a preset instrument position consistency threshold, or the angular deviation exceeds a preset angular consistency threshold, it indicates that the calibration relationship between the tracking coordinate system and the perspective coordinate system may have failed, or the tracker has been affected by abnormal factors such as electromagnetic interference. This determines that the instrument source information is unreliable and reduces the credibility of the instrument-related layers and other layers that depend on the tracking information.

[0053] In endoscopic semantic consistency constraints, the airway bifurcation sequence, lumen direction, or contact state identified by the endoscope can be matched with the corresponding anatomical structures in the preoperative airway tree, and the proportion of mismatches to the total number of valid semantic recognitions can be calculated. When this proportion exceeds a preset endoscopic semantic consistency threshold, it indicates that although the current registration transformation may perform well in fluoroscopic projection geometry, it fails to meet the intraluminal anatomical constraints, and the registration result may have local drift.

[0054] In anatomical pathway continuity constraints, the instrument tip must be positioned within the tubular neighborhood of the pre-planned airway or safety passage. This means the distance from the instrument tip to the centerline of the planned pathway must not exceed the allowable deviation radius. Simultaneously, the movement distance of the instrument tip between adjacent cycles must not exceed the product of the maximum reliable movement speed determined based on physiological and mechanical constraints and the sampling time interval. When the instrument position deviates from the planned pathway or the movement speed exceeds the physiological upper limit, the current state is deemed unreasonable, triggering an abnormal branch.

[0055] In the risk structure distance constraint, the distance from the device tip to each risk structure, such as blood vessels, pleura, interlobar fissures, and non-penetrable tissues, can be calculated. This distance is then divided by the preset safety distance threshold for the corresponding risk structure to obtain the normalized risk distance. The minimum normalized distance among all risk structures is taken as the overall risk distance index of the device. When this minimum normalized distance is less than 1, it indicates that the device has entered the safety boundary of at least one risk structure, posing a risk of damage. In this case, the risk warning level is increased, and the display of low-confidence lesions is restricted.

[0056] This application provides a perspective image enhancement method, which involves: acquiring a current perspective image; acquiring auxiliary observation information of a target structural model and the current perspective image; establishing a projection registration relationship between the target structural model and the current perspective image to generate a set of enhancement candidate layers; generating registration confidence data based on the consistency information of the auxiliary observation information and the residuals of the projection registration relationship; generating a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data; and performing layered display gating on the set of enhancement candidate layers based on the registration confidence data and the registration conflict score. In this application, after establishing the projection registration relationship, it is not directly used to enhance the display. Instead, auxiliary observation information is further acquired, and the consistency of this auxiliary observation information is used to generate registration reliability data, thereby enabling the perception of errors in the registration relationship. Subsequently, the conflict information of the auxiliary observation information and the registration reliability data are considered simultaneously to generate a registration conflict score, which can sensitively identify high-risk misleading scenarios. Finally, since the reliability data reflects whether the registration relationship is reliable, and the registration conflict score reflects whether there are multi-source contradictions and the risk level of the current scenario, if different display strategies are adopted for different layers according to different values ​​of the registration reliability data and the registration conflict score, the registration quality can be transformed from a background evaluation indicator into a control variable that can directly drive the display intensity of the layer, avoiding misleading enhancement of the layer when the registration is uncertain and improving the enhancement accuracy of the perspective image.

[0057] It should be noted that the fluoroscopic image enhancement scheme provided in this application can be adapted to the application scenario. For example, when the auxiliary observation information only includes instrument pose and does not include endoscopic semantics, the source of endoscopic semantics can be marked as unavailable, and the registration confidence can be calculated by projection residual, instrument consistency, image quality, and anatomical rationality. Furthermore, when endoscopic semantics is missing, the normalized error corresponding to endoscopic semantic consistency can be set to a predetermined upper limit, such as 1, instead of 0, to ensure that no data is mistakenly considered a perfect match. This approach is applicable to percutaneous puncture scenarios without endoscopic video interfaces, thereby reducing hardware dependence while maintaining the core gating logic of prioritizing the suppression of high-risk layers.

[0058] For example, when auxiliary observation information includes both instrument pose and endoscopic semantics, in scenarios such as bronchoscopic intervention, the bifurcation type, lumen direction, and contact status identified by the endoscope can be used as additional consistency evidence. When the instrument position is normal in the fluoroscopic image, but the bifurcation sequence identified by the endoscope conflicts with the preoperative airway tree, the endoscopic semantic inconsistency rate increases, thereby increasing the registration conflict score and entering a cautious enhancement or degraded display state. This fully utilizes intraluminal visual information to cross-validate the registration relationship and improves the detection sensitivity of registration conflicts.

[0059] Finally, when the instrument pose is obtained from the fluoroscopic image rather than provided by an external tracker, the instrument pose and the instrument image in the fluoroscopic image come from the same data source, and the consistency error between them theoretically approaches zero. Including this consistency error in the registration confidence calculation would lead to an artificially high confidence level. To avoid this, when the instrument pose is detected to originate from the fluoroscopic image, the instrument consistency error can be excluded from the registration confidence calculation, and the highest achievable registration confidence level can be lowered. In this way, the display of the precise lesion boundary requires manual confirmation, further reducing hardware dependence and making it suitable for scenarios without electromagnetic or optical tracking equipment, while maintaining the core framework of multi-source cross-validation and hierarchical display gating.

[0060] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a perspective image enhancement system provided in an embodiment of this application.

[0061] This application provides a fluoroscopic image enhancement system that can be applied to fluoroscopic image enhancement during procedures such as bronchoscopy, percutaneous puncture, or ablation. It can also be integrated into C-arm X-ray machines, surgical robots, or hybrid operating room navigation platforms, and may include: Image acquisition module 101 is used to acquire the current perspective image; The data acquisition module 102 is used to acquire auxiliary observation information of the target structure model and the current perspective image; The layer generation module 103 is used to establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set; The credibility assessment module 104 is used to generate registration credibility data based on the consistency information of auxiliary observation information and the residual of projection registration relationship; The conflict assessment module 105 is used to generate a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data. The gating display module 106 is used to perform hierarchical gating display of the enhanced candidate layer set based on registration confidence data and registration conflict scores.

[0062] This application provides a fluoroscopic image enhancement system. The auxiliary observation information includes instrument pose and / or endoscopic semantics. The reliability assessment module is specifically used for: determining and normalizing the consistency information between the instrument pose and the instrument image in the current fluoroscopic image to obtain a normalized instrument consistency error; determining and normalizing the consistency information between the endoscopic semantics and the target structural model to obtain a normalized endoscopic semantic inconsistency rate; determining and normalizing the temporal variation of the relatively stable state of the current fluoroscopic image to obtain a normalized temporal variation; and performing projection registration. The residuals of the relationship are normalized to obtain normalized projection residuals. The normalized projection residuals, normalized instrument consistency error, normalized endoscope semantic inconsistency rate, and normalized time series variation are multiplied by their respective negative weights and summed to obtain negative evidence terms. The quality score and reasonableness score of the current fluoroscopic image are multiplied by their respective positive weights and summed to obtain positive evidence terms. Reference confidence is generated based on the baseline bias and the sum of the negative and positive evidence terms. The reference confidence is mapped to a predetermined bounded range to obtain registration confidence data.

[0063] This application provides a perspective image enhancement system, in which a conflict assessment module may include: detecting whether the normalized projection residual exceeds the limit and generating a corresponding first indicator value; detecting whether the normalized instrument consistency error exceeds the limit and generating a corresponding second indicator value; detecting whether the normalized endoscope semantic inconsistency rate exceeds the limit and generating a corresponding third indicator value; determining and normalizing the distance from the instrument tip to each target risk structure in the instrument pose to obtain a normalized risk distance; generating a risk indicator value based on the normalized risk distance; generating a registration indicator value based on registration confidence data; and performing weighted calculations and truncation on the first indicator value, second indicator value, third indicator value, risk indicator value, and registration indicator value to obtain a registration conflict score.

[0064] This application provides a perspective image enhancement system. The gated display module can be specifically used to: determine the source credibility index, structural display importance index, and risk sensitivity index of a candidate layer in an enhancement candidate layer set; generate a fourth index value based on the product of the risk sensitivity index and the registration conflict score; generate a fifth index value based on the difference between a set value and the fourth index value; generate a display intensity base value based on the product of the registration credibility data, source credibility index, structural display importance index, and the fifth index value; truncate the display intensity base value to a preset range to obtain the target display intensity value of the candidate layer; and control the display intensity of the candidate layer according to the target display intensity value.

[0065] This application provides a perspective image enhancement system in which a gated display module can be specifically used to: fully display the enhancement candidate layer set in response to registration confidence data being greater than or equal to a set high confidence threshold and registration conflict score being less than a conflict triggering threshold; fade the enhancement candidate layer set in response to registration confidence data being greater than or equal to a set medium confidence threshold and less than a set high confidence threshold; coarsely display the enhancement candidate layer set in response to registration confidence data being greater than or equal to a set low confidence threshold and less than a set medium confidence threshold; and hide the enhancement candidate layer set in response to registration confidence data being less than a set low confidence threshold and registration conflict score being greater than or equal to a conflict triggering threshold.

[0066] The perspective image enhancement system provided in this application embodiment may further include: The layer restoration module is used to restore the enhanced candidate layer set layer by layer after the gating display module performs layered display gating on the enhanced candidate layer set, and after the enhanced candidate layer set is displayed in a faded, coarse, or hidden manner, in response to the registration confidence data being greater than or equal to the set confidence threshold and the registration conflict score being less than the conflict trigger threshold within N consecutive cycles, the enhanced candidate layer set is restored layer by layer. The order of layer-by-layer restoration is as follows: instrument real image and text prompt layer, instrument axis layer, airway path layer, lesion rough area layer, lesion precise boundary and risk structure boundary layer.

[0067] This application provides a perspective image enhancement system, in which the data acquisition module is specifically used to: use the frame timestamp of the current perspective image as the master clock; select the target structure model and auxiliary observation information of the current perspective image with the smallest time difference from the master clock and not exceeding a preset synchronization window.

[0068] This application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the perspective image enhancement method provided in the embodiments of this application. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0069] An electronic device provided in this application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to implement the steps of the perspective image enhancement method described in any of the above embodiments.

[0070] Please see Figure 4Another electronic device provided in this application embodiment may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside. The display unit 204 may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes, but is not limited to, Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth Low Energy communication technology, and communication technology based on IEEE 802.11s.

[0071] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the perspective image enhancement method described in any of the above embodiments.

[0072] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (compact disc read-only memory), or any other form of storage media known in the art.

[0073] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the perspective image enhancement method described in any of the above embodiments.

[0074] For descriptions of relevant parts of the perspective image enhancement system, electronic device, and computer-readable storage medium provided in the embodiments of this application, please refer to the detailed descriptions of the corresponding parts in the perspective image enhancement method provided in the embodiments of this application, which will not be repeated here. Furthermore, parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0075] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A perspective image enhancement method, characterized in that, include: Get the current perspective image; Obtain auxiliary observation information of the target structure model and the current perspective image; Establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set; Based on the consistency information of the auxiliary observation information and the residual of the projection registration relationship, registration reliability data is generated; Based on the conflict information of the auxiliary observation information and the registration confidence data, a registration conflict score is generated; Based on the registration confidence data and the registration conflict score, the enhanced candidate layer set is subjected to hierarchical display gating; The auxiliary observation information includes instrument pose and / or endoscopic semantics; based on the consistency information of the auxiliary observation information and the residuals of the projection registration relationship, registration reliability data is generated, including: The consistency information between the device pose and the device image in the current perspective image is determined and normalized to obtain the normalized device consistency error. The consistency information between the endoscopic semantics and the target structural model is determined and normalized to obtain the normalized endoscopic semantics inconsistency rate. Determine the temporal variation of the current perspective image in a relatively stable state and normalize it to obtain the normalized temporal variation. The residuals of the projection registration relationship are normalized to obtain the normalized projection residuals; The negative evidence term is obtained by multiplying the normalized projection residual, the normalized instrument consistency error, the normalized endoscope semantic inconsistency rate, and the normalized time series change by their respective negative weights and then summing them. The positive evidence item is obtained by multiplying the quality score and reasonableness score of the current perspective image by their respective positive weights and then summing them. A reference confidence level is generated based on the sum of the baseline bias and the negative evidence items and the positive evidence items; The reference confidence level is mapped to a predetermined bounded range to obtain the registration confidence level data.

2. The method according to claim 1, characterized in that, Based on the conflict information of the auxiliary observation information and the registration reliability data, a registration conflict score is generated, including: The normalized projection residual is checked for whether it exceeds the limit, and a corresponding first indication value is generated; The normalized instrument consistency error is checked for whether it exceeds the limit, and a corresponding second indication value is generated. The normalized endoscopic semantic inconsistency rate is checked for whether it exceeds the limit, and a corresponding third indicator value is generated. Determine the distance from the end effector of the instrument to each target risk structure in the instrument pose and normalize it to obtain the normalized risk distance; Based on the normalized risk distance, a risk indication value is generated; Based on the registration confidence data, a registration indication value is generated; The first indication value, the second indication value, the third indication value, the risk indication value, and the registration indication value are weighted and truncated to obtain a registration conflict score.

3. The method according to claim 2, characterized in that, Based on the registration confidence data and the registration conflict score, a hierarchical display gating system is applied to the enhanced candidate layer set, including: For candidate layers in the enhanced candidate layer set, determine the source credibility index, structural display importance index, and risk sensitivity index of the candidate layers; A fourth index value is generated based on the product of the risk sensitivity index and the registration conflict score; A fifth indicator value is generated based on the difference between the set value and the fourth indicator value; A display intensity base value is generated based on the product of the registration confidence data, the source confidence index, the structure display importance index, and the fifth index value; The display intensity base value is truncated to a preset range to obtain the target display intensity value of the candidate layer; The display intensity of the candidate layers is controlled according to the target display intensity value.

4. The method according to claim 2, characterized in that, Based on the registration confidence data and the registration conflict score, a hierarchical display gating system is applied to the enhanced candidate layer set, including: If the registration confidence data is greater than or equal to a set high confidence threshold and the registration conflict score is less than a conflict trigger threshold, then the set of enhanced candidate layers is fully displayed. In response to the registration confidence data being greater than or equal to a set confidence threshold and less than a set high confidence threshold, the enhanced candidate layer set is then displayed in a faded manner. In response to the registration confidence data being greater than or equal to a set low confidence threshold and less than a set medium confidence threshold, the enhanced candidate layer set is displayed in a coarser manner. If the registration confidence data is less than a set low confidence threshold and the registration conflict score is greater than or equal to the conflict triggering threshold, then the enhanced candidate layer set is hidden.

5. The method according to claim 4, characterized in that, After performing hierarchical display gating on the enhanced candidate layer set, the method further includes: After the enhanced candidate layer set is displayed in a faded, coarse, or hidden manner, if the registration confidence data is greater than or equal to the set confidence threshold and the registration conflict score is less than the conflict trigger threshold within N consecutive periods, then the enhanced candidate layer set is restored layer by layer. The order of layer-by-layer restoration is as follows: instrument real image and text prompt layer, instrument axis layer, airway path layer, lesion rough area layer, lesion precise boundary and risk structure boundary layer.

6. The method according to claim 1, characterized in that, Obtain auxiliary observation information of the target structure model and the current perspective image, including: Use the frame timestamp of the current perspective image as the master clock; Select the target structural model and auxiliary observation information of the current perspective image with the smallest time difference from the master clock and not exceeding the preset synchronization window.

7. A perspective image enhancement system, characterized in that, include: The image acquisition module is used to acquire the current perspective image; The data acquisition module is used to acquire auxiliary observation information of the target structure model and the current perspective image; The layer generation module is used to establish the projection registration relationship between the target structure model and the current perspective image, and generate an enhanced candidate layer set; The credibility assessment module is used to generate registration credibility data based on the consistency information of the auxiliary observation information and the residual of the projection registration relationship; The conflict assessment module is used to generate a registration conflict score based on the conflict information of the auxiliary observation information and the registration confidence data; The gated display module is used to perform hierarchical display gating on the enhanced candidate layer set based on the registration confidence data and the registration conflict score; The auxiliary observation information includes instrument pose and / or endoscopic semantics; The credibility assessment module is used to: determine and normalize the consistency information between the instrument pose and the instrument image in the current fluoroscopic image to obtain a normalized instrument consistency error; determine and normalize the consistency information between the endoscope semantics and the target structural model to obtain a normalized endoscope semantic inconsistency rate; determine and normalize the temporal variation of the relatively stable state of the current fluoroscopic image to obtain a normalized temporal variation; normalize the residual of the projection registration relationship to obtain a normalized projection residual; multiply the normalized projection residual, the normalized instrument consistency error, the normalized endoscope semantic inconsistency rate, and the normalized temporal variation by their respective negative weights and sum them to obtain a negative evidence item; multiply the quality score and reasonableness score of the current fluoroscopic image by their respective positive weights and sum them to obtain a positive evidence item; generate a reference credibility based on the reference bias and the sum of the negative and positive evidence items; and map the reference credibility to a predetermined bounded range to obtain the registration credibility data.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the perspective image enhancement method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the perspective image enhancement method as described in any one of claims 1 to 6.

Images