Video laryngoscope system with blood occlusion image display function and application

Abstract

The application relates to the technical field of medical devices, and discloses a video laryngoscope system with a blood occlusion image display function and an application, which comprises a hardware acquisition device and a processing device. The hardware acquisition device is provided with a dual-wavelength light source module, an image sensor and an inertial measurement unit. The system acquires dual-band images and inertial data through a data synchronous acquisition module; a three-dimensional map construction module constructs a global three-dimensional environment map when the field of view is clear; an occlusion trend solving module calculates a relative occlusion thickness field and a state matrix; a pose recursive mapping module performs pure inertial pose recursion when visual tracking fails, and generates a reference background frame by reprojecting three-dimensional map points; and a safe fusion reconstruction module performs weighted synthesis on the reference background frame and a near-infrared structure compensation image to generate a field of view reconstruction image. The application solves the problem of field of view loss caused by blood occlusion of a lens, and can maintain continuous pose calculation and reconstruction output of dissection images in an occlusion state.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, specifically to a video laryngoscope system and its application with blood-obscured image display function. Background Technology

[0002] Video laryngoscopes are routine equipment used in clinical endotracheal intubation. They primarily acquire visual images of the larynx through an image acquisition device at the front end, assisting operators in observing airway anatomy. During emergency or complex airway intubation, patients often have blood or tissue secretions in their mouth and airway. Because the laryngoscope lens needs to move within the confined airway space, it is prone to contact with blood and obstruction of the lens surface.

[0003] Existing video laryngoscope systems typically employ a single visible light imaging technology, but visible light has limited penetrating power through media such as blood. When blood obstructs the lens, the acquired field of view becomes blurred or completely lost, making it impossible to identify anatomical features in the image. In such situations, operators cannot determine the laryngoscope's current position and airway direction, usually requiring the laryngoscope to be removed, physically cleaned, and then reinserted. This process prolongs intubation time and increases the risk of establishing an artificial airway.

[0004] To address the issue of obstruction, some devices currently employ a solution of attaching airflow purging or mechanical wiping devices to the laryngoscope. These physical cleaning structures increase the size and mechanical complexity of the laryngoscope blades, and their cleaning effectiveness is limited when faced with continuous or heavy bleeding.

[0005] Meanwhile, in existing image processing and tracking technologies, when severe lens occlusion causes complete failure of visual feature tracking, the system loses its current spatial coordinate reference and lacks a continuous pose recursion mechanism. Because existing devices rely solely on a single visual input and lack physical assessment and multi-source data fusion reconstruction mechanisms for blood occlusion, once occlusion occurs, the system cannot recover the underlying structural image of the occluded area, and therefore cannot continue to provide a safe and effective visual reference for intubation operations under limited field of view conditions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a video laryngoscope system and application with blood-obscured image display function. This solves the problem that when the lens of the video laryngoscope is obscured by blood during intubation, the loss of field of view and failure of visual tracking occur. Existing devices, due to the lack of continuous pose recursion and field of view reconstruction mechanisms, cannot continue to provide effective images of the laryngeal anatomical structure and device pose reference under obscured conditions.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The first aspect of this invention provides a video laryngoscope system with blood-obscured image display function, comprising a hardware acquisition device and a processing device; the hardware acquisition device includes a dual-wavelength light source module, a global shutter CMOS image sensor, and a six-axis inertial measurement unit; the processing device includes: The data synchronization acquisition module is used to acquire dual-band image data and inertial data, perform timestamp alignment and spatial registration on the dual-band image data and the inertial data, and output image and inertial data sets. The 3D map construction module is used to combine the image and inertial data set to perform visual inertial odometry processing to obtain the physical depth of the laryngoscope pose and anatomical feature points, and generate a global 3D environment map. The occlusion trend calculation module calculates the relative occlusion thickness field based on the difference in light intensity attenuation between the dual-band image data, extracts the gravity direction vector of the image plane based on the inertial data, and generates an occlusion change state matrix by combining the time change of the relative occlusion thickness field. The pose recursion mapping module is used to perform pure inertial pose recursion when visual feature tracking fails, and to reproject the 3D map points in the global 3D environment map to the current image plane to generate a reference background frame and the corresponding effective area mask. The secure fusion and reconstruction module calculates a near-infrared structure compensation image based on the dual-band image data, constructs basic fusion weights based on the relative occlusion thickness field and the occlusion change state matrix, and performs weighted synthesis of the reference background frame and the near-infrared structure compensation image to generate a field-of-view reconstruction image.

[0009] This invention employs dual-wavelength imaging combined with inertial measurement, utilizing the differences in transmittance of different wavelengths in the blood medium to calculate the occlusion state of the field of view. The system establishes a 3D map of the laryngeal cavity using visual-inertial odometry when the field of view is clear. When visual feature tracking fails, it switches to pure kinematic integration based on inertial data for pose recursive estimation. Combining this with structural information from the near-infrared band, the system reprojects the established 3D environment map from the current viewpoint and performs image weighted fusion to output a reconstructed anatomical image of the laryngeal cavity when occlusion occurs.

[0010] Furthermore, the data synchronization acquisition module drives the dual-wavelength light source module to sequentially acquire green light reflection images and near-infrared images in adjacent trigger cycles to form a dual-band image pair as the dual-band image data. The inertial data is interpolated and aligned to the time reference of the dual-band image pair. Initial motion compensation is performed based on the inertial data between two exposures. Cross-band spatial registration is performed based on the tissue structure contour to generate the image and inertial data set.

[0011] The principle behind this technology is that it uses time-division multiplexing to acquire green light reflection images and near-infrared images, and uses inertial data interpolation to compensate for the device displacement during the time difference between the two exposures, thereby reducing motion artifacts between cross-band images and achieving spatiotemporal registration of images and inertial data.

[0012] In a preferred embodiment of the present invention, the 3D map construction module extracts anatomical feature points from the continuously acquired green light reflection images, calculates the camera baseline displacement and relative attitude between adjacent image acquisition times using a visual inertial odometry method, and performs multi-view mapping. Figure 3 The physical depth of the anatomical feature points is calculated by angularization, and the anatomical feature points are converted into three-dimensional spatial coordinates in the world coordinate system and written into the global three-dimensional environment map.

[0013] The principle behind this technology is that, when the image is not occluded, the three-dimensional spatial coordinates of the anatomical feature points on the airway surface are calculated through the parallax relationship of multiple views, and a three-dimensional environment map is constructed to provide three-dimensional data reference for field of view reconstruction under occlusion conditions.

[0014] Furthermore, the occlusion trend calculation module performs dark field correction, flat field correction, exposure time normalization, sensor gain normalization, and light source output power correction on the green light reflection image and the near-infrared image, and calculates the logarithmic ratio of the pixel intensity of the near-infrared image to the pixel intensity of the green light reflection image to generate the relative occlusion thickness field; the occlusion trend calculation module performs motion compensation on the relative occlusion thickness field of the previous moment according to the relative pose between the image acquisition moments, and calculates the difference between the current relative occlusion thickness field and the motion-compensated relative occlusion thickness field of the previous moment to obtain the time change of the relative occlusion thickness.

[0015] The principle of this technology is that after image correction of the dual-band image, the difference in the absorption coefficient of the dual-band light in the medium is used to convert the attenuation relationship of pixel intensity into a relative occlusion thickness field, and the time change of occlusion thickness is calculated by the difference of the relative occlusion thickness field of consecutive frames.

[0016] Preferably, the inertial data includes three-axis acceleration data and three-axis angular velocity data. The occlusion trend calculation module estimates the gravitational acceleration vector based on the three-axis acceleration data and the three-axis angular velocity data, transforms the gravitational acceleration vector to the camera coordinate system, and extracts the direction component parallel to the image plane to obtain the image plane gravity direction vector. The occlusion trend calculation module calculates the optical flow of the occluded object between continuous relative occlusion thickness fields, and calculates the consistency parameter between the occluded object flow direction and the gravity direction based on the angle relationship between the optical flow direction of the occluded object and the image plane gravity direction vector, and writes it into the occlusion change state matrix.

[0017] The principle of this technology is to combine the optical flow field of the gravity direction component and the relative occlusion thickness field to calculate the consistency parameter between the flow direction of the occluder and the gravity direction, which reflects the changing characteristics of the occlusion state. The consistency parameter is one of the state variables in the occlusion change state matrix and is used to determine the subsequent fusion control parameters and fusion weights.

[0018] Furthermore, the inertial data includes angular velocity data and linear acceleration data. The pose recursive mapping module reads the laryngoscope pose, motion velocity, gyroscope zero bias, and accelerometer zero bias at the last valid moment before the visual feature tracking fails as the initial state for pure inertial pose recursion. It performs zero bias correction on the angular velocity data and the linear acceleration data, updates the camera attitude based on the zero bias-corrected angular velocity data, obtains the rotation matrix of the current moment relative to the last valid moment based on the updated camera attitude, removes the gravitational acceleration component from the zero bias-corrected linear acceleration data to obtain a gravity-free motion acceleration sequence, updates the camera velocity and position based on the gravity-free motion acceleration sequence and the motion velocity at the last valid moment, and obtains the three-dimensional translation vector of the current moment relative to the last valid moment.

[0019] The principle of this technology is that when visual feature tracking fails, the state parameters at the last valid moment are used to perform zero-bias correction and time integration of inertial data, and the continuous calculation of spatial pose is maintained through pure inertial pose recursion.

[0020] Preferably, the pose recursive mapping module reprojects the 3D map points in the global 3D environment map onto the current image plane based on the rotation matrix, the 3D translation vector, and the camera intrinsic parameters. It then performs visibility judgment based on the projection depth to remove map points with non-positive depth values ​​in the camera coordinate system. The module performs gridded resampling and local interpolation on the effective projection points to generate the reference background frame. Finally, it marks areas without effective map point coverage as invalid areas to generate the effective area mask.

[0021] The principle of this technology is that, based on the current pose obtained by calculation, the 3D map points are directly projected onto the current 2D image plane, invalid depth points are removed and a reference background frame is generated, and the area not effectively covered by map points is written into the mask as an invalid area.

[0022] Furthermore, the secure fusion reconstruction module performs exponential compensation calculations based on the relative occlusion thickness field, the corrected near-infrared image pixel intensity, and the offline calibrated green light image domain compensation coefficient to generate the near-infrared structure compensation image, and performs amplitude limiting processing on the pixel values ​​of the near-infrared structure compensation image; when the relative occlusion thickness exceeds the calibration range or the near-infrared image pixel intensity is overexposed or underexposed, the secure fusion reconstruction module marks the near-infrared structure compensation result as invalid.

[0023] The principle behind this technology is to use near-infrared images to perform structural compensation on occluded areas. By using amplitude limiting and invalid state marking, it avoids generating abnormal compensated images under conditions such as relative occlusion thickness exceeding the limit or pixel overexposure / underexposure.

[0024] In a preferred embodiment of the present invention, the secure fusion reconstruction module determines fusion control parameters based on the occlusion change state matrix and converts the fusion control parameters into basic fusion weights. It constructs an inertial recursive confidence time decay function based on the zero-bias instability and random walk variance obtained from the calibration of the six-axis inertial measurement unit. The secure fusion reconstruction module uses the inertial recursive confidence time decay function, the effective region mask of the reference background frame, and the current pose confidence to constrain the basic fusion weights to obtain the final fusion weights. Based on the final fusion weights, it performs pixel-level weighted synthesis of the reference background frame and the near-infrared structure-compensated image.

[0025] The principle of this technology is to construct a time decay function based on the calibration parameters of the inertial measurement unit, constrain the basic fusion weights by combining the mask and pose confidence, and use the final fusion weights to realize the weighted synthesis calculation of the reference background frame and the near-infrared structure compensation image.

[0026] The second aspect of the present invention provides an application of a video laryngoscope system with blood-obscured image display function, the system being used to provide dual-band image acquisition of the laryngeal cavity and reconstruction display of the blood-obscured field of view during laryngoscope intubation.

[0027] This invention provides a video laryngoscope system and its application with blood-obscured image display function. It has the following beneficial effects: 1. This invention introduces a dual-wavelength light source module to acquire green light reflection images and near-infrared images. It calculates the relative occlusion thickness field by utilizing the difference in absorption rates of the two wavelengths in the blood medium, and combines this with the near-infrared image to generate a near-infrared structure compensation image. This allows for the quantitative calculation of the physical occlusion state of the field of view and the extraction of tissue structure information in the occluded area when the lens is obstructed by blood, solving the problem of complete loss of field of view information in single visible light imaging under blood occlusion.

[0028] 2. This invention employs a pose calculation mechanism combining visual inertial odometry and pure inertial recursion. During periods of clear vision, a global 3D environment map of the throat cavity is constructed. When visual feature tracking fails due to occlusion, the system performs pure inertial pose recursion based on the last valid state before visual failure and data from the six-axis inertial measurement unit, and reprojects the 3D map points onto the current image plane. This ensures the device's continuous spatial pose calculation capability under sudden occlusion conditions, providing a spatial reference and background reference for subsequent field-of-view reconstruction.

[0029] 3. This invention provides an image fusion mechanism based on multi-source state constraints. The system combines an occlusion change state matrix, an inertial recursive confidence time decay function, and an effective region mask for reprojection to dynamically calculate image fusion weights. Coupled with an invalidation marking mechanism for thickness exceeding limits or pixel overexposure / underexposure states, the system can perform pixel-level weighted synthesis of the reference background frame and near-infrared structure compensation image based on the reliability of the current data, preventing erroneous reconstructed images due to sensor data drift under long-term occlusion conditions. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 This is a schematic diagram of the data synchronization acquisition process of the present invention; Figure 4 This is a schematic diagram of the three-dimensional map construction process of the present invention; Figure 5 This is a schematic diagram of the occlusion trend calculation process of the present invention; Figure 6 This is a schematic diagram of the pose recursive mapping process of the present invention; Figure 7 This is a schematic diagram of the secure fusion and reconstruction process of the present invention; Figure 8 This is a 3D data diagram representing the spatial relationship logic of 3D map construction and pure inertial pose derivation in this invention. Figure 9 This is a two-dimensional line graph comparing the changes in time confidence data and the visual reconstruction effect after encountering fluid obstruction, according to the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 The video laryngoscope system provided in this embodiment includes a hardware acquisition device and a processing device that is communicatively connected to the hardware acquisition device.

[0033] The hardware acquisition device is located on the laryngoscope body and is used to acquire dual-band image data of the laryngeal cavity and laryngoscope motion data during intubation. The hardware acquisition device includes a dual-wavelength light source module, a global shutter CMOS image sensor, and a six-axis inertial measurement unit.

[0034] A dual-wavelength light source module is used to alternately project illumination light of different wavelengths into the laryngeal cavity. A global shutter CMOS image sensor is used to receive reflected light and scattered light generated by tissue surfaces and obstructing fluids, forming images of the corresponding wavelengths. Thus, the dual-wavelength light source module and the global shutter CMOS image sensor work together to acquire dual-band image data.

[0035] The dual-wavelength light source module outputs green light with a wavelength of 540nm and near-infrared light with a wavelength of 850nm. The global shutter CMOS image sensor has spectral response in both the 540nm and 850nm bands, and its optical path does not contain an infrared cutoff filter that would block the 850nm band.

[0036] The six-axis inertial measurement unit is rigidly fixed inside the laryngoscope and is used to acquire three-axis acceleration data and three-axis angular velocity data of the laryngoscope.

[0037] The processing equipment receives and processes image data and inertial data output from the hardware acquisition device. The processing equipment includes a data synchronization acquisition module, a 3D map construction module, an occlusion trend calculation module, a pose recursive mapping module, and a safe fusion reconstruction module.

[0038] The data synchronization acquisition module is connected to the hardware acquisition device to receive image data output by the global shutter CMOS image sensor and inertial data output by the six-axis inertial measurement unit.

[0039] The data synchronization acquisition module timestamps and aligns image and inertial data according to a unified system clock; it pairs adjacent green light reflection and near-infrared images in time; it performs initial motion compensation based on the inertial data within the acquisition interval between two images, and completes cross-band spatial registration using tissue edges or tissue structure contours as registration features. The data synchronization acquisition module outputs a time-aligned image-inertial data stream.

[0040] The 3D map construction module is connected to the data synchronization acquisition module. In an unobstructed field of view, the 3D map construction module extracts anatomical feature points from the image and performs visual inertial odometry processing in conjunction with inertial data to obtain the laryngoscope pose and the physical depth of the anatomical feature points.

[0041] The 3D map construction module generates and updates a global 3D environment map based on the physical depth and corresponding pixel coordinates of anatomical feature points. Each 3D map point in the global 3D environment map is associated with at least its 3D spatial coordinates, corresponding green light image intensity attribute, source keyframe identifier, and map point confidence level.

[0042] The occlusion trend calculation module is connected to the data synchronization acquisition module. When occlusion occurs in the field of view, the occlusion trend calculation module calculates the relative occlusion thickness field based on the difference in light intensity attenuation between the registered green light reflection image and the near-infrared image.

[0043] The occlusion trend calculation module also estimates the gravitational acceleration vector based on the three-axis acceleration data and the three-axis angular velocity data, and converts the gravitational acceleration vector into the gravity direction vector of the image plane.

[0044] The occlusion trend calculation module performs motion compensation and temporal difference on the relative occlusion thickness field at consecutive time points, and calculates the optical flow of the occluded object between consecutive relative occlusion thickness fields. Based on the temporal change of the relative occlusion thickness, the occlusion trend calculation module determines whether the local region is in an occlusion deepening state, an occlusion dissipation state, or an occlusion stable state, and determines the flow direction attribute of the occluded object based on the degree of consistency between the optical flow direction and the gravity direction vector of the image plane.

[0045] The occlusion trend calculation module outputs an occlusion change state matrix based on this. The occlusion change state matrix includes at least the time change of relative occlusion thickness, the amplitude of occlusion flow, and the consistency parameter between the occlusion flow direction and the gravity direction of the image plane.

[0046] The pose recursive mapping module is connected to both the data synchronization acquisition module and the 3D map construction module. In the occlusion state where visual feature tracking fails, the pose recursive mapping module reads the laryngoscope pose, motion velocity, and inertial sensor zero bias at the last valid moment before the visual feature tracking fails, and uses this as the initial state for pure inertial pose recursion.

[0047] The pose recursive mapping module performs pure inertial pose recursion based on the zero-bias-corrected angular velocity data and gravity-de-acceleration data to obtain the rotation matrix and three-dimensional translation vector of the current moment relative to the last valid moment.

[0048] The pose recursive mapping module reprojects 3D map points carrying image intensity attributes from the global 3D environment map onto the current image plane based on the rotation matrix, 3D translation vector, and camera intrinsic parameters.

[0049] During the reprojection process, the pose recursive mapping module performs visibility assessment based on the projection depth, discarding invisible map points; it performs mesh resampling on valid projection points; it interpolates small gaps between valid projection points; and it marks areas not effectively covered by the map as invalid regions. The pose recursive mapping module thus generates a reference background frame and a corresponding valid region mask to compensate for depth-related projection misalignment.

[0050] The secure fusion and reconstruction module is connected to the data synchronization acquisition module, the occlusion trend calculation module, and the pose recursive mapping module, respectively.

[0051] The data synchronization acquisition module outputs the registered green light reflection image and near-infrared image to the secure fusion reconstruction module; the occlusion trend calculation module outputs the relative occlusion thickness field, occlusion change state matrix and occlusion area validity marker to the secure fusion reconstruction module; the pose recursive mapping module outputs the reference background frame and its effective area mask to the secure fusion reconstruction module.

[0052] The secure fusion and reconstruction module determines the basic fusion weights of the reference background frame based on the relative occlusion thickness field, and adjusts the basic fusion weights based on the occlusion deepening, occlusion dissipation, and occlusion flow state represented by the occlusion change state matrix.

[0053] The safety fusion and reconstruction module also constructs an inertial recursion confidence time decay function based on the zero-bias instability and random walk variance obtained from the calibration of the six-axis inertial measurement unit. The inertial recursion confidence time decay function is used to characterize the relationship between the reliability of the pure inertial pose recursion result and the decrease in recursion time after the failure of visual feature tracking.

[0054] The secure fusion reconstruction module uses the inertial recursive confidence time decay function, the effective region mask of the reference background frame, and the current pose confidence to constrain the basic fusion weights. Based on the constrained fusion weights, it performs pixel-level weighted synthesis of the reference background frame and the near-infrared structure compensation image to generate a field-of-view reconstruction image.

[0055] When the reference background frame does not have effective image coverage, or the current pose confidence is lower than a preset safety threshold, the safe fusion reconstruction module sets the fusion weight of the reference background frame for the corresponding region to zero. The safe fusion reconstruction module then outputs the generated field-of-view reconstruction image to an external display terminal.

[0056] See attached document Figure 2 This embodiment also provides a video laryngoscope application method with blood-obscured image display function, the method including steps S1 to S5.

[0057] Step S1: Data synchronous collection.

[0058] The processing equipment outputs a timing trigger signal based on a unified system clock, driving the dual-wavelength light source module to emit light in a time-division multiplexed manner according to a preset timing sequence, and sequentially acquiring green light reflection images and near-infrared images of the same field of view within adjacent trigger cycles. A green light reflection image and a near-infrared image acquired adjacently are paired into a dual-band image pair.

[0059] The processing device synchronously receives the three-axis acceleration data and three-axis angular velocity data output by the six-axis inertial measurement unit, and assigns system timestamps to the image data and inertial data respectively.

[0060] The processing device uses the midpoint of the image exposure time window as the image time reference and aligns the inertial data to the acquisition time of the corresponding image through timestamp matching and interpolation. For green light reflection images and near-infrared images that make up the same dual-band image pair, the processing device performs initial motion compensation based on the inertial data between the two exposures and performs cross-band spatial registration using tissue edges or tissue structure contours as registration features, thereby generating time-aligned image and inertial data sets.

[0061] Step S2: 3D map construction.

[0062] During the laryngoscope insertion phase, when the field of view is not largely obstructed by fluid, the processing device extracts and tracks static anatomical feature points from continuous green light reflection images.

[0063] The processing device combines inertial data and feature observations from continuous images, employing a visual inertial odometry method to calculate the camera baseline displacement and relative attitude between adjacent image acquisition times. Once visual inertial initialization is complete, and the camera baseline displacement and feature point disparity meet preset conditions, the processing device uses multi-view... Figure 3 Keratinization is used to calculate the physical depth corresponding to each anatomical feature point.

[0064] The processing device converts the anatomical feature points into three-dimensional spatial coordinates based on the two-dimensional pixel coordinates, physical depth, and camera intrinsic parameters, and then transforms the three-dimensional spatial coordinates to a unified world coordinate system to construct a global three-dimensional environment map.

[0065] The processing device updates the global 3D environment map with the input of continuous data frames. For a 3D map point written into the global 3D environment map, the processing device associates and stores its 3D spatial coordinates, corresponding green light image intensity attributes, source keyframe identifier, number of observations, and map point confidence level, so that it can be used for subsequent reprojection to generate a reference background frame with image content.

[0066] Step S3: Occlusion change calculation.

[0067] When blood or other fluids obstruct the field of view, the processing device normalizes the exposure time, sensor gain, and light source output power of the spatiotemporally registered green light reflection image and near-infrared image, and calculates the logarithmic ratio of the dual-band corrected light intensity of the corresponding pixel by referring to the Lambert-Beer optical attenuation relationship, thereby generating a relative occlusion thickness field.

[0068] The relative occlusion thickness field is used to represent the relative occlusion degree of each pixel region. When the system has established a calibration relationship between the logarithmic ratio of dual-band corrected light intensity and fluid thickness in advance, the processing device can also convert the relative occlusion thickness into a thickness estimate with physical dimensions within the applicable range of the calibration relationship.

[0069] The processing device estimates the gravitational acceleration vector based on the three-axis acceleration data and the three-axis angular velocity data. Through the external parameter rotation relationship between the six-axis inertial measurement unit and the image sensor, the gravitational acceleration vector is transformed into the camera coordinate system, and its direction component parallel to the image plane is extracted to obtain the gravity direction vector of the image plane.

[0070] The processing device performs motion compensation on the relative occlusion thickness field of the previous moment based on the relative pose between consecutive image acquisition moments, and calculates the time difference between the current relative occlusion thickness field and the motion-compensated relative occlusion thickness field of the previous moment to obtain the time change of the occlusion degree.

[0071] The processing device also calculates the optical flow of the occluder between the continuous relative occlusion thickness fields, and calculates the consistency parameters between the occluder flow direction and the gravity direction based on the angle between the optical flow direction and the gravity direction vector of the image plane.

[0072] The processing device uses the temporal change in relative occlusion thickness as the primary basis for determining whether occlusion is deepening or dissipating, and uses the flow amplitude of the occluding object and its consistency parameter with the gravity direction as the basis for correcting the flow direction attribute and the confidence level, generating an occlusion change state matrix. The occlusion change state matrix includes at least the temporal change in relative occlusion thickness, the flow amplitude of the occluding object, and the consistency parameter between the flow direction of the occluding object and the gravity direction of the image plane.

[0073] Step S4, pose recursive mapping.

[0074] When the field of view is severely obstructed, causing visual feature tracking in the visual inertial odometry to fail, the processing device determines the last valid image frame before the visual feature tracking fails and its corresponding time.

[0075] The processing device reads the camera position, camera attitude, motion speed, gyroscope bias, accelerometer bias, and state covariance output by the visual inertial odometry at the last valid moment, and uses the above parameters as the initial state for pure inertial pose recursion.

[0076] The processing device performs zero-bias correction on the angular velocity and linear acceleration data, and removes the gravitational acceleration component from the corrected linear acceleration data. The processing device updates the camera attitude based on the corrected angular velocity data, and updates the camera velocity and position based on the degravity-free motion acceleration data and the motion velocity at the last valid moment, thereby obtaining the rotation matrix, three-dimensional translation vector, and corresponding state covariance of the current moment relative to the last valid moment.

[0077] The processing device reprojects the 3D map points in the global 3D environment map and their associated image intensity attributes onto the current image plane based on the rotation matrix, 3D translation vector, and camera intrinsic parameters.

[0078] During the reprojection process, the processing device makes a visibility judgment based on the projection depth, removes map points with non-positive depth values ​​in the camera coordinate system, and retains map points with smaller depths and map point confidence that meet the requirements when multiple map points are projected onto the same pixel area.

[0079] The processing device performs mesh resampling on the effective projection points and local interpolation on small gaps between the effective projection points. For areas where there is no effective point coverage or where the gap range exceeds a preset threshold, the processing device does not perform extrapolation filling but marks them as invalid areas. This generates a reference background frame and a corresponding effective area mask for compensating for depth-related projection misalignment.

[0080] Step S5, Secure Fusion Reconstruction.

[0081] The processing equipment calculates a near-infrared structure-compensated image for semi-transparent occluded areas using scattering return structural information from near-infrared images and a pre-calibrated dual-band response relationship. The near-infrared structure-compensated image enhances the visibility of tissue structures within the occluded area; it does not represent the restoration of the true visible light color of the occluded area.

[0082] The processing device determines the basic fusion weights of the reference background frame based on the relative occlusion thickness field, and adjusts the basic fusion weights based on the occlusion deepening, occlusion dissipation and occlusion flow state represented by the occlusion change state matrix.

[0083] Based on the zero-bias instability and random walk variance obtained from the calibration of the six-axis inertial measurement unit, the processing equipment constructs an inertial recursion confidence time decay function that decreases with the increase of the pure inertial pose recursion time. The processing equipment also determines the current pose confidence based on the state covariance obtained during the pure inertial pose recursion process.

[0084] The processing device uses the inertial recursive confidence time decay function, the current pose confidence, and the effective region mask of the reference background frame to constrain the basic fusion weights and obtain the final fusion weights.

[0085] The processing device performs pixel-level weighted synthesis of the reference background frame and the near-infrared structure compensation image according to the final fusion weight, generates a field-of-view reconstruction image, and outputs the field-of-view reconstruction image to an external display terminal.

[0086] When the reference background frame does not have effective image coverage, the current pose confidence is lower than the preset safety threshold, or the pure inertial pose recursion duration exceeds the preset maximum duration, the processing device sets the reference background frame fusion weight of the corresponding area to zero and outputs a near-infrared structure-compensated image or the original near-infrared image, while simultaneously outputting a low confidence prompt message to the external display terminal.

[0087] See attached document Figure 3 The processing device executes step S1 through the data synchronization acquisition module. Step S1 includes sub-step S11 and sub-step S12.

[0088] Sub-step S11: Time-division multiplexing acquisition of dual-band images.

[0089] In a structure employing a single image sensor, the data synchronization acquisition module controls the dual-wavelength light source module and the global shutter CMOS image sensor to operate according to a preset timing sequence via hardware triggering, so as to acquire green light reflection images and near-infrared images respectively, and reduce spectral aliasing between the two illumination bands.

[0090] The dual-wavelength light source module includes a green light-emitting diode with a main emission peak wavelength of 540nm and a near-infrared light-emitting diode with a main emission peak wavelength of 850nm.

[0091] Among them, green light near 540nm corresponds to the higher absorption band of hemoglobin in blood; compared with green light at 540nm, near-infrared light near 850nm has lower absorption attenuation in blood, and the resulting near-infrared image can retain more tissue scattering return structural information.

[0092] The global shutter CMOS image sensor employs a monochrome image sensor with spectral response to both the 540nm and 850nm bands. The lens optical path of the global shutter CMOS image sensor incorporates a dual-bandpass filter that allows both the 540nm and 850nm bands to pass through, but does not include an infrared cutoff filter that blocks the 850nm band.

[0093] The data synchronization acquisition module controls the microcontroller to output timed trigger pulses according to a fixed trigger cycle. These timed trigger pulses control the two LEDs to light up alternately and trigger the exposure of the global shutter CMOS image sensor. When it is necessary to adjust the irradiance of the LEDs, the microcontroller can separately adjust the drive duty cycle of the corresponding LED via a pulse width modulation signal.

[0094] During odd-numbered trigger cycles, the data synchronization acquisition module drives the green LED to light up and simultaneously triggers the global shutter CMOS image sensor to expose, thereby acquiring a green light reflection image. .

[0095] During even-numbered trigger cycles, the data synchronization acquisition module drives the near-infrared LED to illuminate and simultaneously triggers the global shutter CMOS image sensor to expose, thereby acquiring a near-infrared image. .

[0096] in, This represents the pixel coordinates in the image sensor coordinate system. This indicates the system time corresponding to the midpoint of the exposure time window of the corresponding image frame.

[0097] The global shutter CMOS image sensor integrates signals for each pixel within essentially the same exposure time window, which helps reduce rolling shutter distortion caused by laryngoscope movement.

[0098] Sub-step S12: Time alignment of image data and inertial data, and registration of dual-band images.

[0099] In one implementation, the total acquisition frame rate of the global shutter CMOS image sensor is set to 30Hz to 60Hz. Since green light reflection images and near-infrared images are acquired alternately with adjacent trigger cycles, the effective acquisition frame rate for each band is half the total acquisition frame rate, i.e., 15Hz to 30Hz. The sampling frequency of the six-axis inertial measurement unit is set to 200Hz to 1000Hz.

[0100] The data synchronization acquisition module reads the three-axis angular velocity vector output by the six-axis inertial measurement unit via a serial peripheral interface bus. and the three-axis acceleration vector .

[0101] When the six-axis inertial measurement unit generates a data ready interrupt, the data synchronization acquisition module uses the system clock of the processing device to record the system timestamp of the current inertial data. If the timestamp is recorded after the serial data reading is completed, the data synchronization acquisition module will correct the timestamp according to the pre-calibrated fixed communication delay.

[0102] Because image data and inertial data have different sampling frequencies, their sampling times are usually inconsistent. The data synchronization acquisition module maintains an inertial data buffer queue arranged in timestamp order in memory, and aligns the inertial data to the time reference of the corresponding image frame according to the exposure midpoint time of the image frame.

[0103] For the time of collection The image frame is retrieved, and the data synchronization acquisition module searches for a matching frame in the inertial data buffer queue. ≤ ≤ Preceding inertial sampling time and subsequent inertial sampling time .

[0104] When the time range of the inertial data buffer queue has not yet covered the image acquisition time. If the preceding inertial sampling data and the following inertial sampling data are not obtained simultaneously, the data synchronization acquisition module will temporarily store the current image frame in the waiting queue until the following inertial sampling data is obtained.

[0105] If the preset waiting time has elapsed and a surround is still not obtained... If there are two inertial sampling points, the data synchronization acquisition module discards the current image frame. At the most recent inertial sampling moment and... When the time difference between the two is less than the preset maximum time difference, the most recent inertial sampling data can also be used as approximate time alignment data.

[0106] When adjacent inertial sampling data are successfully acquired, the data synchronization acquisition module assumes that the angular velocity and linear acceleration change approximately linearly within the adjacent inertial sampling interval, and calculates the time-aligned inertial data corresponding to the midpoint of the image exposure through linear interpolation.

[0107] Aligned angular velocity vector Calculate according to the following formula: ; Aligned linear acceleration vector Calculate according to the following formula: ; In the formula, and They represent Time and The three-axis angular velocity vectors collected at all times; and They represent Time and The three-axis acceleration vectors collected at all times; This indicates the system time corresponding to the midpoint of the exposure time window of the corresponding image frame; Indicates the time of subsequent inertial sampling; Indicates the preceding inertial sampling time; This represents the aligned linear acceleration vector; This represents the aligned angular velocity vector.

[0108] Through the above timestamp matching and interpolation calculation, the data synchronization acquisition module obtains the time-aligned angular velocity vector and time-aligned linear acceleration vector corresponding to the exposure midpoint time of each image frame.

[0109] For a green light reflection image and a near-infrared image acquired adjacent to each other, the data synchronization acquisition module pairs the two images to form a dual-band image pair, and uses the exposure midpoint of the green light reflection image as the unified time reference for the dual-band image pair.

[0110] The data synchronization acquisition module calculates the relative rotation of the laryngoscope based on the angular velocity data within the acquisition interval of the two images, and estimates the relative translation by combining the linear acceleration data and the motion state at the previous moment, so as to perform initial motion compensation on the near-infrared image.

[0111] After initial motion compensation is completed, the data synchronization acquisition module uses the tissue edges or tissue structure contours in the two images as registration features, calculates the residual registration transformation through mutual information matching or cross-band edge matching, and resamples the near-infrared image to the pixel coordinate system where the green light reflection image is located.

[0112] When the registration error of a dual-band image pair exceeds the preset registration error threshold, or when the effective tissue structure area that can participate in registration is lower than the preset proportion, the data synchronization acquisition module marks the dual-band image pair as invalid data and prohibits it from entering the subsequent relative occlusion thickness field calculation.

[0113] After completing time alignment and spatial registration, the data synchronization acquisition module generates image and inertial data sets. Each image and inertial data set includes at least the registered green light reflection image, near-infrared image, corresponding time-aligned angular velocity vector, time-aligned line acceleration vector, unified timestamp, and data validity marker, and outputs the image and inertial data sets to the subsequent processing module.

[0114] See attached document Figure 4 The processing device executes step S2 through the 3D map construction module. Step S2 includes sub-steps S21 and S22.

[0115] Before executing step S2, the processing device determines whether the current field of view meets the conditions for constructing a three-dimensional map based on the local contrast and the number of effective feature points of the green light reflection image.

[0116] Specifically, the processing device divides the green light reflection image into multiple local image regions, calculates the grayscale contrast of each local image region, and counts the proportion of local image regions with grayscale contrast higher than a preset contrast threshold. When the proportion of local image regions is higher than a preset proportion threshold, and the number of effective feature points that can be stably tracked in the current image is not lower than a preset feature number threshold, the processing device determines that the current field of view is not largely obscured by the fluid and allows the execution of step S2.

[0117] When the above conditions are not met, the processing device will pause the addition of new 3D map points, but will retain the already constructed global 3D environment map.

[0118] Sub-step S21: Extraction, tracking, and physical depth calculation of anatomical feature points.

[0119] The 3D map building module extracts anatomical feature points from continuously acquired green light reflection images and tracks the pixel positions of these anatomical feature points in subsequent image frames.

[0120] In this embodiment, anatomical feature points refer to image feature points located at tissue edges, areas of texture change, or areas with significant local grayscale changes, and that meet the motion consistency requirements in consecutive image frames. For feature points that do not conform to the camera motion model due to tissue deformation, fluid flow, specular reflection, or other local changes, the 3D map construction module marks them as dynamic or anomalous feature points and does not use them for 3D map construction.

[0121] The 3D map construction module first performs lens distortion correction on the green light reflection image, and then uses a corner feature detection method on the current green light reflection image. Candidate anatomical feature points are extracted. Subsequently, the 3D map construction module uses optical flow tracing to determine the corresponding pixel positions of each candidate anatomical feature point in subsequent green light reflection images.

[0122] The 3D map construction module eliminates incorrectly matched feature points based on the feature matching error between consecutive image frames, the forward and backward consistency error of optical flow, and the reprojection error corresponding to visual inertial pose, and determines the feature points that pass the consistency check as valid feature points.

[0123] When the number of valid feature points in the current image frame is lower than the preset feature number threshold, the 3D map construction module re-executes feature point extraction in the image area where no valid feature points have been distributed, in order to supplement new candidate anatomical feature points.

[0124] The preset feature number threshold is determined based on the image resolution, field of view, and processing capability of the processing device. For example, the preset feature number threshold can be set to 50 to 150.

[0125] When the number of valid feature points within a consecutive preset number of frames is lower than the preset minimum threshold, the processing device pauses the 3D map update and outputs a prompt message to the external display terminal to adjust the laryngoscope observation angle.

[0126] After completing feature point tracking, the 3D map building module combines image feature observations and time-aligned inertial data to perform visual inertial odometry processing.

[0127] During the visual inertial odometry (VIO) initialization phase, the processing device determines the initial scale, gravity direction, initial velocity, gyroscope zero bias, and accelerometer zero bias based on feature observations in continuous images and the angular velocity and linear acceleration output by the six-axis inertial measurement unit. After initialization, the VIO outputs the camera position, camera attitude, and state covariance corresponding to each image acquisition time.

[0128] The 3D map construction module determines the translation vector and relative rotation matrix between adjacent image frames based on the camera position and camera orientation between adjacent image acquisition times.

[0129] When the translation between adjacent image frames is lower than the preset baseline threshold, the pixel disparity of the same effective feature point is lower than the preset disparity threshold, or the state covariance of the visual inertial pose is higher than the preset threshold, the 3D map construction module will not perform triangulation calculation on the corresponding feature points.

[0130] When the camera translation, feature point disparity, and pose confidence all meet preset conditions, the 3D map construction module, based on the pixel coordinates of the same valid feature point in at least two image frames and the corresponding camera pose, performs multi-view mapping. Figure 3 Angular calculation is used to determine the physical depth of the feature point in the current camera coordinate system.

[0131] Since visual inertial odometry can provide camera translation information with physical scale, the feature point depth obtained by triangulation calculation has corresponding physical dimensions.

[0132] Sub-step S22: 3D map point generation and global 3D environment map update.

[0133] The 3D map construction module converts the effective feature points into 3D map points in the current camera coordinate system based on the physical depth of the feature points, the corresponding 2D pixel coordinates, and the camera intrinsic parameters obtained in sub-step S21.

[0134] For physical depth is The homogeneous pixel coordinates after lens distortion correction are: The feature points, whose two-dimensional pixel coordinates and three-dimensional spatial coordinates in the current camera coordinate system. The following relationship exists between them: ; Therefore, three-dimensional spatial coordinates Calculate according to the following formula: ; In the formula, This represents the calculated physical depth of the feature point; This represents the homogeneous pixel coordinate column vector after lens distortion correction, and its form is: ; This represents a column vector of three-dimensional spatial coordinates of the feature point in the current camera coordinate system, in the form of: ; This represents the camera intrinsic parameter matrix corresponding to the global shutter CMOS image sensor; Represents the camera intrinsic parameter matrix The inverse matrix.

[0135] Camera intrinsic matrix It includes at least the lateral focal length, longitudinal focal length, and principal point coordinates. Camera intrinsic parameters and lens distortion parameters are calibrated offline using a calibration board before the device leaves the factory and stored in non-volatile memory readable by the processing device.

[0136] The 3D map construction module performs validity checks on the calculated 3D map points. Validity checks include at least positive depth checks, preset depth range checks, triangulation viewpoint checks, and reprojection error checks.

[0137] When the depth value of a 3D map point in the current camera coordinate system is less than or equal to zero, exceeds the preset throat depth range, the triangulation viewpoint is less than the preset angle threshold, or the reprojection error is greater than the preset error threshold, the 3D map construction module will not write the 3D map point into the global 3D environment map.

[0138] For example, the preset laryngeal cavity depth range can be set from 10mm to 150mm. The depth range can be adjusted according to the focal length of the laryngoscope, the working distance, and the actual application.

[0139] For 3D map points that pass the validity check, the 3D map building module transforms them from the current camera coordinate system to the world coordinate system with the laryngoscope insertion start position as the origin, based on the camera global pose output by the visual inertial odometry, and writes them into the global 3D environment map.

[0140] Each 3D map point in the global 3D environment map is associated with at least the following stored attributes: 3D spatial coordinates in the world coordinate system, corresponding green light image intensity attribute, source keyframe identifier, number of observations, and map point confidence. The map point confidence is determined based on at least one of the following: feature tracking duration frames, triangulation viewpoint, reprojection error, and pose state covariance.

[0141] When an existing 3D map point is observed again in subsequent image frames, the 3D map construction module updates the image intensity attribute, observation count, and map point confidence of that point based on the new feature observations. For duplicate map points that are spatially close and have consistent feature descriptions, the 3D map construction module merges them; for map points that have not been effectively observed for multiple consecutive frames, whose reprojection error continuously exceeds a preset threshold, or that no longer meet the static consistency requirements, the 3D map construction module marks them as invalid map points or deletes them from the global 3D environment map.

[0142] As green light reflection images that meet the mapping conditions are continuously input, the 3D map building module repeatedly performs feature point tracking, physical depth calculation, 3D coordinate transformation, and map point updating to improve the global 3D environment map.

[0143] The global 3D environment map provides historical anatomical data with physical depth and image intensity attributes for 3D map reprojection under occlusion conditions, thereby reducing depth-related projection misalignment caused when only two-dimensional historical images are used for fusion.

[0144] See attached document Figure 5 When the processing device detects that the proportion of the area in the field of view where the image contrast is reduced due to blood or other fluid secretions exceeds a preset ratio, or the number of effectively tracked anatomical feature points is lower than a preset threshold, the processing device executes step S3 through the occlusion trend calculation module. Step S3 includes sub-steps S31, S32 and S33.

[0145] Sub-step S31: Calculate the relative shading thickness field.

[0146] The occlusion trend calculation module calculates the relative occlusion thickness of each pixel region within the field of view based on the difference in light intensity attenuation between the spatiotemporally registered green light reflection image and the near-infrared image.

[0147] Hemoglobin in blood exhibits high absorption of green light around 540 nm; compared to green light, near-infrared light around 850 nm shows relatively low absorption attenuation in blood. Therefore, with imaging parameters such as light source output power, image sensor exposure time, and gain corrected, the intensity ratio of the two image bands can be used to characterize the relative change in the degree of blood occlusion.

[0148] Before calculating the light intensity ratio of the dual-band image, the occlusion trend calculation module performs dark field correction, flat field correction, exposure time normalization, sensor gain normalization, and light source output power correction on the green light reflection image and the near-infrared image, respectively, to obtain the corrected green light pixel intensity and near-infrared pixel intensity.

[0149] The occlusion trend calculation module extracts the corrected pixel intensity of the corresponding pixels after spatiotemporal registration in the near-infrared image and the green light reflection image, and calculates the logarithmic ratio of the dual-band corrected light intensity by referring to the Lambert-Beer optical attenuation relationship, thereby generating the relative occlusion thickness field. The relative shading thickness field is calculated using the following formula: ; In the formula, This indicates that the coordinates in the near-infrared image are... Pixel intensity value; This represents the pixel intensity value at the corresponding coordinate in the green light reflection image; A smaller positive real number compensation term preset for the system; This represents the numerical value of the generated relative occlusion thickness field; The operator represents the natural logarithm operator.

[0150] Positive compensation term This is used to avoid division by zero or logarithmic calculation errors when the corrected pixel intensity is zero. It can be determined based on the normalization range of image intensity; when the image intensity is normalized to 0 to 1, For example, it can be set to 10. -8 Up to 10 -5 Preferably 10 -6 .

[0151] Relative shading thickness field Used to represent the relative occlusion degree of different pixel regions. Without establishing a thickness calibration relationship, the relative occlusion thickness field does not represent the true fluid thickness with physical dimensions.

[0152] In one optional implementation, the processing device pre-acquires multiple dual-band images corresponding to known fluid thicknesses or known occlusion levels, and establishes a calibration curve or lookup table between the logarithmic ratio of the dual-band corrected light intensity and the fluid thickness or occlusion level. Within the applicable range of the calibration curve or lookup table, the processing device can convert the relative occlusion thickness field into a thickness estimation result with physical dimensions.

[0153] Sub-step S32: Extraction of gravitational acceleration vector and determination of image plane orientation.

[0154] The occlusion trend calculation module estimates the gravitational acceleration vector of the laryngoscope in its current posture based on the three-axis acceleration data and three-axis angular velocity data output by the six-axis inertial measurement unit.

[0155] Specifically, the occlusion trend calculation module performs low-pass filtering on the three-axis acceleration data and combines the attitude change obtained by integrating the three-axis angular velocity data to correct the direction of the low-pass filtering results, so as to reduce the impact of motion acceleration generated by the laryngoscope pushing, pulling, rotating or shaking on the gravity estimation results.

[0156] The cutoff frequency of the low-pass filter can be set from 0.5Hz to 2Hz. The processing equipment determines the specific cutoff frequency based on the actual sampling frequency of the inertial measurement unit, the noise level, and the laryngoscope operation speed.

[0157] The occlusion trend calculation module utilizes the extrinsic rotation matrix between the six-axis inertial measurement unit (IMU) and the global shutter CMOS image sensor to transform the gravitational acceleration vector from the IMU coordinate system to the camera coordinate system, and determines the direction of gravitational acceleration within the image plane. Image plane gravity direction vector. The calculation formula is as follows: ; In the formula, This represents the direction vector of gravity in the image plane. This represents the camera intrinsic parameter matrix corresponding to the global shutter CMOS image sensor; The external parameter rotation matrix represents the relationship between the six-axis inertial measurement unit coordinate system and the camera coordinate system; The estimated gravitational acceleration vector is represented by the following form: .

[0158] Extrinsic rotation matrix After the six-axis inertial measurement unit and the global shutter CMOS image sensor are fixedly assembled, the data is obtained through joint calibration of the camera and the inertial measurement unit and stored in a non-volatile memory that can be read by the processing device.

[0159] Based on the above calculation results, the occlusion trend calculation module takes the direction component parallel to the image plane and normalizes it to obtain the gravity direction vector of the image plane. This is done to facilitate pixel-level calculations.

[0160] When the projection amplitude of the gravitational acceleration vector in the image plane is lower than a preset projection threshold, it indicates that the gravity direction is close to the camera's optical axis, and the stability of the normalization result decreases. The occlusion trend calculation module marks the current image plane gravity direction vector as invalid and does not use gravity direction information to correct the occlusion change judgment in the current image frame.

[0161] When the flow amplitude of the occluder is lower than the preset minimum effective flow threshold or the gravity direction vector of the image plane is marked as invalid, the occlusion trend calculation module will mark the corresponding flow direction consistency parameter as invalid and will not use this parameter to correct the current occlusion state judgment.

[0162] Sub-step S33: Calculate the occlusion change state.

[0163] The occlusion trend calculation module generates an occlusion change state matrix based on the temporal change of the relative occlusion thickness field, the optical flow of the occluded object, and the gravity direction information at continuous time intervals.

[0164] First, the occlusion trend calculation module performs motion compensation on the relative occlusion thickness field of the previous moment based on the relative pose between consecutive image acquisition moments, so that it is in the same image coordinate system as the current relative occlusion thickness field.

[0165] Subsequently, the occlusion trend calculation module calculates the difference between the current relative occlusion thickness field and the motion-compensated relative occlusion thickness field of the previous moment, obtaining the time change of the relative occlusion thickness. The time change is used to determine whether the occlusion degree of the corresponding pixel region increases, decreases, or remains stable over time.

[0166] The occlusion trend calculation module also performs optical flow calculations between continuous relative occlusion thickness fields to obtain the optical flow of the occluded object. The optical flow of the occluded object includes at least the flow direction and flow amplitude at each pixel position.

[0167] For image frames where the gravity direction vector of the image plane is valid, the occlusion trend calculation module calculates the degree of consistency between the flow direction of the occluder and the gravity direction of the image plane, and defines the parameter used to characterize the degree of consistency as the flow direction consistency parameter to determine whether the occluder has a flow component in the direction of gravity or the direction of gravity.

[0168] In addition, the occlusion trend calculation module can utilize discrete difference operators to calculate the current relative occlusion thickness field. Spatial gradient calculation is performed to obtain the spatial gradient field. The spatial gradient field represents the spatial direction in which the relative occlusion thickness increases rapidly in the current image.

[0169] The occlusion trend calculation module will calculate the spatial gradient field. Gravity direction vector in the image plane Pixel-level inner product calculations are performed to obtain the thickness gradient direction consistency parameter. This is used to characterize the degree of consistency between the spatial gradient direction of the relative occlusion thickness field and the gravity direction of the image plane. Its calculation formula is as follows: ; In the formula, This represents the consistency parameter of the thickness gradient direction; This represents the direction vector of gravity in the image plane. The spatial gradient field representing the relative occlusion thickness field; This represents the inner product of two-dimensional vectors.

[0170] Thickness gradient direction consistency parameter It is only used to characterize the relationship between the current spatial distribution of relative occlusion thickness and the direction of gravity, and is not used alone to determine whether the occlusion is moving, the occlusion is deepening, or the occlusion is dissipating.

[0171] The occlusion trend calculation module uses the time change of relative occlusion thickness as the main parameter for judging whether occlusion is deepening or dissipating, and uses the flow amplitude and flow direction of the occluding object as motion state parameters, and utilizes the flow direction consistency parameter and the thickness gradient direction consistency parameter. Adjust the confidence level of the corresponding judgment result.

[0172] The generated occlusion change state matrix includes at least the time variation of relative occlusion thickness, the flow amplitude of the occluded object, the consistency parameter between the flow direction of the occluded object and the gravity direction, and the consistency parameter between the spatial distribution of relative occlusion thickness and the thickness gradient direction of the gravity direction. .

[0173] To reduce image noise and frequent state changes caused by minor disturbances, the occlusion trend calculation module sets thickness variation thresholds and optical flow amplitude thresholds. These thresholds are calibrated based on the background noise variance and static optical flow amplitude of consecutive dual-band images in an unoccluded state.

[0174] When the time change in relative occlusion thickness exceeds the preset thickness change threshold, the occlusion trend calculation module determines that the corresponding area is in a state of deepening occlusion.

[0175] When the time change of relative occlusion thickness is less than the negative preset thickness change threshold, the occlusion trend calculation module determines that the corresponding area is in the occlusion dissipation state.

[0176] When the time change of the relative occlusion thickness is between the positive and negative preset thickness change thresholds, and the flow amplitude of the occluded object is lower than the preset optical flow amplitude threshold, the occlusion trend calculation module determines that the corresponding area is in a stable occlusion state.

[0177] When the time change of the relative occlusion thickness is between the positive and negative preset thickness change thresholds, but the flow amplitude of the occluder is not lower than the preset optical flow amplitude threshold, the occlusion trend calculation module determines that the corresponding area is in a flowing state, and records its flow attributes in the direction of gravity or against gravity based on the degree of consistency between the flow direction of the occluder and the gravity direction of the image plane.

[0178] In one alternative implementation, the occlusion trend calculation module requires that the same judgment state be maintained for a preset number of consecutive frames before updating the final state of the corresponding pixel region, so as to reduce the frequent switching of judgment results between adjacent image frames.

[0179] See attached document Figure 6 The processing device executes step S4 through the pose recursive mapping module. Step S4 includes sub-steps S41 and S42.

[0180] When the field of view is severely obstructed, causing visual feature tracking in the visual inertial odometry to fail, the processing device initiates step S4.

[0181] Specifically, when the number of effectively matched anatomical feature points in a consecutive preset number of frames is lower than a preset tracking failure threshold, or when the average reprojection error of the visual features is higher than a preset error threshold, the processing device determines that the visual feature tracking has failed. The preset tracking failure threshold can be set according to the pose calculation method and processing redundancy requirements, for example, set to 10 to 15 effective feature points.

[0182] Sub-step S41, pure inertial pose derivation.

[0183] The pose recursive mapping module determines the last valid image frame before visual feature tracking fails, and records the system time corresponding to the last valid image frame as the start time. .

[0184] The pose recursive mapping module reads the visual inertial odometry at the initial moment. The output includes camera position, camera attitude, motion speed, gyroscope bias, accelerometer bias, gravitational acceleration vector, and state covariance, and these states are used as the initial states for pure inertial pose recursion.

[0185] For the starting time Up to the current moment The pose recursive mapping module obtains the time-aligned angular velocity vector within the time interval between these intervals. and linear acceleration vector And deduct the zero bias of the gyroscope and the zero bias of the accelerometer respectively.

[0186] The pose recursive mapping module updates the current camera pose based on the bias-corrected angular velocity vector and, based on the updated camera pose, transforms the gravitational acceleration vector to the same coordinate system as the linear acceleration vector. The pose recursive mapping module then removes the gravitational acceleration component from the bias-corrected linear acceleration vector to obtain a degravity-free motion acceleration sequence. .

[0187] The pose recursive mapping module employs either the Lie algebra pose update method or the quaternion pose update method for angular velocity vectors. Perform time integration to obtain the current time relative to the starting time. rotation matrix .

[0188] The pose recursive mapping module uses the starting time... The velocity is used as the initial value for the velocity integral, and the acceleration sequence of gravity-free motion is processed. Perform time integration to obtain the current velocity; then further integrate the velocity over time to obtain the current moment relative to the initial moment. 3D translation vector .

[0189] During the pure inertial pose estimation process, the pose estimation mapping module synchronously updates the state covariance to characterize the impact of angular velocity noise, linear acceleration noise, and sensor zero bias error on the current pose estimation result as the integration time increases.

[0190] Because visual observations are lacking to correct inertial errors after visual feature tracking fails, the cumulative pose error generated by pure inertial pose recursion typically increases with the recursion time. Therefore, pure inertial pose recursion is used for short-term pose estimation within a preset maximum duration. The preset maximum duration is determined based on the zero-bias instability of the six-axis inertial measurement unit, the random walk variance, and the maximum allowable pose error of the system.

[0191] Sub-step S42: 3D map reprojection.

[0192] The pose recursive mapping module uses the rotation matrix obtained in sub-step S41 to... and three-dimensional translation vector This reprojects 3D map points from the global 3D environment map onto the current image plane.

[0193] The 3D map construction module pre-stores the 3D spatial coordinates, green light image intensity attributes, map point confidence scores, and source keyframe identifiers of each 3D map point in the global 3D environment map. The pose recursive mapping module calculates the pose based on the starting time. The corresponding global pose of the camera transforms the 3D map points to be reprojected into a coordinate system that is compatible with the relative pose transformation matrix.

[0194] The pose recursive mapping module is based on the camera intrinsic parameter matrix. Rotation matrix and three-dimensional translation vector Calculate the projected coordinates of historical 3D map points within the current image plane. Projected coordinates The mapping calculation formula is as follows: ; To obtain the corresponding two-dimensional pixel coordinates, the above relationship is expressed as: ; In the formula, This represents a column vector of historical 3D map point coordinates read from the global 3D environment map and converted to the adapted coordinate system, in the form of: ; This represents the relative rotation matrix obtained from substep S41; This represents the relative three-dimensional translation vector obtained by sub-step S41; This represents the camera intrinsic parameter matrix corresponding to the global shutter CMOS image sensor; This represents a column vector of homogeneous pixel coordinates corresponding to historical 3D map points in the current image plane, in the form of: ; This represents the homogeneous coordinate normalization factor.

[0195] The pose recursive mapping module performs depth validity checks on 3D map points after rigid body transformation. When the depth value of a 3D map point in the current camera coordinate system is less than or equal to zero, the pose recursive mapping module discards the 3D map point and does not project it onto the current image plane.

[0196] For 3D map points whose projected coordinates exceed the effective pixel range of the image, or whose map point confidence is lower than the preset map point confidence threshold, the pose recursive mapping module will not write them into the reference background frame.

[0197] When multiple 3D map points are projected onto the same pixel location or the same local pixel region, the pose recursive mapping module determines the visibility of each 3D map point based on its depth in the current camera coordinate system, and prioritizes retaining 3D map points with smaller depths and map point confidence that meet the requirements.

[0198] The pose recursive mapping module writes the green light image intensity attribute associated with the valid 3D map points into the corresponding projection positions to form valid projection points of the reference background frame.

[0199] For non-integer pixel coordinates obtained from reprojection, the pose recursive mapping module uses bilinear interpolation or nearest neighbor interpolation to resample the pixel position. For small holes between effective projection points, the pose recursive mapping module uses local interpolation based on adjacent effective projection points to fill them.

[0200] When there is no effective map point coverage, or when the hole range exceeds the preset hole range threshold, the pose recursive mapping module will not extrapolate and fill the area, and will mark the area as an invalid area.

[0201] Through the aforementioned depth validity verification, visibility judgment, image intensity attribute writing, and local resampling, the pose recursive mapping module generates a reference background frame corresponding to the current camera viewpoint. Simultaneously, a mask for the effective area of ​​the reference background frame is generated.

[0202] The reference background frame is used to provide the safe fusion and reconstruction module with historical anatomical images compensated for the current relative pose during visual feature tracking failures. The reference background frame only participates in subsequent image fusion in regions where the effective region mask is valid and the current pose confidence meets preset conditions.

[0203] See attached document Figure 7 The processing device executes step S5 through the secure fusion reconfiguration module. Step S5 includes sub-steps S51, S52, and S53.

[0204] Sub-step S51: Near-infrared structure-compensated image generation.

[0205] The secure fusion and reconstruction module determines the semi-transparent occlusion area that can be structurally compensated based on the relative occlusion thickness field, the validity marker of the dual-band image, and the pixel intensity of the near-infrared image.

[0206] When the relative occlusion thickness of the corresponding pixel is within the pre-calibrated effective range, and the pixel intensity of the near-infrared image is not overexposed or underexposed, the safe fusion reconstruction module determines the area where the pixel is located as the effective compensation area.

[0207] Because the absorption attenuation of near-infrared light near 850nm in blood is typically lower than that of green light near 540nm, near-infrared images can retain some tissue scattering and structural information even under partial occlusion conditions. The secure fusion reconstruction module utilizes this structural information to compensate for the reduced tissue visibility in green light reflection images due to blood absorption attenuation.

[0208] The secure fusion reconfiguration module is based on the relative occlusion thickness field. and corrected near-infrared images Perform exponential compensation calculations to generate near-infrared structure-compensated images. The calculation formula is as follows: ; In the formula, Represents a near-infrared structure-compensated image; This represents the pixel intensity of the near-infrared image after dark field correction, flat field correction, exposure time normalization, sensor gain normalization, and light source output power correction. This represents the green light image domain compensation coefficient obtained through offline calibration; This represents the relative occlusion thickness field output by the occlusion trend calculation module; This represents an exponential function with the natural constant as its base.

[0209] Green light image domain compensation coefficient Offline calibration is performed using standard blood samples with known thickness or blood-like media with equivalent optical parameters.

[0210] During the calibration process, light source output power, exposure time, sensor gain, imaging distance, and image preprocessing parameters corresponding to the actual working state of the video laryngoscope were used to acquire green light reflection images and near-infrared images corresponding to different known occlusion thicknesses. Based on the intensity difference between the near-infrared image compensation result and the corresponding unoccluded green light reference image, the green light image domain compensation coefficient μg and its applicable relative occlusion thickness range were determined. The calibration results were stored in non-volatile memory that could be read by the processing device.

[0211] When the relative occlusion thickness exceeds the calibrated range, or when the near-infrared image pixels are overexposed or underexposed, the safety fusion reconstruction module will mark the near-infrared structure compensation result of the corresponding pixel as invalid.

[0212] Near-infrared structure-compensated images are used to enhance the visibility of tissue structures within occluded areas, but do not represent the restoration of the true green light reflectance intensity or true visible light color of the occluded area.

[0213] Considering that exponential compensation may cause the calculation results to exceed the range of pixel values ​​corresponding to the image bit depth, the secure fusion reconstruction module performs amplitude limiting processing on the pixel values ​​of the near-infrared structure-compensated image. For example, when the output image is in 8-bit grayscale format, the calculation results are limited to the range of 0 to 255.

[0214] Sub-step S52: Calculate the basic fusion weights and inertial recursive confidence.

[0215] The secure fusion reconstruction module reads the relative occlusion thickness field and occlusion change state matrix output by the occlusion trend calculation module, and reads the reference background frame, effective region mask and current pose confidence output by the pose recursive mapping module.

[0216] The secure fusion and reconstruction module determines the initial reference ratio of the reference background frame based on the current relative occlusion thickness, and adjusts the initial reference ratio based on the occlusion deepening state, occlusion dissipation state, occlusion stabilization state and occlusion flow state represented by the occlusion change state matrix.

[0217] In one implementation, the secure fusion reconstruction module reads the time change of relative occlusion thickness, the flow amplitude of the occlusion object, the consistency parameter of the flow direction, and the consistency parameter of the thickness gradient direction from the occlusion change state matrix, and determines the fusion control parameters used to control the fusion weights by combining the corresponding judgment confidence levels. The fusion control parameters are the result obtained after comprehensively processing the above state variables, and are not equivalent to any single consistency parameter, and are normalized to a preset value range before weight calculation.

[0218] The secure fusion reconfiguration module uses a logistic mapping function to convert the thickness gradient direction consistency parameters. Convert to the base blending weights at the corresponding pixel positions of the reference background frame Its mapping formula is as follows: ; In the formula, Indicates the basic fusion weights; Represents an exponential function with the natural constant as its base; This represents the preset mapping gain coefficient, used to adjust the slope of the basic fusion weights as the consistency parameter changes; This represents the consistency parameter in the thickness gradient direction. The mapping gain coefficient can be determined based on the normalization range of the consistency parameter and the desired rate of change of the weights, for example, set to 10 to 50.

[0219] When the consistency parameter indicates that the occlusion level in a local region is high and the occlusion is worsening, the basic fusion weights... Increase the reference ratio of the background frame; when the consistency parameter indicates that the occlusion is dissipating, increase the base fusion weights. Reduce the amount of near-infrared structure-compensated images to increase the citation ratio.

[0220] When a local area is in a stable occlusion state, the secure fusion and reconstruction module maintains the corresponding basic fusion weight according to the current relative occlusion thickness, avoiding fixing the fusion weight to an intermediate value simply because the time change of the occlusion thickness is close to zero.

[0221] When the effective region mask of the reference background frame indicates that the corresponding pixel has no effective map coverage, the secure fusion reconstruction module sets the basic fusion weight of that pixel to zero.

[0222] When the confidence level of the current pose is lower than the preset pose confidence level threshold, the safety fusion reconstruction module reduces the basic fusion weight according to the confidence level of the current pose; when the confidence level of the current pose is lower than the preset safety threshold, the basic fusion weight is set to zero.

[0223] Considering that the cumulative pose error of pure inertial pose recursion increases with the recursion time, the safe fusion reconstruction module constructs an inertial recursion confidence time decay function based on the calibration parameters of the six-axis inertial measurement unit. The calculation formula is as follows: ; In the formula, This represents the time decay function of the inertial recursive confidence level. Represents an exponential function with the natural constant as its base; Indicates the current system time; This indicates the starting moment when visual feature tracking fails and pure inertial pose recursion begins; This represents the attenuation factor determined based on the zero-bias instability and random walk variance of the six-axis inertial measurement unit.

[0224] when and Attenuation factor in seconds The unit is negative first power of seconds. Attenuation factor. It is positively correlated with the inertial measurement error of the six-axis inertial measurement unit and can be determined based on the Allen variance calibration results, for example, set to 0.1 to 1.0.

[0225] Inertial recursive confidence time decay function This is used to characterize the relationship between the reliability of pure inertial pose recursion results and the decrease in recursion duration.

[0226] Sub-step S53: Pixel-level fusion and failure handling.

[0227] The secure fusion reconstruction module utilizes an inertial recursive confidence time decay function. For basic fusion weights Apply time constraints to obtain the final fusion weights. The calculation formula is as follows: ; In the formula, Indicates the final fusion weight; This indicates the base fusion weights that have been adjusted based on the relative occlusion thickness, occlusion change state, effective region mask of the reference background frame, and current pose confidence. This represents the time decay value of the inertial recursive confidence level.

[0228] As the duration of pure inertial pose recursion increases, Gradually reduce the reference background frame in the reconstructed image, thereby reducing the proportion of reference background frames in the field of view and reducing depth-related projection misalignment caused by cumulative pose error.

[0229] The secure fusion reconfiguration module is based on the final fusion weight. For the reference background frame Near-infrared structure-compensated images Pixel-level weighted synthesis is performed. Field of view reconstructed image. The synthesis formula is as follows: ; In the formula, Indicates the reference background frame; Represents a near-infrared structure-compensated image; Represents the reconstructed image of the field of view; This indicates the final fusion weight.

[0230] When both the reference background frame and the near-infrared structure compensation image are valid at their respective pixel positions, the secure fusion reconstruction module performs weighted synthesis according to the final fusion weight.

[0231] When the reference background frame is invalid at the corresponding pixel position while the near-infrared structure compensation image is valid, the secure fusion reconstruction module sets the final fusion weight to zero and outputs the near-infrared structure compensation image.

[0232] When the near-infrared structure-compensated image is invalid at the corresponding pixel position while the reference background frame is valid, the reference background frame is output only if the current pose confidence is not lower than the preset safety threshold; otherwise, the corresponding pixel of the original near-infrared image is output and a low confidence marker is generated.

[0233] When both the reference background frame and the near-infrared structure compensation image are invalid at the corresponding pixel position, the secure fusion reconstruction module outputs the corresponding pixel of the original near-infrared image and marks the pixel as a low-confidence region.

[0234] The secure fusion reconstruction module converts the field-of-view reconstruction image into a video data format supported by the external display terminal, and outputs the field-of-view reconstruction image, low-confidence area markers, and system status prompts to the external display terminal.

[0235] When the duration of pure inertial pose recursion exceeds the preset maximum duration, the confidence level of the current pose is lower than the preset safety threshold, or the effective coverage ratio of the reference background frame is lower than the preset coverage ratio threshold, the safe fusion reconstruction module stops referencing the reference background frame of the corresponding area and outputs the near-infrared structure compensation image or the original near-infrared image, while outputting a low confidence prompt message to the external display terminal.

[0236] In an intubation procedure for patients with traumatic airway bleeding, the video laryngoscope system alternately acquires green light and near-infrared images at a rate of 30Hz. The inertial measurement unit synchronously outputs triaxial acceleration and triaxial angular velocity data at a frequency of 500Hz. The system extracts 120 anatomical feature points, including the glottic margin and epiglottic cartilage texture, from the green light image under clear field of view. Combined with the inertial baseline displacement parameters of the laryngoscope advancing 20mm within the continuous image acquisition interval, the system utilizes multi-view... Figure 3 The angular calculation determined that the physical depth of each feature point from the camera center was 45mm. The system continuously writes and updates the global 3D environment map with 3D map points containing spatial coordinate parameters and corresponding green light image intensity attributes.

[0237] When fluid overflows from the patient's airway and covers the laryngoscope lens, causing fluid obstruction of the field of view, the system simultaneously extracts pixel intensity parameters from the registered dual-band images at the same time. The green light reflection intensity value of the specified center pixel decreases from 200 to 20, and the near-infrared pixel intensity value at the corresponding coordinate position decreases from 200 to 120. The system calculates the ratio of the corrected near-infrared pixel intensity to the green light pixel intensity with an added positive compensation term and takes the natural logarithm, outputting the numerical result of the increase in the relative obstruction thickness parameter. Based on the time difference of the continuous relative obstruction thickness field, the system calculates that the optical flow direction of the obstruction is downward towards the image plane. The system determines that the optical flow direction component is completely consistent with the gravity direction vector of the image plane calculated based on inertial data. The system outputs an obstruction change state matrix that includes the flow attributes along the gravity direction and the obstruction deepening state.

[0238] Due to large-area occlusion of the field of view, the number of valid feature points that meet the consistency check in the current image drops to 5. The system determines that the number of current feature points is below the preset tracking failure threshold of 10 and starts pure inertial pose recursive mapping processing. The system records the laryngoscope 3D pose corresponding to the last valid image frame before visual tracking failure and reads the zero bias parameters of the inertial sensor at that time. In the 0.2s time interval after visual feature tracking failure, the system performs integral calculation based on the pure inertial data after zero bias correction and removal of gravitational acceleration components to obtain the relative pose transformation matrix of the laryngoscope rotating 1.5 degrees along the longitudinal axis and translating 3mm forward at the current moment. The system reprojects the pre-stored 3D map points in the global 3D environment map onto the current image plane based on the relative pose transformation matrix and camera intrinsic parameters. The system combines bilinear interpolation processing to output a reference background frame and effective area mask to compensate for depth-related projection misalignment.

[0239] The system constructs an inertial recursive confidence time decay function with a confidence level decreasing by 50% per second based on pre-calibrated inertial measurement unit error parameters. After visual feature tracking fails for 0.2 seconds, the system calculates that the current inertial recursive confidence level has decayed to 90%. The system uses the inertial recursive confidence time decay function and the occlusion change state matrix to numerically constrain the basic fusion weights. The system performs pixel-level weighted synthesis of a reference background frame with a final fusion weight of 80% and a near-infrared structure compensation image with a final fusion weight of 20%. The external display terminal receives and displays a field-of-view reconstruction image with glottal edge coverage provided by the reference background frame and enhanced tissue structure visibility provided by the near-infrared structure compensation image. The operator completes the insertion of the endotracheal tube into the airway based on the field-of-view reconstruction image output by the external display terminal.

[0240] Two sets of control and verification experiments were designed for the aforementioned video laryngoscope system with blood-obscured image display function. The test object was a medical-grade intubation training model with internal anatomical structural features. The fluid obscuration medium was optical bionic blood with absorption and scattering coefficients equivalent to human venous blood in the 540nm and 850nm bands. The control group test device was a conventional single-band white light video laryngoscope, and the experimental group test device was the video laryngoscope system provided by this invention. The operator placed the control group test device and the experimental group test device into the intubation training model until the glottic structure was exposed on the external display terminal. A micro-injection pump pre-installed inside the trachea of ​​the model sprayed 5ml of optical bionic blood into the lens of the test device to create fluid obscuration. The operator observed the screen on the display terminal under the fluid obscuration state and continued to perform the endotracheal tube insertion step.

[0241] The experimental data recording module simultaneously collected the duration of complete visual field loss and anatomical structure visibility scores for both the control and experimental groups, and calculated the total first-time intubation success rate. The duration of complete visual field loss for the control group's test device after fluid obstruction was greater than 12 seconds, while the duration of complete visual field loss for the experimental group's test device after fluid obstruction was 0 seconds. Within 3 seconds after fluid obstruction was triggered, the anatomical structure visibility score for the control group's test device decreased from 95 points to 5 points, while the anatomical structure visibility score for the experimental group's test device remained at 82 points within the same time period. The system statistics showed that the first-time intubation success rate for the experimental group's test device under fluid obstruction was 95%, while the first-time intubation success rate for the control group's test device under fluid obstruction was 30%. The above data verified the operational usability of the system of the present invention in maintaining visual feature display and assisting in safe navigation under fluid obstruction conditions.

[0242] Figure 8 A global 3D environment map anatomical feature point cloud is constructed using a scatter set, exhibiting a spatial distribution resembling airway channels. Figure 8 The solid black line in the figure represents the joint pose space movement trajectory output by the visual inertial odometry when it is in an effective calculation state. Figure 8 The square scatter markers in the diagram represent the three-dimensional spatial coordinates where the laryngoscope lens encounters severe fluid obstruction, causing visual feature tracking to fail. Figure 8 The dotted line in the figure represents the pure inertial pose recursive trajectory independently calculated and output by the system based on the corrected inertial data after visual feature tracking fails. Figure 8 The filled polygonal plane and corresponding projection beam connection constructed at the end represent the system's process of reprojecting pre-stored global 3D map points to the reference background frame mapping process of recursively calculating the current endpoint's corresponding camera frustum plane.

[0243] Figure 9 The horizontal axis represents the number of seconds the system continues to advance after determining that visual feature tracking has failed, and the vertical axis represents the percentage or score value converted from the corresponding evaluation index. Figure 9 The dashed lines in the figure represent the inertial recursion confidence time decay data, which shows a decreasing trend in parameters as the recursive time variable of pure inertial pose increases. Figure 9 The solid gray line in the figure represents the data showing that the visibility score of the single-band white light control test equipment dropped from 95 points to 5 points the instantaneously when it encountered fluid obstruction. Figure 9 The solid black line in the figure represents the data performance of the system of the present invention, which maintains the visibility score above 82 points after reconstructing the field of view through pixel-level weighted synthesis. Figure 9 The filled area between the medium gray solid line and the black solid line represents the time distribution range of high-confidence safe navigation additionally provided by the system of the present invention under continuous occlusion.

[0244] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result using substantially the same method falls within the scope of the present invention.

Claims

1. A video laryngoscope system with blood-obscured image display function, characterized in that, This includes hardware acquisition devices and processing equipment; The hardware acquisition device includes a dual-wavelength light source module, a global shutter CMOS image sensor, and a six-axis inertial measurement unit; the processing device includes: The data synchronization acquisition module is used to acquire dual-band image data and inertial data, perform timestamp alignment and spatial registration on the dual-band image data and the inertial data, and output image and inertial data sets. The 3D map construction module is used to combine the image and inertial data set to perform visual inertial odometry processing to obtain the physical depth of the laryngoscope pose and anatomical feature points, and generate a global 3D environment map. The occlusion trend calculation module calculates the relative occlusion thickness field based on the difference in light intensity attenuation between the dual-band image data, extracts the gravity direction vector of the image plane based on the inertial data, and generates an occlusion change state matrix by combining the time change of the relative occlusion thickness field. The pose recursion mapping module is used to perform pure inertial pose recursion when visual feature tracking fails, and to reproject the 3D map points in the global 3D environment map to the current image plane to generate a reference background frame and the corresponding effective area mask. The secure fusion and reconstruction module calculates a near-infrared structure compensation image based on the dual-band image data, constructs basic fusion weights based on the relative occlusion thickness field and the occlusion change state matrix, and performs weighted synthesis of the reference background frame and the near-infrared structure compensation image to generate a field-of-view reconstruction image.

2. The video laryngoscope system with blood-obscured image display function according to claim 1, characterized in that, The data synchronization acquisition module drives the dual-wavelength light source module to sequentially acquire green light reflection images and near-infrared images in adjacent trigger cycles to form a dual-band image pair as the dual-band image data. The inertial data is interpolated and aligned to the time reference of the dual-band image pair. Initial motion compensation is performed based on the inertial data between two exposures. Cross-band spatial registration is performed based on the tissue structure contour to generate the image and inertial data set.

3. The video laryngoscope system with blood-obscured image display function according to claim 2, characterized in that, The 3D map construction module extracts anatomical feature points from the continuously acquired green light reflection images, calculates the camera baseline displacement and relative attitude between adjacent image acquisition times using the visual inertial odometry method, calculates the physical depth of the anatomical feature points through multi-view triangulation, converts the anatomical feature points into 3D spatial coordinates in the world coordinate system, and writes them into the global 3D environment map.

4. The video laryngoscope system with blood-obscured image display function according to claim 2, characterized in that, The occlusion trend calculation module performs dark field correction, flat field correction, exposure time normalization, sensor gain normalization, and light source output power correction on the green light reflection image and the near-infrared image. It calculates the logarithmic ratio of the pixel intensity of the near-infrared image to the pixel intensity of the green light reflection image to generate the relative occlusion thickness field. The occlusion trend calculation module performs motion compensation on the relative occlusion thickness field of the previous moment based on the relative pose between image acquisition moments, and calculates the difference between the current relative occlusion thickness field and the motion-compensated relative occlusion thickness field of the previous moment to obtain the time change of the relative occlusion thickness.

5. The video laryngoscope system with blood-obscured image display function according to claim 1, characterized in that, The inertial data includes three-axis acceleration data and three-axis angular velocity data. The occlusion trend calculation module estimates the gravitational acceleration vector based on the three-axis acceleration data and the three-axis angular velocity data, transforms the gravitational acceleration vector to the camera coordinate system, and extracts the direction component parallel to the image plane to obtain the gravity direction vector of the image plane. The occlusion trend calculation module calculates the optical flow of the occluded object between continuous relative occlusion thickness fields, and calculates the consistency parameter between the occluded object flow direction and the gravity direction vector of the image plane based on the angle relationship between the optical flow direction of the occluded object and the gravity direction vector, and writes it into the occlusion change state matrix.

6. The video laryngoscope system with blood-obscured image display function according to claim 1, characterized in that, The inertial data includes angular velocity data and linear acceleration data. The pose recursive mapping module reads the laryngoscope pose, motion velocity, gyroscope zero bias, and accelerometer zero bias at the last valid moment before the visual feature tracking fails as the initial state for pure inertial pose recursion. It performs zero bias correction on the angular velocity data and the linear acceleration data, updates the camera attitude based on the zero bias-corrected angular velocity data, obtains the rotation matrix of the current moment relative to the last valid moment based on the updated camera attitude, removes the gravitational acceleration component from the zero bias-corrected linear acceleration data to obtain a gravity-free motion acceleration sequence, updates the camera velocity and position based on the gravity-free motion acceleration sequence and the motion velocity at the last valid moment, and obtains the three-dimensional translation vector of the current moment relative to the last valid moment.

7. The video laryngoscope system with blood-obscured image display function according to claim 6, characterized in that, The pose recursive mapping module reprojects the 3D map points in the global 3D environment map onto the current image plane based on the rotation matrix, the 3D translation vector, and the camera intrinsic parameters. It performs visibility judgment based on the projection depth to remove map points with non-positive depth values ​​in the camera coordinate system. It performs gridded resampling and local interpolation on the effective projection points to generate the reference background frame. It marks areas without effective map point coverage as invalid areas to generate the effective area mask.

8. The video laryngoscope system with blood-obscured image display function according to claim 4, characterized in that, The secure fusion reconstruction module generates the near-infrared structure compensation image by performing exponential compensation calculation based on the relative occlusion thickness field, the corrected near-infrared image pixel intensity, and the offline calibrated green light image domain compensation coefficient, and performs amplitude limiting processing on the pixel values ​​of the near-infrared structure compensation image. When the relative occlusion thickness exceeds the calibrated range or the pixel intensity of the near-infrared image is overexposed or underexposed, the safety fusion reconstruction module marks the near-infrared structure compensation result as invalid.

9. The video laryngoscope system with blood-obscured image display function according to claim 5, characterized in that, The secure fusion reconstruction module determines fusion control parameters based on the occlusion change state matrix and converts these parameters into basic fusion weights. It then constructs an inertial recursive confidence time decay function based on the zero-bias instability and random walk variance obtained from the six-axis inertial measurement unit calibration. The secure fusion reconstruction module uses the inertial recursive confidence time decay function, the effective region mask of the reference background frame, and the current pose confidence to constrain the basic fusion weights to obtain the final fusion weights. Finally, it performs pixel-level weighted synthesis of the reference background frame and the near-infrared structure-compensated image based on these final fusion weights.

10. Application of a video laryngoscope system with blood-obscured image display function, the system according to any one of claims 1-9, characterized in that, The system is used to provide dual-band image acquisition of the laryngeal cavity and reconstruction display of the field of view when blood obscures the view during laryngoscopy intubation.

Images