A dual-light source fluorescent nasal skull base mirror image processing system and device
The light source parameters of the dual light source are adjusted through predefined evaluation functions and simulated annealing algorithm, and deep learning technology is used to fuse fluorescence and white light imaging images, which solves the problem of insufficient light source adjustment and imaging fusion optimization in the existing technology, and improves the nasal skull base Surgical image quality and surgical efficiency.
Patent Information
- Application Number
- CN202410979497.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The prior art lacks light source adjustment for fluorescence and white light dual light sources and fluorescence and white light dual light source imaging fusion processing optimization, making it difficult to achieve the optimal light configuration and automatic adjustment parameters, affecting the image quality and efficiency of nasal skull base surgery.
The light source parameters of fluorescence and white light sources are adjusted by predefined evaluation functions and simulated annealing algorithm, and the fluorescence and white light imaging images are fused through deep learning technology to determine the lesion boundary, combining corner detection algorithm and homography transformation The matrix performs image registration and fusion.
Real-time optimization and automatic adjustment of light source parameters are achieved, image quality is improved, and doctors can more accurately identify and locate lesion tissue, and improve surgical success rate and safety.
Smart Images

Figure CN118512135B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing optimization, and discloses a dual-light source fluorescent nasal skull base mirror image processing system and device. Background Art
[0002] The field of modern medicine has extremely high requirements for the clarity and accuracy of surgical and diagnostic images, especially in nasal skull base surgery, where the performance of the endoscopic imaging system directly affects the effect and safety of the surgery. Traditional endoscopic systems mainly rely on white light sources for illumination. Although they can provide basic visual information, it is difficult to distinguish between normal tissue and diseased tissue. For this reason, fluorescence imaging technology was introduced to excite fluorescent substances with light of a specific wavelength, making the diseased tissue more prominent in the image, thereby enhancing the accuracy of diagnosis and surgery. However, there are still deficiencies in light source adjustment flexibility, intelligent optimization, real-time and versatility, making it difficult to achieve optimal light configuration and automatic adjustment of parameters, affecting the quality and efficiency of surgical images. However, the existing technology lacks light source adjustment for fluorescent and white light dual light sources, as well as optimization of the fusion processing of fluorescent and white light dual light source imaging.
[0003] For example, the Chinese patent with the authorization announcement number CN114430463B discloses an adaptive medical-grade imaging effect optimization method and system, which decomposes the collected image into R channel, G channel, B channel, and W channel; marks the pixel feature value of each channel data, calculates the current reference value and current weight value of each channel, calculates the optimal adjustment parameters of each channel according to the current pixel feature value, current weight value and current reference value, and controls the light source and camera according to the optimal adjustment parameters. The adaptive medical-grade imaging effect optimization method and system of the invention automatically learns according to the on-site environment, and automatically and intelligently adjusts the camera and light source parameters to achieve satisfactory medical-grade imaging effects.
[0004] The above method has the problems raised by the background technology, that is, lacks light source adjustment for the dual light source of fluorescence and white light, and lacks optimization for the fusion processing of the dual light source of fluorescence and white light. Summary of the invention
[0005] In response to the shortcomings of the prior art, the present application proposes a dual-light source fluorescent nasal skull base endoscope image processing system and device, which is used to use a pre-defined evaluation function to adjust the light source parameters of the fluorescence and white light dual light sources through a simulated annealing algorithm, and use an image fusion processing method based on deep learning technology to fuse the imaging images of the two types of light sources obtained after adjusting the light source parameters to determine the lesion boundary.
[0006] In a first aspect, an embodiment of the present application provides a dual-light source fluorescent nasal skull base endoscope image processing system, comprising:
[0007] An illumination unit is provided with at least one fluorescent light source and at least one white light source, for irradiating a target area; wherein the fluorescent light source is used to emit light of a target wavelength to excite the fluorescent substance in the target area to emit light, and the white light source is used to provide a broadband spectrum to illuminate the target area;
[0008] An imaging unit, used to collect light from the irradiated target area, the imaging unit includes independently or simultaneously collecting white light and fluorescent light;
[0009] a signal conversion unit, configured to receive the fluorescent image and the white light image reflected by the target area after being irradiated by the fluorescent light source and the white light source, and convert them into digital signals and send them to the image processing unit;
[0010] An image processing unit, used for processing the digital signal sent by the signal conversion unit to generate a fluorescence image and a white light image, and determining the tumor boundary by fusing and analyzing the fluorescence image and the white light image;
[0011] The interaction unit is used to collect user feedback information and send it to the imaging unit and the image processing unit.
[0012] As a possible implementation manner, each light source in the lighting unit is used to illuminate the target area independently or jointly; and the imaging unit further includes:
[0013] A light source optimization unit is used to optimize the light path parameters of the light path in the target area based on a preset target evaluation function; wherein the optimization of the light path parameters of the target area is achieved by adjusting the light source parameters of each light source in the lighting unit.
[0014] As a possible implementation, the light source parameters include: light source position, light source intensity, light source angle and light source focal length; the light path parameters include: light transmission path length, image clarity and lighting uniformity; the mathematical expression of the target evaluation function is:
[0015] ;
[0016] in, represents the length of the light transmission path, Represents image clarity, Represents the uniformity of illumination, , , It is a pre-set weighting factor based on medical needs. It is the attenuation coefficient that characterizes the loss of light during transmission. is the gain factor of image clarity, Represents the standard deviation of light intensity.
[0017] As a possible implementation manner, the functional expression of the illumination uniformity a is:
[0018] ;
[0019] in, and represent the cosine values of the angles between the light of the fluorescent light source and the light of the white light source and the imaging plane, respectively, and Respectively represent the light source intensities of the fluorescent light source and the white light source;
[0020] The function expression of the image definition i is:
[0021] ;
[0022] in, and are the imaging clarity of the fluorescent light source and the white light source, respectively, and is the corresponding weight factor.
[0023] As a possible implementation manner, the imaging unit further includes: an adjustment unit, configured to adjust light source parameters of each light source in the lighting unit; wherein the adjustment unit includes:
[0024] A mechanical arm unit, comprising a mechanical arm having at least one joint and a telescopic portion, wherein the mechanical arm moves the light source in multiple directions and angles based on an input control signal, and the light source covers the target area by presetting an initial position of the mechanical arm;
[0025] An angle adjustment unit, comprising a servo motor and an angle sensor; wherein the servo motor adjusts the emission angle of the light source based on an input control signal, and the angle sensor detects the angle information of the current light source and feeds back to the control system to adjust the angle of the light source;
[0026] An intensity control unit, comprising a digital regulator and an intensity sensor; wherein the digital regulator adjusts the current or voltage of the light source based on an input control signal to change the luminous intensity of the light source; the intensity sensor detects the brightness of the current light source and feeds back to the control system;
[0027] The focus adjustment unit includes an automatic zoom lens and a focus adjustment motor; wherein the automatic zoom lens is fixedly installed on the light path in front of the light source, and the focus adjustment motor adjusts the relative position between the light source and the automatic zoom lens based on an input control signal to change the focusing or divergence characteristics of the light.
[0028] The light source optimization unit, when measuring the efficiency and configuration of the optical path parameters of the nasal skull base mirror based on the target evaluation function and adjusting the optical path parameters of the nasal skull base mirror, also includes:
[0029] Based on the simulated annealing algorithm, the optical path parameters of the nasal skull base mirror that maximize the function value of the target evaluation function are determined, and the target light source parameters are generated.
[0030] As a possible implementation manner, the imaging unit further includes: a control unit, configured to send a control signal to the adjustment unit based on the target light source parameter and user feedback information, and perform at least one of the following steps to adjust the light source parameter:
[0031] Based on the control signal, the robot arm unit is controlled to adjust the position of the light source to the target light source position, and the position status is fed back in real time;
[0032] The angle adjustment unit adjusts the emission angle of the light source through the servo motor and the angle sensor based on the control signal, and feeds back the current angle value through the angle sensor;
[0033] The intensity control signal adjusts the current or voltage of the light source through a digital regulator based on the control signal, adjusts the intensity of the light source, and feeds back the current intensity of the light source in real time through an intensity sensor;
[0034] Based on the control signal, the focus adjustment unit moves forward and backward along the optical axis direction through the focus adjustment motor to adjust the distance between the light source and the zoom lens, and provides real-time feedback of the current focus state.
[0035] As a possible implementation, determining the tumor boundary includes:
[0036] preprocessing the digital signals of the fluorescent image and the white light image sent by the signal conversion unit;
[0037] Based on a corner detection algorithm, performing image registration on the fluorescence image and the white light image at the same time, and marking the registered fluorescence image and the white light image at the same time as the same group;
[0038] Based on the deep learning method, the registered fluorescence image and the white light image marked as the same group are fused to generate a fused image;
[0039] Based on the user's feedback information, the fused image is optimized to determine the tumor boundary.
[0040] As a possible implementation, the corner detection algorithm includes: Harris corner detection algorithm; performing image registration based on the corner detection algorithm includes:
[0041] The Harris corner response function is defined as:
[0042] ;
[0043] Among them, R is the Harris corner response value, M is the gradient covariance matrix of the local area of the image,
[0044] det(M) represents the determinant of the matrix M, trace(M) represents the trace of the matrix M, and k is the sensitivity constant for corner detection;
[0045] Based on the ORD algorithm, feature points are extracted from the fluorescence image and the white light image at the same time for matching, so as to establish a first corresponding relationship;
[0046] Based on the first corresponding relationship, generating a homography transformation matrix between the two images at the same moment;
[0047] The calculated homography transformation matrix between the two images at the same time is applied to any one of the two images to align it to the coordinate system of the other image, thereby registering the fluorescence image with the white light image.
[0048] As a possible implementation manner, the first corresponding relationship includes: n pairs of matching points between two images at the same time; based on the first corresponding relationship, generating a homography transformation matrix between the two images at the same time, including:
[0049] Based on n pairs of matching points between two images at the same time, a homography transformation model is established. For each matching point i, the following relationship exists:
[0050] ;
[0051] Among them, there are n pairs of matching points between the two images at the same time, and the corresponding point of the fluorescence image corresponding to each matching point i in the n pairs of matching points is , the corresponding point of the white light image is , arrive are the elements of the homography transformation matrix H;
[0052] Based on the least square method, the homography transformation matrix is determined.
[0053] As a possible implementation, performing image fusion on the registered fluorescence image and the white light image marked as the same group based on the deep learning method to generate a fused image includes:
[0054] Constructing a target deep learning model; wherein the target deep learning model is constructed based on multiple groups of paired fluorescence images and white light images in a historical data set, and the labeled ideal fusion results corresponding to each group of paired fluorescence images and white light images;
[0055] Preprocessing the registered fluorescence image and white light image of the same group to be fused; wherein the preprocessing includes: converting the original image into a grayscale image and scaling the pixel value to between 0 and 1;
[0056] For the pre-processed registered fluorescence images and white light images of the same group, image fusion is performed through the target deep learning model to generate a fused image.
[0057] As a possible implementation, the target deep learning model is a convolutional neural network model with two input branches, wherein:
[0058] Each of the input branches includes two convolutional layers for processing one of the fluorescence image and the white light image.
[0059] The convolution layer is further used to: extract a first target feature of the processed image by applying at least one filter, and send the first target feature to the fusion layer; wherein the first target feature is automatically obtained by the convolution neural network model through a back propagation algorithm during the training process;
[0060] The fusion layer is used to integrate the first target features from the two input branches by weighted averaging.
[0061] As a possible implementation, it also includes:
[0062] The water mist removal unit is used to detect the water mist humidity of the target area and perform water mist removal processing when the water mist humidity of the target area is greater than a preset water mist threshold.
[0063] The water mist removal unit comprises:
[0064] a humidity detection unit, configured with at least one humidity sensor, for detecting the mist humidity of the target area, and outputting a release electrical signal to the demisting unit when the mist humidity of the target area is greater than a preset mist humidity; wherein the preset mist humidity is set based on the feedback information of the user;
[0065] The demisting unit comprises: a controller and a MOSFET; wherein, after receiving the electrical signal released by the humidity detection unit, the controller uses the MOSFET as an electronic switch to remove water mist.
[0066] In a second aspect, the present application also provides a dual-light source fluorescent nasal skull base mirror image processing device, comprising:
[0067] An illumination module, provided with at least one fluorescent light source and at least one white light source, each light source being equipped with an independent control mechanism capable of adjusting the light source parameters of each light source individually or jointly based on image analysis requirements;
[0068] Imaging modules, including a fluorescence imaging module and a white light imaging module, each module is equipped with an independent image sensor and an optical filtering system for capturing images of target areas illuminated by each light source of the illumination module;
[0069] a signal conversion module, configured with a signal conversion device, for converting the captured optical signal into a digital signal to generate image data of the target area;
[0070] Image processing module, with built-in image processor and software, used to process image data from fluorescence imaging module and white light imaging module, and optimize the composite image using algorithm;
[0071] The control interface provides a user operation interface, allowing the user to adjust the parameters of the lighting module, imaging module and image processing module.
[0072] The device also includes: a defogger module; the defogger module includes: a controller, a humidity detection device, a MOSFET and a silica gel hygroscopic material; wherein, after receiving the electrical signal released by the humidity detection device, the controller uses the MOSFET as an electronic switch to control the current to pass through the silica gel hygroscopic material to remove the water mist.
[0073] Compared with the prior art, the beneficial effect of the present application is that the dual-light source fluorescence nasal skull base mirror image processing system is used to optimize the light source imaging and image fusion. In the imaging part, a dual-light source system is formed for the fluorescent light source and the white light source, and the position, intensity, angle and focal length of the light source are optimized by using the evaluation function and the simulated annealing algorithm, so as to realize the real-time optimization and automatic adjustment of the light source parameters and ensure the best lighting effect and image quality. In the image fusion part, the corner detection algorithm and the homography transformation matrix are used to realize the high-precision registration of the fluorescent image and the white light image, and the registered images are fused based on the deep learning technology to generate a high-quality image that combines the high brightness characteristics of the diseased tissue and the overall anatomical structure. In addition, the system and the device also add a water mist removal function to optimize the imaging effect. This comprehensive processing and adjustment system of light source imaging and image fusion improves the image quality during surgery, helps doctors better plan and operate the surgery, reduces damage to normal tissues, and improves the success rate of surgery. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solutions of the present application. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0075] Figure 1 A schematic diagram of the structure of a dual-light source fluorescent nasal skull base mirror image processing system provided in an embodiment of the present application;
[0076] Figure 2 A schematic diagram of the structure of an imaging unit provided in an embodiment of the present application;
[0077] Figure 3 A schematic diagram of the structure of an adjustment unit provided in an embodiment of the present application;
[0078] Figure 4 A schematic diagram of the structure of a water mist removal unit provided in an embodiment of the present application;
[0079] Figure 5 A schematic structural diagram of a dual-light source fluorescent nasal skull base mirror image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0081] Example 1
[0082] See also Figure 1As shown, it is a structural schematic diagram of a dual-light source fluorescent nasal skull base mirror image processing system provided by an embodiment of the present application. The present application provides a dual-light source fluorescent nasal skull base mirror image processing system, which utilizes at least one fluorescent light source and at least one white light source in the lighting unit 60 to illuminate the target area respectively and form a light path in the nasal skull base; utilizes the imaging unit 10 to collect light from the irradiated target area, and the imaging unit 10 can collect white light and fluorescent light independently or simultaneously; the signal conversion unit 20 receives the fluorescent signal and white light signal reflected by the target area after irradiation by the fluorescent light source and the white light source, and converts them into digital signals and sends them to the image processing unit 30; the image processing unit 30 processes the digital signal sent by the signal conversion unit 20, generates a fluorescent image and a white light image, and determines the tumor boundary through fusion analysis; the interactive unit 40 is used to collect user feedback information, and send the feedback information to the imaging unit 10 and the image processing unit 30, so as to allow the user to perform more flexible image processing.
[0083] For lighting unit 60:
[0084] The lighting unit 60 is provided with at least one fluorescent light source and at least one white light source, which are used to illuminate the target area respectively; forming a light path in the nasal skull base; wherein the fluorescent light source is used to emit light of a target wavelength to excite the fluorescent substance in the target area to emit light, and the white light source is used to provide a broadband spectrum to illuminate the surgical area;
[0085] The nasal skull base refers to the area where the top of the nasal cavity and the skull base meet in the human anatomy, including the nasopharynx and part of the skull base. This area is located in the middle of the head, connecting the nasal cavity and the skull cavity, and is a common operation area in otolaryngology and neurosurgery. The nasal skull base has a complex structure and contains important blood vessels and nerves. It is a common site for nasal, sinus and skull base lesions such as tumors and infections. The surgical operation is difficult and requires high-precision image guidance.
[0086] In medical imaging, fluorescent substances can selectively mark diseased tissues, such as tumors. By irradiating excitation light of a specific wavelength, the fluorescent substance emits light, highlighting the target area and helping doctors observe the lesion more clearly.
[0087] In specific implementations, using a fluorescent light source to emit light of a target wavelength to excite fluorescent substances in the target area to emit light is not only applicable to tumor boundary identification during tumor surgery, but can also be used to identify lesion areas within blood vessels in atherosclerotic plaques. In infectious diseases, specific fluorescent probes can be bound to pathogens or infection sites to help identify and locate the source of infection. The marking of inflammatory sites can be targeted to inflammatory cells or molecules through fluorescent dyes, making the inflammatory area clearly displayed in the image.
[0088] During surgery, a broadband white light source can provide good brightness and color reproduction, ensuring adequate illumination of the surgical area, allowing doctors to clearly see the color and structure of the tissue, and improving the accuracy and safety of the surgery.
[0089] In a specific implementation, there may be multiple fluorescent light sources and white light sources in the lighting unit. The system and device in the embodiments of the present application will be described in the application scenario of one fluorescent light source and one white light source, but it can be understood that if the method provided in the present application is only used to expand the imaging range and increase the number of fluorescent or white light sources to improve the imaging effect, it can still be applied.
[0090] Regarding the imaging unit 10: used to collect light from the irradiated target area, the imaging unit 10 can collect white light and fluorescent light independently or simultaneously;
[0091] In a specific implementation, the optical components used in the imaging unit 10 include, but are not limited to, multi-layer coated lenses and high-efficiency optical filters. The lenses in the imaging unit 10 use multi-layer coating technology, which can maximize the transmittance of light and reduce the reflection and absorption of light to ensure good light transmittance within the wide frequency range of the fluorescence excitation wavelength and white light. For the capture of fluorescence signals, the imaging unit 10 is equipped with filters of specific wavelengths, which can effectively prevent the passage of light of non-fluorescent wavelengths while ensuring the maximum transmission of fluorescence signals. Filter materials with high transmittance and low self-luminescence characteristics can be selected to further improve the imaging quality. The photosensitive layer of the sensor of the imaging unit 10 is optimized for the spectral characteristics of fluorescence and white light to ensure that high sensitivity and wide dynamic range image capture can be achieved under different lighting conditions.
[0092] Among them, the filters used in the imaging unit 10 can isolate or pass both fluorescence and white light signals when needed. These filters can be customized to allow fluorescence signals of specific wavelengths to pass while also passing white light, or to selectively pass certain light by adjusting the filter configuration. The design can include a mechanical or electronic switching mechanism that allows the user to choose to use fluorescence or white light sources independently, or to use both at the same time as needed.
[0093] As an optional implementation, see Figure 2 FIG. 1 is a schematic diagram of the structure of an imaging unit provided in an embodiment of the present application. Each light source in the imaging unit 10 is used to act on the target area independently or in combination. The imaging unit 10 also includes:
[0094] The light source optimization unit 11 is used to optimize the light path parameters of the light path in the target area based on a preset target evaluation function; wherein the optimization of the light path parameters of the target area is achieved by adjusting the light source parameters of each light source in the lighting unit 60.
[0095] In a specific implementation, various types of light sources are not limited to being integrated in the same position or controlled by the same control system. Each light source in the imaging unit 10 can be separated individually to illuminate the target area. This means that for scenes where no fluorescent light source is required, only a white light source can be used to provide a broadband spectrum to illuminate the surgical area. Similarly, for certain specific tasks, only a fluorescent light source can be used to emit light of a target wavelength to excite the fluorescent substance in the target area to emit light.
[0096] For example, when examining intravascular atherosclerotic plaques, the plaques are distinguished from the normal blood vessel wall. A specific fluorescent probe is used to mark the plaque area, and only the fluorescent light source is turned on for irradiation. The plaque area emits obvious fluorescence, which forms a sharp contrast with the normal blood vessel wall, making it easier for doctors to identify and evaluate the plaque condition.
[0097] For example, in conventional surgery, clear visible light illumination is required to observe the anatomical structure of the surgical area. Only white light sources are turned on to provide a wide-spectrum light to illuminate the surgical area. The surgical area is bright, the color and structure of the tissue are clearly visible, and the doctor can perform the surgical operation accurately. In addition, in conventional endoscopic examinations, doctors need clear visible light images to examine lesions or abnormalities. Only white light sources are used for illumination, providing comprehensive visible spectrum illumination. The target area is clearly displayed under white light illumination, which is convenient for doctors to observe and diagnose various lesions or abnormalities.
[0098] In a specific implementation, the embodiment of the present application sets a light source optimization unit 11, which is a component for adjusting and optimizing light path parameters to ensure that the light transmission, image clarity and lighting uniformity in the nasal skull base mirror system are in the best state. The target evaluation function is a mathematical model used to measure and evaluate the effect of the light path configuration. In the setting of the target evaluation function, multiple key parameters can be comprehensively considered, such as the length of the light transmission path, image clarity and lighting uniformity, and these parameters can be converted into a comprehensive score, called the evaluation function value.
[0099] As an optional implementation, the light source parameters include: light source position, light source intensity, light source angle and light source focal length; the light path parameters include: light transmission path length, image clarity and lighting uniformity; the mathematical expression of the target evaluation function is:
[0100] ;
[0101] in, represents the length of the light transmission path, Represents image clarity, Represents the uniformity of illumination, , , It is a pre-set weighting factor based on medical needs. It is the attenuation coefficient that characterizes the loss of light during transmission. is the gain factor of image clarity, Indicates the standard deviation of light intensity;
[0102] in, , , As weight factors, they reflect the length of the light transmission path during surgery. , image clarity , represents the uniformity of illumination For example, in brain tumor surgery, image clarity can be given the highest weight. =0.5, because in brain tumor surgery, whether the tumor boundary can be accurately identified directly affects the surgical results; lighting uniformity is also critical and can be given a weight = 0.3, ensuring that there is no visual blind spot in the surgical area. The weight of the path length is set to = 0.2, because the distance has been set close enough, its effect is relatively small.
[0103] For attenuation coefficient , a lower attenuation coefficient means less light loss in short-path transmission, which is consistent with a closer light source distance; for the gain coefficient , a higher gain coefficient emphasizes the importance of image clarity, ensuring that the contribution of clarity improvement to surgical outcomes is fully reflected during image processing.
[0104] It is understandable that the above parameters can be adjusted in a targeted manner in combination with specific surgical tasks, and the specific setting method of weights and coefficients is not limited to the above description.
[0105] In a specific implementation, light source parameters refer to the parameters of the light source itself that emits fluorescence or white light, rather than the parameters of the emitted light or light path; wherein, the light source position refers to the position of the light source in three-dimensional space, represented by coordinates (x, y, z). Adjusting the light source position can change the incident angle and path length of the light, thereby affecting the lighting effect of the target area and the clarity of the image. In the optimization of the light source position, the target evaluation function can be used to find the best position so that the light can evenly cover the target area and reduce shadows and light losses. The light source intensity is the intensity of the light emitted by the light source, which can be measured by power or brightness. Adjusting the light source intensity can directly affect the brightness of the target area and the contrast of the image. In the optimization of the light source intensity, the target evaluation function can be used to find the best light source intensity. The source intensity can provide sufficient brightness without causing overexposure or glare; the light source angle is the direction angle of the light emitted by the light source, which is the angle relative to the reference line (for example, perpendicular to the observation plane). Adjusting the light source angle can change the incident angle and coverage of the light, thereby affecting the lighting uniformity and the lighting effect of the target area. In the optimization of the light source angle, the objective evaluation function can be used to select an appropriate angle so that the light can evenly illuminate the target area and reduce shadows and bright spots; the light source focal length is the focal length of the lens in front of the light source, which determines the degree of focusing or divergence of the light. Adjusting the focal length can change the convergence or divergence characteristics of the light, thereby affecting the spot size and brightness distribution of the target area. The objective evaluation function can be used to select a suitable focal length so that the light can be optimally focused or dispersed to achieve the desired lighting effect and image clarity.
[0106] The optical path parameters are the parameters of the optical path emitted by the fluorescent light source and the white light source, among which the light transmission path length is the distance that the light is transmitted from the light source to the target area and then to the imaging sensor. The path length affects the light transmission efficiency and energy loss. A shorter path usually means higher light transmission efficiency and less energy loss. This can be adjusted by the target evaluation function to the light source parameters mentioned above, so as to obtain a suitable path length to ensure high light transmission efficiency while reducing energy loss. Image clarity is the clarity of the image, which can be measured by the image contrast, edge sharpness or resolution. High-definition images help doctors observe the details and structures of the target area more accurately. This can be adjusted by the target evaluation function to the light source parameters mentioned above, so as to improve the clarity of the image and make the details of the target area clearer and sharper.
[0107] Illumination uniformity refers to the degree of uniform distribution of light in the target area, which is usually measured by the standard deviation of light intensity. Uniform lighting helps to eliminate shadows and bright spots and provide consistent imaging effects. This can be achieved by adjusting the light source parameters mentioned above through the target evaluation function to ensure high lighting uniformity and reduce uneven brightness.
[0108] In specific implementation, by adjusting these light source parameters (position, intensity, angle, focal length) and optical path parameters (transmission path length, image clarity, lighting uniformity), the light source optimization unit can optimize the optical path configuration of the nasal skull base endoscope in real time to ensure that the system provides the best imaging quality and lighting effects in various surgical scenarios. These optimizations not only improve the performance of the imaging system, but also improve the safety and accuracy of the surgery.
[0109] For example, in brain base tumor surgery, the surgical area is usually very small and located deep, and very precise operations are required to avoid damaging important nerves or blood vessels. The present application can be used to adjust the fluorescent light source closer to the surgical area to improve the intensity and focus of the fluorescent lighting and enhance the recognition of the fluorescence of tumor markers; the emission angle of the light source can also be adjusted to reduce shadows in the surgical scene and ensure uniform lighting; the intensity of the fluorescent light source can also be increased to highlight the fluorescent markers in the tumor area, while adjusting the white light intensity to ensure sufficient visibility.
[0110] For example, when performing sinus surgery, it is necessary to clearly distinguish between diseased tissue and normal tissue. The focus of the light source can be adjusted according to the depth of the surgical area to ensure clear imaging of the surgical area. The intensity of the fluorescent and white light sources can also be adjusted to make the contrast between diseased tissue and normal tissue more obvious, facilitating precise resection during surgery.
[0111] For example, when examining inflammatory diseases such as the nasopharynx, it is necessary to observe the condition, position and angle of the inflamed area in detail: adjust the position and angle of the light source to optimize the illumination of the inflamed area, especially in areas where the field of view may be limited; the light source intensity can also be adjusted according to the reflective characteristics of the inflamed area to avoid overexposure or too dark a view, ensuring that details can be seen clearly.
[0112] As an optional implementation manner, the functional expression of the illumination uniformity a is:
[0113] ;
[0114] in, and represent the cosine values of the angles between the light of the fluorescent light source and the light of the white light source and the imaging plane, respectively, and Respectively represent the light source intensities of the fluorescent light source and the white light source;
[0115] It can be understood that the above functional expression of illumination uniformity a is applicable to the case where the number of light sources in the lighting unit 60 is 2, and they are one fluorescent light source and one white light source. As a possible implementation, the number of light sources in the lighting unit 60 may be greater than 2, and the number of fluorescent light sources and white light sources may not be the same.
[0116] Exemplarily, the total number of light sources in the lighting unit 60 may be 3, wherein the number of fluorescent light sources is 2 and the number of white light sources is 1, then the functional expression of the lighting uniformity a may be adjusted to:
[0117] ;
[0118] Wherein, f1 and f2 represent the first fluorescent light source and the second fluorescent light source respectively. and denote the light source intensities of the first fluorescent light source and the second fluorescent light source respectively, and and Respectively represent the cosine values of the angles between the light of the first fluorescent light source and the second fluorescent light source and the imaging plane.
[0119] The function expression of image clarity i is:
[0120] ;
[0121] in, and are the imaging clarity of the fluorescent light source and the white light source, respectively, and is the corresponding weight factor.
[0122] In a specific implementation, the functional expression of the above image clarity i can also be adaptively adjusted according to the change in the number of light sources. The specific adjustment method can refer to the adjustment method of the above illumination uniformity a, which will not be repeated here.
[0123] The weight factor determines the contribution of each light source in the final image clarity calculation. The weight setting represents the relative importance of fluorescence imaging and traditional white light imaging in a specific procedure or examination.
[0124] For example, in a surgery that requires a high degree of identification of diseased tissue, the weight of fluorescence imaging may be higher. In a routine examination or surgery, the weight of a white light source may be higher because a wider area needs to be clearly seen.
[0125] By adjusting and , the system can optimize image output in specific surgical or diagnostic scenarios. For example, in tumor resection surgery, increasing (The weight of fluorescence) can help doctors see the marked tumor boundaries more clearly, and in general endoscopic surgery, increasing (The weight of white light) helps provide a clearer view.
[0126] For example, when performing tumor margin identification surgery, it is important to identify the exact location and boundaries of the tumor, especially in brain or nasal skull base surgery. At this time, the role of fluorescence imaging is to highlight the fluorescently labeled tumor cells. In the imaging clarity setting, Can be set very high, as tumor cells need to be viewed precisely to ensure that minute details are captured; It can be set to medium to provide enough background detail to allow doctors to identify the boundary between tumor and normal tissue. In the weight factor setting, fluorescence imaging can be given a higher , such as 0.7, because it is critical for identifying tumors; lower , for example, 0.3 reflects that white light imaging is mainly used to provide background and auxiliary information.
[0127] For example, during routine sinus surgery, a good overall view is required to observe and operate the internal structures of the sinuses. In this case, the role of white light imaging is even more important.
[0128] In the image clarity setting, Can be set to low, as over-reliance on fluorescence imaging may not be necessary in routine surgery. Can be set very high to ensure that the doctor can see the entire structure of the sinus, including subtle anatomical features; in the weight factor setting: can be set lower , for example 0.2, because fluorescence imaging is not the main visual tool in this scenario, you can set , such as 0.8, emphasizes the importance of white light imaging in routine surgery to ensure adequate illumination and clarity.
[0129] As an optional implementation, see Figure 3 As shown, it is a structural schematic diagram of an adjustment unit provided in an embodiment of the present application, and the imaging unit 10 also includes: an adjustment unit 12, which is used to adjust the light source parameters of each light source in the lighting unit 60; wherein, the adjustment unit 12 includes: a mechanical arm unit 121, which is used to move and position the light source to ensure the adjustability of the light source position; an angle adjustment unit 122, including a servo motor and an angle sensor, which is used to adjust the emission angle of the light source; an intensity control unit 123, including a digital regulator and an intensity sensor, which adjusts the luminous intensity of each light source through a control interface; a focal length adjustment unit 124, including an automatic zoom lens and a focal length adjustment motor, which is used to adjust the focal length of the light source, and the automatic zoom lens is fixedly installed on the light path in front of the light source, and the position relationship between the light source and the automatic zoom lens can be adjusted through the focal length adjustment motor.
[0130] The imaging unit 10 further includes: an adjusting unit 12, which is used to adjust the light source parameters of each light source in the lighting unit 60; wherein the adjusting unit 12 includes:
[0131] The mechanical arm unit 121 includes a mechanical arm having at least one joint and a telescopic part, wherein the mechanical arm moves the light source in multiple directions and angles based on an input control signal, and by presetting an initial position of the mechanical arm, the light source substantially covers the entire target area;
[0132] Angle adjustment unit 122, including a servo motor and an angle sensor; wherein the servo motor adjusts the emission angle of the light source based on the input control signal, and the angle sensor detects the angle information of the current light source and feeds back to the control system to ensure the accuracy of the angle setting;
[0133] The intensity control unit 123 includes a digital regulator and an intensity sensor; wherein the digital regulator adjusts the current or voltage of the light source based on the input control signal to change the luminous intensity of the light source; the intensity sensor detects the brightness of the current light source and feeds back to the control system;
[0134] The focus adjustment unit 124 includes an automatic zoom lens and a focus adjustment motor; wherein the automatic zoom lens is fixedly installed on the light path in front of the light source, and the focus adjustment motor adjusts the relative position between the light source and the automatic zoom lens based on the input control signal to change the focusing or divergence characteristics of the light.
[0135] In a specific implementation, the mechanical arm unit 121 in this embodiment is used to move and position the light source to ensure the adjustability of the light source position. The mechanical arm can be used to accurately position the light source at a specific position in the surgical area. The mechanical arm can be composed of multiple joints and telescopic parts, and can move the light source in multiple directions and angles; the angle adjustment unit 122 is used to adjust the emission angle of the light source, including a servo motor and an angle sensor. The servo motor is responsible for accurately adjusting the angle of the light source, and the angle sensor feeds back the angle information of the current light source to ensure the accuracy of the angle setting; the intensity control unit 123 is used to adjust the luminous intensity of each light source through a control interface, including a digital regulator and an intensity sensor. The digital regulator allows the user or the automatic control system to accurately adjust the current or voltage of the light source, thereby changing the brightness of the light source. The intensity sensor provides feedback to ensure that the set brightness meets the surgical requirements; the focal length adjustment unit 124 is used to adjust the focal length of the light source, wherein the automatic zoom lens is fixedly installed on the optical path in front of the light source, and the focal length adjustment motor adjusts the relative position between the light source and the zoom lens, thereby changing the focus range. This allows the light to be concentrated or diffused as needed to optimize the illumination of the surgical area.
[0136] For example, the robotic arm unit 121 can be installed on a stable support next to the operating table to ensure that it can cover the entire operating area. Before the operation, the initial position of the robotic arm is preset through the system control interface so that the light source can basically cover the entire predetermined operating area. The robotic arm should be equipped with sufficient joints and telescopic functions to adjust the precise position of the light source.
[0137] The angle adjustment unit 122 should be mounted directly near the light source, allowing the servo motor to control the rotation and tilt of the light source. The system interface should allow the doctor or technician to input the optimal angle of the light source, and the servo motor should adjust the light source based on this data to best illuminate the critical surgical area. The angle sensor provides real-time feedback to ensure that the light source angle reaches the preset value.
[0138] The intensity control unit 123 should be integrated into the light source management system so that it can control the current or voltage of each light source through a digital regulator. The light intensity of each light source is preset through the operating room console. During the operation, according to actual needs, the doctor can adjust the brightness of the light source to improve the visual effect. The intensity sensor ensures that the actual light intensity is consistent with the set value.
[0139] The focus adjustment unit 124 should be installed between the light source and the automatic zoom lens. The focus adjustment motor is connected to the automatic zoom lens to adjust the distance between the light source and the lens. Before the operation begins, the focus is preset according to the type of operation and the expected visual needs. The doctor can adjust the focus through the control interface to adapt to the visual needs of different surgical stages and ensure that the surgical area is always in the best focal plane.
[0140] As an optional implementation, the light source optimization unit 11 is used to optimize the light path parameters by using a simulated annealing algorithm in the process of adjusting the light path parameters of the nasal skull base mirror based on the target evaluation function. The algorithm aims to maximize the value of the target evaluation function, thereby determining the optimal light path parameters and generating corresponding target light source parameters;
[0141] As an optional implementation, the imaging unit 10 further includes: a control unit 13, configured to send a control signal to the adjustment unit 12 based on the target light source parameters and user feedback information, and perform at least one of the following steps to adjust the light source parameters:
[0142] Based on the control signal, the mechanical arm unit 121 is controlled to adjust the position of the light source to the target light source position, and the position status is fed back in real time;
[0143] The angle adjustment unit 122 adjusts the emission angle of the light source through the servo motor and the angle sensor based on the control signal, and feeds back the current angle value through the angle sensor;
[0144] The intensity control unit 123 adjusts the current or voltage of the light source through a digital regulator based on the control signal, adjusts the intensity of the light source, and feeds back the current intensity of the light source in real time through an intensity sensor;
[0145] Based on the control signal, the focus adjustment unit 124 moves forward and backward along the optical axis through the focus adjustment motor to adjust the distance between the light source and the zoom lens, and provides real-time feedback of the current focus state.
[0146] When using the simulated annealing algorithm, parameter adjustments include not only the position and focal length of the light source, but also the relative intensity and angle of the two light sources. Doing so can maximize the visual contrast of tumor tissue while maintaining the clarity and details of the overall image.
[0147] In the specific implementation, in order to use the simulated annealing algorithm to optimize the light source position, focal length, intensity and angle of the dual-light source nasal skull base mirror to ensure that the visual contrast of the tumor tissue is maximized and the clarity and details of the overall image are maintained, the fluorescent light source position can be set to and the white light source position is ; Fluorescent light source angle and white light source angle ; Fluorescence light source intensity and white light intensity Fluorescent light source focal length and focal length of white light source .
[0148] Initialization based on the simulated annealing algorithm principle, setting a high initial temperature T and cooling rate. Random initialization parameters and .
[0149] In each iteration, one or more parameters are randomly selected and subjected to small random perturbations.
[0150] Calculate the new value of the evaluation function F:
[0151] ;
[0152] Whether to accept the new parameter configuration is determined based on the difference between the old value of the evaluation function F and the new value of the evaluation function F and the temperature T.
[0153] Gradually reduce the temperature and repeat the above steps until the stop condition is met (such as the temperature is lower than the threshold or the maximum number of iterations is reached). During the whole process, track and record the parameter settings that achieve the maximum F value and output the target light source parameters.
[0154] In a specific implementation, the control unit 13 can be integrated into the imaging unit 10 of the system to ensure real-time communication and coordination with various adjustment units (mechanical arm unit 121, angle adjustment unit 122, intensity control unit 123, focal length adjustment unit 124).
[0155] The control unit 13 initializes the control signal and parameter settings based on the target light source parameters and the user's feedback information. When the system starts, the control unit 13 reads the preset parameter configuration from the system's memory, or initializes according to the target light source parameters input in real time. Based on the target light source parameters, the control unit 13 sends a control signal to the adjustment unit 12, and dynamically adjusts the light source parameters according to the actual situation during the operation and the user's real-time feedback to ensure the best imaging effect.
[0156] In summary, the control unit 13 ensures that the system can efficiently and accurately adjust the light source parameters during the operation, provide the best lighting and imaging quality, and improve the accuracy and safety of the operation.
[0157] Through this method, the parameters of the dual-light source nasal skull base endoscope can be effectively adjusted to achieve the best visual effect required during surgery and diagnosis. This method not only helps to achieve accurate imaging, but also can be flexibly adjusted according to the needs of actual surgical scenarios.
[0158] The above content can be regarded as a nasal skull base endoscope imaging system with dual light sources of fluorescence and white light. Accurate fluorescence images and white light images can be obtained through the above system method. The fusion imaging system of these two images will be explained below.
[0159] For the signal conversion unit 20:
[0160] The signal conversion unit 20 is used to receive the fluorescent image and the white light image reflected by the target area after being irradiated by the fluorescent light source and the white light source, and convert them into digital signals and send them to the image processing unit;
[0161] In a specific implementation, the main function of the signal conversion unit 20 in the fluorescent skull base endoscope image processing system is to convert the light signals (fluorescent and white light signals) received from the surgical area into digital signals that can be processed by the image processing unit 30 .
[0162] The signal conversion unit 20 first receives the light signals reflected by the target area after being illuminated by the fluorescent light source and the white light source. These light signals include the fluorescence emitted by the specific biomarkers labeled with fluorescent markers and the ordinary white light reflection signals. The fluorescent signal is usually used to highlight specific biological or chemical substances, such as tumor cells, while the white light signal provides a general view of the surgical area.
[0163] The received optical signal needs to be converted into a digital signal, which can be accomplished by a set of photoelectric conversion devices such as photodiodes or photomultiplier tubes. The converted digital signal can be processed by a computer system for image reconstruction, analysis and display. The digital signal is sent to the image processing unit 30, where it will be further processed to generate high-quality images for doctors' diagnosis and surgical navigation.
[0164] Exemplarily, the main function of the signal conversion unit 20 is to receive the optical signal reflected by the surgical area after being illuminated by the fluorescent light source and the white light source, and convert it into a digital signal for further processing by the image processing unit 30. To this end, it is first necessary to install high-sensitivity photodiodes at key locations in the surgical area. The location selection of these sensors is intended to maximize the capture of direct and reflected light from the light source to ensure full coverage of the surgical area. In order to improve the quality of the received optical signal, a set of signal amplification and filtering systems can be configured. The goal of this system is to enhance the strength of the signal and reduce the interference of environmental noise, thereby optimizing the clarity and availability of the signal. In terms of signal conversion, a digital signal processor (DSP) is used to receive the analog signal output by the photoelectric sensor. The DSP is responsible for converting the analog signal into a digital format and performing preliminary processing such as signal enhancement and denoising. A communication interface is designed to transmit the processed digital signal to the image processing unit 30. Optical fiber or high-speed USB connection can be selected as the transmission medium, because these transmission media can provide high-speed data transmission while reducing interference and ensuring data integrity and security.
[0165] For the image processing unit 30:
[0166] The image processing unit 30 is used to process the digital signal sent by the signal conversion unit 20, generate a fluorescent image and a white light image, and determine the tumor boundary through fusion analysis;
[0167] As an optional implementation, the image processing unit 30, when processing the digital signal sent by the signal conversion unit 20 to generate a fluorescence image and a white light image, and determining the tumor boundary by fusing and analyzing the two images, is used to: preprocess the fluorescence image and the white light image; based on a corner detection algorithm, perform image registration on the fluorescence image and the white light image at the same time, and mark the registered fluorescence image and the white light image at the same time as the same group; based on a deep learning method, perform image fusion on the registered fluorescence image and the white light image marked as the same group to generate a fused image; based on user feedback information, perform optimization processing on the fused image to determine the tumor boundary;
[0168] In a specific implementation, the image processing unit 30 of the fluorescent nasal skull base endoscope imaging processing system has the ability to fuse and analyze fluorescent images and white light images. The fluorescent image highlights specific biomarkers (such as tumor cells) through light irradiation of a specific wavelength, while the white light image provides a detailed anatomical view of the surgical scene. By fusing these two images, doctors can see the fluorescently marked tumor and the surrounding normal tissue structure in one image at the same time, which helps to define the tumor boundary more accurately. The fused image enables doctors to more accurately distinguish between tumors and normal tissues during surgical cutting, thereby improving the success rate and safety of the operation and reducing damage to normal tissues.
[0169] Among them, preprocessing steps such as denoising and contrast enhancement can significantly improve image quality, especially in images taken in low-light or high-noise environments. This is the basis for ensuring the effectiveness and accuracy of the image fusion process. Preprocessing can also include adjusting the size and resolution of the image so that the fluorescence and white light images have a consistent format before fusion. Doing so can reduce possible errors and deviations in subsequent processing steps.
[0170] Image registration is the process of aligning images taken at different time points or from different perspectives into the same coordinate system. For fluorescence and white light images, although both images are taken of the same surgical area, there may be differences in angle and position due to the position of the light source or the movement of the surgical instrument. Registration ensures the spatial correspondence between the two images, so that the fused image is accurate in anatomical structure. Good image registration is a prerequisite for high-quality image fusion. Only when the fluorescence and white light images are accurately aligned can the fused image accurately reflect the boundary between the tumor and normal tissue, thereby providing important surgical navigation information.
[0171] As an optional implementation, the corner detection algorithm includes: Harris corner detection algorithm; based on the corner detection algorithm, performing image registration on the fluorescence image and the white light image at the same time, including:
[0172] The Harris corner response function is defined as:
[0173] ;
[0174] Among them, R is the Harris corner response value, M is the gradient covariance matrix of the local area of the image,
[0175] det(M) represents the determinant of the matrix M, which reflects the directional dispersion of the gradient. trace(M) represents the trace of the matrix M, which reflects the total amount of the gradient. k is the sensitivity constant for corner detection. This definition can determine the corners by balancing the dispersion and total amount of local image gradients.
[0176] Based on the ORD algorithm, feature points are extracted from the fluorescence image and the white light image at the same time for matching, so as to establish a first corresponding relationship;
[0177] Based on the first corresponding relationship, generating a homography transformation matrix between the two images at the same moment;
[0178] The calculated homography transformation matrix between the two images at the same time is applied to any one of the two images to align it to the coordinate system of the other image, thereby achieving registration of the fluorescence image and the white light image.
[0179] Among them, the Harris corner detection algorithm is one of the widely used feature point detection algorithms in image processing. Corner points are points where two edges in an image meet, and usually have a higher local change intensity, so they are suitable feature markers. In the process of image registration, these corner points can serve as significant landmarks to help accurately match different images. Corner points are usually located in information-intensive areas of the image and are ideal features for distinguishing images and finding correspondences between images. The Harris response function is used to evaluate the possibility of each pixel in the image being a corner point. This response value is calculated by calculating the gradient covariance matrix of the local area of the image. It can be concluded that it reflects the strength of the corner point characteristics of this point.
[0180] The ORD (Oriented FAST and Rotated BRIEF) algorithm is a feature point description and matching algorithm used to quickly and efficiently find corresponding feature points between images. Compared with other feature description algorithms (such as SIFT or SURF), the ORD algorithm is usually better in computing speed and matching efficiency, especially in medical scenarios with high real-time processing requirements. The ORD algorithm can be used to quickly extract features from the corner points detected by Harris and establish the correspondence between the two images.
[0181] The homography transformation matrix is a mathematical model used for image registration. It describes a transformation that can map points in one image to corresponding points in another image, such as mapping points in a fluorescence image to corresponding points in a white light image, or mapping corresponding points in a white light image to a fluorescence image. Homography transformations include rotation, translation, scaling, and perspective transformations. The homography transformation matrix is solved by minimizing the error (such as Euclidean distance) between point pairs before and after the transformation to ensure the accuracy and stability of the registration. This transformation can accurately align the two types of images when processing the registration of fluorescence images and white light images, helping doctors better identify and locate tumor boundaries.
[0182] In a specific implementation, fluorescence images and white light images can be collected simultaneously during surgery. Ensure that the two images are taken at the same time to maintain time consistency. Perform preprocessing operations on the collected fluorescence images and white light images, such as denoising, brightness and contrast adjustment, to improve image quality and prepare for subsequent processing. Apply the Harris corner detection algorithm independently on the fluorescence image and the white light image. Calculate the Harris corner response value of each pixel , using the response function defined previously:
[0183] ;
[0184] Pixels with response values higher than the preset threshold are selected as corner points. The ORD algorithm is used to extract local feature descriptors at the corner points of the two images, and the feature points in the fluorescence image and the white light image are matched according to the similarity of the descriptors to establish the correspondence between the feature points.
[0185] As a possible implementation manner, the correspondence between the feature points is regarded as a first correspondence in the application, and the first correspondence includes: n pairs of matching points between two images at the same time; based on the first correspondence, generating a homography transformation matrix between the two images at the same time, including:
[0186] Based on n pairs of matching points between two images at the same time, a homography transformation model is established. For each matching point i, the following relationship exists:
[0187] ;
[0188] Among them, there are n pairs of matching points between the two images at the same time, and the corresponding point of the fluorescence image corresponding to each matching point i in the n pairs of matching points is , the corresponding point of the white light image is , arrive are the elements of the homography transformation matrix H;
[0189] Based on the matched feature point pairs, the homography transformation matrix is calculated using mathematical methods such as the least squares method.
[0190] This matrix describes a transformation by which feature points in the fluorescence image can be accurately mapped to corresponding feature points in the white light image. The calculated homography transformation matrix is applied to either the fluorescence image or the white light image to align it to the coordinate system of the other image. This step achieves spatial registration of the fluorescence image with the white light image, making the corresponding feature points of the two images consistent in position.
[0191] The registered images are fused to combine the advantages of the two images to generate a comprehensive image containing rich surgical visual information. The fused image is analyzed to determine the tumor boundary and assist the doctor in making surgical decisions. The image fusion post-processing is optimized based on the doctor's feedback, and the image processing parameters are further adjusted to improve the image quality and surgical effect.
[0192] As an optional implementation, the present application performs image fusion on registered fluorescence images and white light images marked as the same group based on a deep learning method to generate a fused image, including: constructing a target deep learning model; wherein the target deep learning model is constructed based on multiple groups of paired fluorescence images and white light images in a historical data set, and the labeled ideal fusion results corresponding to each group of paired fluorescence images and white light images; preprocessing the registered fluorescence images and white light images of the same group to be fused; wherein the preprocessing includes: converting the original image into a grayscale image and scaling the pixel values to between 0 and 1; for the preprocessed registered fluorescence images and white light images of the same group, performing image fusion through the target deep learning model to generate a fused image.
[0193] In a specific implementation, multiple sets of fluorescence images and white light images can be collected, which have been used in clinical practice and have a paired relationship. Each set of image pairs is equipped with annotated ideal fusion results, which serve as the "true value" data for training models. Among them, the ideal fusion result refers to an image generated by expert visual evaluation or image processing technology that can best display the tumor boundary and surrounding important structures. In the selection of neural network models, a suitable deep learning architecture can be selected, such as a convolutional neural network (CNN), which is good at processing image data and can extract complex features from images.
[0194] After designing the network structure, the image pairs and corresponding fusion results in the historical data set are used to train the model, so that the model can learn how to extract effective information from the fluorescence image and the white light image and generate a high-quality fused image. In addition, the original fluorescence image and the white light image can be converted into a grayscale image, because the grayscale image simplifies the data processing process and reduces the computational complexity, while still retaining the key structural information in the image. The pixel values of the grayscale image are scaled to between 0 and 1. This step is to unify the dynamic range of the image data, which helps the model learn more stably and effectively.
[0195] The preprocessed fluorescence images and white light images are input into the trained deep learning model. The model processes the input images through its network structure and automatically generates a fused image using the learned features and patterns. This fusion process takes into account the specificity of the fluorescence image (such as highlighting specific markers such as tumors) and the comprehensiveness of the white light image (providing a wide field of view and detailed anatomical information). The generated fused image combines the advantages of both images and provides richer information to help doctors more accurately identify and locate tumor boundaries during surgery.
[0196] As an optional implementation, the target deep learning model is a convolutional neural network model with two input branches:
[0197] Each of the input branches includes two convolutional layers for processing one of the fluorescence image and the white light image.
[0198] In a specific implementation, the embodiment of the present application adopts a convolutional neural network (CNN) model with two input branches, which is suitable for processing two different types of images (fluorescence images and white light images). This network structure design is intended to extract and learn specific features from each image type, and then effectively fuse these features to generate the final fused image. The model contains two independent input branches, each branch specializing in one type of image: one branch processes fluorescence images and the other branch processes white light images. The model can independently learn and extract the most useful features for each image type, thereby achieving more effective information integration while maintaining the characteristics of each image.
[0199] Each input branch contains two convolutional layers. Convolutional layers are basic components for image processing in deep learning. They extract local features in images through convolution operations. An activation function, such as ReLU, can be set after each convolution layer to increase the nonlinear processing capabilities of the model. The first convolutional layer can be set to extract finer-grained features (such as edges, textures, etc.), while the second convolutional layer can be set to unprocess higher-level combinations of these features (such as specific shapes, patterns, etc.). Each branch independently processes its corresponding image type (fluorescence or white light), extracts meaningful features from the original image data, and ensures that the unique information of each image is fully utilized.
[0200] After being processed by their respective convolutional layers, the outputs of the two branches are usually merged in the subsequent stages of the network. This fusion process can be achieved in a variety of ways, such as direct addition, splicing, etc., with the goal of integrating the features of the two images, taking advantage of the high contrast markers of the fluorescence image and the structural information of the white light image. The merged features can be further processed by additional convolutional layers or fully connected layers to learn how to most effectively combine the information from the fluorescence and white light images to ultimately generate a fused image.
[0201] The output of the model is a fused image that combines the strengths of both input images, aiming to provide doctors with a more comprehensive and clearer view, particularly in identifying and locating tumor boundaries.
[0202] This dual-input branch-based CNN model provides an efficient and targeted solution for the fusion of fluorescence and white light images, optimizes the quality and speed of image processing, and is suitable for situations where high-precision medical image processing is required.
[0203] As an optional implementation, the convolution layer is further used to: extract a first target feature of the processed image by applying at least one filter, and send the first target feature to the fusion layer; wherein the first target feature is automatically obtained by the convolutional neural network model through a back propagation algorithm during the training process;
[0204] The fusion layer is used to integrate the first target features from the two input branches by weighted averaging.
[0205] In practice, a convolutional layer is responsible for extracting features from an input image. This is done by applying multiple filters, also called convolution kernels. Each filter is specifically designed to recognize certain types of features in an image, such as edges, textures, or more complex patterns. When a convolutional layer applies its filters to an input image, it creates activations in different parts of the image, and these activations represent the presence of the features recognized by the filter. These activations constitute the first target features of the image.
[0206] During the training phase, CNN automatically adjusts the weights of the filters in its convolutional layer through the back-propagation algorithm. This process is based on the error between the actual output in the training data set and the model prediction, and the purpose is to minimize this error, thereby optimizing the filter to better capture features that are helpful for image fusion.
[0207] The main responsibility of the fusion layer is to integrate the features from the two input branches (i.e., the fluorescence image and the white light image processing branch). In this application, it is implemented by weighted averaging, where the weight of each feature can be automatically learned during training or manually set according to specific application requirements. Features from fluorescence and white light images will be merged according to their relative importance (determined by their weights). This not only retains the key information in the two images, but also enhances the accuracy of tumor recognition and localization.
[0208] It is understood that the weighted averaged features can also be further processed to generate a fused image that combines the high-contrast marking characteristics of the fluorescence image with the detailed anatomical structure of the white light image, providing doctors with a more comprehensive view for precise surgical navigation.
[0209] For example, for the setting of convolution filters, each convolution layer can be set to use 32 filters (also called convolution kernels), and the size of each filter is 3x3. The stride is set to 1, that is, the filter moves on the image with a step length of 1 pixel. The padding uses the "same" mode to ensure that the output image size remains unchanged after the convolution operation.
[0210] The initial weights of the fluorescence image features and the white light image features can be set to 0.5, which means that at the beginning of the fusion, the two types of features are considered equally important.
[0211] During model training, these weights can be adjusted through the back-propagation algorithm to find the best weight configuration so that the fused image performs best on the validation set. For example, the weight of the fluorescence image feature may be adjusted to 0.6, while the weight of the white light image feature is adjusted to 0.4, and the optimization is carried out according to the actual image content and surgical requirements.
[0212] As a possible implementation, the embodiment of the present application may further include: a water mist removal unit 50, which is used to detect the water mist humidity of the target area and perform water mist removal processing when the water mist humidity of the target area is greater than a preset water mist threshold.
[0213] The purpose of adding a mist removal unit to the fluorescent nasal skull base imaging processing system is to improve the quality and clarity of surgical images, so that doctors can make more accurate diagnoses and surgical operations. Visual obstructions caused by mist are often encountered during surgery, especially when using an endoscope for minimally invasive surgery. Moisture and mist in the surgical area can significantly reduce the clarity of the image, which is particularly common in small spaces such as sinus or skull base surgery. Mist can scatter and refract light, resulting in blurred images and difficulty observing details. Clear images are the basis for accurate analysis and surgical navigation. Mist can affect the effectiveness of image processing algorithms (such as edge detection, feature extraction, etc.), thereby affecting the overall surgical effect. Frequent interruptions to clean up mist will prolong the operation time and increase the risk to the patient. An effective mist removal system can reduce such interruptions and make the surgical process more continuous and efficient.
[0214] In a specific implementation, by detecting the water mist humidity in the target area, the system can automatically determine when to start the water mist removal process. This automated function reduces human intervention and improves the timeliness and accuracy of the process. Once it is detected that the water mist humidity exceeds a preset threshold, the water mist removal unit 50 will be activated. This can be achieved through a variety of technologies, such as heating, using air purge, ultrasonic vibration, etc. The specific method depends on the surgical environment and the configuration of the equipment.
[0215] As a possible implementation, see Figure 4 FIG. 1 is a schematic diagram of the structure of a water mist removal unit provided in an embodiment of the present application. The water mist removal unit 50 includes:
[0216] The humidity detection unit 51 is configured with at least one humidity sensor, and the humidity detection unit 51 is used to detect the water mist humidity of the target area, and when the water mist humidity of the target area is greater than the preset water mist humidity, output a release electrical signal to the defogger unit 52; wherein the preset water mist humidity is set based on the feedback information of the user.
[0217] The demisting unit 52 includes: a controller, a MOSFET and a silica gel hygroscopic material; wherein, after receiving the electrical signal released by the humidity detection unit 51, the controller uses the MOSFET as an electronic switch to control the current to pass through the silica gel hygroscopic material to remove the water mist.
[0218] In a specific implementation, the mist removal unit 50 may include a humidity detection unit 51 and a defogger unit 52. The humidity detection unit 51 is equipped with at least one humidity sensor for real-time monitoring of the humidity of the mist in the surgical target area. The humidity sensor can accurately measure the amount of water vapor in the air, thereby evaluating the mist situation in the field of view. When the detected mist humidity exceeds a preset threshold, the humidity detection unit 51 sends an electrical signal to the defogger unit 52. This preset threshold can be adjusted based on user feedback to ensure that it adapts to different surgical environments and the specific needs of doctors.
[0219] The demisting unit 52 may include a controller and a metal-oxide-semiconductor field-effect transistor (MOSFET). After receiving the electrical signal from the humidity detection unit 51, the controller controls the current through the MOSFET as an electronic switch. The MOSFET allows or blocks the current to flow, thereby activating or shutting down the demisting process.
[0220] Through this technology, the fluorescent nasal skull base endoscope image processing system can automatically remove the water mist in the field of view at critical moments, which is essential for performing delicate and high-risk operations, such as brain or skull base operations. It not only improves the safety of the operation, but also improves the efficiency and success rate of the operation.
[0221] For example, during brain tumor resection surgery, the patient is diagnosed with a brain tumor located in the inaccessible skull base area, and the surgery requires extremely high precision and clear vision. The surgery is performed using a fluorescent nasal skull base endoscope to help identify tumor tissue and important surrounding structures. The small space and abundant blood vessels in the surgical area make it easy for water mist to accumulate in the surgical environment, affecting the clarity of the surgical lens. The presence of water mist will significantly reduce the quality of the surgical image and bring risks to precise operations.
[0222] At the beginning and during the operation, the humidity sensor embedded in the surgical instrument can continuously monitor the humidity level in the surgical area. When the humidity level reaches or exceeds the preset water mist threshold, the humidity detection unit 51 detects a high humidity signal and triggers the defog process. The humidity detection unit 51 outputs an electrical signal to the controller of the defog unit 52. The controller allows or blocks the flow of current through the MOSFET, thereby activating or shutting down the defog process. As the water mist is removed, the field of view of the surgical lens quickly becomes clear. The clear field of view enables doctors to accurately identify the edge of the tumor and the surrounding important anatomical structures. Doctors can perform tumor resection more safely and accurately, reducing damage to normal tissue. The system significantly improves the success rate and safety of surgery, and the operation time is shortened due to the reduction of interruptions to clear the water mist, thereby improving surgical efficiency.
[0223] In addition, the preset water mist threshold can be adjusted according to the needs of the doctor. For example, a lower water mist threshold is required in certain operations. The doctor can input his or her needs through the interactive unit 40. After collecting the doctor's feedback information, the interactive unit 40 can send the feedback information to the water mist removal unit 50.
[0224] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0225] Based on the same inventive concept, the embodiment of the present application also provides a dual-light source fluorescent nasal skull base mirror image processing device corresponding to a dual-light source fluorescent nasal skull base mirror image processing system. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned system in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the system, and the repeated parts will not be repeated.
[0226] Example 2
[0227] See also Figure 5, which is a schematic diagram of a dual-light source fluorescent nasal skull base mirror image processing device provided by an embodiment of the present disclosure, the device includes: an imaging module 110, a signal conversion module 120, an image processing module 130, a control interface 140, a defog module 150, and an illumination module 160; wherein,
[0228] An illumination module 160, provided with at least one fluorescent light source and at least one white light source, each light source being equipped with an independent control mechanism capable of adjusting the light source parameters of each light source individually or jointly based on image analysis requirements;
[0229] An imaging module 110, including a fluorescent imaging module and a white light imaging module, each module is equipped with an independent image sensor and an optical filter system for capturing an image of a target area illuminated by each light source of the illumination module;
[0230] The signal conversion module 120 is configured with a signal conversion device for converting the captured optical signal into a digital signal to generate image data of the target area;
[0231] An image processing module 130 , having a built-in image processor and software, for processing image data from the fluorescence imaging module and the white light imaging module, and optimizing the synthesized image using an algorithm;
[0232] The control interface 140 provides a user operation interface, allowing the user to adjust the parameters of the lighting module, the imaging module and the image processing module.
[0233] It also includes a defogger module 150, which includes: a controller, a humidity detection device, a MOSFET and a silica gel hygroscopic material; wherein, after receiving the electrical signal released by the humidity detection device, the controller uses the MOSFET as an electronic switch to control the current to pass through the silica gel hygroscopic material to remove the water mist.
[0234] Silica gel is a highly efficient hygroscopic material that is often used to remove moisture from the air. In this system, when an electric current is passed through the hygroscopic silica gel material, it can accelerate its hygroscopic effect, thereby effectively removing water mist from the surgical area. This process is not only fast but also environmentally friendly because silica gel is a very safe substance and is harmless to the human body.
[0235] In a specific implementation, the controller allows or blocks the flow of current through MOSFET to control the silica gel hygroscopic material, and the current passes through the silica gel to accelerate its hygroscopic effect. The silica gel hygroscopic material is designed as a ring structure surrounding the lens to ensure that the water mist in front of the lens can be removed evenly and quickly.
[0236] Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0237] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned system embodiments and will not be repeated here.
[0238] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0239] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A dual-light source fluorescent nasal skull base mirror image processing system, characterized in that: include: An illumination unit is provided with at least one fluorescent light source and at least one white light source, for irradiating a target area; wherein the fluorescent light source is used to emit light of a target wavelength to excite the fluorescent substance in the target area to emit light, and the white light source is used to provide a broadband spectrum to illuminate the target area; An imaging unit, used to collect light from the irradiated target area, the imaging unit includes independently or simultaneously collecting white light and fluorescent light; a signal conversion unit, configured to receive the fluorescent image and the white light image reflected by the target area after being irradiated by the fluorescent light source and the white light source, and convert them into digital signals and send them to the image processing unit; An image processing unit, used for processing the digital signal sent by the signal conversion unit to generate a fluorescence image and a white light image, and determining the tumor boundary by fusing and analyzing the fluorescence image and the white light image; An interaction unit, used for collecting user feedback information and sending it to the imaging unit and the image processing unit; Each light source in the lighting unit is used to illuminate the target area independently or jointly; the imaging unit also includes: A light source optimization unit, configured to optimize the light path parameters of the light path in the target area based on a preset target evaluation function; wherein the optimization of the light path parameters of the target area is achieved by adjusting the light source parameters of each light source in the lighting unit; The light source parameters include: light source position, light source intensity, light source angle and light source focal length; the light path parameters include: light transmission path length, image clarity and lighting uniformity; the mathematical expression of the target evaluation function is: ; in, represents the length of the light transmission path, Represents image clarity, Represents the uniformity of illumination, It is a pre-set weighting factor based on medical needs. It is the attenuation coefficient that characterizes the loss of light during transmission. is the gain factor of image clarity, Indicates the standard deviation of light intensity; The functional expression of the illumination uniformity a is: ; in, and represent the cosine values of the angles between the light of the fluorescent light source and the light of the white light source and the imaging plane, respectively, and Respectively represent the light source intensities of the fluorescent light source and the white light source; The function expression of the image definition i is: ; in, and are the imaging clarity of the fluorescent light source and the white light source, respectively, and is the corresponding weight factor.
2. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 1, characterized in that: The imaging unit further includes: an adjusting unit for adjusting light source parameters of each light source in the lighting unit; wherein the adjusting unit includes: A mechanical arm unit, comprising a mechanical arm having at least one joint and a telescopic portion, wherein the mechanical arm moves the light source in multiple directions and angles based on an input control signal, and the light source covers the target area by presetting an initial position of the mechanical arm; An angle adjustment unit, comprising a servo motor and an angle sensor; wherein the servo motor adjusts the emission angle of the light source based on an input control signal, and the angle sensor detects the angle information of the current light source and feeds back to the control system to adjust the angle of the light source; An intensity control unit, comprising a digital regulator and an intensity sensor; wherein the digital regulator adjusts the current or voltage of the light source based on an input control signal to change the luminous intensity of the light source; the intensity sensor detects the brightness of the current light source and feeds back to the control system; A focus adjustment unit, comprising an automatic zoom lens and a focus adjustment motor; wherein the automatic zoom lens is fixedly mounted on the optical path in front of the light source, and the focus adjustment motor adjusts the relative position between the light source and the automatic zoom lens based on an input control signal to change the focusing or divergence characteristics of the light; The light source optimization unit, when measuring the efficiency and configuration of the optical path parameters of the nasal skull base mirror based on the target evaluation function and adjusting the optical path parameters of the nasal skull base mirror, also includes: Based on the simulated annealing algorithm, the optical path parameters of the nasal skull base mirror that maximize the function value of the target evaluation function are determined, and the target light source parameters are generated.
3. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 2, characterized in that: The imaging unit further includes: a control unit, configured to send a control signal to the adjustment unit based on the target light source parameter and user feedback information, and perform at least one of the following steps to adjust the light source parameter: Based on the control signal, the robot arm unit is controlled to adjust the position of the light source to the target light source position, and the position status is fed back in real time; The angle adjustment unit adjusts the emission angle of the light source through the servo motor and the angle sensor based on the control signal, and feeds back the current angle value through the angle sensor; The intensity control signal adjusts the current or voltage of the light source through a digital regulator based on the control signal, adjusts the intensity of the light source, and feeds back the current intensity of the light source in real time through an intensity sensor; Based on the control signal, the focus adjustment unit moves forward and backward along the optical axis direction through the focus adjustment motor to adjust the distance between the light source and the zoom lens, and provides real-time feedback of the current focus state.
4. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 3, characterized in that: Determining the tumor boundaries includes: preprocessing the digital signals of the fluorescent image and the white light image sent by the signal conversion unit; Based on a corner detection algorithm, performing image registration on the fluorescence image and the white light image at the same time, and marking the registered fluorescence image and the white light image at the same time as the same group; Based on the deep learning method, the registered fluorescence image and the white light image marked as the same group are fused to generate a fused image; Based on the user's feedback information, the fused image is optimized to determine the tumor boundary.
5. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 4, characterized in that: The corner detection algorithm includes: Harris corner detection algorithm; performing image registration based on the corner detection algorithm includes: The Harris corner response function is defined as: ; Among them, R is the Harris corner response value, M is the gradient covariance matrix of the local area of the image, det(M) represents the determinant of the matrix M, trace(M) represents the trace of the matrix M, and k is the sensitivity constant for corner detection; Based on the ORD algorithm, feature points are extracted from the fluorescence image and the white light image at the same time for matching, so as to establish a first corresponding relationship; Based on the first corresponding relationship, generating a homography transformation matrix between the two images at the same moment; The calculated homography transformation matrix between the two images at the same time is applied to any one of the two images to align it to the coordinate system of the other image, thereby registering the fluorescence image with the white light image.
6. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 5, characterized in that: The first corresponding relationship includes: n pairs of matching points between the two images at the same time; based on the first corresponding relationship, generating a homography transformation matrix between the two images at the same time, including: Based on n pairs of matching points between two images at the same time, a homography transformation model is established. For each matching point i, the following relationship exists: ; Among them, there are n pairs of matching points between the two images at the same time, and the corresponding point of the fluorescence image corresponding to each matching point i in the n pairs of matching points is , the corresponding point of the white light image is , arrive are the elements of the homography transformation matrix H; Based on the least square method, the homography transformation matrix is determined.
7. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 6, characterized in that: Performing image fusion on the registered fluorescence image and the white light image marked as the same group based on the deep learning method to generate a fused image includes: Constructing a target deep learning model; wherein the target deep learning model is constructed based on multiple groups of paired fluorescence images and white light images in a historical data set, and the labeled ideal fusion results corresponding to each group of paired fluorescence images and white light images; Preprocessing the registered fluorescence image and white light image of the same group to be fused; wherein the preprocessing includes: converting the original image into a grayscale image and scaling the pixel value to between 0 and 1; For the pre-processed registered fluorescence images and white light images of the same group, image fusion is performed through the target deep learning model to generate a fused image.
8. The dual-light source fluorescent nasal skull base mirror image processing system according to claim 7, characterized in that: The target deep learning model is a convolutional neural network model with two input branches, where: Each of the input branches comprises two convolutional layers for processing one of the fluorescence image and the white light image; The convolution layer is further used to: extract a first target feature of the processed image by applying at least one filter, and send the first target feature to the fusion layer; wherein the first target feature is automatically obtained by the convolution neural network model through a back propagation algorithm during the training process; The fusion layer is used to integrate the first target features from the two input branches by weighted averaging.
9. The dual-light source fluorescent nasal skull base endoscope image processing system according to any one of claims 1 to 8, characterized in that: Also includes: A water mist removal unit, used to detect the water mist humidity of the target area, and perform water mist removal processing when the water mist humidity of the target area is greater than a preset water mist threshold; The water mist removal unit comprises: a humidity detection unit, configured with at least one humidity sensor, for detecting the mist humidity of the target area, and outputting a release electrical signal to the demisting unit when the mist humidity of the target area is greater than a preset mist humidity; wherein the preset mist humidity is set based on the feedback information of the user; The demisting unit comprises: a controller and a MOSFET; wherein, after receiving the electrical signal released by the humidity detection unit, the controller uses the MOSFET as an electronic switch to remove water mist.
10. A dual-light source fluorescent nasal skull base mirror image processing device, which is implemented based on the dual-light source fluorescent nasal skull base mirror image processing system according to any one of claims 1 to 9, characterized in that it includes: An illumination module, provided with at least one fluorescent light source and at least one white light source, each light source being equipped with an independent control mechanism capable of adjusting the light source parameters of each light source individually or jointly based on image analysis requirements; Imaging modules, including a fluorescence imaging module and a white light imaging module, each module is equipped with an independent image sensor and an optical filtering system for capturing images of target areas illuminated by each light source of the illumination module; a signal conversion module, configured with a signal conversion device, for converting the captured optical signal into a digital signal to generate image data of the target area; Image processing module, with built-in image processor and software, used to process image data from fluorescence imaging module and white light imaging module, and optimize the composite image using algorithm; The control interface provides a user operation interface, allowing the user to adjust the parameters of the lighting module, imaging module and image processing module; The device also includes: a defogger module; the defogger module includes: a controller, a humidity detection device, a MOSFET and a silica gel hygroscopic material; wherein, after receiving the electrical signal released by the humidity detection device, the controller uses the MOSFET as an electronic switch to control the current to pass through the silica gel hygroscopic material to remove the water mist.
