A multi-dimensional image acquisition method, feature fusion method and system

Abstract

This application discloses a multi-dimensional image acquisition method, a feature fusion method, and a system. The multi-dimensional image acquisition method is applicable to clinical auxiliary acquisition scenarios for surgical instruments, used to avoid light and environmental interference and prevent image fusion distortion. The feature fusion method of the multi-dimensional image acquisition method is used to avoid image fusion distortion of surgical instruments. The system provided by this disclosure includes a multi-dimensional image acquisition module, a real-time quality assessment and analysis module, an adaptive adjustment decision module, an equipment control module, a feature fusion module, and a control center module. This disclosure can effectively and automatically adjust its working state according to the quality of the real-time acquired images to adapt to the complex and changing lighting and environmental conditions in the operating room.

Description

Technical Field

[0001] This application relates to the technical field of image acquisition, and in particular to a multi-dimensional image acquisition method, feature fusion method and system. Background Technology

[0002] Existing surgical instrument image acquisition systems typically employ multiple cameras (such as visible light, infrared, and depth cameras) with fixed positions and parameters for synchronous or asynchronous acquisition. Under complex lighting conditions in the operating room (strong light from shadowless lamps, localized shadows, bloodstain reflections, etc.) and the influence of instrument reflections and obstructions, images acquired by different cameras suffer from uneven exposure, loss of detail, noise interference, and color distortion.

[0003] Currently, existing technologies typically attempt to address these issues in the later fusion stages (such as feature point matching and image registration). However, inconsistencies in the quality of the underlying images place a heavy burden on the fusion algorithm and result in large matching errors. Ultimately, this leads to defects such as artifacts, ghosting, and structural distortion in the fused images, affecting the accuracy of surgical navigation or recording. This makes it difficult to meet the needs of clinical surgical assistance and precise instrument detection, posing potential risks to medical procedures. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-dimensional image acquisition method, feature fusion method, and system that can solve the aforementioned problems existing in the prior art.

[0005] To achieve the above objectives, this application adopts the following technical solution: On the one hand, a multi-dimensional image acquisition method is provided, suitable for clinical auxiliary acquisition scenarios of surgical instruments, to avoid light and environmental interference and avoid image fusion distortion, which includes: Step S10: Synchronously or asynchronously acquire multi-dimensional image data of the same surgical instrument target using at least two heterogeneous image sensors; Step S20: Perform real-time quality assessment on each raw image data obtained in step S10, generate a real-time quality assessment map corresponding to each frame of the image, and identify the type of defect region in the image; the type of defect region corresponds to the lighting and environmental interference of the surgical scene, and is used to guide subsequent parameter adjustment; Step S30: Based on the real-time quality assessment results and the type of defect area, dynamically generate control instructions for at least one image sensor or its associated auxiliary device, and adjust the operating parameters of the corresponding device according to the instructions; the control instructions are used to suppress light and environmental interference in the surgical scene and improve image acquisition quality. Step S40: Based on the adjusted parameters, continue image acquisition and repeat the real-time evaluation step S20 and dynamic control step S30 to form a closed-loop optimization until a multi-dimensional image set that meets the preset quality requirements is obtained; the preset quality requirements are adapted to the detailed acquisition needs of surgical instruments to ensure that the images are free from obvious interference defects. Step S50: Obtain the multi-dimensional image set that meets the preset quality requirements and its corresponding real-time quality evaluation map, and perform feature-level fusion on the multi-dimensional image set based on the real-time quality evaluation map.

[0006] On the other hand, this disclosure also provides a feature fusion method for multi-dimensional images, applied to any of the multi-dimensional image acquisition methods described above, to avoid distortion in surgical instrument image fusion, comprising: Step S10: Obtain a multi-dimensional image set of the same surgical instrument target acquired by at least two heterogeneous image sensors, as well as a real-time quality assessment map generated for each image during the acquisition phase. Step S20: Based on the real-time quality assessment map, calculate the fusion weight of each image at each pixel position or feature region; the fusion weight is positively correlated with the image quality, the better the quality, the greater the weight, and is used to preferentially retain high-quality region features; Step S30: Based on the fusion weights, a multi-scale transformation method or a neural network method is used to perform feature-level fusion on the multi-dimensional image set and output a fused image; the fused image can accurately restore the surface morphology and detailed features of the surgical instruments.

[0007] Furthermore, this disclosure also provides a multi-dimensional image acquisition system for implementing the aforementioned multi-dimensional image acquisition method and feature fusion method, adapted to surgical instrument acquisition scenarios, and characterized by: A multi-dimensional image acquisition module is used to acquire image data of surgical instrument targets through at least two heterogeneous image sensors; the heterogeneous image sensors are equipped with anti-fog and anti-interference lenses and are associated with auxiliary lighting equipment; The real-time quality assessment and analysis module is connected to the multi-dimensional image acquisition module and is used to perform real-time quality assessment on the acquired raw image data, generate a real-time quality assessment map, and identify the type of defect area. An adaptive control decision module is signal-connected to the real-time quality assessment and analysis module, and is used to generate control instructions for the image sensor and auxiliary equipment based on the assessment results and the type of defect area. The equipment control module is connected to the adaptive regulation decision module, the multi-dimensional image acquisition module and the auxiliary lighting equipment signal respectively, and is used to execute the regulation command, adjust the corresponding equipment parameters, and form a closed-loop optimization. The feature fusion module is connected to the multi-dimensional image acquisition module and the real-time quality assessment and analysis module respectively. It is used to perform feature-level fusion on the optimized multi-dimensional image set based on the real-time quality assessment map and output a distortion-free final fused image. The control center module is connected to the signals of each of the above modules to coordinate the collaborative work of each module, ensuring a smooth process of data acquisition, evaluation, control, and fusion, and adapting to the real-time requirements of surgical scenarios.

[0008] The beneficial effects of this application are as follows: By employing at least two heterogeneous image sensors and equipping them with high-transmittance anti-fog and anti-interference lenses, complementary information of surgical instruments can be captured from different physical dimensions (such as color, texture, depth, and polarization state), providing a rich and comprehensive data foundation for subsequent accurate analysis and fusion, and significantly improving the information content and fidelity of the images. Simultaneously, it can instantly identify and locate defects caused by surgical lights, instrument reflections, and tissue obstruction, such as overexposure of highlights, shadow occlusion, and excessive noise, and automatically adjust sensor parameters (such as exposure time) and auxiliary lighting equipment (such as brightness and angle), thereby proactively suppressing interference at its source, rather than merely performing passive repairs in the later stages.

[0009] By directly using the real-time quality assessment map generated during the acquisition process to guide the final feature-level fusion (such as weight calculation or attention mechanism), it is ensured that feature information of high-quality image regions is preferentially preserved and enhanced during the fusion process, while information of low-quality or defective regions is reasonably suppressed. This fundamentally avoids the problems of detail loss, ghosting, or distortion caused by uneven source image quality in traditional fusion methods, enabling the fused image to more accurately restore the surface morphology and details of surgical instruments.

[0010] The entire method forms an intelligent closed-loop system that can automatically adjust its working state based on the quality of real-time acquired images to adapt to the complex and ever-changing lighting and environmental conditions within the operating room. This results in optimized and high-quality fused images that are rich in detail, clear in features, and free from significant interference, enabling them to more effectively support subsequent clinical applications such as automatic identification of surgical instruments, posture tracking, wear detection, surgical navigation, and precise robotic operation, thereby improving the safety and accuracy of surgery. Attached Figure Description

[0011] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0012] Figure 1 This is a flowchart illustrating a multi-dimensional image acquisition method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a feature fusion method for multi-dimensional images according to an embodiment of this application; Figure 3 This is a schematic diagram of a system for a multi-dimensional image acquisition method and a feature fusion method according to an embodiment of this application.

[0013] In the picture: 100. Multi-dimensional image acquisition module; 200. Real-time quality assessment and analysis module; 300. Adaptive control decision module; 400. Equipment control module; 500. Feature fusion module; 600. Control center module. Detailed Implementation

[0014] To make the technical problems solved by this application, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this application are further described in detail below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the description of this application, unless otherwise expressly specified and limited, the terms "connected," "linked," and "fixed" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0016] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0017] This invention provides a multi-dimensional image acquisition method, feature fusion method, and system, which are suitable for high-fidelity, interference-resistant image acquisition and fusion of surgical instruments in an operating room environment. It can effectively solve problems such as image overexposure, shadows, blurring, and fusion distortion caused by surgical shadowless lamps, instrument reflections, blood and tissue obstruction in the prior art.

[0018] like Figure 1As shown, this embodiment provides a multi-dimensional image acquisition method, which is suitable for clinical auxiliary acquisition scenarios of surgical instruments, and is used to avoid light and environmental interference and avoid image fusion distortion.

[0019] Specifically, multi-dimensional image acquisition methods include: Step S10: Synchronously or asynchronously acquire multi-dimensional image data of the same surgical instrument target using at least two heterogeneous image sensors.

[0020] Specifically, the heterogeneous image sensor includes at least two of the following: visible light camera, infrared camera, depth camera, polarization camera, and spectral camera, and the heterogeneous image sensor is equipped with an anti-fog and anti-interference lens with a light transmittance of ≥98%.

[0021] For example, by employing at least two heterogeneous image sensors (such as visible light and depth cameras) and equipping them with anti-fog and anti-interference lenses with high light transmittance (≥98%), complementary information of surgical instruments can be captured from different physical dimensions (such as color, texture, depth, and polarization state), providing a data foundation for subsequent analysis and fusion, and significantly improving the information content and fidelity of the images.

[0022] Step S20: Perform real-time quality assessment on each raw image data obtained in step S10, generate a real-time quality assessment map corresponding to each frame of the image, and identify the type of defect region in the image; the type of defect region corresponds to the lighting and environmental interference of the surgical scene, which is used to guide subsequent parameter adjustment.

[0023] Furthermore, the real-time quality assessment map is generated by evaluating one or more quality indicators of the image, including at least one of brightness uniformity, local contrast, detail entropy, and noise level; the evaluation threshold of the quality indicators is adapted to the surgical instrument acquisition scenario, wherein the detail entropy is ≥7.5 and the noise level is ≤0.02.

[0024] Furthermore, the defect area types include at least one of the following: overexposed highlight areas, shadow occlusion areas, motion blur areas, and noise exceeding limits areas.

[0025] Specifically, by generating a real-time quality assessment map corresponding to the spatial dimensions of each image frame and structurally identifying specific defect types such as overexposure of highlights, occlusion of shadows, motion blur, and excessive noise, the system can perform pixel-level localization and causal classification of image problems. This approach not only identifies images with poor quality but also precisely identifies the location and cause of these defects. For example, whether overexposure is caused by instrument reflection or shadows formed by blood occlusion provides precise information for subsequent targeted interventions. Simultaneously, multi-dimensional indicators such as brightness uniformity, local contrast, detail entropy, and noise level are used for quantitative evaluation, and strict thresholds are set for detailed acquisition of surgical instruments, such as detail entropy ≥ 7.5 and noise level ≤ 0.02. This effectively identifies image degradation critical to clinical analysis, ensuring that the evaluation results are highly consistent with clinical needs.

[0026] Furthermore, through step S20, on the one hand, interference is actively suppressed at the acquisition end to obtain high-quality source images; on the other hand, intelligent weighting is performed at the fusion end to avoid defect information from polluting the final result, thereby ensuring high fidelity and distortion-free output images.

[0027] Step S30: Based on the real-time quality assessment results and the type of defect area, dynamically generate control instructions for at least one image sensor or its associated auxiliary device, and adjust the operating parameters of the corresponding device according to the instructions; the control instructions are used to suppress light and environmental interference in the surgical scene and improve image acquisition quality.

[0028] Furthermore, the control command adjusts the equipment parameters, including at least one of the following: exposure time, gain, aperture, and focus distance of the image sensor, and / or brightness, color temperature, and illumination angle of the auxiliary lighting device; wherein the exposure time adjustment range is 50μs-300μs, the brightness adjustment range of the auxiliary lighting device is 0-500lux, and the illumination angle adjustment range is 0-90°, to suppress reflections and local shadows from the surgical shadowless lamp.

[0029] Understandably, based on the real-time diagnostic results of step S20, control commands are actively and dynamically issued to change the working state of the image sensor and lighting equipment.

[0030] At the same time, the control commands directly affect the core physical parameters of the imaging link, including: Sensor-side: Exposure time (50μs-300μs), gain, aperture, and focusing distance directly control the amount of light entering the sensor and the image focus. Lighting: brightness (0-500 lux), color temperature, and illumination angle (0-90°) directly shape the lighting of the shooting environment.

[0031] Therefore, different parameters can be combined to precisely intervene in response to the different defects identified in step S20. For example, for specular highlights (overexposure) caused by surgical shadowless lamps, the exposure time can be shortened simultaneously (e.g., adjusted to 80μs) and the angle of auxiliary illumination can be adjusted to change the light path and reduce positive reflections. For local shadows, the brightness of auxiliary illumination can be increased or the illumination angle can be finely adjusted to provide supplementary lighting.

[0032] It should be noted that the 0-500 lux supplemental lighting range is sufficient to improve local lighting without interfering with the main shadowless lamp; the 0-90° angle adjustment provides ample operating space to avoid glare from the curved surfaces of instruments.

[0033] In step S40, image acquisition continues based on the adjusted parameters, and the real-time evaluation step S20 and the dynamic adjustment step S30 are repeated to form a closed-loop optimization until a multi-dimensional image set that meets the preset quality requirements is obtained. The preset quality requirements are adapted to the detailed acquisition needs of surgical instruments to ensure that the images are free from obvious interference defects.

[0034] Understandably, in dynamic surgical scenarios, a single parameter change can introduce new problems (such as reducing highlights while deepening shadows). The closed-loop mechanism allows the system to continuously monitor and adjust multiple times, enabling the imaging parameters to dynamically converge to the optimal solution for the current environment. This ensures that regardless of initial conditions or intermediate disturbances, the final acquired image set consistently meets high-quality requirements. Simultaneously, it can track environmental changes in real time and automatically compensate for their impact on image quality through closed-loop feedback, ensuring that image quality continuously adapts to the "detailed acquisition requirements of surgical instruments" throughout the entire acquisition process.

[0035] Step S50: Obtain a multi-dimensional image set that meets the preset quality requirements and its corresponding real-time quality assessment map, and perform feature-level fusion on the multi-dimensional image set based on the real-time quality assessment map.

[0036] Furthermore, guided by a "real-time quality assessment map," selective deep fusion is performed on multi-dimensional image sets at the feature level. This allows for the optimal fusion of the highest quality and most reliable feature information from each source image. For example, in a certain region, a visible light image may lose detail due to reflection (low quality assessment value), but an infrared or polarized image in that region may have very clear texture information (high quality assessment value). The fusion process automatically assigns greater weight to features from high-quality images, thereby synthesizing a "super image" in the final image that maintains high detail and high fidelity globally, surpassing any single input image in terms of information integrity and clarity.

[0037] Please see Figure 2This disclosure also provides a feature fusion method for the multi-dimensional image acquisition method mentioned in the above embodiments, used to avoid image fusion distortion of surgical instruments.

[0038] Specifically, the methods provided in this disclosure include: Step S10: Acquire a multi-dimensional image set of the same surgical instrument target acquired by at least two heterogeneous image sensors, as well as a real-time quality assessment map generated for each image during the acquisition phase.

[0039] Step S20: Based on the real-time quality assessment map, calculate the fusion weight of each image at each pixel location or feature region; the fusion weight is positively correlated with the image quality, and the better the quality, the greater the weight, which is used to preferentially retain high-quality region features.

[0040] Specifically, in step S20, the fusion weight W at pixel position (x,y) of the k-th image... The calculation method for (x,y) is as follows:

[0041] Among them, Q (x,y) represents the quality assessment value of the k-th image at position (x,y), ranging from 0 to 1. γ is an adjustable positive exponential factor used to control the sensitivity to quality differences. K is the total number of images in the multi-dimensional image set, and satisfies the following: .

[0042] Specifically, for each image (k=1 for the visible light map, k=2 for the depth map), based on its quality assessment map Q... (x,y), calculate the fusion weight W for each pixel position (x,y) according to the formula. (x,y). γ is set to 2 to enhance the weighting advantage of high-quality regions. The calculation results show that in regions with clear surgical forceps texture, the visible light map has a higher weight; in regions with accurate depth information and clear boundaries, the depth map may have a higher weight; and in regions where defects such as original highlights or shadows have been repaired, the weights of the two maps tend to be averaged.

[0043] Step S30: Based on the fusion weights, a multi-scale transformation method or a neural network method is used to perform feature-level fusion on the multi-dimensional image set and output a fused image; the fused image can accurately restore the surface morphology and detailed features of the surgical instruments.

[0044] It should be noted that step S30 employs a multi-scale transformation method, including: For each input image, multi-scale decomposition is performed to obtain the low-frequency sub-band coefficients of the overall contour of the corresponding image and the high-frequency sub-band coefficients of the detailed features of the corresponding surgical instruments. For the low-frequency sub-band coefficients, a weighted average method based on fusion weights is used for fusion. The fusion formula is as follows: ; in The low-frequency subband coefficients after fusion. Let be the low-frequency subband coefficients of the k-th image.

[0045] Specifically, for the low-frequency sub-band coefficients, a weighted average method based on the fusion weights calculated in step 1 is used for fusion. This ensures that the overall illumination and contour of the fused image are a smooth transition of information from two high-quality images.

[0046] For high-frequency subband coefficients, a fusion method based on the maximum value of the real-time quality assessment image can be used, that is, the high-frequency coefficient corresponding to the image with the highest quality assessment value is selected at each position to ensure that the detailed features of the surgical instruments are not lost.

[0047] For high-frequency subband coefficients, the "larger method" is allowed, but the criterion for "larger method" is to directly compare the quality evaluation values ​​of the two graphs at that position (x, y). The high-frequency coefficients of the image with the higher Q value at that location are selected as the high-frequency coefficients in the fused image. For example, at the fine teeth of a surgical forceps, if the quality evaluation value of the visible light image is higher, its high-frequency details are preferentially preserved; at the depth boundary between the instrument and the background, if the quality evaluation value of the depth image is higher, its sharp edge information is preferentially preserved. This ensures that the most valuable details are not lost due to averaging.

[0048] Furthermore, the coefficients of each scale subband after fusion are reconstructed by inverse transformation to obtain the fused image.

[0049] Understandably, it is permissible to perform an inverse Laplacian pyramid transform on the fused low-frequency and high-frequency subband coefficients to reconstruct the final fused image. This image retains the rich color and texture details of the visible light image while incorporating the spatial geometric information of the depth image, accurately restoring the surface morphology and three-dimensional details of the surgical forceps, without ghosting or blurring distortion caused by improper fusion.

[0050] Furthermore, in some implementations, step S30 employs a neural network method. The neural network takes a multi-dimensional image set and its corresponding real-time quality assessment map as common input, and outputs a fused image after feature extraction by the encoder, attention weight generation based on the quality assessment map, feature weighted fusion, and reconstruction by the decoder. The neural network is an improved version of U-Net, and the activation function of the feature fusion layer is the ReLU function.

[0051] For example, an improved U-Net network can be used, taking a visible light image, a depth image, and their respective quality assessment maps as input. The encoder part of the network extracts features from each image separately; in the feature fusion layer, the network uses information from the quality assessment maps to generate an attention weight map, which weights and fuses features from different source images, giving greater attention to features in high-quality regions; finally, a high-quality fused image is reconstructed through the decoder. The ReLU activation function is used. This network can be trained on a large amount of paired "multimodal input-ideal fused output" data.

[0052] Understandably, by actively suppressing surgical environment interference through real-time quality assessment and closed-loop control, and by using the quality assessment map generated during the acquisition process to guide the final feature-level fusion, fusion distortion can be effectively avoided, and high-quality fused images suitable for precise analysis and recording of surgical instruments can be obtained.

[0053] Please see Figure 3 This disclosure also provides a system for implementing the multi-dimensional image acquisition method and feature fusion method mentioned in the above embodiments, adapted to surgical instrument acquisition scenarios.

[0054] The system provided in this disclosure includes a multi-dimensional image acquisition module 100, a real-time quality assessment and analysis module 200, an adaptive control decision module 300, an equipment control module 400, a feature fusion module 500, and a control center module 600.

[0055] Specifically, the multi-dimensional image acquisition module 100 is used to acquire image data of surgical instrument targets through at least two heterogeneous image sensors; the heterogeneous image sensors are equipped with anti-fog and anti-interference lenses and are associated with auxiliary lighting equipment.

[0056] Furthermore, the real-time quality assessment and analysis module 200 is signal-connected to the multi-dimensional image acquisition module 100, and is used to perform real-time quality assessment on the acquired raw image data, generate a real-time quality assessment map, and identify the type of defect area.

[0057] Furthermore, the adaptive control decision module 300 is signal-connected to the real-time quality assessment and analysis module 200, and is used to generate control commands for the image sensor and auxiliary equipment based on the assessment results and the type of defect area.

[0058] Furthermore, the equipment control module 400 is connected to the adaptive regulation decision module 300, the multi-dimensional image acquisition module 100, and the auxiliary lighting equipment signal, respectively, to execute regulation commands, adjust the corresponding equipment parameters, and form a closed-loop optimization.

[0059] Furthermore, the feature fusion module 500 is connected to the multi-dimensional image acquisition module 100 and the real-time quality assessment and analysis module 200 respectively, and is used to perform feature-level fusion on the optimized multi-dimensional image set based on the real-time quality assessment map, and output a distortion-free final fused image.

[0060] Furthermore, the control center module 600 is connected to the signals of each of the above modules to coordinate the collaborative work of each module, ensuring the continuity of the acquisition, evaluation, control, and fusion process, and adapting to the real-time requirements of the surgical scenario.

[0061] As an example, a practical application scenario is provided, including: First, the control center module 600 initializes the multi-dimensional image acquisition module 100. This embodiment employs two heterogeneous sensors: a high-resolution visible light RGB camera and a depth camera (a polarized camera can be added as an additional sensor). Both lenses have anti-fog coatings. Initial parameters are set as follows: visible light camera exposure time 150μs, auxiliary illumination brightness 300 lux, and angle 45°. Under the synchronization signal from the control center, both cameras simultaneously capture images of the surgical forceps entering the field of view, acquiring a pair of visible light and depth images.

[0062] Secondly, the real-time quality assessment and analysis module 200 simultaneously receives two raw images.

[0063] For the visible light image: its global detail entropy is calculated to be 6.8 (below the threshold of 7.5), and its local noise level is 0.025 (above the threshold of 0.02). Further analysis of the generated quality assessment map revealed a small area of ​​pixel saturation in the center of the image, identified as an "overexposed highlight area" (caused by the reflection of the shadowless lamp light source on the smooth surface of the instrument); the brightness values ​​of the image edges were low and the texture was blurred, identified as "shadow occlusion areas".

[0064] For depth images: Calculate the confidence level (a quality indicator) of its depth data. If it is found that the depth values ​​of the regions corresponding to the highlights have a lot of noise or holes, the quality assessment map shows that the quality rating of the region is low.

[0065] Furthermore, the adaptive control decision module 300 receives the aforementioned evaluation results.

[0066] For “overexposed high-light areas”: The decision module generates control instructions to reduce the exposure time of the visible light camera to 80μs, while reducing the brightness of the auxiliary lighting equipment to 200 lux and adjusting the illumination angle from 45° to 60° to change the incident angle of the light and reduce specular reflection.

[0067] For “shadow-occluded areas”: the decision module generates another instruction – slightly increase the gain of the visible light camera and instruct the auxiliary lighting device to turn on a weak side supplement light (brightness 50 lux) while maintaining the 60° main light.

[0068] Finally, the equipment control module 400 immediately executes these instructions and adjusts the relevant parameters.

[0069] After adjustments, the system continued with a new round of image acquisition and performed the real-time quality assessment in step two again. This assessment showed that the highlight areas of the visible light image were suppressed (information entropy increased to 7.6), the shadow areas were improved, and the noise level decreased to 0.018. The quality of the depth image also improved accordingly.

[0070] Based on this, the “acquisition-evaluation-adjustment” cycle continues, forming a closed-loop optimization process, until all quality indicators of the acquired set of multi-dimensional images (visible light and depth maps) meet the preset requirements for surgical instrument detail acquisition (such as information entropy ≥ 7.5, noise ≤ 0.02, and no obvious overexposure / shadow defects).

[0071] In summary, this disclosure provides a multi-dimensional image acquisition method, feature fusion method, and system. By employing at least two heterogeneous image sensors and equipping them with high-transmittance anti-fog and anti-interference lenses, it can capture complementary information of surgical instruments from different physical dimensions (such as color, texture, depth, and polarization state). This provides a rich and comprehensive data foundation for subsequent accurate analysis and fusion, significantly improving the information content and fidelity of the images. Simultaneously, it can instantly identify and locate defects caused by surgical lights, instrument reflections, and tissue occlusion, such as overexposure, shadow occlusion, and excessive noise, and automatically adjust sensor parameters (such as exposure time) and auxiliary lighting equipment (such as brightness and angle), thereby proactively suppressing interference at its source, rather than merely performing passive repairs later.

[0072] By directly using the real-time quality assessment map generated during the acquisition process to guide the final feature-level fusion (such as weight calculation or attention mechanism), it is ensured that feature information of high-quality image regions is preferentially preserved and enhanced during the fusion process, while information of low-quality or defective regions is reasonably suppressed. This fundamentally avoids the problems of detail loss, ghosting, or distortion caused by uneven source image quality in traditional fusion methods, enabling the fused image to more accurately restore the surface morphology and details of surgical instruments.

[0073] Understandably, this method forms an intelligent closed-loop system that can automatically adjust its operating status based on the quality of real-time acquired images to adapt to the complex and ever-changing lighting and environmental conditions within the operating room. This results in optimized and high-quality fused images that are rich in detail, clear in features, and free from significant interference, enabling them to more effectively support subsequent clinical applications such as automatic identification of surgical instruments, posture tracking, wear detection, surgical navigation, and precise robotic operation, thereby improving the safety and accuracy of surgery.

[0074] In the description herein, it should be understood that the terms "upper," "lower," "left," "right," and other orientations or positional relationships are used only for ease of description and simplification of operation, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used merely for descriptive distinction and have no special meaning.

[0075] In the description of this specification, references to terms such as "an embodiment," "example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0076] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0077] The technical principles of this application have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of this application and should not be construed as limiting the scope of protection of this application in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of this application without inventive effort, and these embodiments will all fall within the scope of protection of this application.

Claims

1. A multi-dimensional image acquisition method, suitable for clinical auxiliary acquisition scenarios of surgical instruments, used to avoid light and environmental interference and avoid image fusion distortion, characterized in that, include: Step S10: Synchronously or asynchronously acquire multi-dimensional image data of the same surgical instrument target using at least two heterogeneous image sensors; Step S20: Perform real-time quality assessment on each channel of raw image data obtained in step S10, generate a real-time quality assessment map corresponding to each frame of image, and identify the types of defective regions in the image. The types of defective areas correspond to the lighting and environmental interference in the surgical scene, and are used to guide subsequent parameter adjustments. Step S30: Based on the real-time quality assessment results and the type of defect area, dynamically generate control instructions for at least one image sensor or its associated auxiliary device, and adjust the operating parameters of the corresponding device according to the instructions; the control instructions are used to suppress light and environmental interference in the surgical scene and improve image acquisition quality. Step S40: Based on the adjusted parameters, continue image acquisition and repeat the real-time evaluation step S20 and dynamic control step S30 to form a closed-loop optimization until a multi-dimensional image set that meets the preset quality requirements is obtained; the preset quality requirements are adapted to the detailed acquisition needs of surgical instruments to ensure that the images are free from obvious interference defects. Step S50: Obtain the multi-dimensional image set that meets the preset quality requirements and its corresponding real-time quality evaluation map, and perform feature-level fusion on the multi-dimensional image set based on the real-time quality evaluation map.

2. The method according to claim 1, characterized in that, The real-time quality assessment map is generated by evaluating one or more quality indicators of the image, including at least one of brightness uniformity, local contrast, detail entropy, and noise level; the evaluation threshold of the quality indicators is adapted to the surgical instrument acquisition scenario, wherein the detail entropy is ≥7.5 and the noise level is ≤0.

02.

3. The method according to claim 1, characterized in that, The defect area types include at least one of the following: overexposed highlight area, shadow occlusion area, motion blur area, and noise exceeding limit area.

4. The method according to claim 1, characterized in that, The device parameters adjusted by the control command include at least one of the following: exposure time, gain, aperture, and focus distance of the image sensor, and / or brightness, color temperature, and illumination angle of the auxiliary lighting device; wherein the exposure time adjustment range is 50μs-300μs, the brightness adjustment range of the auxiliary lighting device is 0-500lux, and the illumination angle adjustment range is 0-90°, which are used to suppress reflections and local shadows from the surgical shadowless lamp.

5. The method according to claim 1, characterized in that, The heterogeneous image sensor includes at least two of the following: a visible light camera, an infrared camera, a depth camera, a polarization camera, and a spectral camera. The heterogeneous image sensor is equipped with an anti-fog and anti-interference lens, and the light transmittance of the lens is ≥98%.

6. A feature fusion method for multi-dimensional images, applied to the multi-dimensional image acquisition method according to any one of claims 1-5, for avoiding distortion in surgical instrument image fusion, characterized in that, include: Step S10: Obtain a multi-dimensional image set of the same surgical instrument target acquired by at least two heterogeneous image sensors, as well as a real-time quality assessment map generated for each image during the acquisition phase. Step S20: Calculate the fusion weight of each image at each pixel location or feature region based on the real-time quality assessment map; The fusion weight is positively correlated with image quality; the better the quality, the greater the weight, which is used to prioritize the preservation of high-quality region features. Step S30: Based on the fusion weights, a multi-scale transformation method or a neural network method is used to perform feature-level fusion on the multi-dimensional image set and output a fused image; the fused image can accurately restore the surface morphology and detailed features of the surgical instruments.

7. The method according to claim 6, characterized in that, In step S20, the fusion weight of the k-th image at pixel position (x, y) The calculation method is as follows: ； in, Let be the quality assessment value of the k-th image at position (x, y), with a value ranging from 0 to 1; γ be an adjustable positive exponential factor used to control the sensitivity to quality differences; and K be the total number of images in the multi-dimensional image set, satisfying the following: 。 8. The method according to claim 6 or 7, characterized in that, Step S30 employs a multi-scale transformation method, including: For each input image, multi-scale decomposition is performed to obtain the low-frequency sub-band coefficients of the overall contour of the corresponding image and the high-frequency sub-band coefficients of the detailed features of the corresponding surgical instruments. For the low-frequency sub-band coefficients, a weighted average method based on the fusion weights is used for fusion, and the fusion formula is as follows: ； in The low-frequency subband coefficients after fusion. Let be the low-frequency subband coefficients of the k-th image; For high-frequency subband coefficients, the method of taking the largest value based on the real-time quality assessment map is used for fusion, that is, the high-frequency coefficient corresponding to the image with the highest quality evaluation value is selected at each position to ensure that the detailed features of the surgical instruments are not lost. The fused image is obtained by inverse transformation and reconstruction of the coefficients of each scale subband after fusion.

9. The method according to claim 6 or 7, characterized in that, Step S30 employs a neural network method. The neural network takes the multi-dimensional image set and its corresponding real-time quality assessment map as common inputs. After feature extraction by the encoder, attention weight generation based on the quality assessment map, feature weighted fusion, and reconstruction by the decoder, the fused image is output. The neural network is an improved version of U-Net, and the activation function of the feature fusion layer is the ReLU function.

10. A multi-dimensional image acquisition system, used to implement the multi-dimensional image acquisition method according to any one of claims 1-5 and the feature fusion method according to any one of claims 6-9, adapted to surgical instrument acquisition scenarios, characterized in that, include: A multi-dimensional image acquisition module is used to acquire image data of surgical instrument targets through at least two heterogeneous image sensors; The heterogeneous image sensor is equipped with an anti-fog and anti-interference lens and is associated with an auxiliary lighting device; The real-time quality assessment and analysis module is connected to the multi-dimensional image acquisition module and is used to perform real-time quality assessment on the acquired raw image data, generate a real-time quality assessment map, and identify the type of defect area. An adaptive control decision module is signal-connected to the real-time quality assessment and analysis module, and is used to generate control instructions for the image sensor and auxiliary equipment based on the assessment results and the type of defect area. The equipment control module is connected to the adaptive regulation decision module, the multi-dimensional image acquisition module and the auxiliary lighting equipment signal respectively, and is used to execute the regulation command, adjust the corresponding equipment parameters, and form a closed-loop optimization. The feature fusion module is connected to the multi-dimensional image acquisition module and the real-time quality assessment and analysis module respectively. It is used to perform feature-level fusion on the optimized multi-dimensional image set based on the real-time quality assessment map and output a distortion-free final fused image. The control center module is connected to the signals of each of the above modules to coordinate the collaborative work of each module, ensuring a smooth process of data acquisition, evaluation, control, and fusion, and adapting to the real-time requirements of surgical scenarios.

Images