Mirror optical system lens state online monitoring method and device based on AI vision

Abstract

The application provides an AI vision-based galvanometer optical system lens state online monitoring method and device, and belongs to the technical field of image processing. The method comprises the following steps: a processing device acquires real-time characteristic information of a galvanometer optical system lens in multiple modes, the multiple modes comprising at least two modes in visible light, short-wave infrared, thermal force or motion state, and the real-time characteristic information in multiple modes is obtained by real-time sensing of the galvanometer optical system lens under the condition that the multiple modes are spatially aligned; the processing device performs multi-modal fusion on the real-time characteristic information in multiple modes in a spatially aligned manner to obtain fused real-time feature information; and the processing device analyzes the fused real-time feature information through an AI network model based on an attention mechanism multi-task spatio-temporal fusion to obtain the real-time state of the galvanometer optical system lens.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for online monitoring of lens status in a galvanometer optical system based on AI vision. Background Technology

[0002] Galvanometers are core optical components in laser processing, precision measurement, and other fields. Operating under continuous high loads and dynamic conditions for extended periods, their surface and internal properties are susceptible to various complex factors, leading to changes in their operational state that are difficult to discern visually. If these changes are not detected and assessed in a timely manner, they will directly affect the output quality and execution accuracy of the optical system, thereby impacting the stability of the entire process and the final result.

[0003] Currently, monitoring the operational status of such precision optical lenses generally relies on periodic offline inspections or simple threshold judgments based on a single physical dimension. Offline inspections cannot reflect the real-time status during dynamic operation and have monitoring blind spots; while judgment methods based on single-dimensional signals are insufficient for comprehensive analysis and in-depth evaluation of the status due to a lack of information. Summary of the Invention

[0004] This invention provides a method and apparatus for online monitoring of lens status in a galvanometer optical system based on AI vision, in order to solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, an AI vision-based online monitoring method for the state of lenses in a galvanometer optical system is provided, applied to a processing device. The method includes: the processing device acquiring real-time multimodal characteristic information of the lenses in the galvanometer optical system, wherein the multimodal characteristics include at least two modes selected from visible light, shortwave infrared, thermal, or motion states; the real-time characteristic information of the multimodal characteristics is obtained by real-time perception of the lenses in the galvanometer optical system under spatial alignment; the processing device performing multimodal fusion of the real-time characteristic information of the multimodal characteristics in a spatially aligned manner to obtain fused real-time feature information; and the processing device analyzing the fused real-time feature information through an AI network model based on attention mechanism multi-task spatiotemporal fusion to obtain the real-time state of the lenses in the galvanometer optical system.

[0007] This achieves a leap from single-dimensional, qualitative judgment to multi-dimensional, quantitative analysis of galvanometer lens condition. By forcibly requiring spatial alignment and fusion of multimodal information, the problem of data incompatibility between different sensors is solved, laying the data foundation for building a unified and comprehensive lens health profile. The multi-task AI model based on the attention mechanism can perform multiple tasks in parallel, such as defect identification, localization, and severity assessment, significantly improving the automation level of monitoring and the accuracy of comprehensive judgment.

[0008] Optionally, the processing device acquires real-time multimodal characteristic information of the lens of the galvanometer optical system, including: the processing device acquires real-time high-resolution polarized RGB image, real-time short-wave infrared image, real-time thermal image of the lens of the galvanometer optical system, and real-time vibration characteristic information of the lens of the galvanometer optical system. The real-time multimodal characteristic information of the lens includes real-time high-resolution polarized RGB image, real-time short-wave infrared image, real-time thermal image, and real-time vibration characteristic information.

[0009] Thus, by specifically defining four types of information—polarized RGB, shortwave infrared, thermal imaging, and vibration—a comprehensive and simultaneous perception of microscopic defects on the lens surface, the chemical composition of contaminants, temperature field distribution, and mechanical stability is achieved. This combination can effectively distinguish between damages that look similar but have different causes (such as oil stains vs. water stains) and capture hidden dangers that cannot be detected by pure vision (such as internal heat accumulation).

[0010] Optionally, the processing device acquires real-time high-resolution polarized RGB images, real-time short-wave infrared images, real-time thermal images, and real-time vibration characteristic information of the galvanometer optical system lens, including: the processing device acquires a real-time high-resolution polarized RGB image captured by a high-resolution polarization camera at a first moment and from a first viewing angle under bright / dark field illumination; the processing device acquires a real-time short-wave infrared image captured by a short-wave infrared camera at a first moment and from a first viewing angle; the processing device acquires a real-time thermal image captured by a miniature thermal imager at a first moment and from a first viewing angle; and the processing device acquires a real-time video stream captured by a high-speed global shutter CMOS at a first viewing angle and from the galvanometer optical system lens, and extracts real-time vibration characteristic information from the real-time video stream, wherein the time period of the real-time video stream includes the first moment.

[0011] In this way, strictly limiting all image information to be acquired at the same time and from the same perspective ensures that multimodal data is strictly aligned in time and space. This fundamentally eliminates data misalignment and miscorrelation caused by different sampling times or angles, making subsequent pixel-level or feature-level fusion physically authentic, which is a prerequisite for the algorithm to obtain high-precision analysis results.

[0012] Optionally, the processing device performs multimodal fusion of the real-time characteristic information of the multimodal modes in a spatially aligned manner to obtain fused real-time feature information, including: the processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in a spatial dimension; the processing device segments the real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features in a spatial dimension; and the processing device segments the real-time thermal image into multiple real-time thermal image sub-image features in a spatial dimension; the processing device determines multiple RGB sub-image features, multiple real-time shortwave infrared sub-image features, and multiple real-time thermal image sub-image features into multiple sub-feature groups in a spatially aligned manner, wherein each sub-feature group includes one RGB sub-image feature, one real-time shortwave infrared sub-image feature, and one real-time thermal image sub-image feature at the same spatial location, and different sub-feature groups in the multiple sub-feature groups correspond to different spatial locations; the processing device fuses the real-time vibration feature information into the multiple sub-feature groups respectively to obtain multiple fused sub-feature groups, and the fused real-time feature information is the multiple fused sub-feature groups.

[0013] Thus, a specific fusion path of "segmentation and alignment first, then group fusion" is proposed. The global image is decomposed into local sub-feature groups for processing, significantly reducing the complexity of the model directly processing high-dimensional data while preserving spatial details. An independent fusion unit is established for each spatial location, enabling the model to analyze the independent states of different regions on the lens in detail, providing the possibility for locating local damage.

[0014] Optionally, the processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension, including: the processing device extracts RGB sub-images from the real-time high-resolution polarized RGB image containing only the region where the galvanometer optical system lens is located, the region where the galvanometer optical system lens is located is a circular region; the processing device segments the RGB sub-image into multiple RGB honeycomb sub-images using a honeycomb structure, each RGB honeycomb sub-image being a hexagonal pattern; the multiple RGB honeycomb sub-images can be stitched together into an RGB sub-image using a honeycomb structure; the processing device extracts the feature information of each of the multiple RGB honeycomb sub-images through a first convolutional layer to obtain multiple RGB sub-image features; the processing device also segments the real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features in the spatial dimension, including: the processing device extracts infrared sub-images from the real-time shortwave infrared image containing only the region where the galvanometer optical system lens is located, the region where the galvanometer optical system lens is located is a circular region; the processing device segments the infrared sub-image into multiple infrared honeycomb images using a honeycomb structure, each infrared honeycomb image being a hexagonal pattern; the multiple infrared honeycomb sub-images being hexagonal patterns; the multiple RGB ... A hexagonal pattern and multiple infrared honeycomb sub-images can be stitched together in a honeycomb structure to form an infrared sub-image. The processing device extracts the feature information of each of the multiple infrared honeycomb sub-images through a second convolutional layer to obtain multiple real-time shortwave infrared sub-image features. Furthermore, the processing device segments the real-time thermal image into multiple real-time thermal image sub-image features in the spatial dimension, including: the processing device extracts thermal image sub-images from the real-time thermal image that only contain the region where the galvanometer optical system lens is located; the region where the galvanometer optical system lens is located is a circular region; and the processing device segments the thermal image into multiple thermal honeycomb sub-images using a honeycomb structure. The thermal image is a honeycomb sub-image, where each thermal honeycomb sub-image is a hexagonal pattern. Multiple thermal honeycomb sub-images can be stitched together in a honeycomb structure to form a thermal sub-image. The processing device extracts the feature information of each of the multiple thermal honeycomb sub-images through a third convolutional layer to obtain multiple real-time thermal sub-image features. Among them, the number of multiple RGB honeycomb sub-images, multiple infrared honeycomb sub-images, and multiple thermal honeycomb sub-images is the same, and correspondingly, the number of multiple RGB sub-image features, multiple real-time shortwave infrared sub-image features, and multiple real-time thermal sub-image features is also the same.

[0015] Thus, using a cellular network (hexagonal) to segment the circular mirror sub-image, compared to a traditional rectangular grid, can more closely fit the circular area of ​​the mirror, reduce interference from background or invalid areas, and improve computational efficiency. The hexagonal structure has better isotropy, allowing adjacent features to be better correlated, and it does not have the "directional bias" of a rectangular grid, enabling it to process defect features (such as radial scratches) from different directions more evenly.

[0016] Optionally, any one of the multiple sub-feature groups includes the following three features corresponding to the same hexagonal pattern in the cellular network structure: a target RGB sub-image feature, a target real-time shortwave infrared sub-image feature, and a target real-time thermal image sub-image feature. The processing device fuses the real-time vibration feature information into the multiple sub-feature groups to obtain multiple fused sub-feature groups, including: For the target sub-feature group: the processing device establishes a target star-shaped connection relationship between the real-time vibration feature information and the target sub-feature group to obtain a fused target sub-feature group. The target star-shaped connection relationship indicates that the real-time vibration feature information is associated with the target RGB sub-image feature, the target real-time shortwave infrared sub-image feature, and the target real-time thermal image sub-image feature, respectively. The target star-shaped connection relationship is used by the AI ​​network model to associate and analyze the real-time vibration feature information with the target RGB sub-image feature, the target real-time shortwave infrared sub-image feature, and the target real-time thermal image feature, respectively. Thus, multiple fused sub-feature groups are obtained.

[0017] Thus, the above scheme proposes a new "star-shaped sub-network" fusion structure, using vibration characteristics representing the overall motion state as the core hub, and associating them with the RGB, infrared, and thermal characteristics of the same local region. This structural design guides AI models to explicitly explore the causal relationship between dynamic behavior and static appearance / thermal characteristics, which is particularly beneficial for discovering coupled fault modes caused by minute deformations due to vibration or by changes in the vibration spectrum caused by local overheating.

[0018] Optionally, the processing device analyzes the fused real-time feature information through an AI network model based on attention mechanism multi-task spatiotemporal fusion to obtain the real-time state of the galvanometer optical system lens. This includes: the processing device establishing the association relationship of multiple fused sub-feature groups based on the positional adjacency relationship of each hexagonal pattern in the honeycomb structure. In the association relationship, the two hexagonal patterns corresponding to two associated sub-feature groups in the multiple fused sub-feature groups are adjacent in the honeycomb structure. The processing device inputs the association relationship and the multiple fused sub-feature groups into the AI ​​network model for analysis to obtain the real-time state of the galvanometer optical system lens.

[0019] In this way, by leveraging the natural geometric adjacency relationships of the cellular mesh to reorganize the fused local features, the AI ​​model can understand the spatial continuity and propagation of defects (for example, an ablation point may lead to abnormal heat conduction in adjacent areas). This structured feature organization incorporates prior physical knowledge of the lens as a continuum, which helps improve the analytical rationality of the model and the consistency in identifying large-area continuous defects.

[0020] Optionally, the method further includes: the processing device performing lifecycle management on the galvanometer optical system lens based on the real-time status of the lens. For example, the processing device performing lifecycle management on the galvanometer optical system lens based on the real-time status includes: the processing device determining the remaining lifespan score of the galvanometer optical system lens based on the real-time status of the lens; and the processing device performing lifecycle management on the galvanometer optical system lens based on the remaining lifespan score and the preset total lifespan score of the galvanometer optical system lens.

[0021] Thus, by introducing the quantitative indicator of "remaining lifespan score," the abstract "state" is transformed into an intuitive, measurable, and traceable lifespan budget. Combined with the preset "total lifespan score," the degree of lens health degradation can be dynamically assessed, providing users with clear replacement expectations and decision-making basis (such as "30% remaining lifespan, recommended for planned purchase"), greatly improving the planning of maintenance management.

[0022] Secondly, an AI vision-based online monitoring device for the lens status of a galvanometer optical system is provided. The device is configured to: acquire real-time multimodal characteristic information of the lens of the galvanometer optical system, wherein the multimodal characteristics include at least two modes among visible light, short-wave infrared, thermal, or motion states, and the real-time characteristic information of the multimodal characteristics is obtained by real-time perception of the lens of the galvanometer optical system under the condition of spatial alignment of the multimodal characteristics; fuse the real-time characteristic information of the multimodal characteristics in a spatially aligned manner to obtain fused real-time feature information; and analyze the fused real-time feature information through a multi-task spatiotemporal fusion AI network model based on an attention mechanism to obtain the real-time state of the lens of the galvanometer optical system.

[0023] It should be understood that the specific implementation of the device described in the second aspect can be referred to the relevant introduction of the above method, and will not be repeated here.

[0024] Thirdly, an electronic device is provided, comprising: a processor and a memory; the memory is used to store a computer program, which, when executed by the processor, causes the electronic device to perform the method described in the first aspect.

[0025] In one possible design, the electronic device described in the third aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the electronic device described in the third aspect and other electronic devices.

[0026] In the embodiments of the present invention, the electronic device described in the third aspect may be a terminal, or a chip (system) or other component or assembly disposed in the terminal, or a system containing the terminal. Attached Figure Description

[0027] Figure 1This is a schematic diagram of the architecture of the online monitoring system provided in an embodiment of the present invention;

[0028] Figure 2 A flowchart illustrating the online monitoring method for lens status of a galvanometer optical system based on AI vision, provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0031] In this embodiment of the invention, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a specific time. They do not require the device to make a judgment action during implementation, nor do they imply any other limitations.

[0032] In embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. Furthermore, in embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0033] To facilitate understanding of the embodiments of the present invention, firstly, let's take... Figure 1 The online monitoring system shown in the figure is used as an example to illustrate in detail the online monitoring method for lens status of the galvanometer optical system based on AI vision applicable to the embodiments of the present invention.

[0034] For example, Figure 1 This is a schematic diagram of the architecture of an online lens status monitoring device for a galvanometer optical system based on AI vision, provided as an embodiment of the present invention. Figure 1 As shown, the system may include: a processing device and an imaging device. The imaging device may include: a high-resolution polarization camera, a short-wave infrared camera, a miniature thermal imager, and a high-speed global shutter CMOS.

[0035] The processing device can be a terminal with processing capabilities or a chip or chip system that can be installed on the terminal. This terminal device can also be referred to as user equipment (UE), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device. In the embodiments of this application, the terminal device can be a mobile phone, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, vehicle-mounted terminal, RSU with terminal functionality, etc. The terminal device of this application may also be an on-board module, on-board component, on-board chip, or on-board unit that is built into a vehicle as one or more components or units. The vehicle can implement the method provided in this application through the built-in on-board module, on-board component, on-board chip, or on-board unit.

[0036] High-resolution polarization cameras typically rely on integrating pixel-level polarization filter arrays onto the surface of traditional CMOS image sensors. This array uses subwavelength metal nanowire gratings or microstructures arranged regularly at specific angles (e.g., 0°, 45°, 90°, and 135°) on the sensor surface, allowing each pixel to permit light only in a specific polarization direction. This design enables the simultaneous acquisition of light intensity information from the target under different polarization states in a single exposure, allowing subsequent calculations to determine the target's degree of polarization and polarization angle. The core advantage of this approach lies in seamlessly integrating polarization sensing capabilities into a standard imaging chain, resulting in a compact structure while maintaining a high spatial resolution similar to that of the underlying sensor.

[0037] The mainstream technology for short-wave infrared cameras uses uncooled indium gallium arsenide (InGaAs) focal plane arrays as the core detector. This material system typically allows its spectral response range to cover the short-wave infrared band from 900 nm to 1700 nm. The front end of the device needs to be equipped with germanium or chalcogenide glass lenses optimized for transmission in this band, and often integrates thermoelectric coolers to stabilize the sensor's operating temperature and reduce dark noise. Such cameras can detect the unique reflection, absorption, and emission characteristics of objects in this band, have penetration capabilities for many materials that are opaque under visible light (such as silicon and certain plastics), and can distinguish substances with different chemical compositions (such as water and oil).

[0038] A typical form of miniature thermal imager is a thermal imaging module based on uncooled microbolometer technology. The core sensor consists of a pixel array made of vanadium oxide (VOx) or amorphous silicon (α-Si), with pixel size miniaturized to below 12 micrometers. Each pixel absorbs long-wave infrared radiation (8-14 micrometers) emitted by the target, resulting in a temperature rise and resistance change, which is then converted into an electrical signal. Through advanced wafer-level packaging and optical system miniaturization technology, power consumption is extremely low, and it can directly output calibrated two-dimensional temperature field data, achieving true non-contact area array temperature measurement.

[0039] The key technological feature of high-speed global shutter CMOS sensors lies in their global shutter pixel design. Unlike rolling shutters that use line-by-line exposure, each pixel contains a dedicated charge storage node, allowing all pixels to start and end exposure simultaneously, temporarily storing the signal, and then reading it out line by line. This structure fundamentally eliminates the "jelly effect" distortion when photographing fast-moving objects. To achieve high frame rates and high sensitivity, advanced sensors of this type often employ back-illuminated or stacked processes, separating and optimizing the photodiode area and readout circuitry in three-dimensional space. This allows them to achieve a global shutter while maintaining high frame rates, low noise, and large full-well capacity.

[0040] Figure 2 This is a flowchart illustrating the method provided in an embodiment of the present invention. This AI vision-based online monitoring method for the lens state of a galvanometer optical system is applicable to the aforementioned system, and the specific process is as follows:

[0041] S201, The processing device acquires real-time characteristic information of the multimodal properties of the galvanometer optical system lens. The multimodal properties include at least two modes among visible light, short-wave infrared, thermal, or motion states. The real-time characteristic information of the multimodal properties is obtained by real-time sensing of the galvanometer optical system lens under the condition that the multimodal properties are spatially aligned.

[0042] For example, the processing device acquires real-time high-resolution polarized RGB images, real-time short-wave infrared images, real-time thermal images, and real-time vibration characteristic information of the galvanometer optical system lenses. The multimodal real-time characteristic information of the lenses includes real-time high-resolution polarized RGB images, real-time short-wave infrared images, real-time thermal images, and real-time vibration characteristic information. Thus, by specifically limiting the information to four types—polarized RGB, short-wave infrared, thermal imaging, and vibration—comprehensive and simultaneous perception of microscopic defects, chemical composition of contaminants, temperature field distribution, and mechanical stability of the lens surface is achieved. This combination can effectively distinguish between damages that appear similar but have different causes (such as oil stains vs. water stains) and capture hidden dangers that cannot be detected by pure vision (such as internal heat accumulation).

[0043] Specifically, the processing device can acquire a real-time high-resolution polarized RGB image captured by a high-resolution polarization camera under bright / dark illumination, at a first moment and from a first-viewpoint towards the lens of the galvanometer optical system. For example, the processing device establishes a communication connection with the high-resolution polarization camera through a preset data interface (such as GigE Vision, USB3 Vision, or Camera Link). To achieve shooting under bright / dark illumination, the processing device is configured to send control commands to a programmable multi-angle ring LED lighting device; for example, the processing device sends a command to "turn on all LEDs and illuminate from the front" to form bright illumination, or sends a command to "turn on only the side LEDs and turn off the front light source" to form dark illumination. Upon receiving a synchronization trigger signal from the processing device, the polarization camera, at the first moment, performs a single exposure on the polarized light incident through the first viewing angle and reflected or transmitted by the lens, based on the pixel-level polarization filter array integrated on its sensor surface (e.g., periodically arranged in four directions: 0°, 45°, 90°, and 135°). Finally, the image data stream containing the original polarization information is transmitted to the processing device through the data interface, and the processing device calculates and outputs the real-time high-resolution polarized RGB image.

[0044] Furthermore, the processing device acquires a real-time short-wave infrared image captured by the short-wave infrared camera at a first moment, facing the lens of the galvanometer optical system from a first-viewpoint perspective. For example, the processing device connects to the short-wave infrared camera and completes the drive configuration through its integrated corresponding data interface. To achieve spatial alignment with the polarization camera "at a first moment, from a first-viewpoint perspective," the short-wave infrared camera and the high-resolution polarization camera are physically fixed on the same rigid bracket, and their optical axes are parallel or their optical paths are unified through a beam splitter to ensure consistent viewing angles. At the first moment, the same synchronization trigger signal emitted by the processing device simultaneously triggers the short-wave infrared camera. The camera exposes the short-wave infrared band (e.g., 900-1700 nm) radiation received from the first-viewpoint perspective on its indium gallium arsenide sensor target surface and transmits the generated digital image data to the processing device in real time to form the real-time short-wave infrared image.

[0045] Furthermore, the processing device can acquire real-time thermal images captured by the miniature thermal imager at a first-viewpoint angle towards the galvanometer optical system lens. For example, the processing device can connect to the miniature thermal imager via a digital communication bus (such as USB or Ethernet) and load its temperature database and emissivity correction parameters. To achieve spatial alignment, the miniature thermal imager and the aforementioned camera are co-calibrated and fixed on the same observation platform. At the first moment, a synchronization signal emitted by the processing device triggers the thermal imager. The microbolometer focal plane array inside the thermal imager captures long-wave infrared radiation (e.g., 8-14 micrometers) emitted from the lens surface received via the first-viewpoint angle. Its internal processor converts the radiation signal into temperature values ​​based on preset emissivity (e.g., 0.8 for the lens coating material) and ambient temperature parameters, and sends a two-dimensional matrix containing temperature data for each pixel to the processing device in real time to generate the real-time thermal image.

[0046] Furthermore, the processing device can also acquire a real-time video stream captured by a high-speed global shutter CMOS camera from a first-view perspective towards the lens of the galvanometer optical system, and extract real-time vibration feature information from the real-time video stream. The time period of the real-time video stream includes the first moment. For example, the processing device can connect to a high-speed global shutter CMOS camera via a high-speed data interface (such as CoaXPress or 10 GigE) and configure it to a high frame rate continuous shooting mode (e.g., more than 1000 frames per second). The camera continuously acquires a real-time video stream of the lens from a first-view perspective. To extract real-time vibration feature information, after receiving the video stream data containing the first moment, the processing device performs the following steps: First, it selects the lens edge or a specific marker point as a feature region in the initial frame of the video stream; then, for each subsequent frame, it calculates the sub-pixel displacement vector of the feature region using a digital image correlation algorithm or optical flow method; finally, it analyzes the change of the displacement vector sequence over time, for example, by extracting its dominant frequency component and amplitude through Fourier transform. These frequency and amplitude data characterizing the microscopic motion state of the lens constitute the real-time vibration feature information.

[0047] In this way, strictly limiting all image information to be acquired at the same time and from the same perspective ensures that multimodal data is strictly aligned in time and space. This fundamentally eliminates data misalignment and miscorrelation caused by different sampling times or angles, making subsequent pixel-level or feature-level fusion physically authentic, which is a prerequisite for the algorithm to obtain high-precision analysis results.

[0048] S202, the processing device performs multimodal real-time characteristic information fusion in a spatially aligned manner to obtain fused real-time feature information.

[0049] The processing device can segment a real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension. For example, the processing device can extract an RGB sub-image from a real-time high-resolution polarized RGB image that contains only the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device segments the RGB sub-image into multiple RGB honeycomb sub-images with a honeycomb structure. Each RGB honeycomb sub-image is a hexagonal pattern. Multiple RGB honeycomb sub-images can be stitched together with a honeycomb structure to form an RGB sub-image. The processing device extracts the feature information of each of the multiple RGB honeycomb sub-images through the first convolutional layer to obtain multiple RGB sub-image features.

[0050] The processing device can spatially segment a real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features. For example, the processing device extracts an infrared sub-image from the real-time shortwave infrared image that contains only the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device segments the infrared sub-image into multiple infrared honeycomb sub-images using a honeycomb structure. Each infrared honeycomb sub-image is a hexagonal pattern. Multiple infrared honeycomb sub-images can be stitched together using a honeycomb structure to form an infrared sub-image. The processing device extracts the feature information of each of the multiple infrared honeycomb sub-images through a second convolutional layer to obtain multiple real-time shortwave infrared sub-image features.

[0051] Furthermore, the processing device can segment real-time thermal images into multiple real-time thermal image sub-image features in the spatial dimension. For example, the processing device can extract a thermal image sub-image from a real-time thermal image that only contains the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device segments the thermal image sub-image into multiple thermal image honeycomb sub-images using a honeycomb structure. Each thermal image honeycomb sub-image is a hexagonal pattern. Multiple thermal image honeycomb sub-images can be stitched together in a honeycomb structure to form a thermal image sub-image. The processing device extracts the feature information of each of the multiple thermal image honeycomb sub-images through a third convolutional layer to obtain multiple real-time thermal image sub-image features.

[0052] The number of multiple RGB honeycomb sub-images, multiple infrared honeycomb sub-images, and multiple thermal honeycomb sub-images is the same. Correspondingly, the number of multiple RGB sub-image features, multiple real-time shortwave infrared sub-image features, and multiple real-time thermal sub-image features is also the same.

[0053] Then, the processing device spatially aligns multiple RGB sub-image features, multiple real-time shortwave infrared sub-image features, and multiple real-time thermal image sub-image features into multiple sub-feature groups. Each sub-feature group includes one RGB sub-image feature, one real-time shortwave infrared sub-image feature, and one real-time thermal image sub-image feature at the same spatial location. Different sub-feature groups correspond to different spatial locations. Thus, using a hexagonal cellular network to segment the circular mirror sub-image, compared to traditional rectangular grids, allows for a tighter fit to the circular region of the mirror, reducing interference from background or invalid areas and improving computational efficiency. The hexagonal structure also exhibits better isotropy, allowing for better correlation between adjacent features, and avoids the "directional bias" of rectangular grids, enabling more uniform processing of defect features from different directions (such as radial scratches).

[0054] Thus, the above scheme proposes a new "star-shaped sub-network" fusion structure, using vibration characteristics representing the overall motion state as the core hub, and associating them with the RGB, infrared, and thermal characteristics of the same local region. This structural design guides AI models to explicitly explore the causal relationship between dynamic behavior and static appearance / thermal characteristics, which is particularly beneficial for discovering coupled fault modes caused by minute deformations due to vibration or by changes in the vibration spectrum caused by local overheating.

[0055] Finally, the processing device fuses the real-time vibration feature information into multiple sub-feature groups, resulting in multiple fused sub-feature groups. The fused real-time feature information is thus represented by these multiple fused sub-feature groups. This proposes a specific fusion path of "segmentation and alignment first, then group fusion." Decomposing the global image into local sub-feature groups for processing significantly reduces the complexity of directly processing high-dimensional data while preserving spatial details. Establishing an independent fusion unit for each spatial location enables the model to analyze the independent states of different regions on the lens in detail, providing the possibility of locating local damage.

[0056] For example, taking any one of the target sub-feature groups from multiple sub-feature groups as an example, this target sub-feature group includes the following three features corresponding to the same hexagonal pattern in the cellular network structure: a target RGB sub-image feature, a target real-time shortwave infrared sub-image feature, and a target real-time thermal image sub-image feature. For the target sub-feature group: the processing device can establish a target star-shaped connection relationship between the real-time vibration feature information and the target sub-feature group to obtain a fused target sub-feature group. The target star-shaped connection relationship indicates that the real-time vibration feature information is associated with the target RGB sub-image feature, the target real-time shortwave infrared sub-image feature, and the target real-time thermal image feature, respectively. The target star-shaped connection relationship is used by the AI ​​network model to associate and analyze the real-time vibration feature information with the target RGB sub-image feature, the target real-time shortwave infrared image feature, and the target real-time thermal image feature, respectively; thus, multiple fused sub-feature groups are obtained.

[0057] S203, the processing device analyzes the fused real-time feature information through an AI network model based on attention mechanism multi-task spatiotemporal fusion to obtain the real-time status of the lens of the galvanometer optical system.

[0058] For example, the processing device can establish associations among multiple fused sub-feature groups based on the positional adjacency of hexagonal patterns in the cellular mesh structure. In these associations, the two hexagonal patterns corresponding to two associated sub-feature groups are adjacent within the cellular mesh structure. This allows multiple fused sub-feature groups to also establish associations according to the cellular mesh structure. Specifically, the processing device assigns a unique structured index coordinate to each of the multiple fused sub-feature groups based on a preset mathematical model of the cellular mesh structure. Specifically, the cellular mesh structure is modeled as a two-dimensional cellular grid, where the center point of each hexagonal pattern is uniquely determined by its row index i and column index j, and the adjacency relationship is predefined as follows: if the Euclidean distance between the center points of two hexagonal patterns in the two-dimensional plane is equal to the side length of a single hexagon, then they are considered adjacent. Based on this definition, the processing device automatically identifies all coordinate pairs that satisfy the adjacency condition by traversing the index coordinates (i, j) of all sub-feature groups and calculating their Euclidean distances to all other coordinates (m, n) in the grid. For example, for the sub-feature group with index (0,0), the processing device calculates that its possible adjacent coordinates include (0,1), (1,0), and (1,-1). Subsequently, for each identified pair of adjacent coordinates, the processing device creates an undirected edge or an association record in its internal association graph data structure (such as an adjacency list or adjacency matrix) to explicitly indicate that there is an association relationship between the sub-feature groups corresponding to these two coordinates, thereby completely constructing an association network based on the location adjacency relationship of the cellular mesh structure.

[0059] The processing device inputs the correlation relationships and multiple fused sub-feature groups into the AI ​​network model for analysis, obtaining the real-time state of the galvanometer optical system lens. In this way, by utilizing the natural geometric adjacency relationships of the honeycomb grid to reorganize the fused local features, the AI ​​model can understand the spatial continuity and propagation relationships of defects (e.g., an ablation point may cause abnormal heat conduction in adjacent areas). This structured feature organization incorporates prior physical knowledge of the lens as a continuum, helping to improve the model's analytical rationality and consistency in identifying large-area continuous defects.

[0060] Specifically, the processing device can input the output vector of an AI network model based on attention mechanism multi-task spatiotemporal fusion into a state decoding and synthesis module. Specifically, this module first parses the predefined different dimensions in the output vector: for example, it maps the values ​​of the first few dimensions of the vector to predefined state categories (such as "clean," "dust contamination," "oil adhesion," "localized ablation," "thermal deformation," "film damage," "abnormal vibration," etc.) and their corresponding confidence probabilities through a Softmax classifier; simultaneously, it calculates a quantified comprehensive health score (e.g., a value between 0 and 100, where 100 represents intact and 0 represents complete failure) by passing the subsequent dimensions of the vector through a linear regression layer or a pre-calibrated mapping function. Furthermore, the module also parses the spatial location-encoded portion of the output vector, combining it with the index coordinates of the cellular structure to determine the specific lens region corresponding to any identified abnormal state (e.g., outputting "moderate ablation exists within the annular region covered by cellular coordinates (3,5) to (3,7)"). Finally, the processing device integrates the status category (and confidence level), comprehensive health score, and abnormal location information obtained from the above analysis to construct a structured data object (e.g., a JSON-formatted dictionary). This data object serves as a complete and operable technical description of the real-time status of the galvanometer optical system lens. For example, a specific status output instance could be: {“Status Category”: {“Oil Adhesion”: 0.92, “Local Overheating”: 0.87}, “Health Score”: 65, “Abnormal Location”: [“Cellular Area A2”, “Cellular Area B2”], “Timestamp”: “First Moment”}.

[0061] Optionally, the method further includes: the processing device performing lifecycle management on the galvanometer optical system lens based on the real-time status of the lens. For example, the processing device determines the remaining lifespan score of the galvanometer optical system lens based on the real-time status of the lens; the processing device performs lifecycle management on the galvanometer optical system lens based on the remaining lifespan score and the preset total lifespan score of the galvanometer optical system lens.

[0062] Thus, by introducing the quantitative indicator of "remaining lifespan score," the abstract "state" is transformed into an intuitive, measurable, and traceable lifespan budget. Combined with the preset "total lifespan score," the degree of lens health degradation can be dynamically assessed, providing users with clear replacement expectations and decision-making basis (such as "30% remaining lifespan, recommended for planned purchase"), greatly improving the planning of maintenance management.

[0063] In summary, this approach represents a leap from single-dimensional, qualitative judgment to multi-dimensional, quantitative analysis of galvanometer lens condition. By mandating spatial alignment and fusion of multimodal information, it solves the problem of data incompatibility between different sensors, laying a data foundation for constructing a unified and comprehensive lens health profile. The attention-based multi-task AI model can perform multiple tasks in parallel, including defect identification, localization, and severity assessment, significantly improving the automation level of monitoring and the accuracy of comprehensive judgment.

[0064] For easier understanding, here is an example:

[0065] Continuous monitoring and defect-induced testing were conducted on a standard 25.4 mm diameter gold-plated galvanometer lens. This was compared in parallel with a traditional monitoring system equipped only with a 5-megapixel visible light camera. In benchmark and single-defect tests, the data clearly demonstrated the depth perception capabilities of multimodal fusion. When approximately 3% of the lens surface was covered with submicron-sized silica dust, the traditional system failed to trigger an alarm due to insufficient contrast. However, this system, through analysis of the scattered light intensity distribution changes in polarized RGB images, identified "slight dust contamination" 1.5 hours after the contamination occurred, reducing the health score from an initial 99 to 85. For trace amounts of silicone oil contamination approximately 1.5 mm in diameter, the traditional system only reported "stain detected," while this system accurately classified it as "oil adhesion" based on the characteristic absorption of this area in the 1200 nm band of short-wave infrared images, achieving a health score of 78. The most significant difference lies in the detection of early laser ablation: a man-made micro-ablation point with a diameter of about 80 micrometers is indistinguishable from a dark stain in a visible light image, and traditional methods completely miss it; however, this method captures the abnormal local temperature rise of the point under continuous operation (1.8°C higher than the surrounding area) using a thermal imager, and combined with high-resolution image features, it diagnoses "micro-ablation" within the first working cycle after the ablation point appears, and the health score drops sharply to 65.

[0066] In tests of composite and latent defects, this approach reveals its core value in enabling predictive maintenance. We simulated a progressive failure: applying a thin adhesive at the lens edge caused micron-level periodic deformation under high-speed scanning. Traditional approaches failed to report any valid anomalies throughout the process. Our approach, however, extracted an anomalous resonant frequency of 125Hz from subpixel-level displacement sequences captured by high-speed CMOS via Fourier transform; simultaneously, a miniature thermal imager showed a sustained temperature rise of 0.5°C in the edge region due to frictional loss. The system correlated these features through a star-shaped network, issuing a high-level warning of "structural anomalous vibration with localized overheating" at the 36th hour. At this point, although the lens's optical performance had not significantly deteriorated, potential failure was already indicated. Furthermore, for changes in thermal stress distribution caused by long-term operation without surface traces, this approach qualitatively identified and quantitatively assessed birefringence fringe distortion captured by a polarization camera.

[0067] Based on all test data, this solution improves the accuracy of state recognition from 67% to 98% compared to the traditional single-modal solution, and advances the average warning time for major defects by 72 hours. More importantly, its output quantitative health score is highly correlated with the actual reflectivity decrease curve of the lens measured by a third-party precision instrument (R²=0.94), which directly proves that the score can serve as a reliable basis for predicting remaining lifetime (RUL). These data fully validate that this solution, through multimodal deep fusion, achieves a leap from superficial to mechanistic understanding of lens state, and from delayed alarms to early prediction.

[0068] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Exemplarily, the electronic device may be a terminal, or it may be a chip (system) or other component or assembly that can be configured for a specific purpose. Figure 3 As shown, the electronic device 400 may include a processor 401. Optionally, the electronic device 400 may also include a memory 402 and / or a transceiver 403. The processor 401 is coupled to the memory 402 and the transceiver 403, for example, via a communication bus.

[0069] The following is combined with Figure 3 A detailed description of each component of the electronic device 400 is provided below:

[0070] The processor 401 is the control center of the electronic device 400. It can be a single processor or a collective term for multiple processing elements. For example, the processor 401 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0071] Optionally, the processor 401 can perform various functions of the electronic device 400 by running or executing software programs stored in the memory 402 and calling data stored in the memory 402, such as performing the aforementioned functions. Figure 2 The method shown is an online monitoring method for the lens status of a galvanometer optical system based on AI vision.

[0072] In a specific implementation, as one example, processor 401 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0073] In a specific implementation, as one example, the electronic device 400 may also include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0074] The memory 402 is used to store the software program that executes the solution of the present invention, and is controlled by the processor 401 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0075] Optionally, the memory 402 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 402 may be integrated with the processor 401 or exist independently, and may be accessed through the interface circuit of the electronic device 400. Figure 3 (Not shown in the image) is coupled to processor 401, and this embodiment of the invention does not specifically limit this.

[0076] Transceiver 403 is used for communication with other electronic devices. For example, if electronic device 400 is a terminal, transceiver 403 can be used to communicate with a network device or with another terminal device. As another example, if electronic device 400 is a network device, transceiver 403 can be used to communicate with a terminal or with another network device.

[0077] Alternatively, transceiver 403 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0078] Alternatively, the transceiver 403 can be integrated with the processor 401, or it can exist independently and be connected via the interface circuit of the electronic device 400. Figure 3 (Not shown in the image) is coupled to processor 401, and this embodiment of the invention does not specifically limit this.

[0079] Understandable, Figure 3 The structure of the electronic device 400 shown does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] Furthermore, the technical effects of the electronic device 400 can be referred to the technical effects of the methods described in the above method embodiments, and will not be repeated here.

[0081] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0082] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0083] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0084] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0085] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0086] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0087] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0088] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for online monitoring of lens status in a galvanometer optical system based on AI vision, characterized in that, Applied to a processing device, the method includes: The processing device acquires real-time characteristic information of the multimodal properties of the lens of the galvanometer optical system. The multimodal properties include at least two modes among visible light, short-wave infrared, thermal, or motion states. The real-time characteristic information of the multimodal properties is obtained by real-time sensing of the lens of the galvanometer optical system under the condition that the multimodal properties are spatially aligned. The processing device performs multimodal fusion on the real-time characteristic information of the multimodal modes in a spatially aligned manner to obtain fused real-time feature information; The processing device analyzes the fused real-time feature information through an AI network model based on attention mechanism multi-task spatiotemporal fusion to obtain the real-time state of the lens of the galvanometer optical system. The processing device acquires real-time characteristic information of the multimodal properties of the galvanometer optical system lenses, including: The processing device acquires real-time high-resolution polarized RGB images, real-time short-wave infrared images, real-time thermal images, and real-time vibration characteristic information of the lens of the galvanometer optical system. The real-time multimodal characteristic information of the lens includes the real-time high-resolution polarized RGB images, the real-time short-wave infrared images, the real-time thermal images, and the real-time vibration characteristic information. The processing device performs multimodal fusion on the real-time characteristic information of the multimodals in a spatially aligned manner to obtain fused real-time feature information, including: The processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension, the processing device segments the real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features in the spatial dimension, and the processing device segments the real-time thermal image into multiple real-time thermal image sub-image features in the spatial dimension. The processing device spatially aligns the plurality of RGB sub-image features, the plurality of real-time shortwave infrared sub-image features, and the plurality of real-time thermal image sub-image features into a plurality of sub-feature groups. Each sub-feature group includes one RGB sub-image feature, one real-time shortwave infrared sub-image feature, and one real-time thermal image feature at the same spatial location. Different sub-feature groups correspond to different spatial locations. The processing device fuses the real-time vibration feature information into the plurality of sub-feature groups to obtain a plurality of fused sub-feature groups, and the fused real-time feature information is the plurality of fused sub-feature groups; The processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension, including: The processing device extracts an RGB sub-image from the real-time high-resolution polarized RGB image that contains only the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device divides the RGB sub-image into multiple RGB honeycomb sub-images using a honeycomb structure. Each of the multiple RGB honeycomb sub-images is a hexagonal pattern. The multiple RGB honeycomb sub-images can be stitched together using the honeycomb structure to form the RGB sub-image. The processing device extracts the feature information of each of the multiple RGB honeycomb sub-images through a first convolutional layer to obtain the features of the multiple RGB sub-images.

2. The method according to claim 1, characterized in that, The processing device acquires real-time high-resolution polarized RGB images of the galvanometer optical system lenses, real-time short-wave infrared images of the galvanometer optical system lenses, real-time thermal images of the galvanometer optical system lenses, and real-time vibration characteristic information of the galvanometer optical system lenses, including: The processing device acquires a real-time high-resolution polarized RGB image captured by a high-resolution polarization camera under bright / dark field illumination at a first moment and from a first viewing angle toward the lens of the galvanometer optical system. The processing device also acquires a real-time short-wave infrared image captured by a short-wave infrared camera at the first moment and from the first viewing angle toward the lens of the galvanometer optical system. Furthermore, the processing device acquires a real-time video stream captured by a high-speed global shutter CMOS sensor at the first viewing angle toward the lens of the galvanometer optical system, and extracts the real-time vibration feature information from the real-time video stream. The time period of the real-time video stream includes the first moment.

3. The method according to claim 1, characterized in that, The processing device segments the real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features in the spatial dimension, including: The processing device extracts an infrared sub-image from the real-time shortwave infrared image that contains only the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device divides the infrared sub-image into multiple infrared honeycomb sub-images using the honeycomb structure. Each of the multiple infrared honeycomb sub-images is a hexagonal pattern. The multiple infrared honeycomb sub-images can be stitched together using the honeycomb structure to form the infrared sub-image. The processing device extracts the feature information of each of the multiple infrared honeycomb sub-images through a second convolutional layer to obtain the features of the multiple real-time shortwave infrared sub-images. Furthermore, the processing device segments the real-time thermal image into multiple real-time thermal image sub-image features in the spatial dimension, including: The processing device extracts a thermal image sub-image from the real-time thermal image that contains only the area where the galvanometer optical system lens is located. The area where the galvanometer optical system lens is located is a circular area. The processing device divides the thermal image sub-image into multiple thermal image honeycomb sub-images using the honeycomb structure. Each of the multiple thermal image honeycomb sub-images is a hexagonal pattern. The multiple thermal image honeycomb sub-images can be stitched together using the honeycomb structure to form the thermal image sub-image. The processing device extracts the feature information of each of the multiple thermal image honeycomb sub-images through a third convolutional layer to obtain the features of the multiple real-time thermal image sub-images. The number of the plurality of RGB honeycomb sub-images, the plurality of infrared honeycomb sub-images, and the plurality of thermal honeycomb sub-images are the same, and correspondingly, the number of the plurality of RGB sub-image features, the plurality of real-time shortwave infrared sub-image features, and the plurality of real-time thermal sub-image features are also the same.

4. The method according to claim 3, characterized in that, Each of the plurality of sub-feature groups includes the following three features corresponding to the same hexagonal pattern in the cellular structure: a target RGB sub-image feature, a target real-time shortwave infrared sub-image feature, and a target real-time thermal image sub-image feature. The processing device fuses the real-time vibration feature information into the plurality of sub-feature groups to obtain a plurality of fused sub-feature groups, including: For the target sub-feature group: The processing device establishes a target star-shaped connection relationship between the real-time vibration feature information and the target sub-feature group to obtain a fused target sub-feature group. The connection relationship of the target star indicates that the real-time vibration feature information is associated with the target RGB sub-image feature, the target real-time shortwave infrared sub-image feature, and the target real-time thermal image sub-image feature, respectively. The connection relationship of the target star is used by the AI ​​network model to associate and analyze the real-time vibration feature information with the target RGB sub-image feature, the target real-time shortwave infrared sub-image feature, and the target real-time thermal image feature, respectively. Thus, a total of multiple fused sub-feature groups are obtained.

5. The method according to claim 3 or 4, characterized in that, The processing device analyzes the fused real-time feature information using an AI network model based on attention mechanism multi-task spatiotemporal fusion to obtain the real-time state of the lenses in the galvanometer optical system, including: The processing device establishes the association relationship of the multiple fused sub-feature groups based on the positional adjacency of each hexagonal pattern in the cellular network structure. In the association relationship, the two hexagonal patterns corresponding to two associated sub-feature groups in the multiple fused sub-feature groups are adjacent in the cellular network structure. The processing device inputs the correlation and the multiple fused sub-feature groups into the AI ​​network model for analysis to obtain the real-time status of the lens of the galvanometer optical system.

6. The method according to claim 1, characterized in that, The method further includes: The processing device performs lifecycle management on the lenses of the galvanometer optical system based on their real-time status.

7. The method according to claim 6, characterized in that, The processing device performs lifecycle management on the lenses of the galvanometer optical system based on their real-time status, including: The processing device determines the remaining life fraction of the lenses in the galvanometer optical system based on the real-time status of the lenses. The processing device performs lifecycle management on the lenses of the galvanometer optical system based on the remaining life score and the preset total life score of the lenses of the galvanometer optical system.

8. An online monitoring device for the state of a galvanometer optical system lens based on AI vision, the device being configured to: acquire real-time multimodal characteristic information of the galvanometer optical system lens, the multimodality including at least two modes selected from visible light, shortwave infrared, thermal, or motion states, the real-time characteristic information being obtained by real-time sensing of the galvanometer optical system lens under spatial alignment of the multimodality; fuse the real-time characteristic information of the multimodality in a spatially aligned manner to obtain fused real-time feature information; and analyze the fused real-time feature information using a multi-task spatiotemporal fusion AI network model based on an attention mechanism to obtain the real-time state of the galvanometer optical system lens; in, The processing equipment acquires real-time multimodal characteristic information of the lenses in the galvanometer optical system, including: The processing device acquires real-time high-resolution polarized RGB images, real-time short-wave infrared images, real-time thermal images, and real-time vibration characteristic information of the lens of the galvanometer optical system. The real-time multimodal characteristic information of the lens includes the real-time high-resolution polarized RGB images, the real-time short-wave infrared images, the real-time thermal images, and the real-time vibration characteristic information. The processing device performs multimodal fusion on the real-time characteristic information of the multimodals in a spatially aligned manner to obtain fused real-time feature information, including: The processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension, the processing device segments the real-time shortwave infrared image into multiple real-time shortwave infrared sub-image features in the spatial dimension, and the processing device segments the real-time thermal image into multiple real-time thermal image sub-image features in the spatial dimension. The processing device spatially aligns the plurality of RGB sub-image features, the plurality of real-time shortwave infrared sub-image features, and the plurality of real-time thermal image sub-image features into a plurality of sub-feature groups. Each sub-feature group includes one RGB sub-image feature, one real-time shortwave infrared sub-image feature, and one real-time thermal image feature at the same spatial location. Different sub-feature groups correspond to different spatial locations. The processing device fuses the real-time vibration feature information into the plurality of sub-feature groups to obtain a plurality of fused sub-feature groups, and the fused real-time feature information is the plurality of fused sub-feature groups; The processing device segments the real-time high-resolution polarized RGB image into multiple RGB sub-image features in the spatial dimension, including: The processing device extracts an RGB sub-image from the real-time high-resolution polarized RGB image that contains only the region where the galvanometer optical system lens is located. The region where the galvanometer optical system lens is located is a circular region. The processing device divides the RGB sub-image into multiple RGB honeycomb sub-images using a honeycomb structure. Each of the multiple RGB honeycomb sub-images is a hexagonal pattern. The multiple RGB honeycomb sub-images can be stitched together using the honeycomb structure to form the RGB sub-image. The processing device extracts the feature information of each of the multiple RGB honeycomb sub-images through a first convolutional layer to obtain the features of the multiple RGB sub-images.

Images