Dashboard visual inspection system and method based on multi-modal deep learning

Abstract

The application discloses a kind of based on multimodal deep learning's instrument panel visual detection system and method, specifically related to instrument panel visual detection technical field, the system includes: central processing module, hardware diagnostic module, multimodal acquisition module, cross-modal feature alignment network (contains CM-ATT module), adaptive decision fusion module and time sequence verification module.A kind of based on multimodal deep learning's instrument panel visual detection system and method described in the application, the application is fused RGB, IR, Depth feature by multimodal high-precision synchronous acquisition and CM-ATT module dynamic alignment, combine each modal advantage, simultaneously based on real-time confidence dynamic adjustment fusion weight, avoid the limitation of fixed weight, improve the comprehensiveness and accuracy of instrument panel detection;Hardware diagnostic module can offset equipment inherent error by self-calibration algorithm, guarantee high-precision detection demand;Central processing module can improve abnormal response speed by dynamic allocation of computing resources.

Description

Technical Field

[0001] This invention relates to the field of dashboard visual inspection technology, and in particular to a dashboard visual inspection system and method based on multimodal deep learning. Background Technology

[0002] In existing dashboard visual inspection technologies, single-modal detection methods have obvious limitations. They are difficult to fully capture the complex state information of the dashboard. Detection schemes that rely solely on visible light images are easily affected by factors such as changes in illumination and surface reflection, resulting in unstable character recognition accuracy. Temperature detection methods that only use infrared sensing cannot simultaneously acquire the spatial pose information of the dashboard's character display and pointers, making it difficult to achieve multi-dimensional state assessment. Single-depth detection cannot reflect thermal characteristics such as temperature anomalies, and the information dimension is limited.

[0003] While multimodal fusion detection is gradually being applied, key technical bottlenecks remain: First, the time synchronization accuracy of multimodal data acquisition is insufficient. Time misalignment of modal data such as RGB, IR, and Depth can lead to feature association distortion, affecting the fusion effect. Second, cross-modal feature alignment and fusion mechanisms lack dynamic adaptability. Fixed weight allocation methods cannot adjust the fusion strategy according to the real-time confidence of each modality, making it difficult to fully leverage the advantages of high-reliability modalities. Third, inherent errors in hardware devices (such as camera focal length shift, sensor temperature drift, calibration errors, etc.) are not effectively compensated. In high-precision detection scenarios (such as temperature accuracy ≤ ±0.1℃, depth Z-axis resolution ≤ 0.05mm), detection accuracy is easily affected. Fourth, computational resource allocation is mostly static and cannot be dynamically adjusted according to the priority of detection tasks (such as the need for key monitoring during temperature anomalies), resulting in a lag in anomaly response speed. Fifth, the lack of temporal analysis of pointer movement continuity makes it difficult to identify dynamic anomalies such as instantaneous jitter in single-frame detection results, affecting the integrity and reliability of detection. Therefore, there is a need for a dashboard visual inspection system and method based on multimodal deep learning. Summary of the Invention

[0004] The main objective of this invention is to provide a dashboard visual inspection system and method based on multimodal deep learning, which can effectively solve the problems mentioned above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A dashboard visual inspection system based on multimodal deep learning, comprising: The system comprises a central processing module, a hardware diagnostic module, a multimodal acquisition module, a cross-modal feature alignment network (including a CM-ATT module), an adaptive decision fusion module, and a time-series verification module. The central processing module is used to perform dynamic task scheduling, allocate computing resources according to the priority of detection tasks, and generate anomaly location heatmaps based on the fusion results. The hardware diagnostic module is used to monitor the focal length shift of the visible light camera, the temperature drift of the infrared sensor, and the calibration error of the depth camera in real time, and to compensate for the detection error through a self-calibration algorithm. The multimodal acquisition module includes a polarization-filtered visible light camera, a macro infrared sensor, and a structured light depth camera. The three-modal data are acquired synchronously through a hardware trigger signal. The infrared sensor operates in the 8-14μm band with a temperature detection accuracy of ≤±0.1℃. The depth camera has a Z-axis resolution of ≤0.05mm, and the time error for synchronous acquisition of the three-modal data is <1ms. The cross-modal feature alignment network embeds a cross-modal attention module to dynamically align and fuse RGB, IR, and Depth features. The adaptive decision fusion module assigns weights to each modality based on real-time confidence. The timing verification module uses an LSTM network to analyze the continuity of pointer movement.

[0006] Preferably, the dynamic task scheduling of the central processing module includes: Initially, 40% of computing resources are allocated to visible light character recognition, 30% to infrared temperature detection, and 30% to depth pose resolution. When an abnormal temperature is detected, the infrared branch resources are dynamically increased to 50%, while the resource allocation of other branches is reduced accordingly.

[0007] Preferably, the self-calibration algorithm of the hardware diagnostic module includes a focal length compensation formula: ; in The change in temperature This represents the change in humidity. The compensation coefficient is determined through calibration experiments; Infrared temperature drift compensation: Temperature readings are corrected in real time using a blackbody reference source; Depth camera calibration: Dynamically update the intrinsic parameter matrix based on a checkerboard calibration board.

[0008] Preferably, the cross-modal attention (CM-ATT) module performs feature-level fusion, specifically including: The formula for calculating fusion features is: ; in, , These are the feature vectors for the RGB, IR, and Depth modes, respectively. , , These are the fusion weight coefficients for the corresponding modes; this formula is used to weight and integrate the features of the three modes to form a comprehensive fusion feature, in order to combine the advantages of each mode (such as RGB texture information, IR temperature information, and Depth spatial pose information).

[0009] The weighting coefficients are generated using a temperature-shortening confidence mechanism: ; in To score modal confidence, It has an adjustable temperature coefficient (0.1-5.0).

[0010] Preferably, the adaptive decision fusion module includes branches: First branch: YOLOv7 model detects digital screen characters in visible light images; Second branch: ResNet-18 model identifies temperature anomaly regions in infrared images; Third branch: Model analysis of pointer space pose in depth images; The pointer pose analysis in the third branch includes angle deviation calculation: ; in For standard vectors, For the detection vector, At that time, it was judged as qualified.

[0011] Preferably, the LSTM network of the time-series verification module receives a fused feature sequence of 5 consecutive frames as input. If the pointer angle difference between adjacent frames is... If the jitter persists for more than 3 frames, a jitter warning will be triggered. The polarization-filtered visible light camera in the multimodal acquisition module has a dynamically adjustable filter angle, and the adjustment strategy is as follows: ; in These represent the maximum, minimum, and average grayscale values ​​of the image, respectively.

[0012] A dashboard visual inspection method based on multimodal deep learning includes the following steps: S1: Synchronously acquire RGB image, IR thermal image and depth map of the dashboard through hardware trigger signal; S2: Extract visible light character features, infrared temperature anomaly features, and depth space pose features respectively; S3: Cross-modal alignment and weighted fusion of trimodal features are performed using the CM-ATT module; S4: Analysis of the temporal continuity of pointer movement based on the LSTM model; S5: Output detection results and anomaly location heatmap.

[0013] Preferably, in step S1, synchronous acquisition is achieved by generating a hardware trigger pulse through an FPGA, and the rising edge of the pulse triggers synchronous exposure of the three cameras. In step S2, the infrared temperature anomaly detection uses an adaptive threshold: ; in The average temperature of the region. Standard deviation This is an adjustable coefficient (default value 3.0); The fusion weights of S3 The confidence level is updated through online learning. If the submodal detection results are consistent with the manual review, the confidence level is improved. If there are 3 consecutive false positives, the weight will be reset.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves high-precision synchronous acquisition of multimodal data and dynamic alignment and fusion of RGB, IR, and Depth features using the CM-ATT module. It combines the advantages of each modality and dynamically adjusts the fusion weights based on real-time confidence levels, avoiding the limitations of fixed weights and improving the comprehensiveness and accuracy of dashboard detection.

[0015] 2. This invention uses a hardware diagnostic module and a self-calibration algorithm to offset inherent equipment errors, ensuring high-precision detection requirements; the central processing module improves anomaly response speed by dynamically allocating computing resources; and the timing verification module analyzes the continuity of pointer movement based on LSTM, solving the problems of delayed response from static resource allocation and the inability to identify dynamic anomalies in single-frame detection, thus comprehensively improving detection reliability and real-time performance. Attached Figure Description

[0016] Figure 1 This is a flowchart of the overall modules of the present invention; Figure 2 This is a flowchart illustrating the overall method steps of the present invention. Detailed Implementation

[0017] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0018] like Figure 1 and Figure 2 As shown, a dashboard visual inspection system based on multimodal deep learning includes: The system comprises a central processing module, a hardware diagnostic module, a multimodal acquisition module, a cross-modal feature alignment network (including a CM-ATT module), an adaptive decision fusion module, and a time-series verification module. The central processing module is used to perform dynamic task scheduling, allocate computing resources according to the priority of detection tasks, and generate anomaly location heatmaps based on the fusion results. The dynamic task scheduling of the central processing module includes: Initially, 40% of computing resources are allocated to visible light character recognition, 30% to infrared temperature detection, and 30% to depth pose resolution. When an abnormal temperature is detected, the infrared branch resources are dynamically increased to 50%, while the resource allocation of other branches is reduced accordingly. In the above, the central processing module adopts a priority-driven resource allocation strategy. In the initial state, computing resources are allocated to visible light character recognition, infrared temperature detection, and depth pose analysis in a ratio of 40%, 30%, and 30%, respectively. This allocation ratio is based on the importance of each task in a typical detection scenario—visible light character recognition needs to process texture details and consumes the most resources; infrared and depth detection have relatively balanced requirements.

[0019] When the hardware diagnostic module or adaptive decision fusion module detects an abnormal temperature (such as the local temperature of the dashboard exceeding the normal range), the system triggers dynamic adjustments: increasing infrared branch resources to 50%, reducing visible light character recognition to 30%, and reducing depth pose resolution to 20%. This mechanism ensures sufficient computing power for critical detection tasks in abnormal scenarios, improving response speed. By dynamically allocating resources, adaptive switching can be achieved between normal detection and anomaly investigation scenarios, balancing detection efficiency and anomaly sensitivity. In the above, the anomaly location heatmap is generated based on cross-modal fusion results. The central processing module generates a visual heatmap by weighted superposition of the anomaly probability distribution of each modality (such as infrared temperature anomaly area and depth pose deviation area). The darker the color in the heatmap, the higher the anomaly confidence level, which intuitively indicates the anomaly location of the dashboard and assists in manual review and fault location.

[0020] The hardware diagnostic module is used to monitor the focal length shift of the visible light camera, the temperature drift of the infrared sensor, and the calibration error of the depth camera in real time, and to compensate for the detection error through a self-calibration algorithm. The self-calibration algorithm of the hardware diagnostic module includes a focal length compensation formula: ; in The change in temperature This represents the change in humidity. The compensation coefficient is determined through calibration experiments; Infrared temperature drift compensation: Temperature readings are corrected in real time using a blackbody reference source; Depth camera calibration: Dynamically update the intrinsic parameter matrix based on a checkerboard calibration board; In the above, real-time compensation for focal length shift caused by temperature and humidity ensures the clarity of visible light images and the accuracy of character recognition, avoiding misjudgment caused by image blur. Infrared sensor temperature drift compensation: A blackbody reference source (with a known and stable temperature) is used as a reference. The measured value of the blackbody by the infrared sensor is compared with the actual value in real time, the deviation is calculated, and the output is corrected. For example, when the actual temperature of the blackbody is 50℃ and the sensor measurement is 50.2℃, 0.2℃ is uniformly subtracted from the subsequent measurement results.

[0021] Effect: Eliminates temperature drift caused by prolonged operation of infrared sensors, ensuring temperature detection accuracy ≤ ±0.1℃, meeting the high-precision requirements for thermal anomaly detection in instrument panels.

[0022] Depth camera calibration error compensation: Based on a checkerboard calibration board (with known checkerboard dimensions), images of the calibration board are periodically acquired from different angles, and the intrinsic parameter matrix (focal length, principal point coordinates, etc.) is dynamically updated using the Zhang Zhengyou calibration method. For example, when the depth camera's measurement error to a calibration board at a known distance (e.g., 1m) exceeds 0.03mm, an intrinsic parameter update is triggered.

[0023] Effect: It compensates for calibration errors caused by vibration and temperature changes in the depth camera, ensuring Z-axis resolution ≤0.05mm, and providing accurate depth data for pointer spatial pose analysis.

[0024] The multimodal acquisition module includes a polarization-filtered visible light camera, a macro infrared sensor, and a structured light depth camera. The three-modal data are acquired synchronously through a hardware trigger signal. The infrared sensor operates in the 8–14 μm band with a temperature detection accuracy of ≤ ±0.1℃. The depth camera has a Z-axis resolution of ≤ 0.05 mm. The time error of the synchronous acquisition of the three-modal data is < 1 ms. The polarization-filtered visible light camera in the multimodal acquisition module has a dynamically adjustable filter angle, and the adjustment strategy is as follows: ; in These represent the maximum, minimum, and average gray values ​​of the image, respectively. For example, when the dashboard surface reflects light, resulting in large differences in grayscale ( When the value is large, The angle is increased to reduce reflected light and improve the contrast of the character area.

[0025] Effect: Adaptively eliminates surface reflection interference, ensuring the integrity and readability of characters in visible light images.

[0026] Micro infrared sensor: operating band 8-14μm (this band is the heat radiation sensitive range, which can effectively capture the heat of the instrument panel circuit), temperature detection accuracy ≤±0.1℃, ensuring that small temperature anomalies (such as local overheating of 0.2℃) can be identified.

[0027] Structured light depth camera: By projecting coded structured light, it calculates the three-dimensional coordinates of the dashboard surface with a Z-axis resolution of ≤0.05mm, which can accurately capture the tiny displacement of the pointer (such as a tilt of 0.03mm).

[0028] The cross-modal feature alignment network embeds a cross-modal attention module to dynamically align and fuse RGB, IR, and Depth features. The cross-modal attention (CM-ATT) module performs feature-level fusion, specifically including: The formula for calculating fusion features is: ; in, , These are the feature vectors for the RGB, IR, and Depth modes, respectively. , , These are the fusion weight coefficients for the corresponding modalities; this formula is used to weight and integrate the features of the three modalities to form a comprehensive fusion feature, in order to combine the advantages of each modality (such as RGB texture information, IR temperature information, and Depth spatial pose information). The weighting coefficients are generated using a temperature-shortening confidence mechanism: ; in, For modal confidence scoring (such as character recognition accuracy in RGB mode, temperature detection consistency in IR mode), It has an adjustable temperature coefficient (0.1-5.0). The smaller the value, the more significant the weighting shifts towards higher confidence modes; Effect: Weighted fusion achieves complementary advantages of multiple modes: the texture information of RGB, the temperature information of IR, and the spatial information of Depth are comprehensively utilized to avoid the limitations of a single mode (such as RGB being easily affected by lighting, and IR having difficulty recognizing characters).

[0029] Dynamic weighting mechanisms enable high-confidence modes (such as IR under temperature anomalies) to... rise, Increase the dominance of the fusion results and improve feature reliability. For example, when IR detects a clear temperature anomaly, The effect of temperature characteristics is enhanced as the value increases from 0.3 to 0.6.

[0030] The adaptive decision fusion module assigns weights to each modality based on real-time confidence. The adaptive decision fusion module includes branches: First branch: YOLOv7 model detects digital screen characters in visible light images; Second branch: ResNet-18 model identifies temperature anomaly regions in infrared images; Third branch: PointNet++ model parsing pointer space poses in depth images; The pointer pose analysis in the third branch includes angle deviation calculation: ; in For standard vectors, For the detection vector, At that time, it was judged as qualified.

[0031] Effect: The three branches each perform their own functions, covering three key detection indicators of the dashboard: characters, temperature, and position. The angle deviation formula accurately calculates the angle difference through vector dot product, ensuring that even a slight pointer tilt (such as 0.3°) can be detected, thus meeting the requirements for high-precision pose verification.

[0032] The timing verification module uses an LSTM network to analyze the continuity of pointer movement.

[0033] The timing verification module inputs a fused feature sequence of 5 consecutive frames into its LSTM network. If the pointer angle difference between adjacent frames is... If the jitter persists for more than 3 frames, a jitter warning will be triggered. Effect: LSTM's temporal modeling capability can identify non-smooth motion (such as pointer jitter caused by mechanical failure), making up for the limitations of single-frame detection (a single frame cannot determine whether it is an instantaneous error). Triggering conditions for three consecutive frames avoids false alarms caused by accidental noise (such as a 1.2° angle jump in a single frame, which may be due to measurement error), thus improving the reliability of warnings.

[0034] A dashboard visual inspection method based on multimodal deep learning includes the following steps: S1: Synchronous acquisition of multimodal data: Hardware trigger pulses are generated by FPGA, and the rising edge synchronously triggers the exposure of the polarization filter visible light camera, macro infrared sensor, and structured light depth camera to acquire the RGB image (resolution 1920×1080), IR thermal image (resolution 640×512), and depth map (resolution 1280×720) of the instrument panel, with a time error of <1ms. S2: Extracting Unimodal Features - Visible Light Character Features: Extracting features such as the contour and grayscale value of characters on the digital screen through edge detection and threshold segmentation, and inputting them into the YOLOv7 model. - Infrared Temperature Anomaly Features: Calculating the mean and standard deviation of the temperature in each sub-region through region division, and inputting them into the ResNet-18 model. Anomaly detection uses adaptive thresholding. ;in, The average temperature of the region. Standard deviation (Default value, can be adjusted according to noise level).

[0035] Deep spatial pose features: The three-dimensional coordinates and orientation vector of the pointer are extracted by point cloud downsampling and normal estimation and input into the PointNet++ model; S3: Cross-modal feature fusion: The three-modal features extracted by S2 are input into the CM-ATT module, and the fusion weights are calculated using a formula. ( (Enhancing weight discrimination) to generate fused features .

[0036] The fusion weights are updated through online confidence learning: if the single-modal detection result is consistent with the manual review, Increase by 0.1; if there are 3 consecutive false positives, Reset to the initial value of 0.5; S4: Timing continuity verification: Five consecutive frames of fused feature sequences are input into an LSTM network (64 hidden layer dimensions, 100 iterations), which outputs a score for the continuity of pointer motion. If the angle difference between adjacent frames... ° and lasts for 3 frames, triggering a jitter abnormality warning; S5: Output detection results: The central processing module integrates the results of each branch with the time-series verification conclusions, generates a detection report that includes "character recognition results", "temperature status" and "pointer pose deviation", and generates a heat map based on the probability distribution of abnormal areas, which is then output to the display terminal.

[0037] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A dashboard visual inspection system based on multimodal deep learning, characterized in that... The system includes: The system comprises a central processing module, a hardware diagnostic module, a multimodal acquisition module, a cross-modal feature alignment network, an adaptive decision fusion module, and a time-series verification module. The central processing module is used to perform dynamic task scheduling, allocate computing resources according to the priority of detection tasks, and generate anomaly location heatmaps based on the fusion results. The hardware diagnostic module is used to monitor the focal length shift of the visible light camera, the temperature drift of the infrared sensor, and the calibration error of the depth camera in real time, and compensates for the detection errors through a self-calibration algorithm; the self-calibration algorithm of the hardware diagnostic module includes a focal length compensation formula: ;in The change in temperature This represents the change in humidity. The compensation coefficients were determined through calibration experiments; infrared temperature drift compensation: temperature readings were corrected in real time using a blackbody reference source; depth camera calibration: the intrinsic parameter matrix was dynamically updated based on a checkerboard calibration board; The multimodal acquisition module includes a polarization-filtered visible light camera, a macro infrared sensor, and a structured light depth camera. The three-modal data are acquired synchronously through a hardware trigger signal. The infrared sensor operates in the 8-14μm band with a temperature detection accuracy of ≤±0.1℃. The depth camera has a Z-axis resolution of ≤0.05mm, and the time error for synchronous acquisition of the three-modal data is <1ms. The cross-modal feature alignment network embeds a cross-modal attention module to dynamically align and fuse RGB, IR, and Depth features. The cross-modal feature alignment network performs feature-level fusion, specifically including the following feature calculation formula: ;in, , These are the feature vectors for the RGB, IR, and Depth modes, respectively. , , These are the fusion weight coefficients for the corresponding modalities; this formula is used to weight and integrate the features of the three modalities to form a comprehensive fusion feature, combining the advantages of each modality; the weight coefficients are generated through a temperature-based confidence mechanism. ;in To score modal confidence, It has an adjustable temperature coefficient; The adaptive decision fusion module assigns weights to each modality based on real-time confidence; the adaptive decision fusion module includes branches: First branch: YOLOv7 model detects digital screen characters in visible light images; Second branch: ResNet-18 model identifies temperature anomaly regions in infrared images; The third branch: PointNet++ model parses the pointer space pose in the depth image. The pointer pose parsing in the third branch includes angle deviation calculation. ;in For standard vectors, For the detection vector, At that time, it was judged as qualified; The timing verification module uses an LSTM network to analyze the continuity of pointer motion; the LSTM network of the timing verification module is input with a fused feature sequence of 5 consecutive frames, and if the pointer angle difference between adjacent frames is... If the jitter persists for more than 3 frames, a jitter warning will be triggered. The polarization-filtered visible light camera in the multimodal acquisition module has a dynamically adjustable filter angle, and the adjustment strategy is as follows: ;in These represent the maximum, minimum, and average grayscale values ​​of the image, respectively.

2. The dashboard visual inspection system based on multimodal deep learning according to claim 1, characterized in that, The dynamic task scheduling of the central processing module includes: Initially, 40% of computing resources are allocated to visible light character recognition, 30% to infrared temperature detection, and 30% to depth pose resolution. When an abnormal temperature is detected, the infrared branch resources are dynamically increased to 50%, while the resource allocation of other branches is reduced accordingly.

3. A visual inspection method for dashboards based on multimodal deep learning, using the system described in claim 1 or 2, characterized in that, Includes the following steps: S1: Synchronously acquire RGB image, IR thermal image and depth map of the dashboard through hardware trigger signal; S2: Extract visible light character features, infrared temperature anomaly features, and depth space pose features respectively; S3: Cross-modal alignment and weighted fusion of trimodal features are performed through a cross-modal feature alignment network; S4: Analyzing the temporal continuity of pointer movement based on LSTM network; S5: Output detection results and anomaly location heatmap.

4. The dashboard visual inspection method based on multimodal deep learning according to claim 3, characterized in that: In S1, synchronous acquisition is achieved by generating hardware trigger pulses through FPGA, and the rising edge of the pulse triggers synchronous exposure of the three cameras. In step S2, the infrared temperature anomaly detection uses an adaptive threshold: ;in The average temperature of the region. Standard deviation, This is an adjustable coefficient; The fusion weights of S3 The confidence level is updated through online learning. If the submodal detection results are consistent with the manual review, the confidence level is improved. If there are 3 consecutive false positives, the weight will be reset.

Images