Vehicle machine screen two-dimensional coupling detection method and system

Abstract

The invention discloses a vehicle machine screen two-dimensional coupling detection method and system, and belongs to the technical field of computer vision and industrial automation detection. Obtaining a vehicle machine screen image stream through a multi-source video access layer; the vehicle machine screen image flow is input into a pre-trained YOLOv5-Heat multi-mode model, a screen abnormity type is recognized, and the screen abnormity type comprises at least one of color cast, dead pixels, full-screen abnormity and overheat points; analyzing the identified screen abnormity by adopting a two-dimensional coupling algorithm, and generating an early warning signal of a corresponding grade based on a result of the two-dimensional coupling algorithm; and when the early warning signal level is serious abnormity, an abnormal signal is converted into a CAN frame through the weekly standing power module, an instruction is sent to the execution module through the control bus, the relay module is controlled to execute power-off operation, and the vehicle machine is triggered to be restarted. The method is obviously superior to the prior art in five dimensions of detection precision, real-time performance, reliability, easy deployment and later cost.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and industrial automation detection, and particularly relates to a vehicle screen two-dimensional coupling detection method and system. BACKGROUND

[0002] With the popularity of intelligent terminals such as in-vehicle infotainment systems (IVI), smart phones, tablets, etc., the reliability and long-term stability of their screens have become key indicators of product quality. In the production and quality inspection process, the screen usually needs to be tested for a long time to induce and detect potential defects such as color deviation, dead pixels, screen glitches, black screen, etc. However, the traditional screen anomaly detection scheme has many limitations, resulting in low detection efficiency, high false positive and false negative rates, and inability to effectively warn of progressive failures.

[0003] Currently, the existing technology mainly has the following technical defects: The response mechanism is lagging, and there is a lack of hardware protection. Most detection systems can only record, pop up windows or sound warnings at the software level after discovering serious anomalies (such as blue screen, black screen, screen glitches), and still need manual intervention to power off and restart. This manual response method is slow (usually takes tens of seconds or even longer), cannot cut off the power supply at critical moments, and may cause the failure to expand. SUMMARY

[0004] To solve the problems in the related art, the present application provides a vehicle screen two-dimensional coupling detection method and system, which is significantly superior to the prior art in terms of detection accuracy, real-time performance, reliability, ease of deployment, and post-cost in a four-in-one technical combination of "visible light-YOLOv5+ real-time adjustable parameters+ CAN-relay linkage+xlsx report", and has outstanding engineering practical value.

[0005] The technical solution is as follows: An image stream of the vehicle screen is obtained through a multi-source video access layer, the multi-source video access layer synchronously supports an infrared imager and a visible camera, and hardware synchronization and frame alignment of multi-channel image data are realized by using an FPGA; The aligned image stream is input into a pre-trained YOLOv5-Heat multi-modal model to identify the screen anomaly type, the YOLOv5-Heat multi-modal model embeds a chrominance analysis branch at a feature fusion layer of a backbone network, the chrominance analysis branch uses the L*a*b channel of the CIELAB color space and the saturation channel of the HSV space for multi-scale feature splicing, and fuses the feature representations of the infrared thermal map and the visible light image through an adversarial training method to simultaneously detect color deviation, dead pixels, full-screen anomalies, and overheating points; A two-dimensional coupling algorithm is used to analyze the identified screen anomalies. The two-dimensional coupling algorithm includes color difference detection and spatiotemporal coupling. Based on the results of the two-dimensional coupling algorithm, a warning signal of the corresponding level is generated. Generate structured test reports; When the warning signal level is severe abnormal, the abnormal signal is converted into a CAN frame by the Zhouligong module, and a command is sent to the execution module through the control bus to control the relay module to perform a power-off operation and trigger the vehicle system to restart.

[0006] Preferably, the multimodal training of the YOLOv5-Heat multimodal model includes: Joint training was performed using visible light and infrared images, and a chromaticity consistency constraint term was introduced into the loss function to enhance the robustness of color cast and hot spot recognition.

[0007] Preferably, the two-dimensional coupling algorithm includes: Color difference detection calculates the spatial overlap between detection results of different anomalies and identifies color differences. Spatiotemporal coupling involves weighted fusion of correlated anomalous events in the same spatial region within a sliding time window.

[0008] Preferably, the calculation steps for the color difference detection are as follows: ΔE*ab = [(ΔL*)² + (Δa*)² + (Δb*)²]^0.5; In the formula, ΔL* represents the lightness difference, Δa* represents the red-green difference, Δb* represents the yellow-blue difference, and ΔE*ab represents the color difference obtained by taking the square root of the sum of squares. If ΔE≥5, it is judged as a color deviation anomaly.

[0009] Preferably, the spatiotemporal coupling includes the following steps: Record the occurrence time and spatial location of coupled events; Retrieve all historical coupled events that occurred in the same spatial region within a preset time window; The weights of each historical coupled event are calculated using an exponential decay function. If the weighted sum exceeds a threshold, an early warning signal is generated. Preferably, it also includes adjusting parameters in real time through a human-computer interaction interface, including confidence, intersection-over-union ratio (IoU), and system response latency. The adjusted parameters are updated in real time to the YOLOv5-Heat multimodal model and two-dimensional coupling algorithm through a shared memory mechanism to achieve dynamic strategy switching.

[0010] The detection report includes timestamps, anomaly types, confidence levels, screenshot records, and restart records, and supports exporting reports in xlsx format.

[0011] On the other hand, a two-dimensional coupling detection system for vehicle infotainment screens is provided, characterized in that it includes: The image acquisition module is used to acquire images from the vehicle's infotainment screen via a multi-source interface; The image processing module is used to input screen images into a pre-trained YOLOv5-Heat multimodal model to identify anomaly types. The parameter adjustment module is used to adjust parameters in real time through the human-computer interaction interface; The coupling analysis module is used to analyze the anomaly types identified by the two-dimensional coupling algorithm, and generate corresponding warning information based on the results of the two-dimensional coupling algorithm. The report generation module is used to output structured test reports; The hardware control module is used to trigger hardware protection operations through the Zhouligong module and relay module when the warning signal level is severely abnormal.

[0012] Preferably, the hardware control module is further configured as follows: Preferably, before triggering the power-off operation, the current screen state and CAN bus log are saved, and a self-test program is automatically executed after the restart, and the self-test results are fed back to the report generation module.

[0013] Preferably, the image acquisition module uses an FPGA to achieve hardware synchronization of multiple images.

[0014] The technical solution includes at least the following technical effects: A two-dimensional coupling detection method and system for vehicle infotainment screens is disclosed. The system runs on a computer and dynamically connects to multimodal image acquisition devices such as local video cameras, USB cameras, network cameras, and infrared imagers to acquire screen images. It uses a pre-trained anomaly detection model to detect screen anomaly data in real time during stress testing. The anomaly detection model adopts the YOLOv5-Heat multimodal model. The screen anomaly data is identified by a two-dimensional coupling algorithm. Based on the result of the two-dimensional coupling algorithm, the anomaly level is determined. When a severe anomaly occurs, a relay is used to link with Zhou Ligong to achieve hardware-level power-off restart and output a structured report, which significantly improves the reliability and security of the vehicle mobile terminal screen.

[0015] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] Figure 1This is a flowchart of a preferred embodiment of a two-dimensional coupling detection method for vehicle infotainment screens. Figure 2 This is a structural block diagram of a two-dimensional coupling detection system for vehicle infotainment screens, provided as a preferred embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0019] This invention provides a two-dimensional coupling detection method and system for vehicle infotainment screens. The system runs on a computer and dynamically connects to multimodal image acquisition devices such as local video cameras, USB cameras, network cameras, and infrared imagers to acquire screen images. It uses a pre-trained anomaly detection model to detect screen anomaly data in real time during stress testing. This anomaly detection model adopts the YOLOv5-Heat multimodal model. The screen anomaly data is identified through a two-dimensional coupling algorithm. Based on the result of the two-dimensional coupling algorithm, the anomaly level is determined. When a severe anomaly occurs, a relay is used to link with Zhou Ligong to achieve hardware-level power-off restart and output a structured report, which significantly improves the reliability and security of the vehicle mobile terminal screen.

[0020] Example 1 As attached Figure 1 As shown, a two-dimensional coupling detection method for vehicle infotainment screens includes the following steps: S1: The image stream of the vehicle screen is obtained through the multi-source video access layer. The multi-source video access layer synchronously supports infrared imagers and visible cameras, and uses FPGA to realize hardware synchronization and frame alignment of multi-channel image data. The multi-source video access layer supports at least one input source, including local video files, USB cameras, webcams, and infrared imagers.

[0021] S2: The aligned image stream is input into the pre-trained YOLOv5-Heat multimodal model to identify screen anomaly types. The YOLOv5-Heat multimodal model embeds a chromaticity analysis branch in the feature fusion layer of the backbone network. The chromaticity analysis branch uses the L*a*b channels of the CIELAB color space and the saturation channels of the HSV space for multi-scale feature splicing. It also fuses the feature representations of infrared heatmaps and visible light images through adversarial training to simultaneously detect color shift, dead pixels, full-screen anomalies, and hot spots.

[0022] The multimodal training of the YOLOv5-Heat multimodal model includes: Joint training was performed using visible light and infrared images, and a chromaticity consistency constraint term was introduced into the loss function to enhance the robustness of color cast and hot spot recognition.

[0023] In one embodiment, a chromaticity consistency loss term is introduced into the loss function using a visible-infrared paired image dataset: in, and These are the predicted brightness and saturation, respectively. and For the true value, , These are the weighting coefficients; during training, domain adaptation is achieved through a gradient inversion layer, making the backbone network compatible with both infrared and visible light images.

[0024] S3: Parameters are adjusted in real time through the human-computer interaction interface. The anomaly detection model dynamically adjusts its strategy based on the updated parameters. Parameters, including confidence, intersection-over-union ratio (IoU), and system latency, are adjusted in real time through the human-computer interaction interface. The adjusted parameters are updated in real time to the YOLOv5-Heat multimodal model and two-dimensional coupling algorithm through a shared memory mechanism to achieve dynamic strategy switching.

[0025] S4: The identified screen anomalies are analyzed using a two-dimensional coupling algorithm. Based on the results of the two-dimensional coupling algorithm, a warning signal of the corresponding level is generated. The two-dimensional coupling algorithm includes: Color difference detection calculates the spatial overlap between detection results of different anomalies and identifies color differences. The calculation steps for color difference detection are as follows: ΔE*ab = [(ΔL*)² + (Δa*)² + (Δb*)²]^0.5; In the formula, ΔL* represents the lightness difference, Δa* represents the red-green difference, Δb* represents the yellow-blue difference, and ΔE*ab represents the color difference obtained by taking the square root of the sum of squares. If ΔE≥5, it is judged as a color deviation anomaly.

[0026] Spatiotemporal coupling involves weighted fusion of correlated anomalous events in the same spatial region within a preset time window.

[0027] The spatiotemporal coupling includes the following steps: Record the occurrence time and spatial location of coupled events; Retrieve all historical coupled events that occurred in the same spatial region within a preset time window; The influence weights of each historical coupling event are calculated using an exponential decay function. The formula is: Weight_i = exp(-λ×Δt_i); Where Δt_i is the time difference and λ is the decay coefficient, which controls the rate at which the weight decays over time; If the weighted sum exceeds the threshold, an early warning signal is generated. S5: Generate a structured detection report, which includes timestamps, anomaly types, confidence levels, screenshot records, and restart records, and supports exporting reports in xlsx format; S6: When the warning signal level is severe abnormal, the abnormal signal is converted into a CAN frame through the Zhouligong module, and a command is sent to the execution module through the control bus to control the relay module to perform a power-off operation and trigger the vehicle system to restart.

[0028] Example 2 like Figure 2 As shown, a two-dimensional coupling detection system for vehicle infotainment screens includes: The image acquisition module is used to acquire images from the vehicle's infotainment screen through a multi-source interface; the image acquisition module uses an FPGA to achieve hardware synchronization of multiple images.

[0029] The image processing module is used to input screen images into a pre-trained anomaly detection model to identify the anomaly type; The parameter adjustment module is used to adjust parameters in real time through the human-computer interaction interface; The coupling analysis module is used to analyze the anomaly types identified by the two-dimensional coupling algorithm, and generate corresponding warning information based on the results of the two-dimensional coupling algorithm. The report generation module is used to output structured test reports; The hardware control module is used to trigger hardware protection operations when the warning signal level is a severe anomaly.

[0030] The hardware control module is also configured to: Before triggering a power outage, the current screen status and CAN bus log are saved, and a self-test program is automatically executed after the restart, and the self-test results are fed back to the report generation module.

[0031] Example 3 The existing technology employs the following techniques: Mainstream solutions only use visible light cameras to compare RGB color spaces or perform general YOLO detection, resulting in high false alarm / false negative rates. The video source is fixed and singular, and does not support dynamic access from local files, USB, or RTSP / HTTP network cameras. The thresholds for confidence, intersection-over-union (IoU), and system response latency are fixed and cannot be adjusted in real time according to the scenario. After a serious anomaly is detected (blue screen, black screen, screen distortion), only a software pop-up window is displayed, and there is no hardware power-off restart function; The output is in PDF / JPG format and cannot be directly imported into the OEM Excel-MES quality template.

[0032] The difference between the two-dimensional coupling detection method and system for vehicle infotainment screens of this invention and existing technologies is as follows: We employ a pure visible light YOLOv5-Heat multimodal model, focusing on four types of anomalies: blue, black, white, and screen distortion. The training set is optimized for screen display, achieving mAP@0.5≥96%. A plug-in "source manager" allows you to run local videos, USB drives, and webcams. Real-time GUI slider adjustment of confidence, IoU, and system response latency takes effect within 100 ms; On the PC side, the ZLG USBCAN-II + relay module is used to realize the closed loop of abnormality → CAN message → 2 s ACC power failure → restart → recovery; the abnormal signal is converted into a CAN frame by the ZLG module, and the command is sent to the execution module through the control bus to control the relay module to perform the power failure operation and trigger the vehicle's system to restart.

[0033] Generate XLSX reports with timestamps, screenshot paths, and restart records with a single click.

[0034] In summary, this invention is the first to integrate dual-modal information of "visible light", dynamic multi-source access, real-time parameter adjustment, CAN-relay hardware linkage and video reporting into a single computer platform, which significantly improves the accuracy, real-time performance, automation and traceability of long-term screen reliability testing, and has obvious technological progress and engineering practical value.

[0035] The overall plan is as follows: The vehicle infotainment screen two-dimensional coupling detection system consists of computer software (YOLO model PT), a multi-source video access layer, a relay module, and ZLG USBCAN-II.

[0036] The multi-source video access layer includes local video, USB visible light cameras, and RTSP / HTTP webcams. It should be noted that the multi-source video access layer uses miniature infrared thermal imagers and / or visible cameras; that is, both visible light cameras and miniature infrared thermal imagers can be used simultaneously, or a visible camera can be used alone. The miniature infrared thermal imager detects overheating anomalies, while the visible camera detects abnormal information such as blue screens, black screens, and distorted screens.

[0037] The computer software uses the YOLOv5-Heat inference engine, which includes screen best.pt, GUI sliders, and exception output. The GUI sliders are used to adjust parameters in real time, such as confidence, IoU, and system latency. The exceptions output include at least one of the following: blue screen, black screen, white screen, and distorted screen.

[0038] Output anomaly detection results to generate a structure detection report. The detection report includes timestamp, anomaly type, confidence level, screenshot records, and restart records, and supports exporting reports in xlsx format.

[0039] When a serious anomaly is detected, the abnormal signal is converted into a CAN frame by the Zhouligong module, and a command is sent to the execution module via the control bus to control the relay module to perform a power-off operation, triggering the vehicle's infotainment system to restart.

[0040] Hardware layer: It adopts any universal USB2.0 / MIPI-CSI camera, which is plug-and-play without drivers. It includes basic equipment such as network cable router, relays, and ZLG CAN-ZLG; the computing platform only requires ARM Cortex-A53 + 1GB RAM, and can run without a GPU.

[0041] Software layer: including YOLOv5-Heat multimodal model, defect coupling algorithm, and real-time parameter adjustment system, etc.

[0042] The YOLOv5-Heat multimodal model adds a chroma analysis branch to the feature fusion layer of the YOLOv5 backbone network. Specifically, based on the YOLOv5 depthwise separable convolutional architecture, the feature extraction layer includes a C3 module and SPPF spatial pyramid pooling. The C3 module consists of three convolutional layers and residual connections. A chroma analysis branch is added to the third layer of the C3 module in the feature fusion layer (at a feature map size of 80×60). The output categories are hotspot, color shift, dead pixel, and fullscreen.

[0043] Training the model: Data composition: 50,000 1920×1080 industrial inspection images and corresponding graphic matrices; Example of a labeled matrix: # Defect annotation matrix (80×60) [0.0, 1.0, ..., 0.0], # 1.0 represents color_shift defect [0.0, 0.0, ..., 2.0], # 2.0 represents dead_pixel ...].

[0044] The defect coupling algorithm is a two-dimensional coupling algorithm, which includes a color difference detection module and spatiotemporal coupling logic.

[0045] The color difference detection module uses the following calculation formula; ΔE*ab = [(ΔL*)² + (Δa*)² + (Δb*)²]^0.5; In the formula, ΔL* represents the lightness difference, Δa* represents the red-green difference, Δb* represents the yellow-blue difference, and ΔE*ab represents the color difference obtained by taking the square root of the sum of squares. Threshold criteria: CIEDE2000 standard, ΔE≥5 is considered a significant color shift (99% confidence level). Spatiotemporal coupling logic, configurable parameters: Coupling parameters (class CouplingParams): Spatial coupling threshold (iou_threshold) = 0.3; Temporal coupling window (time_window) = 120#, unit: seconds.

[0046] Dynamic weighting: The impact weight of historical defects is calculated using an exponential decay function, as shown in the following formula: Weight_i = exp(-λ * Δt_i); In the formula, Δt_i represents the time difference from the occurrence of the i-th historical event to the present; λ represents the decay coefficient, which controls the rate at which the weight decays over time.

[0047] The real-time parameter tuning system is configured as follows: Communication protocol: gRPC + shared memory hybrid architecture.

[0048] Hot update mechanism: Model parameters are directly modified through the CUDA kernel, the inference thread is separated from the configuration thread, and the average response latency is <15ms.

[0049] Parameters are adjustable in real time: conf: 0.00–1.00; iou: 0.00–1.00; latency: 0–1000 ms.

[0050] The relay and ZLG linkage relationship, hardware configuration is as follows: The relays are Songle 12V 4-channel, with ACC contacts connected in series; ZLG uses USBCAN-II+ RS485-CAN converter.

[0051] The control process is as follows: Anomaly detected → CAN frame → Relay power off after 2 seconds → Vehicle system restarts → Recovery and reporting When a serious anomaly is detected, the abnormal signal is converted into a CAN frame by the Zhouligong module, and a command is sent to the execution module through the control bus to control the relay module to perform a 2-second power-off operation, triggering the vehicle's system to restart and resume reporting.

[0052] Existing solutions rely solely on visible light image comparison, only triggering an alarm when the anomaly is visually apparent. This invention first uses a multimodal model to achieve 96.8% mAP accurate identification of blue / black / white / distorted screens, then converts the abnormal signal into a CAN frame via a Zhou Ligong module. This frame sends a command to the execution module via the control bus, controlling the relay module to perform a 2-second power-off operation, triggering a real-time closed-loop restart of the vehicle's infotainment system. The average time from an anomaly occurrence to the vehicle's power-off is 2.7 seconds, while manually unplugging and replugging the ACC switch takes at least 30 seconds; therefore, the potential risk of burnout can be reduced by more than 90%.

[0053] Existing solutions use fixed local files or a single USB camera, requiring a software restart to switch scenes. This invention employs a plug-in "source manager," allowing hot-swapping of local / RTSP / USB data during runtime, with a measured switching latency of <0.5 seconds. In actual operation, a 4S store can continuously test the central control screens of 8 different car models at the same workstation within a day without restarting, increasing the daily productivity of a single workstation by 3 times.

[0054] The existing SDK fixes conf at 0.5, resulting in a false alarm rate of 12% when the illumination changes. The GUI slider of this invention can be dynamically adjusted between 0.1 and 0.9. Field tests show that when conf=0.75, the false alarm rate drops to 2.1%. After the latency is reduced from 1000ms to 200ms, the false alarm rate drops from 4.5% to 0.6%.

[0055] Traditional systems require manual power-off after detecting screen distortion, which can lead to GPU overheating and burnout due to the absence of on-site personnel. This invention utilizes the ZLG USBCAN-II + relay for a 2-second power-off and restart, and has shown no temperature rise during 100 consecutive 24-hour tests. Hardware-level protection compresses the "anomaly detection → power-off" chain from minutes to seconds, reducing device thermal stress by 70% and extending the lifespan of the tested screen.

[0056] In existing PDF / JPG formats, OEM quality engineers need to manually input 20 fields / exceptions. This invention directly generates an xlsx file containing "time, source name, exception type, confidence level, screenshot path, and restart flag." In actual testing, importing 500 exceptions into MES in batches took 30 seconds. In real-world scenarios, this saves 2.5 man-hours per thousand exceptions. Based on a 4S store's average of 200 exceptions per day, this translates to approximately 182 man-hours saved annually, equivalent to 1.5 man-months.

[0057] In summary, by combining the four technologies of "visible light-YOLOv5 + real-time adjustable parameters + CAN-relay linkage + xlsx reports", this invention significantly outperforms existing technologies in five dimensions: detection accuracy, real-time performance, reliability, ease of deployment, and long-term cost, and has outstanding practical engineering value.

[0058] The two-dimensional coupling anomaly detection system of the present invention runs on a computer and can dynamically connect to local video, USB camera or network camera. It uses a pre-trained multimodal model to detect blue screen, black screen, white screen, screen distortion and hot spot anomalies in long-term stress test in real time, and realizes hardware-level power failure and restart through relay + Zhouligong CAN module in case of serious anomaly.

[0059] Users can adjust the conf, iou, and latency parameters in real time through a graphical interface and export XLSX reports containing temperature curves with a single click. Compared with existing technologies, this invention achieves 6-minute early warning of anomalies, zero automotive-grade hardware deployment, hardware-level protection, and structured reports, significantly improving the efficiency and safety of reliability testing for automotive and mobile terminal screens.

[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams.Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

Claims

1. A two-dimensional coupling detection method for vehicle infotainment screens, characterized in that, include: The image stream from the vehicle's infotainment screen is acquired through a multi-source video access layer, which simultaneously supports infrared imagers and visible cameras, and uses FPGA to achieve hardware synchronization and frame alignment of multi-channel image data. The aligned image stream is input into the pre-trained YOLOv5-Heat multimodal model to identify screen anomaly types. The YOLOv5-Heat multimodal model embeds a chromaticity analysis branch in the feature fusion layer of the backbone network. The chromaticity analysis branch uses the L*a*b channels of the CIELAB color space and the saturation channels of the HSV space for multi-scale feature splicing. It also fuses the feature representations of infrared heatmaps and visible light images through adversarial training to simultaneously detect color shift, dead pixels, full-screen anomalies, and hot spots. A two-dimensional coupling algorithm is used to analyze the identified screen anomalies. The two-dimensional coupling algorithm includes color difference detection and spatiotemporal coupling. Based on the results of the two-dimensional coupling algorithm, a warning signal of the corresponding level is generated. Generate structured test reports; When the warning signal level is severe abnormal, the abnormal signal is converted into a CAN frame by the Zhouligong module, and a command is sent to the execution module through the control bus to control the relay module to perform a power-off operation and trigger the vehicle system to restart.

2. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 1, characterized in that, The multimodal training of the YOLOv5-Heat multimodal model includes: Joint training was performed using visible light images and infrared images, and a chromaticity consistency constraint term was introduced into the loss function.

3. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 1, characterized in that, The two-dimensional coupling algorithm includes: Color difference detection calculates the spatial overlap between detection results of different anomalies and identifies color differences. Spatiotemporal coupling involves weighted fusion of correlated anomalous events in the same spatial region within a sliding time window.

4. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 3, characterized in that, The calculation steps for color difference detection are as follows: ΔE*ab = [(ΔL*)² + (Δa*)² + (Δb*)²]^0.5; In the formula, ΔL* represents the lightness difference, Δa* represents the red-green difference, Δb* represents the yellow-blue difference, and ΔE*ab represents the color difference obtained by taking the square root of the sum of squares. If ΔE≥5, it is judged as a color deviation anomaly.

5. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 3, characterized in that, The spatiotemporal coupling includes the following steps: Record the occurrence time and spatial location of coupled events; Retrieve all historical coupled events that occurred in the same spatial region within a preset time window; The weights of each historical coupled event are calculated using an exponential decay function. If the weighted sum exceeds a threshold, an early warning signal is generated.

6. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 1, characterized in that, It also includes adjusting parameters in real time through the human-computer interaction interface, including confidence, intersection-over-union ratio (IoU), and system latency; The adjusted parameters are updated in real time to the YOLOv5-Heat multimodal model and two-dimensional coupling algorithm via a shared memory mechanism.

7. The two-dimensional coupling detection method for vehicle infotainment screens according to claim 1, characterized in that, The detection report includes timestamps, anomaly types, confidence levels, screenshot records, and restart records, and supports exporting reports in xlsx format.

8. A two-dimensional coupling detection system for vehicle infotainment screens, employing the two-dimensional coupling detection method for vehicle infotainment screens as described in any one of claims 1-7, characterized in that, include: The image acquisition module is used to acquire images from the vehicle's infotainment screen via a multi-source interface; The image processing module is used to input screen images into a pre-trained YOLOv5-Heat multimodal model to identify anomaly types. The parameter adjustment module is used to adjust parameters in real time through the human-computer interaction interface; The coupling analysis module is used to analyze the anomaly types identified by the two-dimensional coupling algorithm, and generate corresponding warning information based on the results of the two-dimensional coupling algorithm. The report generation module is used to output structured test reports; The hardware control module is used to trigger hardware protection operations through the Zhouligong module and relay module when the warning signal level is severely abnormal.

9. The vehicle infotainment screen two-dimensional coupling detection system according to claim 8, characterized in that, The hardware control module is also configured to: Before triggering a power outage, the current screen status and CAN bus log are saved, and a self-test program is automatically executed after the restart, and the self-test results are fed back to the report generation module.

10. The vehicle infotainment screen two-dimensional coupling detection system according to claim 8, characterized in that, The image acquisition module uses an FPGA to achieve hardware synchronization of multiple images.

Images