A deep learning-based intelligent automobile cockpit display and automated testing system

Abstract

The application belongs to the technical field of automobile electronic testing, and relates to an automobile intelligent cockpit display and automatic test system based on deep learning, aiming to solve problems such as low efficiency of traditional manual testing, poor adaptability of automatic tools, insufficient test coverage, lack of full-process closed loop, and difficulty in model deployment, and realize intelligent, high-precision and automatic test of cockpit display functions. The system constructs a special cockpit display dataset, collects images of multiple vehicle instrument screens and central control screens, covers fault icons, ADAS prompt texts and various lighting environments, and improves the model generalization ability through data enhancement. In view of the small target icon detection and text recognition efficiency problems, the YOLOv8 model is improved, the detection precision is improved through optimization of anchor frame; the SVTR text recognition model is improved, a lightweight network structure is introduced, the parameter quantity is reduced and the inference speed is improved. The system integrates data collection, signal simulation, visual recognition, result evaluation and report generation modules, realizes automatic analysis of test cases, CAN signal interaction, display response detection and automatic generation of test reports.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics testing technology, and in particular to a deep learning-based automotive intelligent cockpit display and automated testing system. It is applicable to the functional verification, response speed testing and fault detection of various display devices such as instrument displays and central control displays in automotive intelligent cockpits, and can be widely used in laboratory testing during the automotive R&D stage and off-line testing during the mass production stage. Background Technology

[0002] With the accelerating pace of automotive intelligence, the intelligent cockpit, as the core carrier of human-machine interaction, integrates multiple functions such as instrument display, central entertainment, and ADAS (Advanced Driver Assistance Systems) prompts. The stability and accuracy of its display system directly affect driving safety and user experience. Currently, the testing of cockpit display systems mainly relies on manual visual inspection or traditional automated tools (such as template-matching-based image recognition), which suffers from the following technical shortcomings: low efficiency of manual testing; poor adaptability of traditional automated tools; limited test coverage; and lack of closed-loop testing capabilities. To address these issues, this invention proposes an automated testing system that integrates deep learning algorithms and vehicle bus simulation technology. Through improved target detection and text recognition models, it achieves high-precision recognition of displayed content. Combined with CANoe bus tools, it constructs a fully closed-loop testing process of "test case import - signal simulation - response detection - report generation," significantly improving testing efficiency and accuracy. Summary of the Invention

[0003] This invention aims to address the problems of low efficiency, poor accuracy, and insufficient automation in the testing of existing automotive intelligent cockpit display systems. Specifically, it includes: how to achieve high-precision, real-time recognition of multiple types of responsive content (icons, text) on the cockpit display screen, unaffected by changes in lighting or icon deformation; how to construct a fully automated testing process to achieve an end-to-end closed loop from test case parsing to bus signal simulation and result analysis; and how to balance the versatility and lightweight design of the testing system, making it adaptable to different vehicle cockpit display systems and meeting the high-efficiency requirements of mass production testing. To address the aforementioned problems, this invention provides an automated testing system for automotive intelligent cockpit display systems based on deep learning. The system comprises: a data acquisition module for real-time acquisition of response image data from various displays (instrument display, central control display) in the automotive intelligent cockpit; the data acquisition module includes a high-definition RGB camera that transmits image data to a processing module via a MIPI-CSI interface; the data acquisition module also includes an image preprocessing unit for handling screen highlight reflection, scrolling stripes, and target distortion issues, and for removing invalid images without response icons / text using the Mediapipe algorithm; a storage module for storing deep learning models (including a YOLOv8-A response icon detection model and an SVTR-LM response text recognition model), model training parameters (optimizer configuration, learning rate strategy), self-built datasets (automotive intelligent cockpit display response icon dataset and response text dataset), Excel-formatted test cases, and automated test results (including detection reports and response images); and a processing module including a response icon detection submodule based on an improved YOLOv8-A... The network structure detects display response icons using deep learning algorithms. This network replaces the original Anchor-Free module with an improved K-Means++ anchor box clustering (fitting anchor boxes from 12 suitable icon datasets), introduces a BiFPN bidirectional feature pyramid network for multi-scale feature fusion, and combines a Triplet Attention mechanism to enhance feature representation, reduce model parameters, and suppress prediction box drift. The response text recognition submodule, based on an improved SVTR-LM network structure, is used to recognize ADAS prompt text on the display. This network structure achieves lightweighting by fusing SVTR and the PP-LCNet lightweight network, uses a multi-scale window Local Mixing module to extract character morphological features, combines a One Cycle learning rate decay strategy to accelerate convergence, and introduces an attention-guided CTC training method to optimize text alignment accuracy. The automated testing submodule reads Excel test cases using Python scripts and encapsulates the test cases into a standardized data structure recognizable by CANoe.The system controls the CANoe hardware to simulate CAN bus signals and send them to the cockpit display system, triggering a response from the display screen. A deep learning model is used to detect and recognize the acquired response images, comparing the expected and actual results to generate a test report. A performance evaluation module comprehensively evaluates the system using six metrics: precision (P), recall (R), mAP (mAP50, mAP50-95), inference time, false test rate, and single-test case processing time. Prioritizing test accuracy and real-time performance, the system ensures no missed or false detections in the display screen response. A communication module enables data flow transmission between the processing module, the CANoe hardware, the high-definition camera, and the vehicle display system. It supports the CAN bus protocol and a USB 3.0 high-speed interface, employing a symmetric encryption algorithm to ensure test data security. The system, centered on a computing unit equipped with an NVIDIA GeForce RTX 3060 graphics card and the CANoe hardware, achieves fully automated detection and test result feedback for the cockpit display system response.

[0004] Furthermore, the system's front-end sensing unit acquires images based on the photoelectric conversion principle. The camera's image sensor converts light signals into electrical signals, and then obtains digital image data through analog-to-digital conversion. The module's 30fps frame rate relies on clock signal control and rapid image sensor readout to ensure real-time image data updates. The MIPI-CSI interface uses differential signal transmission to transmit image data to subsequent processing units at high speed and stably. A dimmable ring light utilizes electroluminescence to provide uniform illumination; its 5500K±200K color temperature closely approximates natural sunlight, ensuring accurate color reproduction in the image. The electric translation stage achieves high-precision positioning (accuracy ±0.1mm) through motor drive and precision mechanical transmission. Combined with sensor feedback and control algorithms, it can automatically adjust the shooting angle, fully covering the 12.3-inch instrument panel and 15.6-inch central control screen display areas, providing high-quality and comprehensive image data input for subsequent inspection.

[0005] Furthermore, the preprocessing unit of the data acquisition module includes grayscale processing, normalization processing, and data augmentation operations; the normalization processing can be expressed as: Where I is the original display screen image pixel value, u is the image pixel mean, and σ is the image pixel standard deviation. The values ​​are normalized image pixel values. The self-built dataset includes: a car intelligent cockpit display response icon dataset: containing 12,078 640×640 resolution JPG images, covering 10 types of fault prompt icons including battery, engine oil, engine, thermometer, brake, spark plug, airbag, tire pressure, stability control, and ABS alarm, divided into training, validation, and test sets in a 7:2:1 ratio; and a car intelligent cockpit display response text dataset: containing 7,428 1280×960 resolution JPG plain text images, covering ADAS prompt scenarios such as emergency braking, door opening warning, lane change prompt, and driver fatigue warning, divided into training, validation, and test sets in a 7:2:1 ratio.

[0006] Furthermore, the response icon detection submodule includes a model training unit for training the YOLOv8-A network using a self-built response icon dataset. The model training unit employs the SGD optimizer with an initial learning rate of 0.01, momentum of 0.94, weight decay of 0.0005, batch size of 64, and training epochs of 200. After training, the model achieves an mAP50 of 98.2% and an mAP50-95 of 77.7% on the self-built dataset, representing improvements of 2.8% and 2.6% respectively compared to the original YOLOv8. The number of parameters is reduced to 4.57M, and the inference time is shortened to 7.0ms.

[0007] Furthermore, the response text recognition submodule also includes a model training unit for training the SVTR-LM network using a self-built response text dataset. The model training unit employs the AdamW optimizer with an initial learning rate of 0.0001, L2 regularization, a batch size of 128, 300 training epochs, and a maximum text length of 20. After training, the model achieves a recognition accuracy of 97.3% and a normalized edit distance of 98.7% on the self-built dataset. Compared to the original SVTR model, the inference speed is improved by 42.9%, the number of parameters is reduced to 3.7M, and the inference time is shortened to 1.32s.

[0008] Furthermore, the automated testing submodule includes a test process control unit, used to implement a closed-loop process of "test case import → CAN signal simulation → image acquisition → model detection → result comparison → report generation". The test process control unit automatically reads Excel test cases (including test targets, input signals, and expected responses) through Python scripts, controls CANoe to simulate CAN signals of ECU nodes such as BCM and IPK, and triggers the display screen response. After the test is completed, the actual results (test icons / text, confidence level) are written into the original Excel test cases, and a visual test report containing response images and error analysis is generated.

[0009] Furthermore, the communication module includes a data interaction interface unit that supports CAN bus communication (100Mbps / 1Gbps) between the CANoe and the processing module, MIPI-CSI image transmission between the high-definition camera and the processing module, and LVDS display interface between the processing module and the vehicle display system. The communication module uses the AES symmetric encryption algorithm to encrypt the test data (CAN signal, detection result) to ensure data transmission security, and also supports the ADB debugging interface for system parameter configuration. Attached Figure Description

[0010] To more clearly illustrate the implementation of the present invention or the solutions in the prior art, the drawings used in the description of the implementation examples or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a deep learning-based intelligent cockpit display system for automobiles, provided in one embodiment of the present invention. Detailed Implementation

[0012] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep learning-based automotive intelligent cockpit display and automated testing system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0014] The following description, in conjunction with the accompanying drawings, details a specific solution for a deep learning-based intelligent cockpit display and automated testing system for automobiles provided by this invention.

[0015] Please see Figure 1 This illustrates an embodiment of the present invention, which provides a deep learning-based intelligent cockpit display and automated testing system for automobiles. The system includes: Step S1: Constructing a dedicated dataset for intelligent cockpit display systems for automobiles and collecting data.

[0016] In one embodiment of the present invention, a 20-megapixel high-definition camera (30fps, MIPI-CSI 2.0 interface) was used to capture images of the instrument panel and central control screen of three different models of automobile test benches, collecting 1342 original images (640×640 resolution, JPG format) containing 10 types of fault icons and 682 original images (1280×960 resolution, JPG format) containing ADAS text, covering three lighting scenarios: daytime, nighttime, and high light.

[0017] Preferably, in one embodiment of the present invention, multi-dimensional enhancement is performed to address cabin environment interference, including: geometric transformation: random rotation (-15°~15°), horizontal flip (probability 0.5), affine transformation (scaling 0.9~1.1 times), simulating camera angle deviation; color adjustment: brightness (±20%), saturation (±15%), contrast (±15%), simulating illumination changes; acoustic injection: Gaussian noise (variance 0.01~0.02), salt-and-pepper noise (density 0.01~0.02), simulating electronic interference; specular highlight removal: using the multi-scale Retinex algorithm (MSRCR) to separate illumination and reflection components, the formula is: (I is the original image, R is the target reflection component, and L is the illumination noise).

[0018] Step S2: Improve the YOLOv8-A algorithm to achieve high-precision detection of response icons.

[0019] In one embodiment of the present invention, an improved K-Means++ algorithm is used to cluster the bounding boxes in the training set, and the Dloss distance function (based on IoU) is used instead of the Euclidean distance, as shown in the formula: ,in This represents the actual area of ​​the frame. To generate 12 anchor boxes (e.g., [12×15, 15×18]) to fit the small target, k=12, replacing the original YOLOv8 Anchor-Free module.

[0020] Step S3: Improve the SVTR-LM algorithm to achieve response text recognition and lightweight backbone fusion.

[0021] In one embodiment of the present invention, the first half of the original Transformer of SVTR is replaced with the PP-LCNet lightweight network, reducing the number of parameters from 8.3M to 3.7M, while retaining two Global Mixing modules;

[0022] Preferably, in one embodiment of the present invention, a One Cycle learning rate decay (initially 0.0001, increasing then decreasing) is adopted, and an attention-guided CTC loss function is introduced into the decoding layer to enhance the text region focusing ability.

[0023] Preferably, in one embodiment of the present invention, the AdamW optimizer (L2 regularization 0.001), with a batch size of 128, is trained for 300 rounds, achieving a recognition accuracy of ≥97.3% and an inference speed that is 42.9% faster than the original SVTR.

[0024] Step S4: Build a closed loop for fully automated testing.

[0025] In one embodiment of the present invention, test cases in Excel format (including input CAN signals and expected responses) are read by a Python script and encapsulated into a standardized data structure recognizable by CANoe.

[0026] Preferably, in one embodiment of the present invention, the CANoe COM interface is invoked to generate a CAN message, which is sent to the cockpit domain controller to trigger a display screen response; a high-definition camera captures the display screen response image, which is preprocessed (highlight removal and noise reduction), and then YOLOv8-A detection icons and SVTR-LM recognition text are input; the actual recognition result is matched with the expected response (confidence ≥ 0.85 is considered valid), the test is marked as passed / failed, and the test case is automatically written to an Excel test case and an HTML report (including response image and error analysis) is generated, with an average processing time of ≤ 1.2 seconds per test case.

[0027] An embodiment of the present invention also provides a deep learning-based intelligent cockpit display and automated testing system for automobiles. The system includes a memory, a processor, and a computer program, wherein the memory is used to store the corresponding computer program, the processor is used to run the corresponding computer program, and the computer program can implement the methods described in steps S1-S4 when running in the processor.

[0028] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A deep learning-based intelligent cockpit display and automated testing system for automobiles, characterized in that, The system includes: a data acquisition module for real-time acquisition of response image data from various displays (instrument display and central control display) in the automotive intelligent cockpit; the data acquisition module includes a high-definition RGB camera that transmits image data to the processing module via a MIPI-CSI interface; the data acquisition module also includes an image preprocessing unit for handling screen highlight reflection, scrolling stripes, and target distortion, and for removing invalid images without response icons / text using the Mediapipe algorithm; a storage module for storing deep learning models (including a YOLOv8-A response icon detection model and an SVTR-LM response text recognition model), model training parameters (optimizer configuration and learning rate strategy), self-built datasets (automotive intelligent cockpit display response icon dataset and response text dataset), Excel-formatted test cases, and automated test results (including detection reports and response images); and a processing module including: a response icon detection submodule that detects display response icons based on an improved YOLOv8-A network structure using a deep learning algorithm; the network structure uses improved K-Means++ anchor box clustering (fitting 12... The anchor-free module is replaced with anchor boxes adapted to the icon dataset. A BiFPN bidirectional feature pyramid network is introduced to achieve multi-scale feature fusion, and a Triplet Attention mechanism is used to enhance feature representation, reduce the number of model parameters, and suppress prediction box drift. The response text recognition submodule is based on an improved SVTR-LM network structure to recognize ADAS prompt text on the display screen. The network structure is lightweight by fusing SVTR and PP-LCNet lightweight networks. A multi-scale window Local Mixing module is used to extract character morphological features. A One Cycle learning rate decay strategy is used to accelerate convergence, and an attention-guided CTC training method is introduced to optimize text alignment accuracy. The automated testing submodule reads Excel test cases through Python scripts, encapsulates the test cases into a standardized data structure that CANoe can recognize, and controls CANoe hardware to simulate CAN. The bus signal is sent to the cockpit display system, triggering the display screen to respond; a deep learning model is invoked to detect and recognize the collected response images, and the expected results are compared with the actual results to generate a test report; the performance evaluation module is used to comprehensively evaluate the system using six indicators: precision (P), recall (R), mAP (mAP50, mAP50-95), inference time, false test rate, and single test case processing time, prioritizing test accuracy and real-time performance to ensure that the display screen response has no missed or false detections; the communication module is used to realize data stream transmission between the processing module and CANoe hardware, high-definition camera, and vehicle display system, supporting CAN bus protocol and USB3.The system features a high-speed interface and employs a symmetric encryption algorithm to ensure test data security. Its core is a computing unit equipped with an NVIDIA GeForce RTX 3060 graphics card and CANoe hardware, enabling fully automated detection and feedback of cockpit display system responses.

2. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to claim 1, characterized in that, The preprocessing unit of the data acquisition module further includes grayscale processing, normalization processing, and data augmentation operations; the normalization processing can be expressed as: Where I represents the pixel value of the original display screen image. The average pixel value of the image. The standard deviation of the image pixels. The values ​​are normalized image pixel values. The self-built dataset includes: a car intelligent cockpit display response icon dataset: containing 12,078 640×640 resolution JPG images, covering 10 types of fault prompt icons including battery, engine oil, engine, thermometer, brake, spark plug, airbag, tire pressure, stability control, and ABS alarm, divided into training, validation, and test sets in a 7:2:1 ratio; and a car intelligent cockpit display response text dataset: containing 7,428 1280×960 resolution JPG plain text images, covering ADAS prompt scenarios such as emergency braking, door opening warning, lane change prompt, and driver fatigue warning, divided into training, validation, and test sets in a 7:2:1 ratio.

3. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to claim 1, characterized in that, The response icon detection submodule also includes a model training unit, which is used to train the YOLOv8-A network using a self-built response icon dataset. The model training unit uses the SGD optimizer with an initial learning rate of 0.01, momentum of 0.94, weight decay of 0.0005, batch size of 64, and training epochs of 200. After training, the model achieved an mAP50 of 98.2% and an mAP50-95 of 77.7% on the self-built dataset, representing improvements of 2.8% and 2.6% respectively compared to the original YOLOv8. The number of parameters was reduced to 4.57M, and the inference time was shortened to 7.0ms.

4. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to claim 1, characterized in that, The response text recognition submodule also includes a model training unit for training the SVTR-LM network using a self-built response text dataset. The model training unit uses the AdamW optimizer with an initial learning rate of 0.0001, L2 regularization, a batch size of 128, 300 training epochs, and a maximum text length of 20. After training, the model achieves a recognition accuracy of 97.3% and a normalized edit distance of 98.7% on the self-built dataset. Compared to the original SVTR model, the inference speed is improved by 42.9%, the number of parameters is reduced to 3.7M, and the inference time is shortened to 1.32s.

5. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to claim 1, characterized in that, The automated testing submodule also includes a test process control unit, which is used to realize a closed-loop process of "test case import → CAN signal simulation → image acquisition → model detection → result comparison → report generation". The test process control unit automatically reads Excel test cases (including test targets, input signals, and expected responses) through Python scripts, controls CANoe to simulate CAN signals of ECU nodes such as BCM and IPK, and triggers the display screen to respond. After the test is completed, the actual results (test icons / text, confidence scores) are written into the original Excel test case, and a visual test report containing response images and error analysis is generated.

6. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to claim 1, characterized in that, The communication module also includes a data interaction interface unit, supporting CAN bus communication (100Mbps / 1Gbps) between CANoe and the processing module, MIPI-CSI image transmission between the high-definition camera and the processing module, and LVDS display interface between the processing module and the vehicle display system. The communication module uses the AES symmetric encryption algorithm to encrypt the test data (CAN signal, detection result) to ensure data transmission security, and also supports the ADB debugging interface for system parameter configuration.

7. The deep learning-based intelligent cockpit display and automated testing system for automobiles according to any one of claims 1-6, characterized in that, The system operates on a Windows 11 operating system with CUDA 11.8 for accelerated computing. The system has a false test rate of less than 0.5%, an average processing time of ≤1.2 seconds per test case, supports switching between test scripts for multiple vehicle models (achieved by reading vehicle model configuration files), and can run continuously for 72 hours without interruption, meeting the automated testing requirements of mass-produced automotive intelligent cockpit display systems.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements all the functional steps of the system according to any one of claims 1-7, including response image acquisition and preprocessing, YOLOv8-A model training and response icon detection, SVTR-LM model training and response text recognition, Excel test case parsing, CANoe signal simulation and automated test report generation.

Images