Commercial vehicle system self-diagnosis method and system based on artificial intelligence

Abstract

The invention discloses a commercial vehicle system self-diagnosis method and system based on artificial intelligence, and the method comprises the steps: constructing an SoC-based hardware platform, and integrating video, audio, switching signal, a sensor, a CAN bus, Ethernet and other multi-source data collection modules; deploying a CNN-LSTM hybrid artificial intelligence model, and performing space and time feature fusion analysis on the vehicle environment data and the driving state data; real-time fault detection, automatic diagnosis and maintenance manual generation are realized; user manual fault feedback and AI model self-learning updating are supported, and the fault recognition coverage range and the diagnosis precision are improved. The system comprises a hardware layer, a data acquisition layer, an intelligent analysis layer and an interaction layer, localization processing and diagnosis of data of the whole vehicle are achieved, dependence on a remote server is not needed, and data safety and real-time performance are guaranteed. According to the method, the professional threshold of fault diagnosis of the commercial vehicle is remarkably reduced, the detection capability of hidden faults and accidental faults is improved, and the method has important engineering application value.

Description

Technical Field

[0001] This invention relates to the fields of commercial vehicle electronic technology, vehicle fault diagnosis and predictive maintenance technology, and in particular to a self-diagnosis method and system for commercial vehicle systems based on artificial intelligence. It also relates to a method and system that integrates an artificial intelligence (AI) model into an in-vehicle embedded system and achieves real-time, autonomous, intelligent diagnosis and self-learning of commercial vehicle systems by fusing and analyzing multi-source sensor data (including audio and video, bus data, electrical signals, etc.). Background Technology

[0002] Commercial vehicles (such as trucks, buses, and construction machinery) serve as key carriers of modern logistics and transportation. The complexity of their systems, the continuity of their operation, and the severity of their failures place extremely high demands on vehicle health management. Traditional commercial vehicle fault diagnosis primarily relies on on-board diagnostic (OBD) systems to read fault codes (DTCs) or on remote on-board terminals (T-Boxes) to upload vehicle data to cloud platforms for analysis. However, these methods have significant limitations: First, traditional OBD systems cover a limited range of fault types and rely heavily on preset fault code tables. They are insufficient at identifying complex faults that are not predefined, especially those that are cross-system, latent, or intermittent. Second, professional diagnostic equipment is expensive, and the diagnostic process requires experienced technicians. The analytical logic is complex, leading to low repair efficiency, especially in remote areas lacking professional repair facilities. Third, for intermittent faults, which are difficult to reproduce, diagnosis often relies on the driver's verbal description for vague judgment, or requires prolonged use of an external data recorder to wait for the fault to recur, resulting in long diagnostic cycles and high costs. Finally, while cloud-based remote diagnostic solutions can achieve centralized data analysis, they suffer from limitations such as network latency affecting data real-time performance, high transmission costs, and privacy and information security risks associated with the transfer of vehicle operating data (especially sensitive data such as audio and video).

[0003] In recent years, artificial intelligence (AI) technology, especially deep learning, has made groundbreaking progress in fields such as image recognition, speech processing, and time-series data analysis. Convolutional Neural Networks (CNNs) excel at extracting spatial features from data such as images and spectrograms; Long Short-Term Memory (LSTM) networks, as an improvement on Recurrent Neural Networks (RNNs), can effectively capture long-term dependencies in time-series data. Hybrid models such as CNN-LSTM combine the advantages of both and have shown great potential in scenarios such as predictive maintenance of industrial equipment and environmental perception for autonomous driving. However, deploying these computationally intensive AI models on resource-constrained and harsh-environment in-vehicle embedded terminals for real-time fusion diagnosis of multimodal data at the vehicle level still faces many challenges, including hardware integration, algorithm optimization, and system reliability.

[0004] Meanwhile, the maturity of System-on-Chip (SoC) technology provides the hardware foundation for addressing these challenges. Modern high-performance SoCs (such as automotive-grade chips that integrate CPU, GPU, and NPU) can implement complex control logic and artificial intelligence inference calculations on a single chip. They feature high integration, high performance, low power consumption, and high reliability, and comply with automotive electronic functional safety standards (such as ISO 26262), making it possible to implement complex intelligent diagnostic functions in vehicles.

[0005] Therefore, there is an urgent need for an intelligent diagnostic method and system that can be deployed locally on commercial vehicles, integrate multi-source data acquisition capabilities, utilize advanced artificial intelligence models for real-time and comprehensive analysis, and possess self-learning and evolution capabilities, in order to overcome the shortcomings of existing technologies and improve the operational efficiency, safety, and economy of commercial vehicles. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide an artificial intelligence-based self-diagnosis method and system for commercial vehicles. This aims to solve problems such as the reliance on professional personnel for commercial vehicle fault diagnosis, the difficulty in capturing complex and sporadic faults, and safety and real-time vulnerabilities in remote diagnosis. This invention constructs an integrated hardware and software intelligent diagnostic platform on the vehicle side, enabling localized collection, fusion analysis, and intelligent diagnosis of comprehensive vehicle data. This lowers the diagnostic threshold and improves the real-time performance, accuracy, and coverage of diagnostics.

[0007] The above-mentioned objective of this invention is achieved through the following technical solutions: This invention provides a self-diagnostic method for commercial vehicle systems based on artificial intelligence, comprising the following steps: S1: Construct a hardware platform for self-diagnosis of commercial vehicle systems. The hardware platform includes a system-on-a-chip (SoC) and a multi-channel video acquisition module, a multi-channel audio acquisition module, a switch signal and sensor input module, a power output and control module, a CAN gateway module, and an Ethernet gateway module, which are respectively connected to the SoC. S2: Based on the hardware platform, a multi-task software system is built and run on the system-on-a-chip (SoC). The software system includes system management tasks, data acquisition tasks, data forwarding and recording tasks, status management tasks, artificial intelligence analysis tasks, fault interaction tasks, and log management tasks. S3: Through the data acquisition task, multi-source data during vehicle operation is collected in real time or periodically. The multi-source data includes: vehicle environment data, including video stream data from at least one camera and audio stream data from at least one microphone; vehicle driving status data, including switch signal status, sensor values, power output status, vehicle CAN network data, and vehicle Ethernet network data. S4: Preprocess and extract features from the collected multi-source data, including converting audio time-domain data into frequency-domain data and spatially synthesizing multi-channel video data and multi-channel audio data respectively. S5: Input the pre-processed data into the artificial intelligence model pre-deployed on the system-on-chip (SoC) for analysis. The artificial intelligence model is a CNN-LSTM model that integrates convolutional neural networks and long short-term memory networks. It is used to fuse and model the spatial features of vehicle environmental data and the temporal features of vehicle driving status data in order to identify vehicle system faults. S6: When the artificial intelligence model detects a system fault, it automatically generates a maintenance manual or troubleshooting guide corresponding to the fault and prompts the user. S7: Provides a user manual fault feedback interface. When a user reports an unidentified fault, the system retrieves log data from the fault period, adjusts the detection parameters of the artificial intelligence model for re-analysis, identifies suspicious data, and generates a troubleshooting plan. After the user confirms that the fault has been resolved, the new fault characteristics are updated to the artificial intelligence model to complete the model's self-learning.

[0008] According to one embodiment of the present invention, in step S5, the CNN-LSTM model works as follows: the CNN part is used to extract spatial features from the video data and the audio frequency domain data after Fourier transform; the LSTM part is used to model the temporal correlation of the vehicle driving state data, and further the spatial feature sequence extracted by the CNN is analyzed in the time dimension to achieve traceability and predictive detection of faults.

[0009] According to one embodiment of the present invention, in step S4, the specific method for converting the audio time-domain data into frequency-domain data is as follows: an analog audio signal is acquired through a microphone and a time-domain digital signal is obtained through analog-to-digital conversion (ADC); a fast Fourier transform (FFT) is performed on the time-domain digital signal to extract its feature representation in the frequency domain for feature analysis and recognition by a CNN.

[0010] According to one embodiment of the present invention, in step S4, the spatial synthesis of the multi-channel video data specifically involves: synchronizing and spatially stitching or feature-level fusing video streams from multiple cameras throughout the vehicle to generate a synthetic video stream or synthetic visual feature map reflecting the overall surrounding environment of the vehicle, which is then used by the artificial intelligence model for vehicle-level visual environment status analysis and fault feature identification.

[0011] According to one embodiment of the present invention, in step S4, the spatial synthesis of the multi-channel audio data specifically involves: synchronizing and fusing the multi-channel audio streams from multiple microphones throughout the vehicle to generate a synthesized audio stream or acoustic feature map containing spatial orientation information, which is then used by the artificial intelligence model for vehicle-level acoustic environment status analysis and fault feature identification.

[0012] According to one embodiment of the present invention, in step S7, adjusting the detection parameters of the artificial intelligence model includes improving the sensitivity of the model detection and / or broadening the range of data sources analyzed, so as to identify suspicious patterns in historical log data that were not judged as faults under standard parameters.

[0013] According to one embodiment of the present invention, the fault feature values ​​learned by the artificial intelligence model include at least: fault video feature values ​​based on a single camera, vehicle fault visual feature values ​​based on synthesized video, fault audio feature values ​​based on a single microphone, vehicle fault acoustic feature values ​​based on synthesized audio, and vehicle system fault data feature values ​​based on switch signals, sensor signals, CAN data, and Ethernet data.

[0014] The present invention also provides an artificial intelligence-based self-diagnostic system for commercial vehicles to implement the above method, comprising: The hardware layer includes the system-on-a-chip (SoC) as the core processing unit, and multi-source data acquisition and interface units built around the SoC. The data acquisition and processing layer runs on the system-on-a-chip (SoC) and is used to execute data acquisition and preprocessing tasks in the multi-task software system, realizing real-time acquisition, format conversion and preliminary fusion of multi-source data; The intelligent analysis layer, running on the system-on-a-chip (SoC), integrates the CNN-LSTM artificial intelligence model. It is used to receive feature data from the data acquisition and processing layer and perform fault diagnosis analysis, fault feature extraction, and model self-learning update logic. The interaction and output layer is used to receive the diagnostic results of the intelligent analysis layer, realize fault information prompts, generate and display maintenance manuals, and provide an input interface for users to manually provide fault feedback.

[0015] According to one embodiment of the present invention, the multi-source data acquisition and interface unit of the hardware layer specifically includes: a multi-channel video input interface for connecting an in-vehicle camera and / or video source; a multi-channel audio input interface for connecting an in-vehicle microphone; a digital input / output (DIO) and analog-to-digital converter (ADC) module for acquiring vehicle switching signals and analog / digital sensor signals; a power drive and detection module for controlling vehicle load and having a status readback function; a CAN gateway module for connecting to the vehicle CAN bus network and performing data routing and isolation; and an Ethernet gateway module for connecting to the vehicle Ethernet and performing data exchange.

[0016] According to one embodiment of the present invention, all data acquisition, storage, analysis and diagnosis processes of the system are completed locally in the vehicle without relying on a remote cloud server. The system-on-chip (SoC), data storage module and related circuits are integrated into an in-vehicle embedded device.

[0017] In summary, compared with the prior art, the present invention has at least one of the following beneficial technical effects: Enhanced and proactive diagnostic capabilities: By integrating and analyzing non-traditional diagnostic data such as audio and video, hidden faults that cannot be identified by CAN data alone can be discovered (such as early wear of mechanical parts based on abnormal noises, or loose or leaking parts based on vision). The time-series analysis capabilities of AI models enable them to capture transient characteristic patterns of intermittent faults, elevating diagnosis from "decoding after occurrence" to "real-time identification during occurrence" and even "pre-occurrence trend prediction".

[0018] Significantly reduces reliance on professional personnel: The system automatically generates intuitive maintenance manuals, transforming complex fault logic into actionable steps, enabling ordinary mechanics or drivers to perform preliminary and effective troubleshooting and handling, thus addressing the pain point of untimely professional support in remote areas or emergency situations.

[0019] Ensuring data security and real-time performance: All raw data collection, storage, and analysis are completed locally within the vehicle's equipment. Sensitive data (especially in-vehicle / external audio and video) does not need to leave the vehicle, complying with increasingly stringent data privacy regulations. Local processing also eliminates network latency and bandwidth limitations, enabling true real-time diagnostics and instant alerts.

[0020] The system possesses evolutionary and adaptive capabilities: through a self-learning closed loop of "user feedback - model update", the system can continuously learn new fault modes that appear in new vehicle models and new operating environments, constantly expanding its diagnostic knowledge base, achieving a leap from "static diagnosis" to "dynamic growth", and extending the effective life cycle of the system.

[0021] High integration and reliability: Based on automotive-grade SoC hardware design, it achieves a high degree of integration of multiple gateways, multiple data acquisition and AI computing. The system has a compact structure, high reliability, and meets the requirements of harsh working environments in commercial vehicles.

[0022] In summary, this invention addresses the challenges of high complexity and specialized skill requirements in commercial vehicle system diagnosis. It reduces the difficulty of analyzing intermittent system failures and enhances users' self-troubleshooting capabilities. The commercial vehicle self-diagnostic system, based on comprehensive intelligent analysis of vehicle signals, data, and audio / video data, proactively addresses hidden faults and generates maintenance manuals for all identifiable faults, guiding users in troubleshooting and resolving vehicle malfunctions. For faults not detected by the system, user confirmation enables self-upgrading learning based on artificial intelligence. During the updating and learning of the AI ​​model, suspicious data is identified and fed back to the user, along with a suspicious data investigation plan, guiding users to gradually troubleshoot and handle new faults. All data storage and logical analysis are based locally on the device, ensuring high information security throughout the analysis process and data. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the hardware architecture of a self-diagnostic system for commercial vehicles based on artificial intelligence, provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the software task architecture of a self-diagnostic system for commercial vehicles based on artificial intelligence, provided in an embodiment of the present invention.

[0025] Figure 3 This is a complete logical flowchart of fault identification and self-learning in a self-diagnosis method for commercial vehicle systems based on artificial intelligence, provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] In the description of this application, it should be noted that the terms "upper," "lower," "inner," "outer," "top / bottom," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0028] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed," "equipped with," "sleeved / connected," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0029] The first aspect of this invention provides an artificial intelligence-based self-diagnosis method for commercial vehicle systems, the core of which lies in constructing a closed-loop process of "perception-fusion-diagnosis-learning". Specifically, it includes the following steps: S1: Hardware Platform Construction. Construct an in-vehicle hardware platform centered on an automotive-grade SoC. This platform needs to integrate a rich set of peripheral interfaces, including: multiple video input interfaces (such as MIPI CSI) for connecting surround-view and cabin-mounted cameras; multiple audio input interfaces (including ADCs) for connecting in-vehicle noise monitoring microphones; digital / analog input modules for acquiring signals from sensors such as doors, lights, various switches, and temperature, pressure, and vibration; power drive and detection modules for controlling loads such as lights, fans, and solenoid valves and for real-time detection of their open-circuit, short-circuit, and overload states; CAN gateway modules for accessing multiple CAN subnets in the vehicle to achieve data routing, forwarding, and recording; and Ethernet gateway modules for accessing the in-vehicle Ethernet to achieve high-speed data exchange and recording.

[0030] S2: Software System Architecture. A multi-tasking software architecture based on a real-time operating system (such as QNX, AutoSAR) or a high-performance Linux system is built on the SoC. This architecture includes, but is not limited to, the following key tasks: system management tasks (responsible for power management and task scheduling); video acquisition tasks; audio acquisition and preprocessing tasks (including real-time Fourier transform); switch / sensor signal acquisition tasks; power control and status monitoring tasks; CAN / Ethernet gateway routing and data logging tasks; vehicle driving status (non-audio / video data) management tasks; vehicle environmental status (audio / video data) management tasks; core artificial intelligence analysis tasks; user fault feedback interaction tasks; fault detection mechanism management tasks; and file log read / write tasks.

[0031] S3: Multi-source data acquisition. During vehicle operation, the system continuously and synchronously acquires two main categories of data: 1) Vehicle environmental data: video streams acquired through cameras and audio streams acquired through microphone arrays. 2) Vehicle driving status data: all relevant switch signals (such as brake switches and door switches), sensor signals (such as engine speed, coolant temperature, and air pressure), real-time status of each power output (normal / open circuit / short circuit / overload), all interactive messages on the vehicle's CAN bus (such as data from the engine ECU, transmission TCU, and body BCM), and high-speed data on the vehicle's Ethernet (such as raw data from ADAS cameras and high-precision map update data).

[0032] S4: Data Preprocessing and Feature Engineering. The raw data is standardized to adapt to the AI ​​model input. Key preprocessing includes: for audio data, frame segmentation and windowing, and converting the time-domain waveform into a frequency-domain spectrogram or Mel spectrogram using Fast Fourier Transform (FFT). This is more conducive to CNNs extracting frequency-domain features related to mechanical faults (such as abnormal noises). For video data, image normalization and noise reduction are performed. Furthermore, data fusion is implemented: video streams from multiple cameras are spatiotemporally aligned and stitched together (or feature-level fusion) to form a panoramic "bird's-eye view" or overall vehicle view; audio streams from multiple microphones are beamformed or feature-fused to enhance the ability to determine the direction of sound sources.

[0033] S5: Artificial Intelligence Fusion Diagnosis. Preprocessed multimodal data is input into a pre-trained and optimized CNN-LSTM hybrid model deployed on a SoC. The model's workflow is as follows: CNN branches (which can be multiple parallel CNN sub-networks) receive video frames and audio spectrograms respectively, extracting high-level spatial visual and acoustic features. LSTM branches receive time-series feature vectors composed of switch, sensor, and bus data. Subsequently, the feature sequences extracted by the CNN (which change over time) are fused with other time-series features processed by the LSTM (e.g., concatenated along the feature dimension), and then a comprehensive time-series model is performed by the LSTM network or subsequent fully connected layers. This model learns the boundaries between normal states and various fault states in a multi-dimensional spatiotemporal feature space. Through real-time inference, the model outputs the probability that the current system is in a normal state or a specific fault state. Furthermore, the model can predict the development trends of certain parameters based on time-series patterns, achieving preliminary predictive maintenance.

[0034] S6: Automatic Diagnostic Output. Once the AI ​​model identifies a fault with high confidence, the system immediately alerts the driver or fleet manager via in-vehicle display, voice, or wireless communication. More importantly, based on the identified fault type and characteristics, the system automatically matches and generates a structured repair manual or step-by-step troubleshooting guide from its built-in knowledge base. This guide can include possible faulty components, location diagrams, testing steps, required tools, and safety precautions, greatly assisting on-site repair personnel, especially non-experienced technicians.

[0035] S7: Human-Computer Interaction and Model Self-Learning. The system provides convenient fault feedback entry points (such as touchscreen buttons and voice commands like "Report Fault"). When a vehicle exhibits symptoms but the system does not automatically alarm, the user can manually trigger the system and briefly describe the fault (e.g., "The vehicle makes an abnormal noise when turning"). The system will record this moment and retrieve historical log data from the preceding and following periods (including all multi-source data). Subsequently, the system enters a "high-sensitivity troubleshooting mode": adjusting the AI ​​model's decision threshold (increasing sensitivity, which may reduce specificity), performing retrospective analysis of historical data, and identifying the "suspicious data patterns" or abnormal segments most relevant to the user's description. Based on these suspicious patterns, the system generates targeted data troubleshooting suggestions (e.g., "Please focus on checking the vibration data of the right rear wheel bearing and the right rear camera footage during the XX time period"). Repair personnel perform troubleshooting according to the suggestions, confirm the root cause of the fault, and repair it. Afterward, they can confirm the fault is resolved through the system interface. The system then uses the confirmed fault data sample (including the finally confirmed fault features) as new training data to start an online or periodic incremental model learning process, update the weight parameters of the CNN-LSTM model, and thus incorporate this new fault type into its automatic identification capability, achieving continuous evolution of the system.

[0036] A second aspect of this invention provides an artificial intelligence-based self-diagnostic system for commercial vehicles to implement the above-described method. This system adopts a layered architecture design: Hardware Layer: This is the physical carrier of the system, with the core being a high-performance SoC module that meets automotive-grade standards. Surrounding the SoC are multiple video input interface circuits, multiple audio acquisition and ADC circuits, digital / analog input isolation circuits, high-side / low-side intelligent power driver chips and current / voltage detection circuits, a multi-channel CAN FD controller and physical layer interface, and an automotive Ethernet switch chip and PHY interface. All hardware is integrated into a metal housing that meets automotive installation requirements, featuring wide voltage input, wide operating temperature, and vibration and shock resistance.

[0037] Data Acquisition and Processing Layer: This layer runs as a software task on the SoC. It contains multiple data acquisition drivers and preprocessing threads, responsible for reading raw data from various hardware interfaces in real time, performing format parsing, timestamp synchronization, cache management, and executing feature extraction operations such as audio FFT and video frame processing as described above. The processed standardized feature data stream is then published to the system's data bus or shared memory.

[0038] The intelligent analysis layer is the "brain" of the system. At its core is a lightweight, high-performance AI inference engine (such as TensorRT Lite or TFLite for Microcontrollers), which deploys an optimized CNN-LSTM diagnostic model. This layer subscribes to data streams published by the data acquisition layer and performs real-time fusion inference computation. In addition, this layer includes a fault knowledge base, a model parameter management module, and a self-learning logic controller. The knowledge base stores the mapping relationship between fault codes and maintenance knowledge. The self-learning logic controller is responsible for managing the high-sensitivity analysis process triggered by user feedback, as well as organizing the datasets required for incremental learning and triggering model update tasks.

[0039] Interaction and Output Layer: This includes the local Human-Machine Interface (HMI) and the remote communication interface. The local HMI can be a small touchscreen connected to the system or utilize the vehicle's original central control screen, used to display system status, real-time alarms, maintenance guides, and receive user input. The remote communication interface (such as a 4G / 5G cellular module) can be used as an optional component to upload anonymized diagnostic summaries, fault statistics, and other data to the fleet management platform with user permission, but without transmitting original audio, video, or other sensitive data, ensuring the localization of core diagnostic logic and data.

[0040] A third aspect of this invention provides a method for implementing self-diagnosis of a commercial vehicle system based on artificial intelligence, comprising the following steps: Step 1: Create a basic hardware system to provide the basic environment for AI-based self-diagnosis of commercial vehicle systems. The system consists of multiple parts. The hardware system is composed of a SoC chip for logic control and deploys AI model algorithms through the SoC. This enables AI analysis of vehicle environmental data and vehicle driving data on the device side and implements vehicle system fault self-diagnosis based on software algorithms. Step 2: Create a software system based on the hardware system, running in the SoC; create multiple tasks for the AI-based self-diagnosis logic of the commercial vehicle system; create system tasks to implement power management, power allocation and management in the hardware system; and deploy AI model algorithms. Step 3: Create a vehicle operation video acquisition task to acquire video data during the vehicle operation phase in real time; create a vehicle operation audio acquisition task to acquire audio data during the vehicle operation phase in real time, and perform Fourier transformation on the audio data to generate frequency domain-based audio data. Step 4: Create an input signal acquisition task to continuously acquire the status of external switch signals and sensor values ​​during vehicle operation; create a power output control task to enable user control of power output on / off; and create a power output status detection task to determine open circuit, short circuit, and overload status in real time. Step 5: Create a CAN gateway routing task to enable data forwarding and CAN data logging for multiple subnets of the vehicle's CAN network; create an Ethernet gateway routing task to enable data forwarding and Ethernet data logging for multiple subnets of the vehicle's Ethernet network. Step 6: Create a vehicle driving status management task to record, summarize, and output various non-audio and video data of vehicle driving; create a vehicle environmental status data management task to record, summarize, and output vehicle audio and video data. Step 7: Create an artificial intelligence analysis task, which includes artificial intelligence recognition based on independent image analysis of a single camera, composite image analysis after combining images from all cameras in the vehicle, independent audio analysis of a single microphone, composite audio analysis after combining multiple microphones in multi-channel audio, and data analysis of vehicle driving status. Step 8: Create a user feedback fault task to detect in real time whether there are user feedback faults; create a fault detection mechanism management task to realize the real-time update and management of artificial intelligence algorithm parameters, which is used to adjust the fault detection sensitivity; create a fault status detection task to detect whether the system has a fault, and generate a user maintenance manual after a fault occurs. Step 9: Create a file log read / write task to enable local recording and reading of raw data; The deployment of artificial intelligence algorithms in steps 1 and 2 includes the following steps: deploying a CNN-LSTM model, using CNN to perform spatial data analysis and recognition, mainly camera data and audio data; and using LSTM to perform temporal data analysis and recognition, mainly vehicle operation data, to achieve traceability analysis and predictive detection of fault states.

[0041] The hardware system consists of a SoC (System-on-a-Chip) for logic control and deployment based on RNN and LSTM models; video acquisition to obtain video stream data from the entire vehicle; audio acquisition to obtain audio data stream data from the entire vehicle; signal input to acquire and identify all vehicle switching signals and sensor signals; power output to control the power supply or output signals of the vehicle's loads, and to detect output status, including open circuit, short circuit, and overload conditions; CAN gateway routing to forward and record data from the vehicle's CAN network; Ethernet gateway routing to forward and record data from the vehicle's Ethernet network; and user fault input, which can be achieved through microphone, keyboard, mouse, or wireless input. Since vehicle system faults are always based on three elements—signal, data, and power supply—the commercial vehicle self-diagnostic system needs to simultaneously record vehicle video data, vehicle audio data, vehicle switching and sensor signal inputs, vehicle load power supply status, vehicle CAN network data, and vehicle Ethernet network data. The system also defines basic conditions for vehicle system fault states. Based on the reasonable logic of vehicle form, basic conditions are defined, such as the door should not be opened while the vehicle is in motion; the reversing lights should not be activated while the vehicle is moving forward. After completing the basic conditions for vehicle system fault states, the basic conditions for vehicle system fault states can still be upgraded and improved based on the vehicle design and testing phase. The CNN model performs feature extraction on video data and converted audio data; the LSTM model performs modeling and analysis of feature data to obtain the temporal patterns of system faults; after the user manually inputs fault information, the artificial intelligence model obtains historical data of the fault occurrence from the log file, replaces the artificial intelligence recognition parameters with more sensitive ones, and identifies suspicious vehicle data and operation processes, generating a suspicious data investigation plan; after the system fault is resolved based on user feedback, the system fault feature points are updated to the artificial intelligence model, and the artificial intelligence model updates the fault detection mechanism; during vehicle operation, the artificial intelligence model detects the system fault state in real time, and after detecting a fault state, it proactively informs the user and generates a maintenance manual.

[0042] Step 3, which involves Fourier transforming the audio data, includes the following steps: acquiring the time-domain data of the original audio ADC through the microphone, performing Fourier transform on the time-domain data, extracting and converting it into frequency-domain data, and then using CNN to analyze and recognize the audio data.

[0043] Step 6, which involves Fourier transforming the audio data, includes the following steps: acquiring the time-domain data of the original audio ADC through the microphone, performing Fourier transform on the time-domain data, extracting and converting it into frequency-domain data, and then using CNN to analyze and recognize the audio data.

[0044] Step 7 involves analyzing the synthesized image from all the vehicle cameras, including the following steps: synthesizing all the videos into one video signal, performing artificial intelligence feature analysis and recognition on the overall vehicle environment, and the recognized feature values ​​represent the overall vehicle video environment status when the vehicle system malfunctions. Step 7 involves the following steps for synthesizing multi-channel audio signals from multiple microphones: synthesizing all videos into multi-channel audio signals; performing artificial intelligence feature analysis and recognition on the overall vehicle environment; and identifying the feature values ​​to represent the overall vehicle audio environment status when the vehicle system malfunctions. Step 8, the real-time update and management of the artificial intelligence algorithm parameters, includes the following steps: The identification effectiveness and scope of the artificial intelligence algorithm are analyzed based on the detection sensitivity parameters and the limited data source range. Therefore, by adjusting the detection sensitivity and data range, accurate fault analysis by artificial intelligence can be achieved. Under general parameter conditions, existing types of fault states can be identified, but new types of fault states cannot be automatically identified. When the user provides manual fault feedback, by adjusting the detection sensitivity and data range of the artificial intelligence algorithm, suspicious data of newly added fault types can be identified. After informing the user of the identification results and troubleshooting plan, the customer updates the fault handling results to the artificial intelligence algorithm. The artificial intelligence algorithm then updates the various feature values ​​of the current fault, realizing the automatic identification of subsequent faults of the same type. The feature values ​​of the artificial intelligence algorithm in step 8 mainly include the following: single video feature value of vehicle system fault, composite video feature value of vehicle system fault, single audio feature value of vehicle system fault, composite audio feature value of vehicle system fault, and data feature value of vehicle system fault (switch signal, sensor signal, CAN data, Ethernet data).

[0045] The vehicle operation data aggregation includes the following steps: The system periodically acquires vehicle switch signals and sensor signal inputs; the system periodically detects the power output status to determine if there is an open circuit, short circuit, or overload; the CAN gateway routes the data, enabling the routing and forwarding of vehicle CAN network data, while simultaneously recording the vehicle CAN network data and transmitting it to the artificial intelligence model algorithm for analysis; the Ethernet gateway routes the data, enabling the routing and forwarding of vehicle Ethernet network data, while simultaneously recording the vehicle Ethernet network data and transmitting it to the artificial intelligence model algorithm for analysis. Vehicle fault diagnosis and analysis are all based on offline data identification and local data analysis, without the need for remote data transmission and networking, thus achieving higher information security and timeliness of data identification; The aggregation of vehicle audio and video data includes the following steps: the system continuously acquires camera video data through the camera interface; the system continuously collects audio through the microphone interface, and after Fourier transform, the audio data is converted from time domain data into frequency domain data that can be analyzed by artificial intelligence models; Vehicle environmental status analysis and recording includes the following steps: The system transmits audio and video data during vehicle operation to the SoC chip and artificial intelligence model algorithm for real-time analysis, and the data is recorded into a file log; Vehicle driving status analysis and recording includes the following steps: The system transmits the signal input, power output status, CAN network data, and Ethernet data during vehicle operation to the SoC chip and artificial intelligence model algorithm for real-time analysis, and the data is recorded into the file log; The automatic vehicle system discovery process includes the following steps: The system uses artificial intelligence to detect vehicle environmental data and vehicle driving data in real time, identifies vehicle system fault characteristics, and then proactively alerts the user to the vehicle system fault. Real-time vehicle environmental data detection includes the following steps: real-time detection of data from individual cameras; real-time detection of data from all cameras; real-time detection of data from individual microphones; real-time detection of data from all microphones. Real-time vehicle driving data detection includes the following steps: real-time detection of all CAN network data being transmitted by the vehicle; real-time detection of all Ethernet data being transmitted by the vehicle; real-time detection of all switching signals and sensor signals used by the vehicle; and real-time detection of the open circuit, short circuit, and overload status of all power outputs. Manually detecting vehicle system faults involves the following steps: If the system fails to detect a fault automatically, and the user clearly or suspects a fault, the user can manually input the fault symptom and the time of the fault occurrence; the artificial intelligence model obtains the corresponding fault log data based on the fault feedback information input by the user; the artificial intelligence model updates its analysis parameters to improve sensitivity and reduce accuracy, in order to identify suspicious data that fails to automatically identify faults; and a fault investigation plan is generated for the suspicious data. The vehicle system fault status self-learning includes the following steps: the user manually reports the fault and, with the help of the artificial intelligence troubleshooting suggestions, the fault is resolved; the artificial intelligence model updates the feature values ​​and updates the file logs.

[0046] In summary, the AI-based self-diagnosis method for commercial vehicle systems of this invention adopts a hardware system construction and software system operation approach. By deploying AI algorithms, it classifies and defines all signal data of the entire vehicle, realizing vehicle environmental data composed of audio and video data; and vehicle driving status data composed of switch signals, sensor signals, CAN data, and Ethernet data. Individual audio and video data form local states, allowing for detailed recording of fault states. The synthesized audio and video data form a global state, enabling preliminary identification of vehicle system faults. Preliminary identification of system fault states based on the overall state is performed first, followed by detailed identification of fault states using independent data, achieving a complete analysis of fault states. Combined with the analysis of fault points, corresponding troubleshooting and repair manuals can be generated, reducing the difficulty of fault diagnosis, avoiding data analysis delays caused by remote transmission of vehicle data, improving fault handling efficiency, increasing the identification rate of hidden vehicle faults, and enhancing vehicle driving safety.

[0047] Example 1: System Hardware Architecture like Figure 1 As shown, the system hardware in this embodiment is based on a high-performance automotive-grade SoC (e.g., NVIDIA Xavier NX, TI TDA4VM, or a similar chip with integrated AI acceleration cores). This SoC connects to the following key peripheral modules via an internal bus or high-speed serial interface: Video input module: Includes multiple MIPI CSI-2 receivers, which can be connected to up to 8 2-megapixel vehicle cameras (including front-view, rear-view, side-view, cabin monitoring, etc.), and supports H.264 / H.265 hardware encoding and decoding for video stream acquisition and compression.

[0048] Audio input module: Includes a multi-channel high-performance ADC and digital microphone interface, connecting 4-6 noise-canceling microphones located in key areas such as the engine compartment, driver's cab, and chassis, for collecting the sound environment of the entire vehicle.

[0049] Signal input module: includes multiple opto-isolated digital input (DI) channels for acquiring 24V / 12V switching signals (such as the status of various switches and relays); and multiple high-precision ADC channels for acquiring 4-20mA or 0-5V analog sensor signals (such as temperature and pressure sensors).

[0050] Power output and detection module: Integrates multiple high-side switches or intelligent power ICs (such as Infineon PROFET). Each output has independent current sampling (via sampling resistors and operational amplifiers) and overcurrent, short-circuit, and overheat protection circuits. The GPIO / ADC of the MCU or SoC can read the status (current value, load voltage) of each output in real time and determine whether it is open-circuited, short-circuited, or overloaded.

[0051] CAN Gateway Module: Contains 2-4 independent controllers supporting CAN FD, each connected to a different CAN network of the vehicle (such as powertrain CAN, body CAN, and entertainment CAN) via a CAN isolation transceiver. This module implements message filtering, forwarding, and protocol conversion between different CAN network segments, and bypasses copying all messages to the SoC for recording and analysis.

[0052] Ethernet gateway module: integrates an in-vehicle Ethernet switch (supports 100BASE-T1 / T2), provides multiple Ethernet ports, connects in-vehicle infotainment systems, ADAS domain controllers, T-Boxes and other devices, and enables high-speed data exchange and data recording.

[0053] Storage module: Includes LPDDR4 memory, eMMC flash memory, and a pluggable high-endurance microSD card for storing the operating system, applications, AI models, and cyclically recorded multi-source vehicle data logs.

[0054] Human-computer interaction and communication module: includes an LCD display interface (which may connect to a small touchscreen), an audio codec (for voice prompts), and optional 4G / 5G cellular modules and Wi-Fi / Bluetooth modules for local interaction and limited remote communication.

[0055] All modules are mounted on a multi-layer PCB and housed in a metal housing that meets IP67 protection standards. They are connected to the vehicle's electrical system via vehicle wiring harness connectors.

[0056] Example 2: System Software Architecture and Workflow like Figure 2 As shown, the software system runs on a real-time operating system (RTOS) on top of the SoC, and adopts a multi-threaded / multi-tasking design. The main tasks and workflows are as follows: System management tasks: Initialize all hardware and load AI model parameters during system startup. During runtime, it is responsible for scheduling various tasks, watchdog monitoring, and power mode management (switching working modes according to the vehicle's ignition status).

[0057] Data acquisition task group: Video capture task: Drive the camera to capture YUV or RGB images at a fixed frame rate (e.g., 15fps), perform color space conversion, scaling and other preprocessing, and then put them into the video data buffer.

[0058] Audio acquisition and processing task: Capture audio PCM data at a high sampling rate (e.g., 44.1kHz). Apply a Hamming window and perform FFT on each frame of audio (e.g., 1024 samples) to calculate its amplitude spectrum or Mel spectrum, generate a spectrogram sequence, and place it into the audio feature buffer.

[0059] Signal acquisition task: Periodically (e.g., every 10ms) scan all DI and ADC channels, read the switch status and sensor values, and package them into time series data packets.

[0060] Power status monitoring task: Periodically read the current and voltage of each power output, determine the status, and update the status register.

[0061] Gateway and logging task group: CAN gateway task: Monitor each CAN bus and forward necessary messages according to the preset routing table. At the same time, write all received CAN messages (with precise timestamps) to a circular log file.

[0062] Ethernet gateway tasks: Handling Ethernet frame routing and switching. Parsing and recording important application layer protocol data (such as SOME / IP and DoIP).

[0063] Log management tasks: Unify the management of log files (video, audio, signal, bus data) written from various data sources, ensure time synchronization, and implement a cyclic overwrite strategy to manage storage space.

[0064] Data Fusion and Management Task Group: Vehicle driving status management task: Receive and integrate non-audio and video data from task signal acquisition task, power status monitoring task, CAN gateway task, and Ethernet gateway task, organize them into a unified, timestamped "vehicle status vector" and provide it to AI analysis task.

[0065] Vehicle environmental status management task: Receives and integrates audio and video feature data from video acquisition and audio acquisition and processing tasks. Executes video synthesis algorithms (such as image stitching and feature fusion) and audio synthesis algorithms (such as sound source localization feature calculation) to generate a "vehicle environmental feature vector".

[0066] Core AI and Diagnostics Task Group: Artificial Intelligence Analysis Task: This is the core task. It subscribes to a fused feature stream from both the vehicle driving state management task and the vehicle environmental state management task. Internally, the task runs an optimized CNN-LSTM model inference engine. The model processes video frames and audio spectrograms in parallel (via the CNN branch) while simultaneously processing vehicle state vectors (via the LSTM branch), ultimately outputting a multi-dimensional vector representing the probability of various preset faults and an "overall anomaly score." This task runs at regular intervals (e.g., 1 second).

[0067] Fault detection mechanism management task: Maintain a dynamic fault detection threshold table. In normal mode, standard thresholds are used. When user feedback is received, this task lowers the judgment threshold for the relevant fault category (enters high-sensitivity mode) and notifies the AI ​​task to re-evaluate historical data.

[0068] Fault Status Detection and Output Task: Monitor the probability and anomaly score output by the AI ​​task. When the probability of any fault exceeds its current threshold, or the overall anomaly score exceeds the threshold, a fault is determined to have occurred. Subsequently, based on the fault type code, retrieve the corresponding fault description, possible causes, repair steps, diagrams, etc. from the local SQLite knowledge base, and generate an HTML or PDF repair manual file. Finally, trigger the interactive layer to issue an audible and visual alarm and display the manual.

[0069] User interaction task group: User feedback fault task: Listen for the "Fault Report" button on the touchscreen or specific voice keywords. When triggered, record the current timestamp and pop up a simple interface for the user to select or describe the fault. Package the user feedback information and send it to the fault detection mechanism management task.

[0070] Example 3: Fault Diagnosis and Self-Learning Closed-Loop Process like Figure 3 As shown, the core diagnostic and learning process of this invention is a dynamic closed loop: Step 301: Normal vehicle operation and full data collection. The system continues to operate, and all data collection and recording tasks remain running, recording all dimensions of vehicle data in local storage in a time-synchronized manner.

[0071] Step 302: Real-time AI Fault Detection. The artificial intelligence analysis task periodically performs inference calculations on the latest fused feature data.

[0072] Step 303: Determine if a fault has been detected. The fault state detection task makes a decision based on the model output.

[0073] Step 304: If a fault is detected (yes), the system immediately executes step 305: Generates and pushes a detailed repair manual, displays detailed instructions on the vehicle screen, and issues a warning sound. Simultaneously, the fault characteristic data detected will be automatically labeled and used as training samples to strengthen the model's memory of this type of fault.

[0074] Step 305: Repair Execution and Confirmation. Repair personnel perform troubleshooting and repairs according to the manual. After repair, a "Fault Resolved" confirmation can be obtained through the system interface. This confirmation signal triggers Step 310: Incremental Model Learning. The system adds high-quality data (especially features corresponding to the confirmed root cause of the fault) accumulated during the period from fault occurrence to resolution as positive samples to an incremental learning buffer. When the system is idle (e.g., after the vehicle is turned off), a low-priority background training task is started to fine-tune the CNN-LSTM model using the new data accumulated in the buffer, updating the model parameters and thereby enhancing or expanding its diagnostic capabilities.

[0075] Step 306: If no fault is detected (No), but the vehicle actually has an abnormality (perceived by the user), the user can actively trigger Step 307: Manual fault feedback.

[0076] Step 307: The user reports the fault via interface or voice, briefly describing the phenomenon and time.

[0077] Step 308: The system enters high-sensitivity troubleshooting mode. The fault detection mechanism management task adjusts relevant detection thresholds based on feedback. The system retraces historical log data before and after the user report time (e.g., 5 minutes before and after), and reanalyzes this data using the adjusted high-sensitivity model.

[0078] Step 309: Generate a suspicious data investigation plan. The AI ​​model will output the "most suspicious" abnormal patterns or time periods found in this historical data (e.g., a sudden increase in audio energy at a specific frequency at a certain moment, accompanied by a slight fluctuation in a certain sensor). The system will organize these suspicious points and their corresponding raw data fragments (such as spectrograms, video frames, signal curves) and generate a data-driven investigation suggestion (e.g., "Please check the abnormal sound and vibration data in the right rear wheel area near [timestamp], and visually inspect the components in this area is recommended"), and display it to the user.

[0079] Subsequently, maintenance personnel conducted targeted checks based on this data clue (step 305). After confirming and repairing the fault, they also confirmed it, which ultimately triggered incremental model learning in step 310, learning this new "fault mode-data feature" correlation into the model.

[0080] Through this closed loop, the system can not only diagnose known faults in the early stages of deployment, but also continuously learn and diagnose new and unknown fault types as the vehicle is used, thus continuously improving its intelligence level.

[0081] Example 4: Specific Implementation Details of the CNN-LSTM Model In this embodiment, the CNN-LSTM hybrid model deployed on the SoC has been carefully designed and pruned to adapt to the embedded environment. The model structure is briefly described below: Visual CNN subnetwork: Input is a 224x224x3 RGB image (from a single camera or synthetic view). Lightweight MobileNetV2 or EfficientNet-Lite is used as the backbone, the last fully connected layer is removed, and the output is a 1280-dimensional feature vector (per frame).

[0082] Acoustic CNN subnetwork: Input is a 128x128 single-channel Mel spectrogram (representing approximately 2 seconds of audio). A small 4-layer CNN (3x3 convolutional kernels, pooling layers) is used, which is finally flattened into a 256-dimensional feature vector (every 2 seconds).

[0083] The time-series data LSTM branch: The input is a vehicle state vector, which may have 50 dimensions (including 10 switch values, 15 sensor values, 20 selected CAN signal values, and 5 power status values). After compression through a fully connected layer, it is input into a 2-layer LSTM network with a hidden layer dimension of 128.

[0084] Feature Fusion and Decision: Assuming a 1-second analysis period, each second contains one visual feature vector (interpolated if the video frame rate is low), 0.5 acoustic feature vectors (since the audio segment is 2 seconds long), and multiple vehicle state vectors (sampled at 1Hz). These features of different frequencies are aligned to 1Hz in the time dimension through interpolation or pooling. Then, the aligned visual and acoustic feature vectors are concatenated with the vehicle state vectors in the feature dimension to form a fused feature vector of approximately (1280+256+128) = 1664 dimensions per second. This 1Hz fused feature sequence (e.g., the sequence from the most recent 30 seconds) is fed into a 2-layer fused LSTM network (256 hidden layer dimensions). Finally, the output of the LSTM at the last moment is mapped to the probabilities of N fault categories through a fully connected layer and a Softmax activation function, while another fully connected branch outputs a scalar "anomaly score".

[0085] The model is pre-trained on the server side using a large-scale commercial vehicle multi-source fault dataset, and then compressed and optimized using techniques such as knowledge distillation and quantization (INT8). Finally, it is deployed to run on the NPU or GPU of the SoC to ensure that the inference latency is less than 100ms, meeting the real-time requirements.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

[0091] The implementation principle of this invention is as follows: This invention discloses a self-diagnostic method and system for commercial vehicle systems based on artificial intelligence. The method includes: constructing an on-board hardware platform with a System-on-Chip (SoC) as its core, integrating multi-source interfaces for acquiring vehicle video, audio, switch signals, sensor signals, CAN bus data, and Ethernet data; deploying a CNN-LSTM fusion artificial intelligence model on the SoC to perform spatiotemporal feature fusion analysis on preprocessed vehicle environmental data and driving status data to identify system faults in real time; automatically generating and pushing repair manuals after fault diagnosis; supporting user manual feedback of unidentified faults, with the system assisting in troubleshooting by reviewing historical data and adjusting model sensitivity, and updating the model feature library after fault resolution to achieve self-learning. The corresponding system includes a hardware layer, a data acquisition and processing layer, an intelligent analysis layer, and an interaction layer. This invention achieves localized, real-time, intelligent diagnosis and autonomous evolution of commercial vehicle faults, significantly improving diagnostic coverage, accuracy, and repair efficiency.

[0092] Compared with existing technologies, the beneficial effects of the AI-based self-diagnosis system for commercial vehicles according to the present invention are as follows: System fault prompts are more timely and accurate; fault repair manuals are automatically generated by the equipment, improving vehicle repair efficiency; there is no need to wait for professional personnel to perform data analysis, and ordinary repair personnel can also troubleshoot and handle faults based on the repair manual, improving repair convenience and reducing the risk of hidden system faults; by acquiring all data from the entire vehicle system and inputting it into the system, unified AI performs system fault data analysis and viewing locally, which can solve the security risks caused by data outsourcing and achieve more efficient system fault perception; the data analysis and diagnosis process does not require vehicle networking, avoiding data delays and traffic consumption during remote data transmission; at the same time, the system also retains the method of manual fault feedback, adding another form of auxiliary identification of system faults to the AI ​​algorithm.

[0093] The embodiments described herein are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A self-diagnostic method for a commercial vehicle system based on artificial intelligence, characterized in that, Includes the following steps: S1: Construct a hardware platform for self-diagnosis of commercial vehicle systems. The hardware platform includes a system-on-a-chip and a multi-channel video acquisition module, a multi-channel audio acquisition module, a switch signal and sensor input module, a power output and control module, a CAN gateway module, and an Ethernet gateway module, which are respectively connected to the system-on-a-chip. S2: Based on the hardware platform, a multi-task software system is built and run on the system-on-a-chip. The software system includes system management tasks, data acquisition tasks, data forwarding and recording tasks, status management tasks, artificial intelligence analysis tasks, fault interaction tasks, and log management tasks. S3: Through the data acquisition task, multi-source data during vehicle operation is collected in real time or periodically. The multi-source data includes: vehicle environment data, including video stream data from at least one camera and audio stream data from at least one microphone; vehicle driving status data, including switch signal status, sensor values, power output status, vehicle CAN network data, and vehicle Ethernet network data. S4: Preprocess and extract features from the collected multi-source data, including converting audio time-domain data into frequency-domain data and spatially synthesizing multi-channel video data and multi-channel audio data respectively. S5: Input the preprocessed data into the artificial intelligence model pre-deployed on the system-on-a-chip for analysis. The artificial intelligence model is a CNN-LSTM model that integrates convolutional neural networks and long short-term memory networks. It is used to fuse and model the spatial features of vehicle environmental data and the temporal features of vehicle driving status data in order to identify vehicle system faults. S6: When the artificial intelligence model detects a system fault, it automatically generates a maintenance manual or troubleshooting guide corresponding to the fault and prompts the user. S7: Provides a user manual fault feedback interface. When a user reports an unidentified fault, the system retrieves log data from the fault period, adjusts the detection parameters of the artificial intelligence model for re-analysis, identifies suspicious data, and generates a troubleshooting plan. After the user confirms that the fault has been resolved, the new fault characteristics are updated to the artificial intelligence model to complete the model's self-learning.

2. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, In step S5, the CNN-LSTM model works as follows: the CNN part is used to extract spatial features from the video data and the audio frequency domain data after Fourier transform; the LSTM part is used to model the temporal correlation of the vehicle driving state data, and further the spatial feature sequence extracted by the CNN is analyzed in the time dimension to achieve traceability and predictive detection of faults.

3. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, In step S4, the specific method for converting the audio time-domain data into frequency-domain data is as follows: analog audio signals are collected through a microphone and converted from analog to digital to obtain time-domain digital signals; the time-domain digital signals are subjected to fast Fourier transform to extract their feature representation in the frequency domain for CNN to perform feature analysis and recognition.

4. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, In step S4, the spatial synthesis of the multi-channel video data specifically involves: synchronizing and spatially stitching or feature-level fusing video streams from multiple cameras throughout the vehicle to generate a synthetic video stream or synthetic visual feature map that reflects the overall surrounding environment of the vehicle, which is then used by the artificial intelligence model for vehicle-level visual environment status analysis and fault feature identification.

5. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, In step S4, the spatial synthesis of the multi-channel audio data specifically involves: synchronizing and fusing the multi-channel audio streams from multiple microphones throughout the vehicle to generate a synthesized audio stream or acoustic feature map containing spatial orientation information, which is then used by the artificial intelligence model for vehicle-level acoustic environment status analysis and fault feature identification.

6. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, In step S7, adjusting the detection parameters of the artificial intelligence model includes improving the sensitivity of the model detection and / or broadening the range of data sources analyzed, so as to identify suspicious patterns in historical log data that were not judged as faults under standard parameters.

7. The self-diagnosis method for a commercial vehicle system based on artificial intelligence according to claim 1, characterized in that, The fault feature values ​​learned by the artificial intelligence model include at least: fault video feature values ​​based on a single camera, vehicle fault visual feature values ​​based on synthesized video, fault audio feature values ​​based on a single microphone, vehicle fault acoustic feature values ​​based on synthesized audio, and vehicle system fault data feature values ​​based on switch signals, sensor signals, CAN data, and Ethernet data.

8. A self-diagnostic system for a commercial vehicle system based on artificial intelligence for implementing the method of any one of claims 1 to 7, characterized in that, include: The hardware layer includes a system-on-a-chip (SoC) as the core processing unit, and a multi-source data acquisition and interface unit built around the SoC. The data acquisition and processing layer runs on the system-on-a-chip and is used to execute data acquisition and preprocessing tasks in the multi-task software system, realizing real-time acquisition, format conversion and preliminary fusion of multi-source data; The intelligent analysis layer, running on the system-on-a-chip, integrates the CNN-LSTM artificial intelligence model and is used to receive feature data from the data acquisition and processing layer, and to perform fault diagnosis analysis, fault feature extraction and model self-learning update logic. The interaction and output layer is used to receive the diagnostic results of the intelligent analysis layer, realize fault information prompts, generate and display maintenance manuals, and provide an input interface for users to manually provide fault feedback.

9. The self-diagnostic system for commercial vehicle systems based on artificial intelligence according to claim 8, characterized in that, The multi-source data acquisition and interface unit of the hardware layer specifically includes: a multi-channel video input interface for connecting vehicle cameras and / or video sources; a multi-channel audio input interface for connecting vehicle microphones; a digital input / output and analog-to-digital conversion module for acquiring vehicle switching signals and analog / digital sensor signals; a power drive and detection module for controlling vehicle load and having status readback function; a CAN gateway module for connecting to the vehicle CAN bus network and performing data routing and isolation; and an Ethernet gateway module for connecting to the vehicle Ethernet and performing data exchange.

10. The self-diagnostic system for commercial vehicle systems based on artificial intelligence according to claim 8, characterized in that, All data acquisition, storage, analysis, and diagnostic processes of the system are completed locally in the vehicle, without relying on a remote cloud server. The on-chip system, data storage module, and related circuits are integrated into an in-vehicle embedded device.

Images