Vehicle-mounted camera chip neural network fault diagnosis method and self-repairing redundant circuit

Abstract

The invention relates to the technical field of vehicle-mounted cameras, and particularly discloses a vehicle-mounted camera chip neural network fault diagnosis method and a self-repairing redundant circuit. Chip data of different vehicle-mounted camera chips on a multi-core architecture are collected at the same time; image features are extracted from the chip data by using a convolutional neural network, numeric features are extracted from the chip data by using a full-connection neural network, and complex space structures and modes in the images can be learned through the convolutional neural network, so that the features of the images are described more comprehensively; through the full-connection neural network, factors influencing the image quality can be comprehensively considered, the quality of the image signal is more comprehensively diagnosed, the neural network is continuously utilized to learn and analyze features in the image signal, and new image quality problems or potential faults are timely found, so that comprehensive and continuous diagnosis of the image signal quality is realized, and the diagnosis efficiency is improved. And the accuracy of chip fault diagnosis is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle-mounted cameras, and particularly relates to a neural network fault diagnosis method for vehicle-mounted camera chips and a self-repairing redundant circuit. Background Art

[0002] The chips of vehicle-mounted cameras usually include an image sensor chip, a digital signal processing chip, a power management chip, and a serializer / deserializer chip. The image sensor chip converts optical signals into electrical signals, the digital signal processing chip is used to process the electrical signals output by the image sensor chip, the power management chip provides a stable power supply for other chips in the camera, and the serializer / deserializer chip performs high-speed data transmission between the camera and the vehicle-mounted host, converting the parallel data output by the camera into serial data to reduce the number and cost of transmission cables. In order to ensure driving safety and the stable performance of the camera, it is necessary to diagnose the faults of vehicle-mounted camera chips.

[0003] Currently, the method for diagnosing faults in vehicle-mounted camera chips is usually to evaluate quality indicators such as image clarity, color restoration, and contrast to determine whether the camera is blocked, whether the installation position is normal, and whether the image processing function of the chip is normal. For example, by comparing the actually captured image with a standard image, detecting abnormal areas or distortion in the image.

[0004] However, since different people may have different evaluations of image quality, and for some minor image quality problems, it may be impossible to accurately determine whether they are caused by chip faults or other factors, such as lens dirt, light conditions, etc., resulting in unsatisfactory accuracy of chip fault diagnosis. Therefore, we need to propose a neural network fault diagnosis method for vehicle-mounted camera chips and a self-repairing redundant circuit to solve the above existing problems, enabling it to learn and analyze complex features in the image signal using a neural network, more comprehensively diagnose the quality of the image signal, and improve the accuracy of chip fault diagnosis. Summary of the Invention

[0005] The purpose of the present invention is to provide a neural network fault diagnosis method for vehicle-mounted camera chips and a self-repairing redundant circuit, which can use a neural network to learn and analyze complex features in the image signal, more comprehensively diagnose the quality of the image signal, and improve the accuracy of chip fault diagnosis, so as to solve the problems raised in the above background art.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A neural network fault diagnosis method for vehicle-mounted camera chips includes the following steps:

[0008] S1. Simultaneously collect the chip data of different in-vehicle camera chips on the multi-core architecture. The chip data includes power parameters, image signal data, and image quality-related data.

[0009] S2. Use a convolutional neural network to extract image features from the chip data.

[0010] S3. Use a fully connected neural network to extract numerical features from the chip data.

[0011] S4. Input the image features and numerical features into a pre-trained classification neural network, classify the state of the chip according to the learned feature patterns, and determine whether the chip has a fault.

[0012] If the chip has a fault, use the parallel computing of the multi-core architecture to determine the specific location where the fault occurs; enter S5.

[0013] If the chip has no fault, continue to monitor the in-vehicle camera chip and repeat S1 - S4.

[0014] S5. Perform targeted repair through the self-repair redundant circuit according to the specific location where the fault occurs.

[0015] S6. Conduct a comprehensive test on the repaired chip to evaluate whether the performance of the repaired chip has returned to the normal level. If the performance of the repaired chip has not returned to the normal level, it is necessary to re-determine the fault location and return to S5.

[0016] If the performance of the repaired chip has returned to the normal level, continue to monitor the in-vehicle camera chip and repeat S1 - S4.

[0017] Preferably, the multi-core architecture includes a processor unit with multiple independent cores, a multi-level cache unit, a bus connection unit, and a memory controller; the MESI protocol for ensuring data consistency is integrated in the processor unit, and the processor unit has an arithmetic logic unit, a register bank, and a cache component.

[0018] Preferably, the acquisition process of the chip data is as follows:

[0019] A1. Perform initialization operations on the operating environment of each core of the processor unit and the hardware interface with the in-vehicle camera chip.

[0020] A2. Develop a task allocation strategy according to the number of cores and the number of chips in the multi-core architecture, and clarify the chip numbers and data types responsible for collection by each core.

[0021] A3. Each core collects the power parameter data of the chip using the power monitoring circuit according to the task allocation strategy, collects the image data through the image signal transmission interface, and reads the image quality-related data through the memory controller.

[0022] A4. Aggregate the power parameter data, image data, and image quality-related data and store them in the shared memory area.

[0023] Preferably, the process of the convolutional neural network for image feature extraction is as follows:

[0024] B1. Perform a convolution operation on the input image and the convolution kernel to generate a feature map;

[0025] B2. Use an activation function on the generated feature map to introduce non-linear features into the convolutional neural network;

[0026] B3. Perform a pooling operation on the feature map processed by the activation function to obtain a downsampled feature map.

[0027] Preferably, the process of the numerical feature extraction is as follows:

[0028] C1. Organize the collected numerical data on the in-vehicle camera chip and the statistical parameters of the image signal to form a numerical input vector;

[0029] C2. Perform a matrix multiplication operation on the numerical input vector and the weight matrix of the first fully connected layer, and then add the bias vector of the first fully connected layer to obtain a weighted summation result;

[0030] C3. Input the weighted summation result into an activation function for activation operation to obtain the output of the first fully connected layer;

[0031] C4. Use the output of the previous fully connected layer as the input of the current layer, perform a matrix multiplication operation with the weight matrix of the current layer, add the bias vector of the current layer to obtain the weighted summation result of the current layer, and then obtain the output of the current layer through the activation function operation on the weighted summation result of the current layer;

[0032] C5. Repeat C4 to process multiple hidden layers in sequence to obtain the output of the last hidden layer;

[0033] C6. Perform a matrix multiplication operation on the output of the last hidden layer and the weight matrix of the output layer, add the bias vector of the output layer to obtain the output of the output layer, and then obtain the finally extracted numerical feature after processing the output result of the output layer through the activation function.

[0034] Preferably, the classification neural network is set as a multi-layer perceptron structure, and the classification neural network is composed of a classification input layer, multiple classification hidden layers, and a classification output layer. The classification input layer receives the concatenated image features and numerical features to form a feature vector. Each classification hidden layer contains a number of neurons, and the classification hidden layer performs a non-linear transformation on the input features to learn complex patterns in the data. The classification output layer uses the Softmax function to convert the output into a probability distribution of two categories: fault and normal.

[0035] Preferably, the process of pre-training the classification neural network is as follows:

[0036] D1. Collect a large amount of data of in-vehicle camera chips in normal and faulty states, and at the same time extract image features and numerical features, and form training feature vectors from the image features and numerical features;

[0037] D2. Label each training feature vector with the corresponding class label;

[0038] D3. Divide the training feature vectors into a training set, a validation set, and a test set;

[0039] D4. Determine the bias vector of the classification input layer according to the weight matrix of the classification input layer of the classification neural network. The bias vector of the classification input layer is the number of neurons in the first classification hidden layer * the number of neurons in the classification input layer;

[0040] D5. Input the training feature vectors in the training set into the classification neural network, and perform weighted sum calculation and activation output calculation on each classification hidden layer in turn;

[0041] D6. The classification output layer performs a weighted sum according to the activation output of the last classification hidden layer, then performs an activation output using the activation function ReLU according to the weighted sum result of the classification output layer, and finally converts the activation output value into a probability output using the Softmax function; where,

[0042] The calculation formula for the weighted sum of the classification output layer is:

[0043] z o =W o y o-1 +b o where, z o is the weighted sum result of the classification output layer, W o is the weight matrix of the classification output layer, y o-1 is the output result of the last classification hidden layer, and b o is the bias vector of the classification output layer;

[0044] The calculation formula for the activation output calculation of the classification output layer is:

[0045] y o = σ o (z o ), where y o is the output after the classification output layer is processed by the activation function, and σ o is the ReLU activation function used by the classification output layer, and z o is the weighted sum result of the classification output layer;

[0046] The conversion formula of the Softmax function is:

[0047] where P(c|x) is the probability that the sample x belongs to the category c, and z c is the output of the c-th neuron in the classification output layer, C is the total number of categories, and z k is the output of the k-th category in the classification output layer, and k is the index variable of the category, is the exponential operation with base e for the output of the k-th category in the classification output layer, and e is the base of the exponential function;

[0048] D7. Calculate the loss value of the trained classification neural network using the cross-entropy loss function;

[0049] D8. Calculate the partial derivatives of the weight matrix and bias vector of each layer in the classification neural network according to the loss value, and calculate the gradient of the weight matrix and the gradient of the bias vector;

[0050] D9. Update the weight matrix and bias vector of each layer using the optimization algorithm;

[0051] D10. Repeat D5 - D9 until the loss function converges to the minimum value.

[0052] Preferably, when determining the specific location of the fault, first establish a fault feature library, which contains the feature patterns corresponding to different fault types, and then use the cosine similarity to calculate the similarity between the current fault feature vector and the feature patterns in the fault feature library, and find the corresponding index position with the maximum similarity. The position associated with the fault pattern corresponding to this index position is the possible fault occurrence location.

[0053] Preferably, the process of targeted repair of the self-repairing redundant circuit is as follows:

[0054] S51. Determine the specific location of the fault on the chip according to the possible fault occurrence location and encode it to obtain the encoded fault location information;

[0055] S52. After receiving the fault location information, query the redundant circuit resource mapping table to find the redundant circuit resources that match the fault location. The redundant circuit resources include the redundant storage unit address and the switching control information;

[0056] S53. Control the switching circuit to switch from the connected faulty circuit to the state of preparing to connect the redundant circuit, complete the pre-connection of the line and the preliminary configuration of the signal path;

[0057] S54. Send an isolation signal to the fault area isolation circuit to cut off the electrical connection of the faulty circuit; then trigger the switching circuit to connect the redundant circuit to the chip working circuit, enabling it to undertake the original functions of the faulty circuit;

[0058] S55. Conduct a quick test on the repaired functional area to check whether the functions of the redundant circuit are normal; if the redundant functions are normal, record the successful repair information and notify the system to resume normal operation, and enter S6; if the redundant functions are abnormal, restart the fault location process to troubleshoot problems and return to S51.

[0059] Based on the above-described neural network fault diagnosis method for in-vehicle camera chips, the present invention also provides a self-repairing redundant circuit for neural network fault diagnosis of in-vehicle camera chips, including: a redundant circuit unit, which quickly replaces the corresponding module of the main circuit when a fault occurs in the main circuit to ensure the normal operation of the overall functions of the chip;

[0060] A fault detection module, which monitors the operating state of the main circuit in real time, discovers faults in a timely manner and issues fault signals;

[0061] A switching control module, which isolates the main circuit from the faulty part according to the instructions of the management module when a fault occurs and connects the redundant circuit to the working circuit;

[0062] A management module, which receives the fault signals from the fault detection module, queries the redundant circuit resource mapping table to determine the redundant circuit resources, and sends switching instructions to the switching control module.

[0063] The neural network fault diagnosis method and self-repairing redundant circuit for in-vehicle camera chips proposed by the present invention have the following advantages compared with the prior art:

[0064] 1. The present invention collects chip data of different in-vehicle camera chips on a multi-core architecture simultaneously, then uses a convolutional neural network to extract image features from the chip data and a fully connected neural network to extract numerical features from the chip data. The convolutional neural network can learn complex spatial structures and patterns in the image, thus more comprehensively describing the features of the image. The fully connected neural network can comprehensively consider factors affecting image quality and more comprehensively diagnose the quality of the image signal. By inputting the image features and numerical features into a pre-trained classification neural model, it can accurately determine whether there is a fault in the chip, and further more comprehensively diagnose the quality of the image signal. Using the parallel computing of the multi-core architecture to determine the specific location where the fault occurs can more accurately understand the specific impact of the fault on image quality. Through a self-repairing redundant circuit, targeted repair is carried out on the specific location where the fault occurs, and a comprehensive test is performed on the repaired chip to evaluate whether the performance of the repaired chip has returned to the normal level. Continue to monitor the in-vehicle camera chip, continuously use the neural network to learn and analyze the features in the image signal, and timely discover new image quality problems or potential faults, so as to achieve a comprehensive and continuous diagnosis of the image signal quality and improve the accuracy of chip fault diagnosis;

[0065] 2. The present invention realizes the rapid processing of faults in in-vehicle camera chips by the cooperation of a redundant circuit unit, a fault detection module, a switching control module and a management module. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart according to an embodiment of the present invention is shown;

[0067] Figure 2 A flowchart for collecting chip data according to an embodiment of the present invention is shown;

[0068] Figure 3 A flowchart for extracting numerical features according to an embodiment of the present invention is shown;

[0069] Figure 4 A flowchart for targeted repair by a self-repairing redundant circuit according to an embodiment of the present invention is shown;

[0070] Figure 5 A structural diagram of an automatic repair redundant circuit according to an embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0072] The present invention provides a neural network fault diagnosis method for in-vehicle camera chips as Figures 1-4 shown, including the following steps:

[0073] S1. Simultaneously collect chip data of different in-vehicle camera chips on a multi-core architecture. The chip data includes power supply parameters, image signal data, and image quality-related data;

[0074] The multi-core architecture includes a processor unit with multiple independent cores, a multi-level cache unit, a bus connection unit, and a memory controller; the processor unit integrates the MESI protocol for ensuring data consistency. The processor unit has an arithmetic logic unit, a register set, and a cache component. The arithmetic logic unit is set as an arithmetic logic unit ALU and a floating-point arithmetic unit FPU. The cache component is set as a first-level cache L1 Cache and a second-level cache L2 Cache. Each core of the processor unit can independently execute instructions to complete data operations and logical judgment operations;

[0075] The multi-level cache unit improves the efficiency of core data access by optimizing the cache hierarchy, significantly enhancing the multi-core collaborative working performance; the bus connection unit includes an on-chip bus, which is used to transfer data, instructions, and control signals between different components; the memory controller is used to manage the data interaction between the processor unit and the main memory, which can improve the memory access efficiency and resource utilization rate.

[0076] The acquisition process of the chip data is as follows:

[0077] A1. Initialize the operating environment of each core of the processor unit and the hardware interface with the in-vehicle camera chip. Among them, the initialization operation of the core's operating environment includes allocating memory space, loading necessary driver programs, and acquisition program modules; the initialization operation of the hardware interface of the in-vehicle camera chip includes initialization configurations such as the baud rate, data bits, and parking positions of interface communication;

[0078] A2. Develop a task allocation strategy based on the number of cores and chips in the multi-core architecture, and clearly define the chip number and data type each core is responsible for collecting. When developing the task allocation strategy, if there are four cores and eight onboard camera chips, the eight chips can be evenly distributed among the four cores, with each core responsible for collecting data from two chips. For example, core 1 is responsible for collecting power parameters, image signal data, and image quality-related data from chips 1 and 2.

[0079] A3. The core uses the power monitoring circuit to collect chip power parameter data according to the task allocation strategy, collects image data through the image signal transmission interface, and reads image quality-related data through the memory controller;

[0080] A4. Aggregate the power parameter data, image data, and image quality related data into a shared memory area for storage;

[0081] S2, using convolutional neural networks to extract image features from chip data;

[0082] The process of image feature extraction using the convolutional neural network is as follows:

[0083] B1. Perform convolution operation on the input image and the convolution kernel to generate a feature map. The formula for convolution operation is:

[0084] Among them, F(i,j) is the feature map with coordinates (i,j), K(m,n) is the convolution kernel with coordinates (m,n), I(i+m,j+m) is the image corresponding to the position (m,n) of the input image relative to the current feature map position (i,j) according to the position (m,n) in the convolution kernel, and M and N are the height and width of the convolution kernel respectively;

[0085] B2. Use the activation function to introduce nonlinear features into the convolutional neural network. The formula for introducing nonlinear features is: f(x) = max[0, F(i, j)], where f(x) is the output feature of the Haipai function, F(i, j) is the feature map after the convolution operation, and max[.] is the maximum activation function. The maximum activation function sets input values less than 0 to 0 and keeps input values greater than 0 unchanged, thereby adding nonlinear characteristics to the neural network.

[0086] B3. Perform a pooling operation on the feature map processed by the activation function to obtain the downsampled feature map. The formula for the pooling operation is:

[0087] Among them, S is the pooling size, f(i*S + m, j*S + n) is the image feature of the activated feature within the pooling kernel range starting from (i*S, j*S), m and n are the offsets within the pooling kernel, with values ranging from 0 to S - 1, and P(i, j) is the feature map at position (i, j) after pooling;

[0088] Convolutional neural networks have the characteristics of local perception and weight sharing, and can automatically learn local features at different positions in the image. Through the combination of multiple convolutional layers and pooling layers, convolutional neural networks can gradually extract different levels of features such as edges, textures, and shapes of the image; for example, smaller convolutional kernels can capture fine textures and details in the image, while larger convolutional kernels can extract more macroscopic shape and structural features; through this hierarchical feature extraction method, convolutional neural networks can learn the complex spatial structures and patterns in the image signal, thus more comprehensively describing the features of the image and providing rich information for subsequent image quality diagnosis.

[0089] S3. Use a fully connected neural network to extract numerical features from the chip data;

[0090] The process of the above-mentioned numerical feature extraction is as follows:

[0091] C1. Organize the collected numerical data and statistical parameters of the image signal regarding the in-vehicle camera chip to form a numerical input vector;

[0092] C2. Perform a matrix multiplication operation on the numerical input vector and the weight matrix of the first fully connected layer, and then add the bias vector of the first fully connected layer to obtain the weighted sum result; the formula for the weighted sum is:

[0093] z1 = W1X + b1, where z1 is the weighted sum result, W1 is the weight matrix of the first fully connected layer, X is the numerical input vector, and b1 is the bias vector of the first fully connected layer;

[0094] C3. Input the weighted sum result into the activation function for activation operation to obtain the output of the first fully connected layer; the formula for the activation operation is:

[0095] y1 = σ1(z1), where y1 is the output of the first fully connected layer after being processed by the activation function, σ1 is the activation function ReLU used in the first fully connected layer, and z1 is the weighted sum result;

[0096] C4. Use the output of the previous fully connected layer as the input of the current layer, perform a matrix multiplication operation with the weight matrix of the current layer, add the bias vector of the current layer to obtain the weighted sum result of the current layer, and then obtain the output of the current layer through the activation function operation on the weighted sum result of the current layer;

[0097] C5. Repeat the execution of C4, process multiple hidden layers sequentially, and obtain the output of the last hidden layer;

[0098] C6. Perform a matrix multiplication operation on the output of the last hidden layer and the weight matrix of the output layer, add the bias vector of the output layer to obtain the output of the output layer, and then process the result of the output of the output layer through an activation function to obtain the finally extracted numerical features;

[0099] Through multi-layer weighted summation and activation function transformation, the fully connected neural network gradually extracts more representative features that can better reflect the internal patterns of the data from the original numerical data, providing effective data support for subsequent tasks such as fault diagnosis.

[0100] S4. Input the image features and numerical features into a pre-trained classification neural network, classify the state of the chip according to the learned feature patterns, and determine whether the chip has a fault;

[0101] If the chip has a fault, use the parallel computing of the multi-core architecture to determine the specific location where the fault occurs; enter S5;

[0102] When determining the specific location where the fault occurs, first establish a fault feature library, which contains the feature patterns corresponding to different fault types, and then use the cosine similarity to calculate the similarity between the current fault feature vector and the feature patterns in the fault feature library, find the corresponding index position with the largest similarity, and the position associated with the fault pattern corresponding to this index position is the possible fault occurrence position; the cosine similarity calculation formula is:

[0103] where, s j is the cosine similarity between the current fault feature vector and the j-th feature pattern in the fault feature library, and its value range is [-1, 1]. The closer the value is to 1, the higher the similarity; is the feature vector of the current fault, is the j-th feature pattern in the fault feature library;

[0104] If the chip has no fault, continue to monitor the in-vehicle camera chip and repeat the execution of S1 - S4;

[0105] Among them, the classification neural network is set to a multi-layer perceptron structure. The classification neural network consists of a classification input layer, multiple classification hidden layers, and a classification output layer. The classification input layer receives the feature vector formed by splicing the image features and numerical features; each classification hidden layer contains several neurons, and the classification hidden layer performs a non-linear transformation on the input features to learn the complex patterns in the data; the classification output layer uses the Softmax function to convert the output into the probability distribution of two categories: fault and normal;

[0106] The process of pre-training the classification neural network is as follows:

[0107] D1. Collect a large amount of data on in-vehicle camera chips containing normal and faulty states, and at the same time extract image features and numerical features, and combine the image features and numerical features to form training feature vectors;

[0108] D2. Label each training feature vector with a corresponding class label, that is, set the class label of the faulty feature vector to 1 and the class label of the normal feature vector to 0;

[0109] D3. Divide the training feature vectors into a training set, a validation set, and a test set. The training set is used for training the classification neural network, the validation set is used for tuning the hyperparameters of the classification neural network, and the test set is used for evaluating the final performance of the classification neural network;

[0110] D4. Determine the bias vector of the classification input layer according to the weight matrix of the classification input layer of the classification neural network. The bias vector of the classification input layer is the number of neurons in the first classification hidden layer * the number of neurons in the classification input layer;

[0111] D5. Input the training feature vectors in the training set into the classification neural network, and perform weighted summation calculation and activation output calculation on each classification hidden layer in turn. The formula for weighted summation of the classification hidden layer is:

[0112] z l =W l y l-1 +b l where z l is the weighted summation result of the l-th classification hidden layer, W l is the weight matrix of the l-th classification hidden layer, y l-1 is the output result of the previous layer of the l-th layer, and b l is the bias vector of the l-th classification hidden layer;

[0113] The formula for activation output calculation is:

[0114] y l =σ l (z l ) where y l is the output of the l-th classification hidden layer after being processed by the activation function, σ l is the activation function ReLU used in the l-th classification hidden layer, and z l is the weighted summation result of the l-th classification hidden layer;

[0115] D6. The classification output layer performs a weighted sum based on the activation output of the last classification hidden layer, then uses the ReLU activation function to activate the output according to the result of the weighted sum of the classification output layer, and finally converts the activated output value into a probability output using the Softmax function; among them,

[0116] The calculation formula for the weighted sum of the classification output layer is:

[0117] z o =W o y o-1 +b o , where z o is the result of the weighted sum of the classification output layer, W o is the weight matrix of the classification output layer, y o-1 is the output result of the last classification hidden layer, and b o is the bias vector of the classification output layer;

[0118] The calculation formula for the activation output of the classification output layer is:

[0119] y o =σ o (z o ), where y o is the output of the classification output layer after being processed by the activation function, σ o is the ReLU activation function used by the classification output layer, and z o is the result of the weighted sum of the classification output layer;

[0120] The conversion formula of the Softmax function is:

[0121] Among them, P(c|x) is the probability that the sample x belongs to the category c, z c is the output of the c-th neuron of the classification output layer, C is the total number of categories, z k is the output of the k-th category of the classification output layer, k is the index variable of the category, is the exponential operation with base e for the output of the k-th category of the classification output layer, and e is the base of the exponential function;

[0122] D7. Use the cross-entropy loss function to calculate the loss value of the trained classification neural network. The calculation formula of the cross-entropy loss function is:

[0123] Among them, L is the loss value of the trained classification neural network, N is the training sample; C is the total number of categories, c is the index of the category to which the sample i belongs, y ic is the true label of the sample i belonging to the category c, and P ic is the predicted probability that the sample i belongs to the category c;

[0124] D8. Calculate the partial derivatives of the weight matrix and bias vector of each layer in the classification neural network according to the loss value, and calculate the gradients of the weight matrix and the bias vector. The calculation formula for the gradient of the weight matrix is:

[0125] where D wl is the gradient of the weight matrix of the l-th layer, W l is the weight matrix of the l-th layer, and L is the loss value of the classification neural network;

[0126] The calculation formula for the gradient of the bias vector is:

[0127] where D bl is the gradient of the bias vector of the l-th layer, W l is the bias vector of the l-th layer, and L is the loss value of the classification neural network;

[0128] D9. Use an optimization algorithm to update the weight matrix and bias vector of each layer;

[0129] The formula for updating the weight matrix is:

[0130] where is the weight matrix of the l-th layer after update, w l is the weight matrix of the l-th layer before update, α is the learning rate of the classification neural network, and D wl is the gradient of the weight matrix of the l-th layer;

[0131] The formula for updating the bias vector is:

[0132] where is the bias vector of the l-th layer after update, b l is the bias vector of the l-th layer before update, α is the learning rate of the classification neural network, and D bl is the gradient of the bias vector of the l-th layer;

[0133] D10. Repeat D5 - D9 until the loss function converges to the minimum value;

[0134] The process for determining whether a chip has a fault is as follows:

[0135] S41. Set the threshold for the output probability distribution of the fault categories of the classification neural network, and the threshold is set to 0.5;

[0136] S42. Concatenate the image features and numerical features into a feature vector and input it into the trained classification neural network. After forward propagation calculation, obtain the probability distribution of the output layer;

[0137] S43. When the probability distribution of the output layer is greater than the threshold, it is determined that the chip has a fault; otherwise, it is determined that the chip is normal.

[0138] S5. Perform targeted repair through the self-repair redundant circuit according to the specific location where the fault occurs;

[0139] The process of targeted repair by the self-repair redundant circuit is as follows:

[0140] S51. Determine the specific location of the fault in the chip according to the possible fault occurrence locations and encode it to obtain the encoded fault location information;

[0141] S52. After receiving the fault location information, query the redundant circuit resource mapping table to find the redundant circuit resources that match the fault location. The redundant circuit resources include the redundant storage unit address and the switching control information;

[0142] S53. Control the switching circuit to switch from connecting the faulty circuit to the state of preparing to connect the redundant circuit, and complete the pre-connection of the circuit and the preliminary configuration of the signal path;

[0143] S54. Send an isolation signal to the fault area isolation circuit (such as a fuse, an electronic switch) to cut off the electrical connection of the faulty circuit; then trigger the switching circuit to connect the redundant circuit to the chip working circuit so that it undertakes the original function of the faulty circuit;

[0144] S55. Perform a quick test on the repaired functional area to check whether the redundant circuit function is normal; if the redundant function is normal, record the repair success information and notify the system to resume normal operation, and enter S6; if the redundant function is not normal, restart the fault location process to troubleshoot problems and return to S51.

[0145] Through the self-repair redundant circuit, it can be ensured that when the chip fails, the redundant circuit can be used for repair quickly and accurately to maintain the stable operation of the in-vehicle camera chip.

[0146] S6. Conduct a comprehensive test on the repaired chip to evaluate whether the performance of the repaired chip has returned to the normal level. If the performance of the repaired chip has not returned to the normal level, it is necessary to re-determine the fault location and return to S5;

[0147] If the performance of the repaired chip has returned to the normal level, continue to monitor the in-vehicle camera chip and repeat S1 - S4.

[0148] By simultaneously collecting the chip data of different in-vehicle camera chips on a multi-core architecture, and then using a convolutional neural network to extract image features from the chip data and a fully connected neural network to extract numerical features from the chip data, the convolutional neural network can learn the complex spatial structures and patterns in the image, thus more comprehensively describing the features of the image. The fully connected neural network can comprehensively consider the factors affecting the image quality and more comprehensively diagnose the quality of the image signal. By inputting the image features and numerical features into a pre-trained classification neural model, it is possible to accurately determine whether there is a fault in the chip, and further more comprehensively diagnose the quality of the image signal. Using the parallel computing of the multi-core architecture to determine the specific location where the fault occurs can more accurately understand the specific impact of the fault on the image quality. Through the self-repair redundant circuit, targeted repair is carried out on the specific location where the fault occurs, and a comprehensive test is performed on the repaired chip to evaluate whether the performance of the repaired chip has returned to the normal level. Continuing to monitor the in-vehicle camera chip and continuously using the neural network to learn and analyze the features in the image signal can timely detect new image quality problems or potential faults, thereby achieving a comprehensive and continuous diagnosis of the image signal quality and improving the accuracy of chip fault diagnosis.

[0149] Based on the above-described neural network fault diagnosis method for in-vehicle camera chips, the present invention also provides a self-repair redundant circuit for neural network fault diagnosis of in-vehicle camera chips, as Figure 5 shown, which includes a redundant circuit unit, a fault detection module, a switching control module, and a management module. When a corresponding module in the main circuit fails, the redundant circuit quickly replaces it to work, ensuring the normal operation of the overall function of the chip; the redundant circuit corresponds to the functional modules in the main circuit that are prone to failure and is a standby circuit with the same or similar functions. After the fault detection module detects the main circuit fault and is confirmed by the management module, the switching control module acts to connect the redundant circuit unit to the working circuit, and the redundant circuit unit starts to execute the corresponding function. For example, when the main storage unit fails, the redundant storage unit is connected, and the storage and reading data operations are transferred to the redundant unit.

[0150] The fault detection module continuously monitors the operating state of the main circuit, timely discovers faults and issues fault signals; the fault detection module is composed of various sensors, monitoring circuits, and comparators. The sensors continuously collect the operating parameters of each part of the chip, the monitoring circuits convert the original data into a format convenient for comparison, and the comparators compare it with the normal threshold. When the parameters exceed the normal range, the fault detection module determines that a fault has occurred, generates a fault signal and sends it to the management module; for example, when it is detected that the output voltage of a certain logic unit is abnormal, the fault detection module will issue a fault signal;

[0151] According to the instructions of the management module, the switching control module isolates the main circuit from the faulty part when a fault occurs and connects the redundant circuit to the working circuit; the switching control module mainly includes a multiplexer (MUX), an electronic switch, and a control logic circuit. The multiplexer is responsible for selecting the data path, the electronic switch is used to cut off or connect the circuit, and the control logic circuit receives the instructions of the management module and controls the actions of the multiplexer and the electronic switch. After receiving the switching instruction sent by the management module, the control logic circuit controls the electronic switch to cut off the connection between the faulty circuit and other parts, and at the same time controls the multiplexer to switch the data path and connect the redundant circuit to achieve circuit switching. For example, when a fault occurs in the data transmission line, the multiplexer switches the data transmission path to the redundant line, and the electronic switch disconnects the connection of the faulty line.

[0152] The management module receives the fault signal from the fault detection module, queries the redundant circuit resource mapping table to determine the redundant circuit resources, and sends a switching instruction to the switching control module; the management module includes a storage unit for storing the redundant circuit resource mapping table and a processor for storing the information of the fault record, as well as a communication interface for communicating with the fault detection module and the switching control module; after receiving the fault signal from the fault detection module, the management module queries the redundant circuit resource mapping table, finds the corresponding redundant circuit information, and then sends an instruction including the position of the redundant circuit and the switching operation to the switching control module. After the repair is completed, the information such as the fault occurrence time, position, and repair method is recorded; if the management module receives the fault signal of the image signal processing module, it queries the mapping table to determine the position of the redundant circuit, sends an instruction to the switching control module, and records the fault and repair details after the redundant circuit switching is completed.

[0153] Through the cooperation of the redundant circuit unit, the fault detection module, the switching control module, and the management module, the automatic repair of the redundant circuit realizes the rapid processing of the on-vehicle camera chip fault.

[0154] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A neural network fault diagnosis method for in-vehicle camera chips, characterized in that: The steps include: S1. Simultaneously collect chip data from different vehicle-mounted camera chips on a multi-core architecture. The chip data includes power parameters, image signal data, and image quality-related data. S2, using convolutional neural networks to extract image features from chip data; S3, using a fully connected neural network to extract numerical features from chip data; S4. Input the image features and numerical features into a pre-trained classification neural network, classify the chip status according to the learned feature patterns, and determine whether the chip is faulty; If there is a chip fault, the parallel computing of the multi-core architecture is used to determine the specific location of the fault; Enter S5; If the chip is not faulty, continue to monitor the vehicle camera chip and repeat S1-S4; S5. Perform targeted repairs based on the specific location of the fault using the self-repair redundant circuit; S6. Perform a comprehensive test on the repaired chip to evaluate whether the performance of the chip has returned to normal. If the performance of the chip has not returned to normal, redetermine the fault location and return to S5. If the chip performance returns to normal after repair, continue to monitor the vehicle camera chip and repeat S1-S4.

2. The on-vehicle camera chip neural network fault diagnosis method according to claim 1, wherein: The multi-core architecture includes processor units with multiple independent cores, multi-level cache units, bus connection units and memory controllers; the processor units are integrated with the MESI protocol for ensuring data consistency, and the processor units have an arithmetic logic unit, a register group and a cache unit.

3. The on-vehicle camera chip neural network fault diagnosis method according to claim 2, wherein: The chip data collection process is as follows: A1. Initialize the operating environment of each core of the processor unit and the hardware interface with the vehicle camera chip; A2. Develop a task allocation strategy based on the number of cores and chips in the multi-core architecture, and clearly define the chip number and data type that each core is responsible for collecting. A3. The core uses the power monitoring circuit to collect chip power parameter data according to the task allocation strategy, collects image data through the image signal transmission interface, and reads image quality-related data through the memory controller; A4. The power parameter data, image data, and image quality related data are aggregated and stored in the shared memory area.

4. The on-vehicle camera chip neural network fault diagnosis method according to claim 1, wherein: The process of image feature extraction using the convolutional neural network is as follows: B1. Convolve the input image with the convolution kernel to generate a feature map; B2. Use activation function to introduce nonlinear features into convolutional neural network on the generated feature map; B3. Perform a pooling operation on the feature map processed by the activation function to obtain the downsampled feature map.

5. The on-vehicle camera chip neural network fault diagnosis method according to claim 1, characterized in that: The process of numerical feature extraction is as follows: C1. Arrange the collected numerical data about the vehicle camera chip and the statistical parameters of the image signal to form a numerical input vector; C2. Perform matrix multiplication on the numerical input vector and the weight matrix of the first fully connected layer, and then add the bias vector of the first fully connected layer to obtain the weighted sum result. C3. Input the weighted sum result into the activation function for activation operation to obtain the output of the first fully connected layer; C4. Use the output of the previous fully connected layer as the input of the current layer, perform matrix multiplication with the weight matrix of the current layer, add the bias vector of the current layer to obtain the weighted sum result of the current layer, and then perform an activation function operation on the weighted sum result of the current layer to obtain the output of the current layer; C5. Repeat C4, process multiple hidden layers in sequence, and obtain the output of the last hidden layer; C6. Perform matrix multiplication on the output of the last hidden layer and the weight matrix of the output layer, add the bias vector of the output layer to obtain the output of the output layer, and then process the result of the output layer through an activation function to obtain the finally extracted numerical features.

6. The on-vehicle camera chip neural network fault diagnosis method according to claim 5, wherein: The classification neural network is set as a multi-layer perceptron structure. The classification neural network consists of a classification input layer, multiple classification hidden layers, and a classification output layer. The classification input layer receives the feature vector formed by splicing the image features and numerical features; each classification hidden layer contains several neurons, and the classification hidden layer performs a non-linear transformation on the input features to learn complex patterns in the data; the classification output layer uses the Softmax function to convert the output into the probability distributions of two categories: fault and normal.

7. The on-vehicle camera chip neural network fault diagnosis method according to claim 6, wherein: The process of pre-training the classification neural network is as follows: D1. Collect a large amount of data of in-vehicle camera chips in normal and fault states, extract image features and numerical features at the same time, and form training feature vectors from the image features and numerical features; D2. Label each training feature vector with the corresponding category label; D3. Divide the training feature vectors into a training set, a validation set, and a test set; D4. Determine the bias vector of the classification input layer according to the weight matrix of the classification input layer of the classification neural network. The bias vector of the classification input layer is the number of neurons in the first classification hidden layer * the number of neurons in the classification input layer; D5. Input the training feature vectors in the training set into the classification neural network, and perform weighted sum calculation and activation output calculation on each classification hidden layer in sequence; D6. The classification output layer performs weighted sum according to the activation output of the last classification hidden layer, then performs activation output using the activation function ReLU according to the weighted sum result of the classification output layer, and finally converts the activation output value into a probability output using the Softmax function; among them, The calculation formula for the weighted sum of the classification output layer is: z o = W o y o-1 + b o , where z o is the weighted sum result of the classification output layer, W o is the weight matrix of the classification output layer, y o-1 is the output result of the last classification hidden layer, and b o is the bias vector of the classification output layer; The calculation formula for the activation output calculation of the classification output layer is: y o = σ o (z o ), where y o is the output after the classification output layer is processed by the activation function, σ o is the activation function ReLU used by the classification output layer, and z o is the weighted sum result of the classification output layer; The conversion formula of the Softmax function is: Among them, P(c|x) is the probability that the sample x belongs to the category c, and z c is the output of the c-th neuron in the classification output layer, C is the total number of categories, and z k is the output of the k-th category in the classification output layer, where k is the index variable of the category, is the exponential operation with base e on the output of the k-th category in the classification output layer, and e is the base of the exponential function; D7. Use the cross-entropy loss function to calculate the loss value of the classification neural network after training; D8. Calculate the partial derivatives of the weight matrix and bias vector of each layer in the classification neural network according to the loss value, and calculate the gradient of the weight matrix and the gradient of the bias vector; D9. Use the optimization algorithm to update the weight matrix and bias vector of each layer; D10. Repeat D5 - D9 until the loss function converges to the minimum value.

8. The on-vehicle camera chip neural network fault diagnosis method according to claim 7, characterized in that: When determining the specific location where a fault occurs, first establish a fault feature library that contains feature patterns corresponding to different fault types. Then, use cosine similarity to calculate the similarity between the current fault feature vector and the feature patterns in the fault feature library, and find the index position corresponding to the maximum similarity. The position associated with the fault pattern corresponding to this index position is the possible fault occurrence location.

9. The on-vehicle camera chip neural network fault diagnosis method according to claim 8, wherein: The process of targeted repair by the self-repairing redundant circuit is as follows: S51. Determine the specific location of the fault in the chip based on the possible fault occurrence location and encode it to obtain the encoded fault location information. S52. After receiving the fault location information, query the redundant circuit resource mapping table to find the redundant circuit resource that matches the fault location. The redundant circuit resource includes the redundant storage unit address and the switching control information. S53. Control the switching circuit to switch from connecting to the faulty circuit to the state of preparing to connect to the redundant circuit, and complete the pre-connection of the circuit and the preliminary configuration of the signal path. S54. Send an isolation signal to the fault area isolation circuit to cut off the electrical connection of the faulty circuit; then trigger the switching circuit to connect the redundant circuit to the chip working circuit so that it undertakes the original function of the faulty circuit. S55. Perform a quick test on the repaired functional area to check whether the redundant circuit functions normally; if the redundant function is normal, record the repair success information and notify the system to resume normal operation, and enter S6; if the redundant function is abnormal, restart the fault location process to troubleshoot the problem and return to S51.

10. Self-repairing redundant circuit for neural network fault diagnosis of in-vehicle camera chips, based on the neural network fault diagnosis method for in-vehicle camera chips described in any one of claims 1-9, characterized in that: Including: A redundant circuit unit, which quickly replaces the corresponding module of the main circuit when a fault occurs in the main circuit to ensure the normal operation of the overall function of the chip. A fault detection module, which continuously monitors the operating state of the main circuit, promptly detects faults and issues fault signals. A switching control module, which isolates the main circuit from the faulty part according to the instruction of the management module when a fault occurs and connects the redundant circuit to the working circuit. A management module, which receives the fault signal from the fault detection module, queries the redundant circuit resource mapping table to determine the redundant circuit resource, and sends a switching instruction to the switching control module.

Images