Vehicle software anomaly detection method and system based on visual matching

Abstract

The invention relates to the technical field of video image recognition, and discloses a visual matching-based vehicle-mounted terminal software anomaly detection method and system, and the method comprises the steps: obtaining a real-time monitoring video and all frame images of a vehicle-mounted terminal screen corresponding to vehicle-mounted terminal software; scanning all adjacent frames of the video, and sequentially determining whether image differences exist between all current frames and the adjacent frames so as to judge the possibility of abnormal display of each current frame image; when it is judged that the current frame image is possibly abnormal in display, fault calculation and fault image matching are carried out on the current frame image at the same time to obtain a fault mode inspection result; and determining whether the current frame image is actually abnormal in display and the fault mode according to the fault mode inspection result and the fault judgment condition. According to the method, on the basis of the adjacent frame images, the two-section verification difference possibility and authenticity, the fault matching mode that fault calculation and fault image matching are carried out in the verification process is adopted, and efficient, low-cost and highly-customized vehicle-mounted terminal software anomaly detection is achieved through the visual performance of a vehicle-mounted terminal screen.

Description

Technical Field

[0001] The present invention relates to the technical field of video image recognition, and particularly to a method and system for abnormal detection of in-vehicle software based on visual matching. Background Art

[0002] Today, with the rapid development of intelligent connected vehicle technology, the in-vehicle screen has become an important bridge for communication between the vehicle and the user, carrying multiple functions such as navigation, entertainment, vehicle status monitoring, and information display and control of intelligent driving assistance systems. The integration of these functions not only greatly enriches the driving experience but also poses unprecedented challenges to the stability and reliability of the in-vehicle display system.

[0003] However, in the prior art, visual anomaly monitoring for in-vehicle screens and in-vehicle automation software faces many problems that need to be solved urgently.

[0004] First of all, manual visual inspection, as a traditional monitoring method, its limitations are becoming increasingly prominent. In intelligent connected vehicles, the amount of information on the in-vehicle screen is huge and the update speed is extremely fast. Manual visual inspection not only has difficulty covering all possible abnormal situations but also is prone to missing key information due to visual fatigue or distraction. This inefficient monitoring method not only fails to ensure driving safety but may also cause greater losses due to delays in fault discovery.

[0005] Secondly, although existing visual monitoring systems achieve automated monitoring to a certain extent, they are expensive and lack customization capabilities. These systems often adopt a general technical architecture and are difficult to meet the customization requirements of different vehicle models, different display screen specifications, and specific software functions. In practical applications, such inflexible monitoring systems often cannot accurately identify and warn of potential visual anomalies, and may even cause greater problems due to false alarms or missed alarms. At the same time, the high cost also makes these systems difficult to be widely used in the field of intelligent connected vehicles.

[0006] In addition, the complexity and integration of in-vehicle automation software are increasing, making software failures and hardware damages one of the main causes of visual anomalies. These failures may be caused by various factors such as software programming errors, memory leaks, hardware component aging, or external environmental factor interference. Due to the intertwining of these factors, the work of fault troubleshooting and repair becomes extremely complex and time-consuming. In intelligent connected vehicles, any small visual anomaly may have a serious impact on driving safety and driving efficiency. Therefore, it is particularly important to efficiently and accurately monitor visual anomalies of in-vehicle screens and in-vehicle automation software.

[0007] In summary, in the prior art, there are problems such as low efficiency, high cost, insufficient customization ability, and complex fault diagnosis and repair in the visual anomaly monitoring of in-vehicle infotainment (IVI) screens and automation software. These problems not only affect driving safety and driving efficiency but also increase the complexity and cost of vehicle maintenance. Summary of the Invention

[0008] The present invention aims to provide a method and system for detecting anomalies in IVI software based on visual matching, which can achieve efficient, low-cost, and highly customizable anomaly detection of IVI software based on the visual performance of the IVI screen, ensure the accuracy and reliability of the detection results, and is of great significance for improving the safety and reliability of intelligent connected vehicles and reducing vehicle maintenance costs.

[0009] The basic solution provided by the present invention is: a system for detecting anomalies in IVI software based on visual matching, the system includes:

[0010] A video processing module for acquiring a real-time monitoring video of the IVI software corresponding to the IVI screen and all frame images of the video;

[0011] An adjacent frame difference detection module for scanning all adjacent frames of the acquired video and sequentially determining whether there are image differences between all current frames and their adjacent frames to judge the possibility of display anomalies in each current frame image;

[0012] A fault matching module for performing fault calculation and fault image matching on the current frame image simultaneously when it is judged that the current frame image may have a display anomaly to obtain a fault mode inspection result;

[0013] A fault diagnosis module for determining whether the current frame image actually has a display anomaly and its corresponding fault mode according to the fault mode inspection result and the fault judgment condition.

[0014] The present invention also provides a method for detecting anomalies in IVI software based on visual matching, which is applied to the above system for detecting anomalies in IVI software based on visual matching. The method includes:

[0015] S100, acquiring a real-time monitoring video of the IVI software corresponding to the IVI screen and all frame images of the video;

[0016] S200, scanning all adjacent frames of the acquired video, and judging whether there are image differences between all current frames and their adjacent frames to judge the possibility of display anomalies in each current frame image;

[0017] S300, when it is judged that the current frame image may have a display anomaly, performing fault calculation and fault image matching on the current frame image simultaneously to obtain a fault mode inspection result;

[0018] The S400 determines whether there is a real display anomaly and its fault mode for the current frame image according to the fault mode inspection result and the fault judgment condition.

[0019] The working principle and advantages of the present invention are as follows:

[0020] Currently, with the improvement of the intelligent level of in-vehicle systems, the causes of visual anomalies in in-vehicle head unit screens are very complex. These anomalies may stem from various factors such as software programming errors, memory leaks, aging of hardware components, or interference from external environmental factors. Moreover, these factors are intertwined, making it extremely complicated to directly troubleshoot and repair faults from the visual anomalies of in-vehicle head unit screens. To handle faults quickly and efficiently, in conventional fault detection, only a judgment on whether there is a fault is made and a general solution is adopted to change the operation strategy to ensure the normal operation of the in-vehicle system after the fault is identified. For the judgment of the specific cause type of the fault, conventionally, it is further confirmed by subsequent technicians and additional detection system equipment. Due to the increasing complexity and integration of in-vehicle head unit software, compared with the update of automotive technology, the update speed of in-vehicle head unit software is even more rapid. Even if the further confirmation is completed through the above process, there are not only problems of delayed judgment but also problems of incorrect fault diagnosis caused by delayed judgment and software updates.

[0021] In response to the above existing defects, through analysis, it is found that if starting from the in-vehicle display itself, such as developing a software for detecting software anomalies and installing it into the in-vehicle display system for detecting software anomalies in the in-vehicle head unit, it usually involves issues such as user privacy, system compatibility, and system permissions. This will inevitably make the software-based detection method lack generality (it is necessary to obtain system permissions from different vehicle manufacturers, and different systems have different compatibilities). If starting from the fault phenomenon of the in-vehicle display itself, the display anomaly of the in-vehicle screen has the most direct impact on the user experience and can be used for fault anomaly judgment. Although there are various reasons for the anomaly found in the in-vehicle screen, through in-depth analysis, it is found that the main reason for the current anomaly phenomenon of the in-vehicle screen (including the black screen, white screen, red screen, flower screen, and freezing mentioned in this article) is caused by in-vehicle head unit software faults. And there are certain performance differences between the video display anomaly of the in-vehicle screen caused by in-vehicle head unit software anomalies and the video display anomaly of the in-vehicle screen caused by other reasons. The differences include the manifestation form of the display anomaly and the time limit of the difference. Therefore, it is feasible to detect in-vehicle head unit software anomalies through the visual condition of the in-vehicle screen.

[0022] Based on this, this solution deeply analyzes the differences and characteristics of various fault manifestations of in-vehicle infotainment (IVI) screen videos for characterizing IVI software anomalies, and proposes a fault matching method based on adjacent frame images, the possibility and authenticity of two-stage verification differences, and fault calculation and fault image matching for different faults during the verification process, to achieve efficient, low-cost, and highly customized IVI software anomaly detection through the visual performance of the IVI screen.

[0023] Compared with the prior art, this solution has the following advantages:

[0024] 1. Through the difference detection of adjacent frame images before and after the video stream and two-stage verification of fault matching, this solution first judges the possibility of differences in the IVI screen display caused by IVI software anomalies, and then further verifies the authenticity of the possible differences, ensuring the screening and judgment of the differences characterizing IVI software anomalies based on the IVI screen video, and two-stage verification to ensure the accuracy of the detection results;

[0025] 2. The fault diagnosis process of adjacent frame image difference detection and fault matching in this solution does not require training and has a small sample demand. The completeness of the detection method can be ensured with less time cost and sample size, overcoming the problems that existing deep learning methods require a large number of samples and the deep learning models of big data have incomplete detection results due to their black box characteristics and sample incompleteness;

[0026] 3. Due to the particularity of IVI display failures caused by IVI software anomalies, different verification methods are proposed based on different fault modes (such as four types of faults: black screen, white screen, red screen, and mosaic screen), that is, verification is carried out through fault calculation and fault image matching. Specifically, a method of calculating each pixel by RGB is proposed to distinguish 3 types of faults: black screen, white screen, and red screen, and whether a mosaic screen fault occurs is judged by comparing the image where the fault may occur with the pre-collected different IVI screen mosaic image templates Figure 1 one by one, while completing the authenticity judgment of the current frame image difference and the specific determination of the fault mode. The anomaly diagnosis result is richer and more valuable for reference, and is not limited by vehicle models and IVI software system types, with high generality. Brief Description of the Drawings

[0027] Figure 1 It is a schematic structural diagram of an IVI software anomaly detection system based on visual matching provided by an embodiment of the present invention;

[0028] Figure 2 It is a flowchart of an IVI software anomaly detection method based on visual matching provided by an embodiment of the present invention. Detailed Description of the Invention

[0029] The following is a further detailed description through specific embodiments:

[0030] When deeply analyzing the performance differences of in-vehicle infotainment (IVI) screen videos for various fault causes indicating IVI software anomalies, it is found that although both are abnormal displays of IVI screen videos, there are differences between the abnormal display of IVI screen videos caused by IVI software anomalies and those caused by other reasons.

[0031] For example, when certain faults occur in the system, causing a black rectangle to appear in a certain area of the in-vehicle display (black screen fault), there are obvious differences in the screen display between the frames before and after the fault (no black rectangle before the fault, and there is a black rectangle after the fault). Then, calculate the pixel value differences between these two frames. If there are significant differences, it can be determined that the screen may have a fault, and proceed to the next step of fault matching. Since both fault calculation and fault image matching are proposed for IVI software anomalies, in this case, effective detection of IVI software anomalies can be achieved.

[0032] Another situation is when the user actively clicks on the screen and the screen enters the next normal interface (such as from the home screen to the navigation screen). The adjacent frame difference detection module will also detect the pixel value differences between the two frames for this change. However, in the next step of the fault matching module, since this fault manifestation is not a manifestation of IVI software anomalies, it will not match to the corresponding fault template image, and thus no fault will be detected.

[0033] This is also the necessity of this solution to propose adjacent frame images and two-stage verification, which can achieve efficient, low-cost, and highly customized detection of IVI software anomalies based on the visual performance of the IVI screen.

[0034] The embodiment is basically as shown in the appendix Figure 1 An IVI software anomaly detection system based on visual matching, the system includes:

[0035] A video processing module, configured to obtain the real-time monitoring video of the IVI screen corresponding to the IVI software, and obtain all frame images of the video;

[0036] Specifically, the video processing module includes a video acquisition module and a frame image noise reduction and enhancement module.

[0037] The video acquisition module is configured to obtain the real-time monitoring video of the IVI screen corresponding to the IVI software and all frame images of the video.

[0038] The frame image noise reduction and enhancement module decomposes the real-time monitoring video image into wavelet components of different scales and frequencies, suppresses or eliminates the wavelet coefficients corresponding to the noise according to the different manifestations of noise and signals on the wavelet coefficients, and finally reconstructs the denoised video image through inverse wavelet transform.

[0039] Specifically, the real-time monitoring frame image of the in-vehicle screen is decomposed into wavelet coefficients at different scales, that is, multi-level wavelet transform is performed. Each level generates a set of high-frequency and low-frequency coefficients. In the decomposed wavelet coefficients, outliers in the high-frequency or low-frequency components are identified through the outlier filtering threshold γ, and the noise coefficients are set to 0 or their amplitudes are reduced. After wavelet transform and noise reduction, the real-time monitoring frame image Img(t) of the in-vehicle screen is as follows:

[0040]

[0041] where t is time, ψ(t) is the wavelet function, a is the scale factor, and b is the translation factor.

[0042] Through the above processing, the clarity and quality of the frame image are significantly improved, making the image details sharper, the background noise greatly reduced, and the visual effect and analysis accuracy enhanced.

[0043] The adjacent frame difference detection module is used to scan all adjacent frames of the acquired video, and sequentially determine whether there are image differences between all current frames and their adjacent frames, so as to judge the possibility of display abnormality of each current frame image.

[0044] Specifically, the adjacent frame difference detection module includes a video sliding window module, an image difference detection module, and an abnormality judgment module.

[0045] The video sliding window module is used to sequentially obtain all adjacent frames in the video, and input the two adjacent (front and back) frame images obtained each time into the image difference detection module; a video is actually a time series of images. Generally, it is processed serially, but images for comparison can be selected at intervals to improve efficiency. For example, if the extraction interval is 2, then frames 1 and 3 are compared, frames 3 and 5 are compared. Generally, the middle frame in between does not affect the overall effect.

[0046] The image difference detection module is used to convert the two adjacent frame images obtained into grayscale images, divide each grayscale image into n*n blocks, calculate the average value of the grayscale values of each block area, compare the corresponding average values of the corresponding block areas of the two grayscale images, and comprehensively compare the results of all areas to determine whether there are image differences between the current frame and its adjacent frame;

[0047] The abnormality judgment module is used to judge the possibility of display abnormality of each current frame image according to whether there are image differences between the current frame and its adjacent frame. If there are image differences, it is judged that the current frame image may be abnormal. If there are no image differences, it is judged that the current frame image is normal.

[0048] Specifically, different weights are assigned to the red (R), green (G), and blue (B) of each pixel point in the i-th and (i + 1)-th frame images obtained by the sliding window, and then weighted average is performed to obtain a grayscale image:

[0049] Gray = 0.299 * R + 0.587 * G + 0.114 * B

[0050] The current frame image is divided into n×n blocks, and the block images are traversed by moving from the upper left corner to the right and downwards. For each block, its height block_height and width block_width are respectively:

[0051]

[0052] where H is the height of the original frame image, W is the width of the original frame image, and n is the number of single-dimensional segmentation blocks; for each block, the image region between its upper left corner coordinates (x, y) and its lower right corner coordinates (x + block_weight, y + block_height) is extracted.

[0053] The pixel average value of an image block with size M×N and pixel value P(i, j) is:

[0054]

[0055] where i and j are the indexes of the row and column respectively, and the difference Difference between the u-th block image in the i-th frame image and the image at the corresponding position in the (i + 1)-th frame image is obtained from this. u is:

[0056]

[0057] where, is the pixel average value of the u-th block in the i-th row, is the pixel average value of the u-th block in the j-th column, and Max is the function to take the maximum value.

[0058] Set the block difference threshold to ξ; if the calculated difference Difference of the current block u > ξ, then it is judged that the current block detects a difference and the current frame image has an abnormality; if Difference u < ξ, then continue to accumulate the image difference Sum_Diff of m blocks as:

[0059]

[0060] where m is the total number of blocks; set the accumulated block difference threshold to δ; if Sum_Diff > δ, then it is judged that the current frame image has an abnormality; if Sum_Diff < δ, then it is judged that the current frame image has no abnormality.

[0061] The adjacent frame difference detection module mainly calculates the image difference based on pixel differences to achieve anomaly detection. Compared with convolutional operations, it has a lower time complexity and higher detection efficiency.

[0062] A fault matching module, which is used to perform fault calculation and fault image matching on the current frame image simultaneously when it is determined that the current frame image may have a display anomaly, and obtain the fault mode inspection result;

[0063] A fault diagnosis module, which is used to determine whether the current frame image actually has a display anomaly and its corresponding fault mode according to the fault mode inspection result and the fault diagnosis conditions.

[0064] Specifically, the fault matching module includes a black / white / red screen fault diagnosis module and a color distortion fault diagnosis module.

[0065] The black / white / red screen fault diagnosis module is used to detect black screen, white screen, and red screen on the frame image that may have an anomaly based on fault calculation.

[0066] Fault calculation is to generate at least three candidate boxes of different sizes on the current frame image. For example, generate 3 candidate boxes Anchor S 、Anchor M 、Anchor L Scan on the image. For the n p pixels (R i , G i , B i ) in each candidate box, scan them, and perform corresponding fault mode inspection according to the preset corresponding fault judgment conditions, where Ri, Gi, and Bi are the intensities of the red, green, and blue components of the i-th pixel respectively.

[0067] If |R i | < ε black , |G i | < ε black , |B i | < ε black , ε black →0, ε black is the intensity threshold of the color component for the black screen fault, then accumulate the number of pixels n black . If within this candidate box:

[0068]

[0069] Among them, σ black is the black screen fault threshold, n p is the total number of pixels, and n black is the total number of pixels that meet the black screen fault judgment conditions;

[0070] Meet the black screen fault judgment condition σblack ∈(0,1), it is determined that the current frame image has a display anomaly and is a black screen fault.

[0071] If |R i -255| < ε white , |G i -255| < ε white , |B i -255| < ε white , ε white →0, ε white is the intensity threshold of the color component for the white screen fault, then the cumulative number of pixels n white ; If within the candidate box:

[0072]

[0073] where σ white is the white screen fault threshold, n p is the total number of pixels, n white is the total number of pixels that meet the white screen fault judgment condition;

[0074] Satisfying the white screen fault judgment condition σ white ∈(0,1), it is determined that the current frame image has a display anomaly and is a white screen fault.

[0075] If it satisfies

[0076]

[0077] ε red is the intensity threshold of the color component for the red screen fault, then the cumulative number of pixels n red ; If within the candidate box:

[0078]

[0079] where σ red is the red screen fault threshold, n p is the total number of pixels, n red is the total number of pixels that meet the red screen fault judgment condition;

[0080] Satisfying the red screen fault judgment condition σ red ∈(0,1), it is determined that the current frame image has a display anomaly and is a red screen fault.

[0081] The mosaic fault diagnosis module is used to perform mosaic monitoring on the frame images that may have anomalies based on fault image matching;

[0082] Among them, the fault image matching is to match the frame images that may have anomalies with the template Figure 1 one by one to query whether there is a template image that matches the current frame image.

[0083] Specifically, the manifestation characteristics of the flower screen make it impossible to determine it like the black-and-white, red screen through a single pixel RGB range. Instead, more detailed feature extraction methods are required for detection (such as Gaussian transformation), that is, the flower screen fault diagnosis module requires higher matching accuracy, including:

[0084] The Gaussian function with variable scale is:

[0085]

[0086] where m and n are the dimensions of the Gaussian template, x and y are the position coordinates of the image pixels, and σ is the scale space factor.

[0087] Then the scale space of the in-vehicle screen frame image Img’(x, y) is:

[0088] Scale(x,y,σ)=G(x,y,σ)*Img’(x,y)

[0089] where * represents the convolution operation.

[0090] The scale space coordinates of the sub-level layer are:

[0091]

[0092] where σ0 is the initial scale, s is the sub-level layer coordinate, and S is the number of layers in each group.

[0093] Use Gaussian difference kernels with different scales and convolution to establish a DOG Gaussian difference pyramid to extract key points:

[0094] D(x,y,σ)=[G(x,y,kσ)-G(x,y,σ)]*Img’(x,y)=Scale(x,y,kσ)-Scale(x,y,σ)

[0095] Compare each intermediate detection point with its 8 adjacent points of the same scale, as well as 18 points corresponding to the adjacent scales above and below, to obtain the extreme points in the scale space and the two-dimensional image space. The Taylor expansion of the DOG function in the scale space is:

[0096]

[0097] Interpolate the pixels around the candidate key points to improve the accuracy. Remove the key points with low contrast and excessive edge response through contrast and edge response tests, calculate the gradient direction histogram within the neighborhood of the key points, and find the dominant direction.

[0098] On the main direction of the key point, divide the neighborhood around the key point into several sub-regions, calculate the histogram of gradient directions for each sub-region, and concatenate all the histograms to form the final N-dimensional descriptor. Thus, we can obtain:

[0099] The descriptor of the key point in the screen pattern template image is:

[0100] F i =(f i1 , f i2 , ···, f iN )

[0101] The descriptor of the key point in the in-vehicle screen frame image Img’ is:

[0102] I i =(i i1 , i i2 , ···, i iN )

[0103] The similarity measure between the two descriptors is:

[0104]

[0105] Among them, d(F i , I i ) is the similarity measure between the i-th rows of the two descriptors F i and I i , f ij is the descriptor of the i-th row and j-th column of the key point in the template image, and i ij is the descriptor of the i-th row and j-th column of the key point in the current frame image; N is the descriptor dimension;

[0106] Suppose the minimum value and the second minimum value of d(F i , I i ) are d1 and d2. When the screen pattern fault judgment condition σ glitch is the set threshold, it is considered that the key points are matched, the fault image matching is successful, and it is judged that the current frame image has a display abnormality and is a screen pattern fault.

[0107] For the above thresholds such as the block difference threshold ξ, the cumulative block difference threshold δ, the black screen fault threshold σ black , the white screen fault threshold σ white , the red screen fault threshold σ red , and the screen pattern fault threshold σ glitch , generally, the optimal values need to be selected according to the actual experimental situation.

[0108] The system further includes a sample library module, which is used to add, delete, or store all possible template images after switching the screen, including the loading interface that may appear during the process of entering the software and the interface images after entering each software.

[0109] The sample library module includes a template image storage module and a template image addition / removal module; the template image storage module stores the screen flashing images of different types of in-vehicle device screens; the template image addition / removal module is used to add or delete template images in the template image storage module. Abnormal images of the in-vehicle device screen caused by abnormal in-vehicle device software can be crawled through a crawler, abnormal frames during video recording can be stored, and pictures representing abnormal display of the in-vehicle device software can be obtained by actively injecting faults, so as to enrich the sample library and make the judgment of faults more accurate and thorough.

[0110] As Figure 2 shown, this solution also provides a method for detecting abnormal in-vehicle device software based on visual matching. Using the above system, the method includes:

[0111] S100, obtaining the real-time monitoring video of the in-vehicle device software corresponding to the in-vehicle device screen, and obtaining all frame images of the video;

[0112] S200, scanning all adjacent frames of the obtained video, and judging whether there are image differences between all current frames and their adjacent frames to judge the possibility of abnormal display of each current frame image;

[0113] S300, when it is judged that the current frame image may have abnormal display, performing fault calculation and fault image matching on the current frame image simultaneously to obtain the fault mode inspection result;

[0114] S400, determining whether the current frame image actually has abnormal display and its fault mode according to the fault mode inspection result.

[0115] It can be understood that this method is fully executed during the operation of the above system, and the specific process steps are the same as the implementation content of the above system, so they will not be repeated here.

[0116] An abnormal detection method and system for in-vehicle software based on visual matching provided by this embodiment deeply analyze the differences and performance characteristics of various fault manifestations of the in-vehicle screen video for characterizing the abnormality of the in-vehicle software, and propose a fault matching method based on adjacent frame images, the possibility and authenticity of two-stage verification differences, and the calculation of faults and the matching of fault images for different faults respectively during the verification process. The abnormal detection of in-vehicle software is realized efficiently, with low cost and high customization based on the visual performance of the in-vehicle screen. Ensure that the differences indicating the abnormality of the in-vehicle software are screened and judged based on the in-vehicle screen video, and two-stage verification ensures the accuracy of the detection results; no training is required, the sample demand is small, and the completeness of the detection method can be ensured with less time cost and sample volume; due to the particularity of the in-vehicle display failure caused by the abnormality of the in-vehicle software, different verification methods are proposed based on different fault modes, and at the same time, the authenticity judgment of the current frame image difference and the specific determination of the fault mode are completed. The abnormal diagnosis results are richer and more valuable for reference, and are not limited by vehicle models and software, with high generality.

[0117] Embodiment 2

[0118] Different from Embodiment 1, the system further includes a fault diagnosis re-verification module for performing re-diagnosis of faults when there are multiple fault modes.

[0119] That is, when there is only one fault mode in the final result determined by the fault diagnosis module, it is determined that the current frame image actually has a display abnormality. For example, if only a black screen fault is obtained, it is determined that the current frame image is a black screen fault, and the fault diagnosis re-verification module is not started.

[0120] When there is more than one fault mode in the final result determined by the fault diagnosis module, such as giving a black screen fault and a white screen fault at the same time, the fault diagnosis re-verification module is used to start the secondary verification and the tertiary verification. The secondary verification and the tertiary verification respectively repeat the corresponding functions of the adjacent frame difference detection module, the fault matching module and the fault diagnosis module. If there is only one fault mode and it is the same after the verification is executed, such as giving a black screen fault, it is determined that the current frame image actually has a display abnormality, which is a black screen fault; if there is still more than one fault mode, or the fault modes are different, such as the secondary verification gives a white screen and a flower screen, the tertiary verification gives a black screen and a white screen, or the secondary verification gives a white screen and the tertiary verification gives a black screen, it is determined that the abnormal detection system has a fault and system optimization is required.

[0121] A method and system for abnormal detection of in-vehicle software based on visual matching provided in this embodiment, after the initial diagnosis, if the fault judgment is abnormal, through fault re-diagnosis, the fault is verified again to ensure the accuracy of fault diagnosis. At the same time, secondary verification and tertiary verification are added to judge fault diagnosis errors and system faults, ensuring the accuracy of fault diagnosis and providing a fault detection path for the abnormal detection system of this solution.

[0122] The above are only embodiments of the present invention. Common general knowledge such as specific structures and characteristics known in the art are not described in detail here. Those of ordinary skill in the art know all the general technical knowledge in the technical field to which the invention belongs before the application date or the priority date, are able to obtain all the prior art in this field, and have the ability to apply the conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, combine their own abilities to complete and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can also be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent.

Claims

1. A vehicle-mounted software anomaly detection system based on visual matching, characterized in that The system includes: A video processing module, configured to obtain a real-time monitoring video of the in-vehicle software corresponding to the in-vehicle screen and all frame images of the video; An adjacent frame difference detection module, configured to scan all adjacent frames of the obtained video, and sequentially determine whether there is an image difference between all current frames and their adjacent frames, so as to judge the possibility of display abnormality for each current frame image; A fault matching module, configured to perform fault calculation and fault image matching on the current frame image simultaneously when it is judged that the current frame image may have a display abnormality, so as to obtain a fault mode inspection result; A fault diagnosis module, configured to determine whether the current frame image actually has a display abnormality and its corresponding fault mode according to the fault mode inspection result and the fault judgment condition.

2. The vehicle-mounted software anomaly detection system based on visual matching according to claim 1, wherein The video processing module includes a frame image noise reduction and enhancement module, configured to decompose each frame image into wavelet components of different scales and frequencies, suppress or eliminate the wavelet coefficients corresponding to noise according to the different performances of noise and signals on the wavelet coefficients, and finally reconstruct the denoised frame image through inverse wavelet transform.

3. The vehicle-mounted software anomaly detection system based on visual matching according to claim 1, characterized in that, The adjacent frame difference detection module includes an image difference detection module; The image difference detection module is configured to convert both the current frame and its adjacent frame into grayscale images, divide each grayscale image into n*n blocks, calculate the average value of the grayscale values of each block area, compare the corresponding average values of the corresponding block areas of the two grayscale images, and comprehensively compare the results of all areas to determine whether there is an image difference between the current frame and its adjacent frame.

4. The vehicle-mounted software anomaly detection system based on visual matching according to claim 3, characterized in that, Comparing the corresponding average values of the corresponding block areas of the two grayscale images and comprehensively comparing the results of all areas to determine whether there is an image difference between the current frame and its adjacent frame is as follows: Obtain the difference between the u-th block of the i-th frame image and the image at the corresponding position in the (i + 1)-th frame image u : Among them, is the average value of the pixels in the u-th block of the i-th row, is the average value of the pixels in the u-th block of the j-th column, and Max is the maximum value function; Set the block difference threshold to ξ; if the difference Difference calculated for the current block u > ξ, it is determined that a difference is detected in the current block and there is an abnormality in the current frame image; if Difference u < ξ, continue to accumulate the image differences Sum_Diff of m blocks as follows: Where m is the total number of blocks; a cumulative block difference threshold is set to δ; if Sum_Diff>δ, it is judged that the current frame image has an abnormality.

5. The vehicle-mounted software anomaly detection system based on visual matching according to claim 1, characterized in that, The fault matching module includes a black / white / red screen fault diagnosis module for monitoring black screen, white screen, and red screen for frame images that may have abnormalities based on fault calculation. The fault calculation is as follows: at least three candidate boxes of different sizes are generated on the current frame image, and n p pixels (R i , G i , B i ) of each candidate box image are scanned, and corresponding fault mode checks are performed according to preset corresponding fault judgment conditions, where R i , G i , and B i are the intensities of the red, green, and blue components of the i-th pixel, respectively.

6. The vehicle-mounted software anomaly detection system based on visual matching according to claim 5, characterized in that Performing corresponding fault mode inspection according to the preset corresponding fault judgment conditions includes: If |R i | < ε black , |G i | < ε black , |B i | < ε black , ε black → 0, ε black is the intensity threshold of the color component for the black screen failure, then the cumulative number of pixels n black , if within this candidate bounding box: Among them, σ black is the black screen fault threshold, n p is the total number of pixels, and n black is the total number of pixels that meet the black screen fault judgment condition; Meet the black screen fault judgment condition σ black ∈(0, 1), it is determined that the current frame image has a display anomaly and is a black screen fault.

7. The vehicle-mounted software anomaly detection system based on visual matching according to claim 5, wherein Performing corresponding fault mode inspection according to the preset corresponding fault judgment conditions includes: If |R i - 255| < ε white , |G i - 255| < ε white , |B i - 255| < ε white , ε white →0, ε white is the intensity threshold of the color component for the white screen fault, then the cumulative number of pixels n white , if within this candidate box: where σ white is the white screen fault threshold, n p is the total number of pixels, and n white is the total number of pixels that meet the white screen fault judgment condition; Meet the white screen fault judgment condition σ white ∈(0,1), then it is judged that the current frame image has a display abnormality and it is a white screen fault.

8. An in-vehicle software anomaly detection system based on visual matching according to claim 5, characterized in that, Performing corresponding fault mode inspection according to the preset corresponding fault judgment conditions includes: if it satisfies ε red is the intensity threshold of the color component for the red screen fault, then the cumulative number of pixels n red , if within this candidate bounding box: Among them, σ red is the red screen fault threshold, n p is the total number of pixels, and n red is the total number of pixels that meet the red screen fault judgment condition; Meet the red screen fault judgment condition σ red ∈(0, 1), then it is judged that the current frame image has abnormal display and is a red screen fault.

9. The anomaly detection system for in-vehicle software based on visual matching according to claim 1, wherein The fault matching module includes a screen freeze fault diagnosis module, configured to perform screen freeze monitoring on the frame image that may have an abnormality based on fault image matching; wherein, the fault image matching is that the similarity measure between the key point descriptors in the current frame image and the template image representing the screen freeze fault is: where d(F i , I i ) is the similarity measure between two descriptors F i and I i in the i-th row, f ij is the descriptor of the key point at the i-th row and j-th column in the template image, and i ij is the descriptor of the key point at the i-th row and j-th column in the current frame image; N is the dimension of the descriptor; Let the minimum value and the second minimum value of d(F i , I i ) be d1 and d2. If the condition for judging the screen freeze fault is satisfied, where ξ is a set threshold value, then it is considered that the key points match, the fault image matches successfully, and it is judged that the current frame image has a display abnormality and is a screen freeze fault.

10. A method for abnormal detection of in-vehicle software based on visual matching, characterized in that, Applied to an in-vehicle software abnormality detection system based on visual matching according to any one of claims 1-9, the method includes: S100, obtaining a real-time monitoring video of the in-vehicle software corresponding to the in-vehicle screen and all frame images of the video; S200, scanning all adjacent frames of the obtained video, and judging whether there is an image difference between all current frames and their adjacent frames, so as to judge the possibility of display abnormality for each current frame image; S300, when it is judged that the current frame image may have a display abnormality, performing fault calculation and fault image matching on the current frame image simultaneously to obtain a fault mode inspection result; S400, determining whether the current frame image actually has a display abnormality and its fault mode according to the fault mode inspection result and the fault judgment condition.

Images