Vehicle anomaly detection method and device, data processing method and device, equipment and storage medium
By extracting low-level, intermediate and advanced features of image blocks in vehicle anomaly detection, generating embedding vectors and calculating Gaussian distribution distances, the problem of insufficient abnormal samples is solved, efficient abnormal detection is achieved, and detection accuracy and robustness are improved.
Patent Information
- Application Number
- CN202411551650.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-08-12
AI Technical Summary
In the prior art, due to the small number of abnormal samples in vehicle abnormality detection, it is difficult to collect sufficient data, resulting in the training neural network model being low in accuracy and being unable to achieve accurate abnormality detection.
By acquiring the image to be detected by the vehicle, the low-level, intermediate and advanced features of the image block are extracted using the pre-trained model, the embedding vector is generated, and the Gaussian distribution distance of the image block is calculated based on the feature distribution of multiple normal images, to determine whether the image block is abnormal, and abnormal detection without abnormal samples is achieved.
It effectively realizes abnormal detection of vehicle image blocks when there are fewer abnormal samples, improves detection accuracy and robustness, reduces calculation complexity, and is suitable for fast response application scenarios.
Smart Images
Figure CN120472137A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of anomaly detection technology, and in particular to a vehicle anomaly detection method, data processing method, device, equipment and storage medium. Background Art
[0002] Trains, such as high-speed trains, motor trains, and trams, offer fast, safe, and convenient land transportation, significantly impacting people's mobility. To ensure safe operation, maintenance personnel must regularly perform maintenance on trains.
[0003] Current vehicle anomaly detection methods typically require inputting both normal and abnormal samples into a network model for training, enabling the model to identify anomalies. These methods rely heavily on high-quality training data, requiring significant resources for data collection and annotation. However, abnormal vehicle samples are typically scarce, making sufficient data collection difficult. Therefore, achieving anomaly detection with a limited number of abnormal vehicle samples is an urgent problem. Summary of the Invention
[0004] The present invention provides a vehicle anomaly detection method, a data processing method, an apparatus, a device and a storage medium, which are used to detect vehicle anomalies when there are few abnormal vehicle samples.
[0005] In a first aspect, the present invention provides a vehicle abnormality detection method, the method comprising:
[0006] Obtaining an image of a vehicle to be detected;
[0007] Inputting the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected;
[0008] Based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected, the abnormality of the target detection component in the image to be detected is determined.
[0009] Optionally, the image feature is an embedded vector; inputting the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected includes:
[0010] For any image block of the target detection component in the image to be detected, determine the low-level features, intermediate features and high-level features of the image block through the pre-trained model; the low-level features are features extracted based on the initial layer of the pre-trained model; the intermediate features are features extracted based on the middle layer of the pre-trained model; and the high-level features are features extracted based on the deep layer of the pre-trained model;
[0011] The low-level features, the mid-level features, and the high-level features are concatenated to obtain an embedding vector corresponding to the image block.
[0012] Optionally, the feature distribution is a Gaussian distribution; and determining the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in a plurality of normal images and the image features of each image block of the target detection component in the image to be detected includes:
[0013] For any image block of the target detection component in the image to be detected, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block;
[0014] An abnormality of the target detection component in the image to be detected is determined according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected.
[0015] Optionally, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block includes:
[0016] performing dimensionality reduction on the embedding vector of the image block by using a kernel principal component analysis method;
[0017] Calculating the distance between the reduced embedding vector and a Gaussian distribution corresponding to the image block; the Gaussian distribution is determined based on a target embedding vector; wherein the target embedding vector is determined by performing kernel principal component analysis on the embedding vectors of the image blocks in each normal sample after dimensionality reduction.
[0018] Optionally, the Gaussian distribution is represented by a mean and a covariance matrix of the embedding vectors of the image block in a plurality of normal samples; and calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block includes:
[0019] determining a difference between the embedding vector and the mean;
[0020] The distance is determined based on the difference and the covariance matrix.
[0021] Optionally, determining the abnormality of the target detection component in the image to be detected according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected includes:
[0022] For any image block, when the distance of the image block is greater than a preset value, the image block is determined to be an abnormal area, and the abnormal area is marked.
[0023] Optionally, the method further includes:
[0024] A heat map of the image to be detected is generated according to the distances between the image blocks, and the heat map is displayed to the auditor, so that the auditor can determine the abnormal area according to the heat map.
[0025] Optionally, the first precision is the floating-point precision when determining the Gaussian distribution based on the embedding vector of the image block in the normal sample; the second precision is the floating-point precision of the embedding vector corresponding to each image block of the target detection component in the image to be detected generated by the pre-trained model, and the floating-point precision when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block; the first precision is greater than the second precision.
[0026] Optionally, the pre-trained model is an EfficientNet model.
[0027] Optionally, the vehicle runs on two track beams, and an image acquisition device is provided at a preset position between the two track beams, the image acquisition device being used to acquire images of the vehicle passing by; acquiring the image of the vehicle to be detected includes:
[0028] receiving the image sent by the image acquisition device and generating an image of the vehicle bottom;
[0029] The vehicle bottom image is segmented to obtain the image to be detected.
[0030] In a second aspect, the present invention provides a data processing method, comprising:
[0031] Acquire multiple normal images of the vehicle;
[0032] Inputting the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image;
[0033] For any image block, a feature distribution is determined based on the image features corresponding to the image block in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
[0034] In a third aspect, the present invention provides a vehicle abnormality detection device, the device comprising:
[0035] A first acquisition module is used to acquire an image of a vehicle to be detected;
[0036] A first generating module is used to input the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected;
[0037] The abnormality determination module is used to determine the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected.
[0038] In a fourth aspect, the present invention provides a data processing device, comprising:
[0039] A second acquisition module is used to acquire multiple normal images of the vehicle;
[0040] A second generating module is configured to input the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image;
[0041] The third determination module is used to determine the feature distribution of any image block according to the image features corresponding to the image blocks in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
[0042] In a fifth aspect, the present invention provides an electronic device comprising: at least one processor and a memory;
[0043] The memory stores computer-executable instructions;
[0044] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspect or the second aspect.
[0045] In a sixth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in any one of the first aspect or the second aspect is implemented.
[0046] In a seventh aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspect or the second aspect.
[0047] The present invention provides a vehicle abnormality detection method, data processing method, device, equipment and storage medium. The method includes: obtaining an image of a vehicle to be detected, inputting the image to be detected into a pre-trained model, generating image features corresponding to each image block of a target detection component in the image to be detected, and determining the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected. Based on the description of the feature distribution of each image block of the target detection component in the normal image, abnormality detection can be effectively performed on each image block of the image to be detected, and abnormality detection can be achieved without obtaining abnormal samples of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0049] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention;
[0050] Figure 2 A schematic flow chart of a vehicle abnormality detection method is provided for an embodiment of the present invention;
[0051] Figure 3 Provided is a panoramic view of a passing vehicle in an embodiment of the present invention;
[0052] Figure 4 A schematic diagram of a normal image and an abnormal image is provided for an embodiment of the present invention;
[0053] Figure 5 A schematic diagram of determining an embedding vector of any image block provided by an embodiment of the present invention;
[0054] Figure 6 A schematic diagram of determining the Gaussian distribution of any image block provided by an embodiment of the present invention;
[0055] Figure 7 A thermal map and a determination result map corresponding to two types of images to be detected provided in an embodiment of the present invention;
[0056] Figure 8 A flowchart of a data processing method is provided for an embodiment of the present invention;
[0057] Figure 9 A schematic structural diagram of a vehicle abnormality detection device is provided for an embodiment of the present invention;
[0058] Figure 10 A schematic structural diagram of a data processing device provided by an embodiment of the present invention;
[0059] Figure 11 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.
[0060] The above drawings illustrate specific embodiments of the present invention, which will be described in more detail below. These drawings and the accompanying description are not intended to limit the scope of the present invention in any way, but rather to illustrate the concept of the present invention to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION
[0061] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present invention.
[0062] In this document, it should be understood that the terms involved are only used to facilitate understanding and do not represent any limitation on the meaning. In addition, the number of any element in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.
[0063] Vehicle anomaly detection is crucial for safe vehicle operation. When using a neural network model to detect vehicle anomalies, it is necessary to train the model using both abnormal and normal vehicle samples, enabling it to distinguish between normal and abnormal vehicle images. However, since abnormal vehicle samples are relatively scarce, collecting sufficient data is difficult. When there are few abnormal samples, the trained neural network model is inaccurate and cannot accurately detect vehicle anomalies. Therefore, achieving anomaly detection with a small number of abnormal vehicle samples is an urgent technical challenge.
[0064] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention, such as Figure 1 As shown, the method can be applied in a server, or in an electronic device with data computing capabilities. The electronic device can process the image to be detected, determine the image features corresponding to each image block of the target detection component in the image to be detected, and then calculate the deviation between the image features corresponding to each image block of the target detection component in the image to be detected and the normal feature distribution of the corresponding image block to determine whether the image to be detected has an abnormality.
[0065] Figure 2 A flow chart of a vehicle abnormality detection method is provided in an embodiment of the present invention, as shown in FIG. Figure 2 As shown, the method includes:
[0066] Step S201: Acquire an image of a vehicle to be detected.
[0067] When performing abnormality detection on a vehicle, an image of the vehicle to be detected can be first obtained. Optionally, the vehicle's underbody area, top area, or side area can be inspected. This application does not limit the specific location of the vehicle to be inspected. Due to the large number of components under the vehicle, the following description uses the underbody area as an example.
[0068] Optionally, the vehicle runs on two track beams, and an image acquisition device is provided at a preset position between the two track beams, the image acquisition device being used to acquire images of the vehicle passing by; acquiring the image of the vehicle to be detected includes:
[0069] receiving the image sent by the image acquisition device and generating an image of the vehicle bottom;
[0070] The vehicle bottom image is segmented to obtain the image to be detected.
[0071] When a vehicle runs on the track beams, an image acquisition device can be set in the middle of the two track beams to capture images of the vehicle during operation.
[0072] Optionally, the image acquisition device may be a line scan camera, and the images captured by the line scan camera are multiple local images of the bottom of the vehicle when the vehicle is running. The bottom image of the vehicle can be obtained by splicing the multiple local images. Figure 3 A panoramic view of a passing vehicle is provided for an embodiment of the present invention, such as Figure 3 As shown, the bottom image of the vehicle can be obtained when the vehicle is running.
[0073] Line scan cameras have high resolution and can capture high-resolution images, thereby improving the accuracy of subsequent anomaly detection. Line scan cameras capture partial images, which can be sent to electronic devices. The electronic devices can stitch together multiple partial images to create a stitched image, representing the vehicle's underbody image.
[0074] Optionally, after acquiring multiple partial images, the server may perform preprocessing operations on the acquired images. Preprocessing operations may include blank image removal, filtering, image enhancement, image stitching, etc. Optionally, since the trigger time of the image acquisition device is usually earlier than the time when the vehicle arrives at the installation location of the image acquisition device, the image acquisition device will capture blank images, and the blank images in the acquired images can be removed. The acquired partial images may contain noise, and the area of interest cannot be presented in a single image. Therefore, the acquired images can be filtered. Since the acquired images are images of the vehicle bottom, the overall image is dark. The acquired images can be enhanced to adjust the overall brightness of the image. Since the acquired images are partial images of the vehicle bottom, the multiple acquired images can be stitched together to obtain an image of the vehicle bottom. The acquired panoramic image of the vehicle bottom is segmented to obtain the image to be detected. Specifically, image segmentation can be performed, such as using a threshold method or a grayscale projection method to obtain an area containing certain components, that is, the image to be detected.
[0075] Optionally, the image to be detected may be an image containing only one component, or may be an image containing two or more components. Preferably, the image to be detected is an image containing only one component.
[0076] Figure 4 A schematic diagram of a normal image and an abnormal image is provided for an embodiment of the present invention, such as Figure 4 As shown, the image is obtained after the panoramic image of the vehicle bottom is segmented.
[0077] Step S202: Input the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected.
[0078] After determining the image to be detected, the image to be detected can be input into a pre-trained model. The pre-trained model is a model with feature extraction capabilities that can extract features of the image to be detected. Optionally, the pre-trained model can be a convolutional neural network model.
[0079] Optionally, when determining the features of the image to be detected, the image of the target detection component in the image to be detected can be divided into blocks to obtain individual image blocks of the target detection component, and then the image features corresponding to each image block are determined, so that the abnormal part of the target detection component can be accurately located.
[0080] Exemplarily, the target detection component in the image to be detected can be divided into N image blocks, and for each image block, the image feature of the image block can be determined.
[0081] Through block processing, each image block in the image to be detected can be processed separately and anomaly detection can be performed. When the abnormal area is not in the center of the image to be detected, a high anomaly detection accuracy can be maintained.
[0082] Step S203 : determining abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected.
[0083] After the image features of each image block of the target detection component in the image to be detected are determined, the abnormality of the target detection component in the image to be detected can be determined based on the image features.
[0084] Specifically, the feature distribution of each image block of the target detection component determined based on multiple normal images can be predetermined or obtained. The feature distribution can describe the feature distribution characteristics of the image block in the normal sample. Therefore, for any image block, the image features of the image block under the target detection component in the image to be detected and the feature distribution of the image block are compared. By calculating the deviation between the two, it can be determined whether there are abnormalities in each image block of the target detection component, thereby determining the abnormality of the target detection component.
[0085] Optionally, the feature distribution here is the feature distribution of learned normal samples, which conforms to a type of learning paradigm for anomaly detection tasks. There is no need to learn the feature distribution of abnormal samples and the dependence on abnormal samples is small.
[0086] Exemplarily, when there is an abnormality in any image block of the target detection component in the image to be detected, it can be determined that there is an abnormality in the target detection component in the image to be detected.
[0087] Compared with existing methods, this method can clearly describe the feature distribution of each image block in a normal sample. When an unprecedented defect appears, since its image features are significantly different from the normal feature distribution, this method can effectively detect it as an abnormality. Therefore, it can effectively detect various abnormal defects that have appeared or have not appeared.
[0088] The present invention provides a vehicle abnormality detection method, which includes: obtaining an image to be detected of a vehicle, inputting the image to be detected into a pre-trained model, generating image features corresponding to each image block of a target detection component in the image to be detected, and determining the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected. Based on the description of the feature distribution of each image block of the target detection component in the normal image, abnormality detection can be effectively performed on each image block of the image to be detected, and abnormality detection can be achieved without obtaining abnormal samples.
[0089] Optionally, the image feature is an embedded vector; inputting the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected includes:
[0090] For any image block of the target detection component in the image to be detected, determine the low-level features, intermediate features and high-level features of the image block through the pre-trained model; the low-level features are features extracted based on the initial layer of the pre-trained model; the intermediate features are features extracted based on the middle layer of the pre-trained model; and the high-level features are features extracted based on the deep layer of the pre-trained model;
[0091] The low-level features, the mid-level features, and the high-level features are concatenated to obtain an embedding vector corresponding to the image block.
[0092] When the image features corresponding to each image block of the target detection component in the image to be detected are determined using a pre-trained model, the image features can be embedded vectors. Embedded vectors are vectors that map complex, high-dimensional, or sparse data into a low-dimensional, dense vector space. Embedded vectors can preserve the semantic or structural information of the original data, effectively expressing high-dimensional data in a low-dimensional form.
[0093] When determining the embedding vector corresponding to each image block, features at three different semantic levels can be extracted, such as low-level features, mid-level features, and high-level features. The low-level features, mid-level features, and high-level features corresponding to the same image block are concatenated to obtain a multi-dimensional embedding vector corresponding to the image block. Figure 5 A schematic diagram of determining an embedding vector of any image block provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, an embedding vector can be calculated for each image block in an image. When there are H*W image blocks, there are H*W corresponding embedding vectors.
[0094] Optionally, low-level features are features extracted based on the initial layer of the pre-trained model, that is, features extracted through the first few layers of the pre-trained model, which can be used to capture local information such as edges and textures. Low-level features can be used to detect subtle surface anomalies. Intermediate features are component-level features extracted based on the intermediate layers of the pre-trained model. This feature has a higher level of abstract expression capability and can capture more complex information such as the shape, structure, and spatial relationship of objects. For example, it can recognize simple geometric shapes, texture patterns, and the contours of objects, thereby identifying structural anomalies in a larger range. High-level features are object-level features extracted based on the deep layers of the pre-trained model. High-level features have more abstract and semantic expression capabilities and can capture higher-level semantic information.
[0095] Among them, the embedding vector of the image block is obtained by splicing the three features. These embedding vectors not only contain fine-grained local information, but also global context information, which can improve the accuracy of anomaly detection.
[0096] By extracting and splicing multi-level features, the pre-trained model can focus on local details and global semantic information at the same time, which can improve the accuracy and robustness of anomaly detection.
[0097] Optionally, the pre-trained model is an EfficientNet model.
[0098] The pre-trained model selected in this application can be the EfficientNet network model, which is a lightweight network model that can achieve higher accuracy with fewer parameters, thereby extracting richer and more representative features.
[0099] Optionally, the feature distribution is a Gaussian distribution; and determining the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in a plurality of normal images and the image features of each image block of the target detection component in the image to be detected includes:
[0100] For any image block of the target detection component in the image to be detected, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block;
[0101] An abnormality of the target detection component in the image to be detected is determined according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected.
[0102] The characteristic distribution of each image block of the target detection component, pre-determined based on multiple normal images, can be a Gaussian distribution. The embedding vector corresponding to an image block is multidimensional data. The Gaussian distribution can characterize the distribution of each data in the embedding vector corresponding to an image block in a normal image. When determining whether a certain image block of the target detection component in the image to be detected is abnormal, the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block can be calculated. The distance can be used to quantify the difference between the embedding vector corresponding to the image block and the Gaussian distribution. When the distance is large, it indicates that the image block in the target detection component is abnormal.
[0103] After determining the distances corresponding to the image blocks, the abnormality of the target detection component in the image to be detected can be determined. For example, when there are multiple image blocks with larger distances corresponding to the image blocks, the multiple image blocks in the target detection component are abnormal image blocks.
[0104] Optionally, the distance may be the Mahalanobis distance. By calculating the distance, it is determined whether there is an abnormality in the target detection component in the image to be detected. This method can reduce computational complexity and is suitable for application scenarios requiring a fast response, such as vehicle underbody detection.
[0105] By calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block, the abnormality of each image block in the target detection component in the image to be detected can be effectively measured.
[0106] Optionally, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block includes:
[0107] performing dimensionality reduction on the embedding vector of the image block by using a kernel principal component analysis method;
[0108] Calculating the distance between the reduced embedding vector and a Gaussian distribution corresponding to the image block; the Gaussian distribution is determined based on a target embedding vector; wherein the target embedding vector is determined by performing kernel principal component analysis on the embedding vectors of the image blocks in each normal sample after dimensionality reduction.
[0109] To reduce computational complexity when determining the distance between the embedding vector corresponding to an image block and the Gaussian distribution corresponding to that image block, the embedding vectors for each image block can be subjected to dimensionality reduction. After dimensionality reduction, the dimensionality of the embedding vectors is reduced. For example, before dimensionality reduction, the dimensionality of the embedding vectors is 50, while after dimensionality reduction, the dimensionality of the embedding vectors is 40. By reducing the dimensionality of the embedding vectors, the computational complexity of calculating the distance between the embedding vectors for the image blocks and the Gaussian distribution corresponding to the image blocks is reduced.
[0110] Optionally, kernel principal component analysis can be used for dimensionality reduction. Specifically, a kernel function can be used to map the embedding vector to a high-dimensional feature space. Dimensionality reduction can be performed in this high-dimensional feature space through principal component analysis, which can achieve nonlinear dimensionality reduction of the data and retain valid feature data.
[0111] Correspondingly, when determining the corresponding Gaussian distribution based on the embedding vector of a certain image block in each normal sample, the embedding vector can also be first subjected to dimensionality reduction processing, and then the Gaussian distribution is determined based on the embedded vector after dimensionality reduction, so that when the distance between the embedded vector after dimensionality reduction and the Gaussian distribution corresponding to the image block is subsequently calculated, the data involved in the calculation have the same dimension.
[0112] By embedding each image block into a vector for dimensionality reduction, the computational complexity can be effectively reduced and the output anomaly detection results can be improved. In addition, the kernel principal component analysis method can effectively filter the data. Compared with the random discarding dimensionality reduction method, it can retain effective features and improve the effect of dimensionality reduction processing.
[0113] Optionally, the Gaussian distribution is represented by a mean and a covariance matrix of the embedding vectors of the image block in a plurality of normal samples; and calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block includes:
[0114] determining a difference between the embedding vector and the mean;
[0115] The distance is determined based on the difference and the covariance matrix.
[0116] When determining the Gaussian distribution of any image block in a normal sample, the mean and covariance matrix of the embedding vectors corresponding to that image block are actually calculated. Optionally, a training set is set up for each vehicle component. For example, a training set can be set up for each component under the vehicle, such as the tire, battery, brake, etc., and the training set contains multiple normal samples of that vehicle component. For each normal sample in the training set, the embedding vector of each image block in that normal sample can be determined. For a single image block, the mean and covariance matrix can be determined based on the embedding vectors of that image block in multiple normal samples.
[0117] For example, the training set for the vehicle component of tires contains 50 normal samples, each of which is divided into 30 image blocks. For the first image block, there are 50 such samples. Based on the embedding vectors of these 50 image blocks, the Gaussian distribution of the image block can be determined, that is, the mean and covariance matrix of these 50 embedding vectors can be calculated. Figure 6A schematic diagram of determining the Gaussian distribution of any image block provided by an embodiment of the present invention. For an image block at a certain position, the Gaussian distribution of the image block can be determined based on each embedded vector, that is, the parameters of the Gaussian distribution: mean and covariance matrix can be calculated.
[0118] Optionally, the Gaussian distribution is a multivariate Gaussian distribution, which can be expressed as N(μ,∑), where the mean is μ and the covariance matrix is ∑. The mean and covariance matrix can be calculated using the following formula:
[0119]
[0120] Among them, i and j represent the position of the image block, the position of the image block is the i-th row and j-th column, N represents the number of normal samples, x ij represents the embedding vector, μ ij represents the mean, ∑ ij Represents the covariance matrix, I represents the homography matrix, that is, the data on the diagonal position of the matrix is 1, and the data at other positions is 0. ε is a very small value. By setting the homography matrix, the covariance matrix can be avoided to be 0, thereby avoiding the subsequent inability to calculate the Mahalanobis distance.
[0121] Since the embedding vector contains three layers of features, the covariance matrix can not only describe the correlation within the same semantic level, but also capture the correlation between different semantic levels, which can improve the accuracy of anomaly detection.
[0122] Through the above-calculated mean and covariance matrix, a multivariate Gaussian distribution model can be established for the position of each image block. The multivariate Gaussian distribution model can describe the characteristic distribution of normal image blocks.
[0123] When calculating the distance between the embedding vector of an image block and the Gaussian distribution corresponding to the image block, the Mahalanobis distance can be calculated. The calculation formula of the Mahalanobis distance is:
[0124]
[0125] Specifically, the difference between the embedding vector and the mean can be calculated, and the Mahalanobis distance can be determined based on the difference and the covariance matrix.
[0126] By calculating the Mahalanobis distance, the gap between the features of the image block and the multivariate Gaussian distribution model can be quickly determined, thereby quickly determining whether the image block has an abnormality, and thus quickly determining whether the image to be detected has an abnormality.
[0127] Optionally, determining the abnormality of the target detection component in the image to be detected according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected includes:
[0128] For any image block, when the distance of the image block is greater than a preset value, the image block is determined to be an abnormal area, and the abnormal area is marked.
[0129] After determining the distance between each image block of the target detection component in the image to be detected, it can be determined whether the image to be detected has an abnormality based on the distance. Specifically, a preset value can be set in advance. When the calculated distance is greater than the preset value, it indicates that the image block is an abnormal area. When the calculated distance is less than or equal to the preset value, it indicates that the image block is a normal area.
[0130] After the abnormal area in the image to be detected is determined, the abnormal area can be marked and displayed, so that the reviewer can directly obtain the abnormal area.
[0131] In the early stages, not only can a training set be established for each vehicle component, but a validation set can also be established for each vehicle component. This validation set can include both normal and abnormal samples of that vehicle component. Normal samples can cover a variety of common vehicle underbody conditions, while abnormal samples include various possible defects, such as cracks and the presence of foreign matter. After obtaining the Gaussian distribution model for each image block, the performance of the Gaussian distribution model can be verified based on the normal and abnormal samples in the validation set. Specifically, an initial preset value can be set in advance, and this preset value can be modified based on the Gaussian distribution model's classification of normal and abnormal samples to obtain the final preset value. Therefore, abnormal samples are only used during the testing phase, or simply to set the preset value, and are not relevant to the establishment of the Gaussian distribution model.
[0132] By setting the preset value, the abnormal area in the image to be detected can be accurately obtained.
[0133] Optionally, the method further includes:
[0134] A heat map of the image to be detected is generated according to the distances between the image blocks, and the heat map is displayed to the auditor, so that the auditor can determine the abnormal area according to the heat map.
[0135] For each image block in the image to be detected, after determining the distance between each image block, a distance matrix can also be obtained. Based on the distance matrix, a heat map corresponding to the image to be detected can be generated, and the data can be mapped to different colors to show the distribution of the data. Finally, the generated heat map can be displayed to the reviewer.
[0136] Optionally, after generating the heat map, abnormal areas can be further determined, marked and displayed to the user, so that the user can determine which areas are abnormal areas and determine the heat map information corresponding to each area in the image to be detected.
[0137] Figure 7The embodiment of the present invention provides a thermal map and a judgment result map corresponding to two types of images to be detected, such as Figure 7 As shown, the abnormal area in the heat map corresponding to the abnormal image appears in a darker color. After labeling processing, the darker area in the heat map can be labeled.
[0138] By generating a heat map corresponding to the image to be detected, users can easily determine the heat map information of the entire image to be detected. The auditor can conduct a secondary check on areas that are not marked as abnormal areas and may have abnormalities to determine whether there are actually abnormalities, thereby improving the accuracy of anomaly detection.
[0139] Optionally, the first precision is the floating-point precision when determining the Gaussian distribution based on the embedding vector of the image block in the normal sample; the second precision is the floating-point precision of the embedding vector corresponding to each image block of the target detection component in the image to be detected generated by the pre-trained model, and the floating-point precision when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block; the first precision is greater than the second precision.
[0140] In order to improve the accuracy and efficiency of anomaly detection, when determining the Gaussian distribution based on the embedding vector of the image block in the normal sample, the precision of the floating-point number used can be higher, such as a 32-bit floating-point number, to improve the accuracy of the determined Gaussian distribution. When performing anomaly detection on the image to be detected, such as generating the embedding vector corresponding to each image block of the target detection component in the image to be detected through a pre-trained model, the precision of the floating-point number used can be lower, and the precision of the floating-point number when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block can also be lower, such as a 16-bit floating-point number, which can increase the calculation speed during the anomaly detection process. It should be noted that the number of floating-point numbers used here is not specifically limited.
[0141] By setting different floating-point precisions in the process of generating Gaussian distribution and the process of anomaly detection, the speed of anomaly detection can be improved while maintaining high calculation accuracy.
[0142] Figure 8 A data processing method provided by an embodiment of the present invention is as follows: Figure 8 As shown, the method includes:
[0143] Step S801: Acquire multiple normal images of the vehicle;
[0144] Step S802: Input the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image;
[0145] Step S803: for any image block, determining a feature distribution according to the image features corresponding to the image block in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
[0146] Before performing anomaly detection on the image to be detected, the feature distribution corresponding to each image block in the target detection component can be determined first, so that there is no need to recalculate the feature distribution corresponding to each image block in the target detection component each time anomaly detection is performed. Alternatively, the calculated feature distribution corresponding to each image block in the target detection component can be provided to a third party for use.
[0147] Before determining the feature distribution of each image block of the target detection component, you can first obtain multiple normal images, such as a panoramic image of the vehicle bottom. Perform image enhancement and image segmentation on the panoramic image to obtain normal images corresponding to each component. The normal images corresponding to the same component are constructed into a training set, and a training set can be constructed for each component.
[0148] For each normal image in the training set of the object detection component, image features corresponding to each image block may be determined. Exemplarily, the image features may be embedded vectors.
[0149] For each image block, a feature distribution of the image block is determined based on the embedding vector corresponding to the image block in each normal image. For example, the feature distribution may be a Gaussian distribution. Thus, when performing anomaly detection on each image block of the target detection component in the image to be detected, anomaly detection can be performed based on the feature distribution of each image block of the target detection component.
[0150] The present invention provides a data processing method, which includes: obtaining multiple normal images of a vehicle, inputting the multiple normal images into a pre-trained model, generating image features corresponding to each image block of a target detection component in the normal image, and determining, for any image block, a feature distribution based on the image features corresponding to the image blocks in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected. By determining the feature distribution corresponding to each image block of the target detection component based on the normal image, abnormality detection can be effectively performed on each image block of the image to be detected, and abnormality detection can be achieved without obtaining abnormal samples.
[0151] The above-mentioned vehicle anomaly detection method provided in this application can share the same set of hardware equipment with the existing trackside online detection system, without the need for additional hardware cost expenditure.
[0152] Figure 9A schematic diagram of the structure of a vehicle abnormality detection device provided in an embodiment of the present invention, the device comprising:
[0153] A first acquisition module 901 is used to acquire an image of a vehicle to be detected;
[0154] A first generating module 902 is configured to input the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected;
[0155] The abnormality determination module 903 is used to determine the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected.
[0156] Optionally, the image feature is an embedded vector; when the first generating module 902 inputs the image to be detected into a pre-trained model and generates image features corresponding to each image block of the target detection component in the image to be detected, it is specifically used to:
[0157] For any image block of the target detection component in the image to be detected, determine the low-level features, intermediate features and high-level features of the image block through the pre-trained model; the low-level features are features extracted based on the initial layer of the pre-trained model; the intermediate features are features extracted based on the middle layer of the pre-trained model; and the high-level features are features extracted based on the deep layer of the pre-trained model;
[0158] The low-level features, the mid-level features, and the high-level features are concatenated to obtain an embedding vector corresponding to the image block.
[0159] Optionally, the feature distribution is a Gaussian distribution; when the abnormality determination module 903 determines the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in the multiple normal images and the image features of each image block of the target detection component in the image to be detected, it is specifically configured to:
[0160] For any image block of the target detection component in the image to be detected, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block;
[0161] An abnormality of the target detection component in the image to be detected is determined according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected.
[0162] Optionally, when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block, the abnormality determination module 903 is specifically configured to:
[0163] performing dimensionality reduction on the embedding vector of the image block by using a kernel principal component analysis method;
[0164] Calculating the distance between the reduced embedding vector and a Gaussian distribution corresponding to the image block; the Gaussian distribution is determined based on a target embedding vector; wherein the target embedding vector is determined by performing kernel principal component analysis on the embedding vectors of the image blocks in each normal sample after dimensionality reduction.
[0165] Optionally, the Gaussian distribution is represented by the mean and covariance matrix of the embedding vectors of the image blocks in multiple normal samples; when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block, the abnormality determination module 903 is specifically configured to:
[0166] determining a difference between the embedding vector and the mean;
[0167] The distance is determined based on the difference and the covariance matrix.
[0168] Optionally, when determining the abnormality of the target detection component in the image to be detected based on the distances corresponding to the image blocks of the target detection component in the image to be detected, the abnormality determination module 903 is specifically configured to:
[0169] For any image block, when the distance of the image block is greater than a preset value, the image block is determined to be an abnormal area, and the abnormal area is marked.
[0170] Optionally, the device further includes: a heat map generation module, configured to:
[0171] A heat map of the image to be detected is generated according to the distances between the image blocks, and the heat map is displayed to the auditor, so that the auditor can determine the abnormal area according to the heat map.
[0172] Optionally, the first precision is the floating-point precision when determining the Gaussian distribution based on the embedding vector of the image block in the normal sample; the second precision is the floating-point precision of the embedding vector corresponding to each image block of the target detection component in the image to be detected generated by the pre-trained model, and the floating-point precision when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block; the first precision is greater than the second precision.
[0173] Optionally, the pre-trained model is an EfficientNet model.
[0174] Optionally, the vehicle runs on two track beams, and an image acquisition device is provided at a preset position between the two track beams, the image acquisition device being used to acquire images of the vehicle passing by; when acquiring the image to be detected of the vehicle, the first acquisition module 901 is specifically used to:
[0175] receiving the image sent by the image acquisition device and generating an image of the vehicle bottom;
[0176] The vehicle bottom image is segmented to obtain the image to be detected.
[0177] The vehicle abnormality detection device provided by the embodiment of the present invention can achieve the above Figure 2 The vehicle anomaly detection method of the embodiment shown has similar implementation principles and technical effects, which will not be described again here.
[0178] Figure 10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present invention, the device comprising:
[0179] The second acquisition module 1001 is used to acquire multiple normal images of the vehicle;
[0180] The second generating module 1002 is configured to input the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image;
[0181] The third determination module 1003 is used to determine the feature distribution of any image block according to the image features corresponding to the image block in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
[0182] The data processing device provided by the embodiment of the present invention can realize the above-mentioned Figure 8 The data processing method of the embodiment shown has similar implementation principles and technical effects, which will not be described in detail here.
[0183] Figure 11 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Figure 11 As shown, the server provided by this embodiment includes: at least one processor 1101 and a memory 1102. The processor 1101 and the memory 1102 are connected via a bus 1103.
[0184] During the specific implementation process, at least one processor 1101 executes the computer-executable instructions stored in the memory 1102, so that the at least one processor 1101 executes the method in the above method embodiment.
[0185] The specific implementation process of the processor 1101 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.
[0186] In the above Figure 11 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.
[0187] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.
[0188] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.
[0189] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of the above method embodiment is implemented.
[0190] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of the above method embodiment when executed by a processor.
[0191] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The computer-readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0192] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0193] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.
[0194] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0196] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A vehicle abnormality detection method, characterized in that: The method comprises: Obtaining an image of a vehicle to be detected; Inputting the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected; Based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected, the abnormality of the target detection component in the image to be detected is determined.
2. The method according to claim 1, characterized in that The image feature is an embedded vector; the image to be detected is input into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected, including: For any image block of the target detection component in the image to be detected, determine the low-level features, intermediate features and high-level features of the image block through the pre-trained model; the low-level features are features extracted based on the initial layer of the pre-trained model; the intermediate features are features extracted based on the middle layer of the pre-trained model; and the high-level features are features extracted based on the deep layer of the pre-trained model; The low-level features, the mid-level features, and the high-level features are concatenated to obtain an embedding vector corresponding to the image block.
3. The method according to claim 2, characterized in that The feature distribution is a Gaussian distribution; and determining an abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in a plurality of normal images and the image features of each image block of the target detection component in the image to be detected includes: For any image block of the target detection component in the image to be detected, calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block; An abnormality of the target detection component in the image to be detected is determined according to the distances corresponding to the respective image blocks of the target detection component in the image to be detected.
4. The method according to claim 3, characterized in that Calculating a distance between the embedding vector of the image block and a Gaussian distribution corresponding to the image block includes: performing dimensionality reduction on the embedding vector of the image block by using a kernel principal component analysis method; Calculating the distance between the reduced embedding vector and a Gaussian distribution corresponding to the image block; the Gaussian distribution is determined based on a target embedding vector; wherein the target embedding vector is determined by performing kernel principal component analysis on the embedding vectors of the image blocks in each normal sample after dimensionality reduction.
5. The method according to claim 3, characterized in that The Gaussian distribution is represented by the mean and covariance matrix of the embedding vectors of the image blocks in multiple normal samples; Calculating a distance between the embedding vector of the image block and a Gaussian distribution corresponding to the image block includes: determining a difference between the embedding vector and the mean; The distance is determined based on the difference and the covariance matrix.
6. The method according to claim 3, characterized in that Determining an abnormality of the target detection component in the image to be detected according to distances corresponding to respective image blocks of the target detection component in the image to be detected includes: For any image block, when the distance of the image block is greater than a preset value, the image block is determined to be an abnormal area, and the abnormal area is marked.
7. The method according to claim 3, characterized in that The method further comprises: A heat map of the image to be detected is generated according to the distances between the image blocks, and the heat map is displayed to the auditor, so that the auditor can determine the abnormal area according to the heat map.
8. The method according to claim 4, characterized in that The first precision is the floating point precision when determining the Gaussian distribution based on the embedding vector of the image block in the normal sample; the second precision is the floating point precision when generating the embedding vector corresponding to each image block of the target detection component in the image to be detected through the pre-trained model, and the floating point precision when calculating the distance between the embedding vector of the image block and the Gaussian distribution corresponding to the image block; The first precision is greater than the second precision.
9. The method according to any one of claims 1 to 8, characterized in that The pre-trained model is the EfficientNet model.
10. The method according to any one of claims 1 to 8, characterized in that The vehicle runs on two track beams, and an image acquisition device is provided at a preset position between the two track beams, and the image acquisition device is used to capture images when the vehicle passes; Obtain the image of the vehicle to be detected, including: receiving the image sent by the image acquisition device and generating an image of the vehicle bottom; The vehicle bottom image is segmented to obtain the image to be detected.
11. A data processing method, characterized in that: The method comprises: Acquire multiple normal images of the vehicle; Inputting the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image; For any image block, a feature distribution is determined based on the image features corresponding to the image block in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
12. A vehicle abnormality detection device, characterized in that: The device comprises: A first acquisition module is used to acquire an image of a vehicle to be detected; A first generating module is used to input the image to be detected into a pre-trained model to generate image features corresponding to each image block of the target detection component in the image to be detected; The abnormality determination module is used to determine the abnormality of the target detection component in the image to be detected based on the feature distribution of each image block of the target detection component in multiple normal images and the image features of each image block of the target detection component in the image to be detected.
13. A data processing device, characterized in that: The device comprises: A second acquisition module is used to acquire multiple normal images of the vehicle; A second generating module is configured to input the plurality of normal images into a pre-trained model to generate image features corresponding to each image block of the target detection component in the normal image; The third determination module is used to determine the feature distribution of any image block according to the image features corresponding to the image blocks in each normal sample; the feature distribution is used to determine the abnormality of the target detection component in the image to be detected.
14. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 11.
15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.
16. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when the computer program is executed by a processor.