Abnormality detection method and device, electronic equipment and storage medium
By extracting the frame features of the image taken by the on-board camera, predicting the image, and calculating the similarity, the problem of inaccurate detection under the influence of light changes in the prior art is solved, and the accuracy of abnormalities of the vehicle shooting device is realized, and safety is enhanced.
Patent Information
- Application Number
- CN202410986918.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-03
AI Technical Summary
The existing vehicle-mounted camera abnormal detection technology is sensitive to changes in lighting conditions, resulting in inaccurate detection results, and it is impossible to accurately determine whether the shooting device set on the vehicle is abnormal, which poses safety hazards.
By acquiring multiple continuous image frames arranged in time series captured by the shooting device, extracting features and predicting the predicted image of the next frame, calculating the similarity between the predicted image and the actual image. If the similarity exceeds the threshold, confirming that there is no abnormality. If it is not exceeded, it is confirmed that there is an abnormality, and the lighting change and jitter information are considered.
The accuracy of the predicted image is improved, detection inaccurate caused by the influence of light conditions is avoided, and accurate determination of abnormalities in the vehicle shooting device is achieved, which enhances safety.
Smart Images

Figure CN119068232A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle diagnosis technology, and in particular to an abnormality detection method, device, electronic device and storage medium. Background Art
[0002] With the growth of material culture, driving has become the first choice for the public, and vehicles are becoming more and more intelligent. As an important part of the intelligent driving system, the working status of the on-board camera directly affects the safety of driving. Therefore, real-time monitoring and diagnosis of the working status of the on-board camera is of great significance to ensure the stable operation of the intelligent driving system. In the field of computer vision, image processing and computer graphics are two important subfields. Image processing mainly refers to the conversion of digital images into a form more suitable for computer processing, while computer graphics refers to the process of using computers to generate images.
[0003] The existing vehicle camera anomaly detection technology is mainly achieved by comparing the pixel values of two adjacent frames of images taken by the vehicle camera. However, the above method is sensitive to changes in lighting conditions and can easily lead to inaccurate anomaly detection results. Summary of the invention
[0004] In view of this, the embodiments of the present application provide an abnormality detection method, device, electronic device and storage medium to solve the problem in the prior art that it is impossible to accurately determine whether the shooting device installed on the vehicle is abnormal, resulting in safety hazards.
[0005] According to a first aspect of an embodiment of the present application, a method for detecting an abnormality is provided, the method comprising: obtaining a plurality of continuous image frames taken by a photographing device and arranged in a time series; performing feature extraction on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, respectively, and predicting a predicted image of a next frame of the photographing device based on the key features; obtaining an actual image of the next frame taken by the photographing device, and calculating the similarity between the predicted image and the actual image; if the similarity does not exceed a preset similarity threshold, confirming that an abnormality exists in the photographing device; if the similarity exceeds the similarity threshold, confirming that there is no abnormality in the photographing device.
[0006] According to a second aspect of an embodiment of the present application, an abnormality detection device is provided, which includes: an acquisition module for acquiring a plurality of continuous image frames arranged in a time series and photographed by a photographing device; a prediction module for performing feature extraction on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and predicting a predicted image of a next frame of the photographing device based on the key features; a calculation module for acquiring an actual image of the next frame photographed by the photographing device, and calculating the similarity between the predicted image and the actual image; if the similarity does not exceed a preset similarity threshold, it is confirmed that the photographing device has an abnormality; if the similarity exceeds the similarity threshold, it is confirmed that the photographing device has no abnormality.
[0007] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0008] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0009] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the method in the embodiments of the present application obtains a plurality of continuous image frames arranged in a time series taken by a photographing device; performs feature extraction on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, respectively, and predicts a predicted image of the next frame of the photographing device according to the key features; obtains an actual image of the next frame taken by the photographing device, and calculates the similarity between the predicted image and the actual image; if the similarity exceeds a preset similarity threshold, it is confirmed that the photographing device is abnormal; if the similarity does not exceed the similarity threshold, it is confirmed that the photographing device is not abnormal, wherein, by obtaining a plurality of continuous image frames arranged in a time series taken by the photographing device, the predicted image of the next frame of the photographing device is predicted, and then the timing information such as the illumination change of the image taken by the photographing device is taken into account, the accuracy of the predicted image is improved, and it will not be affected by the illumination conditions, and the abnormal jitter of the photographing device can be taken into account, and subsequently by determining the similarity between the predicted image containing the timing information and the actual image, it is achieved to accurately determine whether the photographing device is abnormal, thereby avoiding the problem in the related art that it is impossible to accurately determine whether the photographing device installed on the vehicle is abnormal, resulting in a safety hazard. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1 It is a flowchart of an anomaly detection method provided in an embodiment of the present application;
[0012] Figure 2 It is a flowchart of another anomaly detection method provided in an embodiment of the present application;
[0013] Figure 3 is a flowchart of another anomaly detection method provided in an embodiment of the present application;
[0014] Figure 4 It is a flowchart of another abnormality detection method provided in an embodiment of the present application;
[0015] Figure 5 It is a flowchart of another abnormality detection method provided in an embodiment of the present application;
[0016] Figure 6 It is a flowchart of another optional anomaly detection method provided in an embodiment of the present application;
[0017] Figure 7 It is a flowchart of another optional anomaly detection method provided in an embodiment of the present application;
[0018] Figure 8 This is a flowchart of another optional anomaly detection method provided in an embodiment of the present application;
[0019] Fig. 9 is a structural schematic diagram of an abnormality detection device provided in an embodiment of the present application;
[0020] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0022] An anomaly detection method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0023] Figure 1 This is an anomaly detection method provided by an embodiment of the present application, such as Figure 1 As shown, the method includes:
[0024] S101, acquiring a plurality of continuous image frames taken by a shooting device and arranged in a time sequence;
[0025] S102, extracting features from a plurality of consecutive image frames to obtain key features corresponding to the plurality of consecutive image frames, and predicting a predicted image of a next frame of the shooting device according to the key features;
[0026] S103, obtaining an actual image of the next frame captured by the shooting device, and calculating the similarity between the predicted image and the actual image;
[0027] S104: If the similarity exceeds a preset similarity threshold, it is confirmed that there is an abnormality in the photographing device; if the similarity does not exceed the similarity threshold, it is confirmed that there is no abnormality in the photographing device.
[0028] It can be understood that the above-mentioned anomaly detection method provided in this example is applied to vehicles, and the above-mentioned vehicles include vehicles with automatic driving or intelligent driving (including vehicles with passenger functions (such as cars, buses, large buses, minibuses, etc.), vehicles with cargo functions (such as ordinary trucks, vans, trailers, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures), special vehicles (such as logistics distribution vehicles, automatic guided transport vehicles AGV, patrol cars, cranes, cranes, excavators, bulldozers, forklifts, rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor washing vehicles, sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.), vehicles with entertainment functions (such as entertainment vehicles, amusement park automatic driving devices, balance cars, etc.), rescue vehicles (such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.)), etc.
[0029] Specifically, the vehicle is provided with at least one shooting device capable of performing continuous shooting, and the multiple continuous image frames acquired in step S101 of this example are image frames continuously shot by the same shooting device.
[0030] It can be understood that this example does not limit the number of image frames acquired, and the number of image frames acquired is determined by the preset acquisition duration and the frame rate of the shooting device; illustratively, the shooting device on the vehicle is a high-definition camera with a resolution of 1280*720, a frame rate of 30 frames / second, and an acquisition duration of 1 minute. In this example, 1800 frames of images collected within one minute are taken as multiple continuous image frames acquired.
[0031] It can be understood that in continuous shooting, images are captured in chronological order, and the changes between previous and next frames are usually small. For example, in a video, the position, shape, and color of an object will not change dramatically in adjacent frames, and the movement of the object is continuous. The position and shape in the next frame can be predicted based on the movement trend of the previous N frames, and in many cases, the background is stable, and the changes in the previous and next frames mainly come from the movement of the foreground objects. Therefore, by analyzing the previous few frames, the background part can be predicted, and on this basis, the changes in the foreground objects can be predicted. Therefore, after acquiring multiple image frames sorted in time series, this example can extract features from multiple consecutive image frames, obtain key features corresponding to the multiple consecutive image frames, and predict the predicted image of the next frame of the shooting device based on the key features.
[0032] It can be understood that the above method obtains multiple continuous image frames arranged in a time series, and then performs feature extraction on the multiple continuous image frames to obtain key features corresponding to the multiple continuous image frames, and predicts the predicted image of the next frame of the shooting device based on the key features, wherein the key features corresponding to the multiple continuous image frames include the timing information of the changes in lighting conditions, so that the generated predicted image can include the changes in lighting conditions over time during the driving process of the vehicle; in addition, since the shooting conditions of multiple continuous image frames should be consistent or tend to be consistent, the key features of multiple continuous image frames can also reflect minor abnormalities of the shooting device (for example, slight jitter abnormalities).
[0033] It can be understood that if there is no abnormality in the shooting device, the predicted image and the actual image should be consistent. On the contrary, if there is an abnormality in the shooting device, there will be a large difference between the predicted image and the actual image. Therefore, this example will obtain the actual image of the next frame taken by the shooting device and calculate the similarity between the predicted image and the actual image. Specifically, this example will obtain the actual image of the next frame taken by the shooting device and calculate the similarity between the predicted image and the actual image.
[0034] If the similarity does not exceed the preset similarity threshold, it is confirmed that there is an abnormality in the shooting device; if the similarity exceeds the similarity threshold, it is confirmed that there is no abnormality in the shooting device; the above similarity threshold is determined based on historical data, and the above similarity threshold can also be determined by relevant personnel based on actual scenarios.
[0035] According to the technical solution provided in the embodiment of the present application, a plurality of continuous image frames arranged in a time series taken by a photographing device are obtained; feature extraction is performed on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and a predicted image of the next frame of the photographing device is predicted based on the key features; the actual image of the next frame taken by the photographing device is obtained, and the similarity between the predicted image and the actual image is calculated; if the similarity exceeds a preset similarity threshold, it is confirmed that the photographing device is abnormal; if the similarity does not exceed the similarity threshold, it is confirmed that the photographing device is normal, wherein, by obtaining a plurality of continuous image frames arranged in a time series taken by the photographing device, the predicted image of the next frame of the photographing device is predicted, and then the timing information such as the illumination change of the image taken by the photographing device is taken into account, the accuracy of the predicted image is improved, it will not be affected by the illumination conditions, and the abnormal jitter of the photographing device can be taken into account, and subsequently by determining the similarity between the predicted image containing the timing information and the actual image, it is achieved to accurately determine whether the photographing device is abnormal, thereby avoiding the problem in the related art that it is impossible to accurately determine whether the photographing device installed on the vehicle is abnormal, resulting in a safety hazard.
[0036] In some embodiments, Figure 2 As shown, feature extraction is performed on a plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and a predicted image of the next frame of the shooting device is predicted according to the key features, including:
[0037] S201, inputting a plurality of continuous image frames into a deep learning model, so that the deep learning model performs feature extraction on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames respectively;
[0038] S202: Determine correlation features between a plurality of consecutive image frames according to the key features, and determine a predicted image based on the correlation features.
[0039] Specifically, this example uses the acquired multiple continuous image frames as the input of the deep learning module. It can be understood that the deep learning model is a convolutional neural network model. The deep learning model includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer. The input layer inputs multiple continuous image frames with timing information, and the convolution layer convolves the multiple continuous image frames through the convolution kernel to complete the local feature extraction of the image; then the pooling layer is used to reduce the dimension and retain the estimated key features, and the fully connected layer performs the final regression task to complete the learning and prediction process of the deep learning model; finally, the output layer outputs the final predicted image.
[0040] Continuing with the above example, multiple consecutive image frames with time series information are input into the deep learning model. The deep learning model captures and learns the correlation features between consecutive image frames through the key features of the multiple consecutive image frames. During the prediction process of the deep learning model, the deep learning model predicts the next frame of the image based on the learned correlation relationship and uses it as the output result.
[0041] It can be understood that the deep learning model contains five convolutional layers and three fully connected layers. The number of neurons in the five convolutional layers are 64, 128, 256, 512, and 512 respectively. In the step where the deep learning model processes multiple continuous input image frames and extracts key features from the image frames, the deep learning model uses the ReLU activation function and the stochastic gradient descent optimizer to implement the above steps. The learning rate of the ReLU activation function and the stochastic gradient descent optimizer is 0.01, and the number of training rounds is 1000, thereby ensuring the accurate extraction of key features from the image frames.
[0042] According to the technical solution provided in the embodiment of the present application, multiple continuous image frames are input into a deep learning model, so that the deep learning model extracts features from the multiple continuous image frames to obtain key features; the correlation features between the multiple continuous image frames are determined based on the key features, and the predicted image is determined based on the correlation features, thereby achieving accurate acquisition of the predicted image, and obtaining the predicted image through the deep learning model reduces the consumption of computing resources.
[0043] In some embodiments, Figure 3 As shown, the similarity between the predicted image and the actual image is calculated, including:
[0044] S301, calculating brightness similarity values, contrast similarity values, and structure similarity values of the predicted image and the actual image;
[0045] S302: Calculate similarity based on brightness similarity value, contrast similarity value and structure similarity value.
[0046] Specifically, in order to accurately obtain the similarity between the predicted image and the actual image, this example calculates the brightness similarity value, contrast similarity value, and structural similarity value of the predicted image and the image, and then calculates the similarity based on the brightness similarity value, contrast similarity value, and structural similarity value.
[0047] The steps of calculating the brightness similarity value of the predicted image and the actual image are as follows: obtaining the brightness mean value corresponding to the predicted image, obtaining the brightness mean value of the actual image, inputting the brightness mean value corresponding to the predicted image and the brightness mean value of the actual image into a preset brightness similarity value determination formula to obtain the brightness similarity value, and the brightness similarity value determination formula is as follows:
[0048]
[0049] Among them, the above l(x, y) is the brightness similarity value, μ x is the brightness mean corresponding to the predicted image, μ y is the brightness mean value corresponding to the actual image, C1 is a constant determined according to the dynamic range corresponding to the actual image, specifically, C1=(K1×L) 2 , where K1 is a constant, L is the dynamic range corresponding to the actual image, and taking K1 as 0.01 and L as 255 as an example, C1=6.5025. It can be understood that the above C1 is set to prevent the denominator of the above brightness similarity value determination formula from being zero, thereby making the brightness similarity value calculated by the above brightness similarity value determination formula more stable and robust.
[0050] In some examples, the steps of calculating the contrast similarity value of the predicted image and the actual image are as follows: obtaining the contrast variance corresponding to the predicted image, obtaining the contrast variance of the actual image, inputting the contrast variance corresponding to the predicted image and the contrast variance of the actual image into a preset contrast similarity value determination formula to obtain the contrast similarity value, and the contrast similarity value determination formula is as follows:
[0051]
[0052] Among them, c(x, y) is the contrast similarity value, σ x is the contrast variance corresponding to the predicted image, σ y is the contrast variance corresponding to the actual image, C2 is a constant determined according to the dynamic range corresponding to the actual image, specifically, C2 = (K2 × L) 2 , where K2 is a constant, and L is the dynamic range corresponding to the actual image. Taking K2 as 0.03 and L as 255 as an example, C2=58.5225. It can be understood that the above C2 is set to prevent the denominator of the above contrast similarity value determination formula from being zero, so that the contrast similarity value calculated by the above contrast similarity value determination formula is more stable and robust.
[0053] In some examples, the steps of calculating the structural similarity value of the predicted image and the actual image are as follows: obtaining the contrast variance corresponding to the predicted image, obtaining the contrast variance of the actual image, and obtaining the covariance between the predicted image and the actual image, and inputting the contrast variance corresponding to the predicted image and the contrast variance of the actual image and the covariance between the predicted image and the actual image into a preset structural similarity value determination formula to obtain the structural similarity value, and the structural similarity value determination formula is as follows:
[0054]
[0055] Among them, the above S(x, y) is the structural similarity value, σ x is the structural variance corresponding to the predicted image, σ y is the structural variance corresponding to the actual image, σ xy is the covariance between the predicted image and the actual image, C3 is a constant determined according to C2, specifically, C3 = C2 / 2. It can be understood that the above C 32 This is a value set in order to prevent the denominator of the above structural similarity value determination formula from being zero, thereby making the structural similarity value calculated by the above structural similarity value determination formula more stable and robust.
[0056] After obtaining the brightness similarity value, contrast similarity value and structure similarity value, the similarity is calculated according to the brightness similarity value, contrast similarity value and structure similarity value, thereby accurately obtaining the similarity between the predicted image and the actual image.
[0057] According to the technical solution provided in the embodiments of the present application, the brightness similarity value, contrast similarity value and structure similarity value of the predicted image and the actual image are calculated; the similarity is calculated based on the brightness similarity value, contrast similarity value and structure similarity value, wherein the similarity between the predicted image and the actual image is comprehensively calculated using the brightness similarity value, contrast similarity value and structure similarity value, thereby improving the accuracy of the similarity between the predicted image and the actual image, thereby making the obtained similarity closer to the actual situation.
[0058] In some embodiments, Figure 4 As shown, the similarity is calculated based on the brightness similarity value, the contrast similarity value and the structure similarity value, including:
[0059] S401, taking the product of the brightness similarity value, the contrast similarity value and the structure similarity value as the similarity;
[0060] S402, obtaining weights corresponding to the brightness similarity value, the contrast similarity value and the structure similarity value respectively; and performing weighted summation on the brightness similarity value, the contrast similarity value and the structure similarity value according to the obtained weights to obtain similarity.
[0061] It can be understood that the above-mentioned step S401 and step S402 are parallel steps, rather than having a sequential relationship. The present method can determine the similarity between the predicted image and the actual image through S401 or step S402. Relevant technicians can also process the brightness similarity value, contrast similarity value and structure similarity value through other methods to obtain the similarity between the predicted image and the actual image, which will not be repeated here.
[0062] In step S401, the product of the brightness similarity value, the contrast similarity value and the structure similarity value is taken as the similarity; specifically, as follows:
[0063] SSIM(x,y)=l(x,y)·c(x,y)·s(x,y).
[0064] Among them, SSIM(x, y) is the similarity between the predicted image and the actual image, l(x, y) is the brightness similarity value, c(x, y) is the contrast similarity value, and s(x, y) is the structural similarity value.
[0065] In step S401, the weights corresponding to the brightness similarity, the contrast similarity value and the structural similarity value are obtained respectively; and the brightness similarity, the contrast similarity value and the structural similarity value are weighted and summed according to the obtained weights to obtain the similarity. For example, taking the weights corresponding to the brightness similarity, the contrast similarity value and the structural similarity value as a, b and c respectively as an example, the brightness similarity, the contrast similarity value and the structural similarity value are weighted and summed according to the obtained weights to obtain the similarity as follows:
[0066] SSIM(x,y)=a·l(x,y)+b·c(x,y)+c·s(x,y).
[0067] Among them, SSIM(x, y) is the similarity between the predicted image and the actual image, l(x, y) is the brightness similarity value, c(x, y) is the contrast similarity value, s(x, y) is the structure similarity value, a is the weight corresponding to the brightness similarity value, b is the weight corresponding to the contrast similarity value, and c is the weight corresponding to the structure similarity value; it can be understood that a, b, and c can be obtained based on historical data, and a, b, and c can also be flexibly set by relevant personnel according to actual scenarios.
[0068] In order to better understand the present method, this embodiment provides a more specific example for illustration. Figure 5As shown, the anomaly detection method provided in this example includes steps S501 to S506. In steps S501 to S506, a plurality of continuous image frames arranged in a time series taken by a shooting device are first obtained, and then the plurality of continuous image frames are input into a deep learning model to obtain a next frame prediction image, and then the similarity between the next frame prediction image and the next frame actual image taken by the shooting device is obtained based on SSIM as the standard. When the similarity is less than a similarity threshold, it is confirmed that there is an abnormality in the next frame actual image and the shooting device has an abnormality; when the similarity is not less than the similarity threshold, it is confirmed that there is no abnormality in the next frame actual image and the shooting device has no abnormality.
[0069] According to the technical solution provided in the embodiment of the present application, the product of the brightness similarity value, the contrast similarity value and the structural similarity value is used as the similarity, so that the similarity between the predicted image and the actual image is jointly determined according to the brightness similarity value, the contrast similarity value and the structural similarity value, thereby improving the stability, robustness and accuracy of the similarity between the predicted image and the actual image, so that the obtained similarity is closer to the actual situation; similarly, the weights corresponding to the brightness similarity value, the contrast similarity value and the structural similarity value are obtained respectively; and the brightness similarity value, the contrast similarity value and the structural similarity value are weightedly summed according to the obtained weights to obtain the similarity, so that the similarity between the predicted image and the actual image is jointly determined according to the brightness similarity value, the contrast similarity value and the structural similarity value, thereby improving the stability, robustness and accuracy of the similarity between the predicted image and the actual image, so that the obtained similarity is closer to the actual situation.
[0070] In some embodiments, Figure 6 As shown, after confirming that the photographing device is abnormal, the method further includes:
[0071] S601, calculating the difference between the predicted image and the actual image;
[0072] S602: When the difference does not exceed a preset difference threshold, correct the shooting device according to the predicted image.
[0073] Specifically, when there is a slight abnormality in the shooting device, the shooting device can be corrected to a certain extent. Therefore, this example first calculates the difference between the predicted image and the actual image, and compares the difference with the preset difference threshold. It can be understood that if the difference exceeds the difference threshold, it means that the shooting device cannot be corrected algorithmically; conversely, if the difference does not exceed the difference threshold, it means that the actual image captured by the shooting device is less different from the predicted image and can be corrected by the algorithm. Therefore, when the difference does not exceed the preset difference threshold, the shooting device is corrected according to the predicted image.
[0074] The specific steps for correcting the shooting device according to the predicted image are as follows: obtaining the difference points between the predicted image and the actual image (that is, the places where the predicted image and the actual image differ), then determining the compensation scheme of the shooting device according to the difference points, and finally correcting the shooting device based on the compensation scheme; illustratively, if the actual image captured by the shooting device is dark, the brightness difference between the actual image captured by the shooting device and the predicted image can be obtained, and the shooting device can be compensated for the brightness according to the brightness difference, so that the brightness of the actual image captured by the shooting device is consistent with or tends to be consistent with the brightness of the predicted image, thereby achieving correction of the shooting device.
[0075] According to the technical solution provided in the embodiment of the present application, the difference between the predicted image and the actual image is calculated; when the difference does not exceed the preset difference threshold, the shooting device is corrected according to the predicted image. The degree of abnormality of the shooting device is determined in the above manner. When the degree of abnormality of the shooting device is low, this example will correct the shooting device to ensure the normal use of the shooting device.
[0076] In some examples, such as Figure 7 As shown, after calculating the difference between the predicted image and the actual image, the method further includes:
[0077] S701, verifying the corrected shooting device, and obtaining the operation mode of the vehicle after the corrected shooting device passes the verification;
[0078] S702: When the operation mode of the vehicle is a preset target mode, an abnormality alarm is issued to the camera.
[0079] It can be understood that after the camera is corrected, in order to ensure that the corrected camera can be used normally, this example will further verify the corrected camera. Specifically, the verification method is to re-execute steps S101 to S104. If there is no abnormality in the corrected camera, it is determined that the camera has passed the verification and the camera continues to be used. Otherwise, it is determined that the camera has failed the verification and the camera cannot be corrected by the algorithm, then the camera is stopped from being used.
[0080] After the revised camera device is verified, the vehicle's operating mode is obtained, and when the vehicle's operating mode is the preset target mode, the vehicle function corresponding to the camera device is obtained, and an abnormal alarm is issued for the vehicle function, wherein the preset target mode includes one of the flameout mode, parking mode, and power-off mode. It is understandable that although the revised camera device can be used normally, due to the abnormality, in order to avoid safety hazards in subsequent use, after the vehicle is in the flameout mode, parking mode, and power-off mode, it is necessary to promptly remind the user that the camera device has an abnormality, so that the user can troubleshoot the problem in a timely manner, thereby improving driving safety.
[0081] According to the technical solution provided in the embodiment of the present application, the corrected camera is verified, and the operating mode of the vehicle is periodically obtained after the corrected camera is verified. When the operating mode of the vehicle is a preset target mode, an abnormal alarm is issued for the camera, thereby ensuring that the camera can be used normally and promptly reminding the user that the camera has an abnormality, allowing the user to troubleshoot in time, thereby improving driving safety.
[0082] In some examples, such as Figure 8 As shown, the environmental data and status data corresponding to the vehicle are obtained, including:
[0083] S801, when the difference exceeds a preset difference threshold, controlling the shooting device to stop shooting;
[0084] S802: Obtain the vehicle-mounted function corresponding to the photographing device, and issue an abnormal alarm for the vehicle-mounted function.
[0085] Specifically, when the difference exceeds the difference threshold, it indicates that the actual image captured by the shooting device is significantly different from the predicted image, and the shooting device cannot be corrected by the algorithm. Since the shooting device now has a large abnormality, the actual image captured may not reflect the real environment. Therefore, in order to prevent the shooting device from continuing to shoot without objection, this example will control the shooting device to stop shooting.
[0086] After controlling the camera to stop shooting, this example will also obtain the vehicle-mounted functions corresponding to the camera, and issue abnormal alarms for the vehicle-mounted functions; specifically, taking the case where the vehicle-mounted functions corresponding to the camera include reversing images and intelligent driving, when an abnormality occurs in the camera, the reversing image and intelligent driving cannot be used normally. At this time, it is necessary to issue an abnormal alarm for the "reversing image and intelligent driving" function abnormality, so that the user can promptly know that the vehicle has abnormal problems with "reversing image and intelligent driving", thereby avoiding safety hazards caused by the user's failure to promptly know about the abnormality of the vehicle.
[0087] It can be understood that the vehicle-mounted function corresponding to the shooting device is the vehicle-mounted function using the shooting device, that is, if any vehicle-mounted function can use the shooting device, then the vehicle-mounted function is the vehicle-mounted function corresponding to the shooting device.
[0088] According to the technical solution provided in the embodiment of the present application, when the difference exceeds a preset difference threshold, the shooting device is controlled to stop shooting; the vehicle-mounted function corresponding to the shooting device is obtained, and an abnormal alarm is issued for the vehicle-mounted function, so that the user can be informed in time that the vehicle has an abnormal problem with the vehicle function, thereby avoiding safety hazards caused by the user's inability to be informed of the abnormality of the vehicle in time.
[0089] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0090] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.
[0091] This embodiment also provides an abnormality detection device, such as Fig. 9 As shown, the device comprises:
[0092] An acquisition module 901 is used to acquire a plurality of continuous image frames arranged in a time sequence taken by a shooting device;
[0093] The prediction module 902 is used to extract features from a plurality of consecutive image frames, obtain key features corresponding to the plurality of consecutive image frames, and predict a predicted image of the next frame of the shooting device according to the key features;
[0094] The calculation module 903 is used to obtain the actual image of the next frame captured by the shooting device and calculate the similarity between the predicted image and the actual image; if the similarity does not exceed the preset similarity threshold, it is confirmed that there is an abnormality in the shooting device; if the similarity exceeds the similarity threshold, it is confirmed that there is no abnormality in the shooting device.
[0095] In some examples, the prediction module 902 is also used to input multiple continuous image frames into a deep learning model, so that the deep learning model performs feature extraction on the multiple continuous image frames to obtain key features corresponding to the multiple continuous image frames; determine the associated features between the multiple continuous image frames according to the key features, and determine the predicted image based on the associated features.
[0096] In some examples, the calculation module 903 is also used to calculate the brightness similarity value, contrast similarity value and structure similarity value between the predicted image and the actual image; and the similarity is calculated based on the brightness similarity value, contrast similarity value and structure similarity value.
[0097] In some examples, the calculation module 903 is also used to take the product of the brightness similarity value, the contrast similarity value and the structural similarity value as the similarity; or, obtain the weights corresponding to the brightness similarity value, the contrast similarity value and the structural similarity value respectively; and perform weighted summation of the brightness similarity value, the contrast similarity value and the structural similarity value according to the obtained weights to obtain the similarity.
[0098] In some examples, the calculation module 903 is further used to calculate the difference between the predicted image and the actual image; when the difference does not exceed a preset difference threshold, the shooting device is corrected according to the predicted image.
[0099] In some examples, the computing module 903 is also used to verify the corrected shooting device, and periodically obtain the vehicle's operating mode after the corrected shooting device is verified; when the vehicle's operating mode is a preset target mode, an abnormal alarm is issued for the vehicle's functions.
[0100] In some examples, the calculation module 903 is also used to control the shooting device to stop shooting when the difference exceeds a preset difference threshold; obtain the vehicle function corresponding to the shooting device, and issue an abnormal alarm for the vehicle function.
[0101] According to the technical solution provided by the embodiment of the present application, the abnormality detection device provided by the embodiment obtains a plurality of continuous image frames arranged in a time series taken by a shooting device; performs feature extraction on the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and predicts a predicted image of the next frame of the shooting device according to the key features; obtains the actual image of the next frame taken by the shooting device, and calculates the similarity between the predicted image and the actual image; if the similarity exceeds a preset similarity threshold, it is confirmed that the shooting device has an abnormality; if the similarity does not exceed the similarity threshold, it is confirmed that the shooting device has no abnormality, wherein, by obtaining a plurality of continuous image frames arranged in a time series taken by the shooting device, the predicted image of the next frame of the shooting device is predicted, and then the timing information such as the illumination change of the image taken by the shooting device is taken into account, the accuracy of the predicted image is improved, it will not be affected by the illumination conditions, and the abnormal jitter of the shooting device can be taken into account, and subsequently by determining the similarity between the predicted image containing the timing information and the actual image, it is achieved to accurately determine whether the shooting device has an abnormality, thereby avoiding the problem in the related art that it is impossible to accurately determine whether the shooting device installed on the vehicle is abnormal, resulting in a safety hazard.
[0102] Fig.10 Schematic diagram of an electronic device 10 provided in an embodiment of the present application. Fig.10As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0103] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or different components.
[0104] Processor 1001 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0105] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.
[0106] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0107] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of regional requirements and patent practice. For example, in some areas, according to regional requirements and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0108] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. An anomaly detection method, It is characterized in that The method comprises: Acquire a plurality of continuous image frames taken by a photographing device and arranged in a time sequence; Extracting features from the plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and predicting a predicted image of the next frame of the shooting device according to the key features; Acquire an actual image of a next frame captured by the shooting device, and calculate the similarity between the predicted image and the actual image; If the similarity does not exceed a preset similarity threshold, it is confirmed that there is an abnormality in the photographing device; if the similarity exceeds the similarity threshold, it is confirmed that there is no abnormality in the photographing device.
2. The method according to claim 1, It is characterized in that Extracting features from a plurality of continuous image frames to obtain key features corresponding to the plurality of continuous image frames, and predicting a predicted image of a next frame of the shooting device according to the key features, including: Inputting the plurality of continuous image frames into a deep learning model, so that the deep learning model performs feature extraction on the plurality of continuous image frames to obtain the key features corresponding to the plurality of continuous image frames respectively; The associated features between the plurality of continuous image frames are determined according to the key features, and the predicted image is determined based on the associated features.
3. The method according to claim 1, It is characterized in that Calculating the similarity between the predicted image and the actual image includes: Calculating brightness similarity values, contrast similarity values, and structure similarity values between the predicted image and the actual image; The similarity is calculated based on the brightness similarity value, the contrast similarity value and the structure similarity value.
4. The method according to claim 3, It is characterized in that The similarity is calculated according to the brightness similarity value, the contrast similarity value and the structure similarity value, including: The similarity is taken as the product of the brightness similarity value, the contrast similarity value and the structure similarity value; or, Obtain weights corresponding to the brightness similarity value, the contrast similarity value, and the structure similarity value, respectively; and perform weighted summation on the brightness similarity value, the contrast similarity value, and the structure similarity value according to the obtained weights to obtain the similarity.
5. The method according to claim 1, It is characterized in that After confirming that the photographing device is abnormal, the method further includes: Calculating the difference between the predicted image and the actual image; When the difference does not exceed a preset difference threshold, the shooting device is corrected according to the predicted image.
6. The method according to claim 5, It is characterized in that After correcting the photographing device according to the predicted image, the method further includes: Verifying the modified photographing device, and periodically acquiring the operation mode of the vehicle after the modified photographing device passes the verification; When the operation mode of the vehicle is a preset target mode, an abnormality alarm is issued to the shooting device.
7. The method according to claim 5, It is characterized in that After calculating the difference between the predicted image and the actual image, the method further includes: When the difference exceeds the preset difference threshold, controlling the shooting device to stop shooting; The vehicle-mounted function corresponding to the photographing device is obtained, and an abnormal alarm is issued for the vehicle-mounted function.
8. An abnormality detection device, It is characterized in that The device comprises: An acquisition module, used to acquire a plurality of continuous image frames arranged in time sequence taken by a shooting device; A prediction module, used to extract features from a plurality of continuous image frames, obtain key features corresponding to the plurality of continuous image frames, and predict a predicted image of a next frame of the shooting device according to the key features; A calculation module is used to obtain the actual image of the next frame taken by the shooting device, and calculate the similarity between the predicted image and the actual image; if the similarity does not exceed a preset similarity threshold, it is confirmed that there is an abnormality in the shooting device; if the similarity exceeds the similarity threshold, it is confirmed that there is no abnormality in the shooting device.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
