Camera anomaly detection method and device

Abstract

The invention provides a camera anomaly detection method and device. According to the method and the device, the image features of the images acquired at different time points are subjected to time sequence fusion analysis processing, the dynamic change condition of the image features of the images acquired at each time point in the time sequence can be captured, and the dynamic change features of stains or shelters on the camera in the time sequence can be captured, so that the image processing efficiency is improved. When detecting whether the camera is abnormal or not, not only are the image features of the images acquired at each time point utilized, but also the dynamic change condition of the image features of the images acquired at each time point in the time sequence is utilized, so that richer information can be captured to assist in detecting whether the camera is abnormal or not; the accuracy of detecting whether the camera is abnormal or not can be improved, and misjudgment is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a camera anomaly detection method and device. Background Art

[0002] With the continuous development of autonomous driving systems and advanced driver assistance systems, the vehicle's environmental perception capability has become one of the important factors affecting driving safety.

[0003] In order to achieve all-round environmental perception, vehicles usually install cameras in multiple locations, such as a front-view camera, a rear-view camera, a left-view camera, and a right-view camera.

[0004] These cameras not only help vehicles identify roads, pedestrians and obstacles, but also play an important role in parking assistance, lane keeping and blind spot monitoring.

[0005] However, since vehicles are in outdoor environments for a long time, rain, dust, dirt, snowflakes, etc. are easily stuck to the camera, causing the camera lens to be blocked, affecting the camera's field of view, and causing the quality of the image taken by the camera to decrease, which may affect the decision-making of the autonomous driving system and the advanced driver assistance system, and may bring potential driving safety hazards.

[0006] Therefore, it is necessary to detect whether the camera is blocked or dirty, and clean the camera in time if the camera is blocked or dirty.

[0007] However, how to detect whether the camera is blocked or dirty is a technical problem that needs to be solved urgently. Summary of the invention

[0008] The present application shows a camera anomaly detection method and device.

[0009] In a first aspect, the present application provides a camera anomaly detection method, the method comprising:

[0010] Acquire multiple target images that are continuous in time sequence according to multiple images that are continuously captured by the camera in time sequence;

[0011] Extract image features of each target image respectively;

[0012] According to the image features of each target image, the overall dynamic change features of multiple target images in time sequence are obtained;

[0013] Whether the camera is abnormal is determined based on the overall dynamic change characteristics of multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

[0014] In an optional implementation, there are multiple cameras, and each camera collects multiple images continuously in time sequence;

[0015] The method of acquiring a plurality of target images that are continuous in time sequence according to a plurality of images that are continuously captured by a camera in time sequence comprises:

[0016] Performing inverse perspective transformation on the images collected by each camera at the same time, respectively, to obtain images from a bird's-eye view corresponding to the images collected by each camera at the same time;

[0017] The bird's-eye view images corresponding to the images collected by the cameras at the same time are stitched together to obtain the wide-area bird's-eye view image corresponding to the same time;

[0018] A plurality of target images that are continuous in time sequence are acquired based on the wide-area bird's-eye view images corresponding to each moment.

[0019] In an optional implementation, the step of obtaining overall dynamic change characteristics of multiple target images in time sequence according to image characteristics of each target image includes:

[0020] For any two target images that are adjacent in time sequence among the plurality of target images, according to image features of the two adjacent target images, acquiring adjacent dynamic change features of the two adjacent target images in time sequence;

[0021] According to the adjacent dynamic change characteristics of each two adjacent target images in time sequence, the overall dynamic change characteristics of multiple target images in time sequence are obtained.

[0022] In an optional implementation, acquiring adjacent dynamic change features of the two adjacent target images in time sequence according to the image features of the two adjacent target images includes:

[0023] Acquire a portion of image features of the first target image in the two adjacent target images;

[0024] Acquire a portion of image features of the second target image in the two adjacent target images;

[0025] Adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to a portion of image features in the image features of the first target image and a portion of image features in the image features of the second target image.

[0026] In an optional implementation, acquiring adjacent dynamic change features of the two adjacent target images in time sequence according to a portion of image features in the image features of the first target image and a portion of image features in the image features of the second target image includes:

[0027] Splicing a portion of the image features of the first target image with a portion of the image features of the second target image to obtain a first spliced ​​feature;

[0028] The adjacent dynamic change feature of the two adjacent target images in time sequence is obtained according to the first splicing feature.

[0029] In an optional implementation, acquiring the adjacent dynamic change feature of the two adjacent target images in time sequence according to the first splicing feature includes:

[0030] Perform average pooling on the first concatenated features to obtain pooled features;

[0031] Perform feature extraction and dimensionality reduction processing on the pooled features to obtain the first dimensionality reduction feature;

[0032] The adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to the first dimensionality reduction feature.

[0033] In an optional implementation, acquiring the adjacent dynamic change feature of the two adjacent target images in time sequence according to the first splicing feature includes:

[0034] Performing feature extraction and dimensionality reduction processing on the first concatenated features to obtain second dimensionality reduction features;

[0035] Performing feature extraction and dimensionality reduction processing on the second dimensionality reduction feature to obtain a third dimensionality reduction feature;

[0036] The adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to the third dimensionality reduction feature.

[0037] In an optional implementation, the step of obtaining overall dynamic change features of multiple target images in time sequence according to adjacent dynamic change features of each two adjacent target images in time sequence includes:

[0038] Calculate the product of the adjacent dynamic change features of each two adjacent target images in time sequence to obtain a first product feature;

[0039] The overall dynamic change feature is obtained according to the first product feature.

[0040] In an optional implementation, determining whether the camera is abnormal based on the overall dynamic change characteristics of the multiple target images in time sequence includes:

[0041] Calculate the product of the overall dynamic change feature and at least part of the image features of the target image located first among the multiple target images to obtain a second product feature;

[0042] Calculate the product of the overall dynamic change feature and at least part of the image features of the target image located at the last position among the multiple target images to obtain a third product feature;

[0043] Whether the camera is abnormal is determined based on the second product feature and the third product feature.

[0044] In an optional implementation, determining whether the camera is abnormal according to the second product feature and the third product feature includes:

[0045] Concatenate the second product feature with the third product feature to obtain a second concatenated feature;

[0046] Check whether the camera is abnormal based on the second stitching feature.

[0047] In a second aspect, the present application provides a camera anomaly detection device, the device comprising:

[0048] A first acquisition module is used to acquire a plurality of target images that are continuous in time sequence according to a plurality of images that are continuously acquired in time sequence by a camera;

[0049] An extraction module, used for respectively extracting image features of each target image;

[0050] The second acquisition module is used to acquire the overall dynamic change characteristics of the multiple target images in time sequence according to the image characteristics of each target image;

[0051] The determination module is used to determine whether the camera is abnormal based on the overall dynamic change characteristics of multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

[0052] In an optional implementation, there are multiple cameras, and each camera collects multiple images continuously in time sequence;

[0053] The first acquisition module includes:

[0054] A transformation unit, used to perform inverse perspective transformation on the images collected by the cameras at the same time, to obtain images of bird's-eye view corresponding to the images collected by the cameras at the same time;

[0055] A stitching unit, used for stitching the bird's-eye view images corresponding to the images collected by the cameras at the same time, to obtain a wide-area bird's-eye view image corresponding to the same time;

[0056] The first acquisition unit is used to acquire a plurality of target images that are continuous in time sequence according to the wide-area bird's-eye view images corresponding to each moment.

[0057] In an optional implementation, the second acquisition module includes:

[0058] A second acquisition unit is used to acquire, for any two target images that are adjacent in time sequence among the plurality of target images, adjacent dynamic change features of the two target images in time sequence according to image features of the two target images;

[0059] The third acquisition unit is used to acquire the overall dynamic change characteristics of multiple target images in time sequence according to the adjacent dynamic change characteristics of each two adjacent target images in time sequence.

[0060] In an optional implementation, the second acquiring unit includes:

[0061] A first acquisition subunit, used for acquiring a part of image features of the first target image in the two adjacent target images;

[0062] A second acquisition subunit, used for acquiring a part of image features of the second target image in the two adjacent target images;

[0063] The third acquisition subunit is used to acquire adjacent dynamic change features of the two adjacent target images in time sequence according to a part of the image features of the first target image and a part of the image features of the second target image.

[0064] In an optional implementation, the third acquisition subunit is specifically used to: splice a part of the image features of the first target image with a part of the image features of the second target image to obtain a first splicing feature; and obtain adjacent dynamic change features of the two adjacent target images in time sequence based on the first splicing feature.

[0065] In an optional implementation, the third acquisition subunit is specifically used to: perform average pooling processing on the first splicing feature to obtain a pooling feature; perform feature extraction processing and dimensionality reduction processing on the pooling feature to obtain a first reduced dimensionality feature; and obtain the adjacent dynamic change features of the two adjacent target images in time series based on the first reduced dimensionality feature.

[0066] In an optional implementation, the third acquisition subunit is specifically used to: perform feature extraction and dimensionality reduction processing on the first splicing feature to obtain a second dimensionality reduction feature; perform feature extraction and dimensionality reduction processing on the second dimensionality reduction feature to obtain a third dimensionality reduction feature; and obtain the adjacent dynamic change features of the two adjacent target images in time series based on the third dimensionality reduction feature.

[0067] In an optional implementation, the third acquisition unit includes:

[0068] A calculation subunit, used for calculating the product between the adjacent dynamic change features of each two adjacent target images in time sequence to obtain a first product feature;

[0069] The fourth acquisition subunit is used to acquire the overall dynamic change feature according to the first product feature.

[0070] In an optional implementation, the determining module includes:

[0071] A first calculation unit is used to calculate the product of the overall dynamic change feature and at least part of the image features of the target image located first among the multiple target images to obtain a second product feature;

[0072] A second calculation unit is used to calculate the product of the overall dynamic change feature and at least part of the image features of the target image located at the last position among the multiple target images to obtain a third product feature;

[0073] A determination unit is used to determine whether the camera is abnormal based on the second product feature and the third product feature.

[0074] In an optional implementation, the determining unit includes:

[0075] A concatenation subunit, used for concatenating the second product feature with the third product feature to obtain a second concatenation feature;

[0076] The determination subunit is used to determine whether the camera is abnormal according to the second stitching feature.

[0077] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0078] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0079] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0080] The technical solution provided by this application may have the following beneficial effects:

[0081] In the present application, based on multiple images continuously captured by a camera in time sequence, multiple target images continuous in time sequence are obtained; image features of each target image are extracted respectively; based on the image features of each target image, overall dynamic change features of the multiple target images in time sequence are obtained; based on the overall dynamic change features of the multiple target images in time sequence, it is determined whether the camera is abnormal, and camera abnormalities include that the camera is dirty and / or the camera is blocked.

[0082] Through the present application, the image features of images collected at different time points are subjected to time-series fusion analysis processing, which can capture the dynamic changes of the image features of the images collected at each time point in the time series, as well as the dynamic changes of stains or obstructions on the camera in the time series. In this way, when detecting whether the camera is abnormal, not only the image features of the images collected at each time point are utilized, but also the dynamic changes of the image features of the images collected at each time point in the time series are utilized, so that richer information can be captured to assist in detecting whether the camera is abnormal, which can improve the accuracy of detecting whether the camera is abnormal and reduce misjudgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flowchart of the steps of a camera anomaly detection method of the present application.

[0084] Figure 2 This is a flowchart of the steps of a method for obtaining overall dynamic change characteristics of the present application.

[0085] Figure 3 This is a flowchart of the steps of a camera anomaly detection method of the present application.

[0086] Figure 4 It is a schematic diagram of a camera anomaly detection method of the present application.

[0087] Figure 5 It is a schematic diagram of a camera anomaly detection method of the present application.

[0088] Figure 6 It is a structural block diagram of a camera anomaly detection device of the present application.

[0089] Figure 7 It is a block diagram of an electronic device of the present application.

[0090] Figure 8 It is a block diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0092] Reference Figure 1 , shows a flowchart of the steps of a camera abnormality detection method of the present application, which can be applied to electronic devices, wherein the method specifically may include the following steps:

[0093] In step S101, a plurality of target images that are continuous in time sequence are acquired according to a plurality of images that are continuously captured by a camera in time sequence.

[0094] The camera can continuously capture images in time sequence, thereby obtaining multiple images continuously captured by the camera in time sequence, and transmit the multiple images continuously captured by the camera in time sequence to the electronic device in real time, so that the electronic device can obtain the multiple images continuously captured by the camera in time sequence in real time.

[0095] In the present application, the camera may be a camera located on the vehicle, for example, a front view camera located at the front of the vehicle, a rear view camera located at the parking space of the vehicle, a left view camera located on the left side of the vehicle, and a right view camera located on the right side of the vehicle, etc. In the case where the electronic device is a vehicle-mounted host, the camera may transmit multiple images continuously captured in time sequence to the vehicle-mounted host in real time, so that the vehicle-mounted host obtains multiple images continuously captured in time sequence by the camera in real time.

[0096] In one embodiment, there is only one camera, and the electronic device can directly use multiple images captured continuously in time sequence by the camera as multiple target images that are continuous in time sequence.

[0097] Alternatively, in another embodiment, there are multiple cameras, and for any one of the multiple cameras, the camera continuously captures multiple images in time sequence, and the multiple images continuously captured by the camera in time sequence can be directly used as multiple target images continuously in time sequence, and the process of steps S102 to S104 is performed separately for the multiple images continuously captured by the camera in time sequence. The same is true for each of the other cameras in the multiple cameras.

[0098] In one embodiment, the multiple images in this step are at least 3 images, for example, 3 images, 4 images or 5 images. The specific number of images can be determined according to actual conditions, and this application does not limit this.

[0099] Alternatively, in another embodiment, there are multiple cameras (such as the multiple cameras on the vehicle mentioned above), and each camera continuously captures multiple images in a time sequence.

[0100] Each camera captures an image at the same time.

[0101] In this way, for any moment when an image is captured, the moment is regarded as the same moment, and the images captured by each camera at the same moment can be subjected to inverse perspective mapping (IPM) to obtain the bird's-eye view images corresponding to the images captured by each camera at the same moment, which is equivalent to converting the perspective of the images captured by each camera at the same moment into a bird's-eye view, and then stitching the bird's-eye view images corresponding to the images captured by each camera at the same moment to obtain the wide-area bird's-eye view images corresponding to the same moment. In one example, an AVM (Around View Monitor) can be used to stitch the bird's-eye view images corresponding to the images captured by each camera at the same moment, so that the edges of the images are seamlessly connected to avoid obvious gaps or distortion in the stitching area.

[0102] In an example, if the multiple cameras are a front-view camera located at the front of the vehicle, a rear-view camera located at the parking space of the vehicle, a left-view camera located on the left side of the vehicle, and a right-view camera located on the right side of the vehicle, then the wide-area bird's-eye view image corresponding to the same moment can cover the entire appearance of the vehicle's surrounding environment, help eliminate the perspective differences between the cameras, and improve the spatial consistency of the image, providing a more accurate basis for subsequent feature extraction.

[0103] The same is true for every other moment when an image is acquired (other same moment).

[0104] Then, multiple target images that are continuous in time sequence may be acquired according to the wide-area bird's-eye view images corresponding to each moment. For example, the wide-area bird's-eye view images corresponding to each moment may be directly determined as multiple target images that are continuous in time sequence.

[0105] It should be noted that the camera of the present application can be a high-resolution camera or a wide-angle camera, which can capture clear images under different lighting conditions. All cameras are connected to the central processing unit (CPU) in the electronic device through a high-speed data bus to ensure real-time data transmission and processing.

[0106] Secondly, before executing step S102 on the image captured by the camera, or before performing inverse perspective transformation on the image captured by the camera, the image captured by the camera can also be preprocessed, including image denoising, color enhancement, contrast adjustment and distortion correction, etc., in order to improve image quality and improve subsequent detection accuracy.

[0107] In step S102, image features of each target image are extracted respectively.

[0108] The purpose of this step is to extract high-dimensional features in the target image that can characterize stains or occlusions on the camera.

[0109] In one embodiment of the present application, for any one of the multiple target images, a convolutional neural network (CNN) can be used to extract features of the target image. For example, ResNet-18 (18-layer deep Residual Network) is used to extract features of the target image. The extracted image features include but are not limited to edge features, texture features, brightness distribution features, color distribution features, and depth features. These image features are the basis for determining whether the camera is abnormal.

[0110] In step S103, overall dynamic change characteristics of the multiple target images in time sequence are obtained according to the image characteristics of each target image.

[0111] The overall dynamic change characteristics of multiple target images in time series can reflect the dynamic change characteristics of stains or obstructions on the camera in time series to assist in detecting whether the camera is abnormal, which can improve the accuracy of detecting whether the camera is abnormal and reduce misjudgment.

[0112] This step can be referred to in the embodiment shown later, and will not be described in detail here.

[0113] In step S104, it is determined whether the camera is abnormal based on the overall dynamic change characteristics of the multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

[0114] The camera being blocked includes that a part of the camera area is blocked, or the entire camera area is blocked.

[0115] In one embodiment of the present application, when a camera abnormality is determined, an alarm signal can be output, for example, the alarm device on the vehicle is controlled to output an alarm signal (for example, through an audible or visual alarm or through an alarm light on the vehicle's central control display) to prompt the driver to clean the dirt and / or obstructions on the camera.

[0116] Alternatively, when there is also a cleaning device (such as a water sprayer) for automatically cleaning the camera, the cleaning device can be automatically controlled to remove stains or obstructions on the camera to ensure that the camera works normally.

[0117] In the present application, based on multiple images continuously captured by a camera in time sequence, multiple target images continuous in time sequence are obtained; image features of each target image are extracted respectively; based on the image features of each target image, overall dynamic change features of the multiple target images in time sequence are obtained; based on the overall dynamic change features of the multiple target images in time sequence, it is determined whether the camera is abnormal, and camera abnormalities include that the camera is dirty and / or the camera is blocked.

[0118] Through the present application, the image features of images collected at different time points are subjected to time-series fusion analysis processing, which can capture the dynamic changes of the image features of the images collected at each time point in the time series, as well as the dynamic changes of stains or obstructions on the camera in the time series. In this way, when detecting whether the camera is abnormal, not only the image features of the images collected at each time point are utilized, but also the dynamic changes of the image features of the images collected at each time point in the time series are utilized, so that richer information can be captured to assist in detecting whether the camera is abnormal, which can improve the accuracy of detecting whether the camera is abnormal and reduce misjudgments.

[0119] In one embodiment of the application, see Figure 2 , step S1013 includes:

[0120] In step S201, for any two target images that are adjacent in time sequence among a plurality of target images, adjacent dynamic change features of the two target images in time sequence are obtained according to image features of the two target images.

[0121] The same is true for other two target images that are adjacent in time sequence among the multiple target images.

[0122] In one embodiment of the present application, this step can be implemented through the following process, including:

[0123] 2011. Obtain a portion of image features of the first target image in the two adjacent target images.

[0124] The first target image is one of the two adjacent target images.

[0125] The image features of the first target image may be a three-dimensional matrix, and the three-dimensional matrix has width, height, and depth. A portion of the image features in the image features of the first target image is a portion of the three-dimensional matrix of the image features of the first target image. For example, the three-dimensional matrix of the image features of the first target image is divided into two halves in the depth direction, and one half of the three-dimensional matrix can be regarded as a portion of the image features in the image features of the first target image.

[0126] 2012. Obtain a portion of image features of the second target image in the two adjacent target images.

[0127] The second target image is one of the two adjacent target images.

[0128] The image features of the second target image may be a three-dimensional matrix, and the three-dimensional matrix has width, height, and depth. A portion of the image features in the image features of the second target image is a portion of the three-dimensional matrix of the image features of the second target image. For example, the three-dimensional matrix of the image features of the second target image is divided into two halves in the depth direction, and one half of the three-dimensional matrix can be regarded as a portion of the image features in the image features of the second target image.

[0129] 2013. Acquire adjacent dynamic change features of the two adjacent target images in time sequence according to a portion of image features in the image features of the first target image and a portion of image features in the image features of the second target image.

[0130] In one embodiment of the present application, this step can be implemented through the following process, including:

[0131] 11) Splicing a portion of the image features of the first target image with a portion of the image features of the second target image to obtain a first spliced ​​feature.

[0132] For example, a part of the image features of the first target image is a three-dimensional matrix, and a part of the image features of the second target image is a three-dimensional matrix. The three-dimensional matrix has width, height, and depth. A part of the image features of the first target image and a part of the image features of the second target image can be superimposed in the depth direction to obtain a first splicing feature.

[0133] For example, the image feature of the first target image is a three-dimensional matrix C*H*W, where H is the dimension of height, W is the dimension of width, and C is the dimension of depth.

[0134] A portion of the image features of the first target image is a three-dimensional matrix C / 2*H*W, and the depth dimension is halved.

[0135] The image feature of the second target image is a three-dimensional matrix C*H*W, where H is the dimension of height, W is the dimension of width, and C is the dimension of depth.

[0136] A portion of the image features of the second target image is a three-dimensional matrix C / 2*H*W, and the depth dimension is halved.

[0137] A portion of the image features C / 2*H*W in the image features of the first target image may be superimposed with a portion of the image features C / 2*H*W in the image features of the second target image in the depth direction to obtain a three-dimensional matrix C*H*W as the first splicing feature.

[0138] 12) Acquire the adjacent dynamic change features of the two adjacent target images in time sequence according to the first splicing feature.

[0139] In one embodiment, the first concatenated feature may be subjected to average pooling to obtain a pooled feature. For example, the first concatenated feature is input into a global average pooling layer (GAP), so that the global average pooling layer performs average pooling on the first concatenated feature and outputs the pooled feature.

[0140] Then, feature extraction and dimensionality reduction are performed on the pooled features to obtain the first dimensionality reduction features.

[0141] For example, the pooling feature is input into the convolutional layer (Conv). In one example, the convolutional layer (Conv) can be a convolutional layer with kernel=1, etc., so that the convolutional layer processes the pooling feature, and the result after the convolutional layer is processed is input into the batch normalization layer (BN, Batch Normalization), so that the batch normalization layer processes the result after the convolutional layer is processed, and the result after the batch normalization layer is processed is input into the ReLU activation function, so that the ReLU activation function processes the result after the batch normalization layer is processed, and the output result is obtained, and the output result is the first dimensionality reduction feature.

[0142] Afterwards, the adjacent dynamic change feature of the two adjacent target images in time sequence is obtained according to the first dimensionality reduction feature. For example, the first dimensionality reduction feature is directly determined as the adjacent dynamic change feature of the two adjacent target images in time sequence.

[0143] In another embodiment, feature extraction processing and dimensionality reduction processing may be performed on the first concatenated feature to obtain a second dimensionality reduction feature.

[0144] For example, the first splicing feature is input into the convolution layer (Conv). In one example, the convolution layer (Conv) can be a convolution layer with kernel=3, etc., so that the convolution layer processes the first splicing feature, and the result after the convolution layer is processed is input into the batch normalization layer, so that the batch normalization layer processes the result after the convolution layer is processed, and the result after the batch normalization layer is processed is input into the ReLU activation function, so that the ReLU activation function processes the result after the batch normalization layer is processed to obtain an output result, which is the second dimensionality reduction feature.

[0145] Then, feature extraction and dimension reduction are performed on the second dimension reduction feature to obtain a third dimension reduction feature.

[0146] For example, the second dimensionality reduction feature is input into the convolution layer (Conv). In one example, the convolution layer (Conv) can be a convolution layer with kernel=3, etc., so that the convolution layer processes the second dimensionality reduction feature, and the result after the convolution layer is processed is input into the batch normalization layer, so that the batch normalization layer processes the result after the convolution layer is processed, and the result after the batch normalization layer is processed is input into the ReLU activation function, so that the ReLU activation function processes the result after the batch normalization layer is processed to obtain an output result, which is the third dimensionality reduction feature.

[0147] Then, the adjacent dynamic change feature of the two adjacent target images in time sequence is obtained according to the third dimensionality reduction feature. For example, the third dimensionality reduction feature is directly determined as the adjacent dynamic change feature of the two adjacent target images in time sequence.

[0148] In step S202, the overall dynamic change characteristics of multiple target images in time sequence are obtained according to the adjacent dynamic change characteristics of each two adjacent target images in time sequence.

[0149] In one embodiment of the present application, this step can be implemented through the following process, including:

[0150] 2021. Calculate the product of adjacent dynamic change features of each two adjacent target images in time series to obtain a first product feature.

[0151] In one embodiment, it is assumed that there are three target images, namely target image t-2, target image t-1 and target image t.

[0152] The target image t-2 is earlier than the target image t-1 in time sequence, and the target image t-2 and the target image t-1 are adjacent in time sequence. The target image t-1 is earlier than the target image t in time sequence, and the target image t-1 and the target image t are adjacent in time sequence.

[0153] Then, we can obtain the adjacent dynamic change feature 1 between the target image t-1 and the target image t in time sequence, and the adjacent dynamic change feature 2 between the target image t-2 and the target image t-1 in time sequence.

[0154] In this way, the product between the adjacent dynamic change feature 1 and the adjacent dynamic change feature 2 can be calculated and used as the first product feature.

[0155] In another embodiment, it is assumed that there are four target images, namely target image t-3, target image t-2, target image t-1 and target image t.

[0156] Target image t-3 is earlier than target image t-2 in time sequence, target image t-3 and target image t-2 are adjacent in time sequence, target image t-2 is earlier than target image t-1 in time sequence, target image t-2 and target image t-1 are adjacent in time sequence, target image t-1 is earlier than target image t in time sequence, and target image t-1 and target image t are adjacent in time sequence.

[0157] Then we can obtain the adjacent dynamic change feature 1 between the target image t-1 and the target image t in time sequence, the adjacent dynamic change feature 2 between the target image t-2 and the target image t-1 in time sequence, and the adjacent dynamic change feature 3 between the target image t-3 and the target image t-2 in time sequence.

[0158] In this way, the product between the adjacent dynamic change feature 1 and the adjacent dynamic change feature 2 can be calculated, and then the product between the product and the adjacent dynamic change feature 3 can be calculated and used as the first product feature.

[0159] Assuming that the number of target images is 5, 6, 7 or more, examples will not be given one by one.

[0160] 2021. Obtain the overall dynamic change feature based on the first product feature.

[0161] In one embodiment, the first product feature may be directly used as the overall dynamic change feature.

[0162] Alternatively, in another embodiment, the first product feature can be input into a Sigmoid function so that the Sigmoid function processes the first product feature to map the values ​​in the first product feature to the interval (0, 1), and then the result output by the Sigmoid function is used as the overall dynamic change feature.

[0163] In another embodiment of the present application, see Figure 3 , step S104 includes:

[0164] In step S301, the product of the overall dynamic change feature and at least part of the image features of the first target image among the multiple target images is calculated to obtain a second product feature.

[0165] There is a temporal sequence between the multiple target images.

[0166] In one embodiment, the product of the overall dynamic change feature and all image features in the image features of the target image located at the first position may be calculated and used as the second product feature.

[0167] Alternatively, in another embodiment, the product of the overall dynamic change feature and some image features in the image features of the target image that is at the first position may be calculated and used as the second product feature.

[0168] In this case, for the target image located at the first position, the target image located at the first position has an adjacent image, and the adjacent image is after the target image located at the first position in terms of time sequence.

[0169] In step 2011, the first target image among the two adjacent target images may be the target image located in the first position, and a part of the image features of the image features of the target image located in the first position is obtained. Thus, the "part of the image features of the image features of the target image located in the first position" in step S301 may be: another part of the image features of the image features of the target image located in the first position except a part of the image features of the image features of the target image located in the first position obtained in step 2011.

[0170] In step S302, the product of the overall dynamic change feature and at least part of the image features of the last target image among the multiple target images is calculated to obtain a third product feature.

[0171] There is a temporal sequence between the multiple target images.

[0172] In one embodiment, the product of the overall dynamic change feature and all the image features in the image features of the target image located at the last position may be calculated and used as the second product feature.

[0173] Alternatively, in another embodiment, the product of the overall dynamic change feature and a portion of the image features of the target image located at the last position may be calculated and used as the second product feature.

[0174] For the target image at the last position, the target image at the last position has an adjacent image, and the adjacent image precedes the target image at the last position in time sequence.

[0175] In step 2012, the second target image among the two adjacent target images may be the last target image, and a portion of the image features of the last target image is obtained. Thus, the "partial image features in the image features of the last target image" in step S302 may be: another portion of the image features in the image features of the last target image other than a portion of the image features in the image features of the last target image obtained in step 2012.

[0176] Among them, step S301 and step S302 can be executed synchronously.

[0177] In step S303, it is determined whether the camera is abnormal based on the second product feature and the third product feature.

[0178] In one embodiment of the present application, this step can be implemented through the following process, including:

[0179] 3031. Concatenate the second product feature and the third product feature to obtain a second concatenated feature.

[0180] For example, the second product feature is a three-dimensional matrix, the third product feature is a three-dimensional matrix, the three-dimensional matrix has width, height and depth, and the second product feature and the third product feature can be superimposed in the depth direction to obtain the second splicing feature. The second product feature and the third product feature have the same dimension in the width direction, the second product feature and the third product feature have the same dimension in the height direction, and the second product feature and the third product feature can have the same dimension in the depth direction.

[0181] For example, the second product feature is a three-dimensional matrix C / 2*H*W, where H is the dimension of height, W is the dimension of width, and C is the dimension of depth.

[0182] The image feature of the third product feature is a three-dimensional matrix C / 2*H*W, where H is the height dimension, W is the width dimension, and C is the depth dimension.

[0183] The second product feature C / 2*H*W and the third product feature C / 2*H*W may be superimposed in the depth direction to obtain a three-dimensional matrix C*H*W, which is used as the second splicing feature.

[0184] 3031. Determine whether the camera is abnormal based on the second stitching feature.

[0185] In one embodiment of the present application, the second splicing feature can be input into the fully connected layer so that the fully connected layer processes the second splicing feature, and the result of the processing by the fully connected layer is input into the Sigmoid function so that the Sigmoid function processes the result of the processing by the fully connected layer to obtain the probability of camera abnormality, and output the probability of camera abnormality.

[0186] When the probability of camera abnormality is greater than a preset threshold, it can be determined that the camera is abnormal, for example, it can be determined that the camera is dirty or the camera is blocked. Or, when the probability of camera abnormality is less than or equal to the preset threshold, it can be determined that the camera is normal, for example, it can be determined that the camera is not dirty or the camera is not blocked.

[0187] See also Figure 4 , an example is used to illustrate the scheme of the present application, but it is not intended to limit the scope of protection of the present application.

[0188] For example, it is assumed that the target images acquired in step S101 include target image t-2, target image t-1, and target image t. Target image t-2 is earlier than target image t-1 in time sequence, target image t-2 and target image t-1 are adjacent in time sequence, target image t-1 is earlier than target image t in time sequence, and target image t-1 and target image t are adjacent in time sequence.

[0189] The image feature of the target image t-2 acquired through step S102 is image feature t-2, the image feature of the target image t-1 acquired is image feature t-1, and the image feature of the target image t acquired is image feature t.

[0190] The image feature t-2 may be a three-dimensional matrix C*H*W(1).

[0191] The image feature t-1 may be a three-dimensional matrix C*H*W(2).

[0192] The image feature t can be a three-dimensional matrix C*H*W(3).

[0193] H is the height dimension of the three-dimensional matrix, W is the width dimension of the three-dimensional matrix, and C is the thickness dimension of the three-dimensional matrix.

[0194] In the direction of the depth dimension of the three-dimensional matrix, the image feature t-2C*H*W(1) is split into two three-dimensional matrices of the same dimension, namely image feature C / 2*H*W(1) and image feature C / 2*H*W(2). The depth of image feature C / 2*H*W(1) and image feature C / 2*H*W(2) is half the depth of C*H*W(1).

[0195] In the direction of the depth dimension of the three-dimensional matrix, the image feature t-1C*H*W(2) is split into two three-dimensional matrices of the same dimension, namely image feature C / 2*H*W(3) and image feature C / 2*H*W(4). The depth of image feature C / 2*H*W(3) and image feature C / 2*H*W(4) is half the depth of C*H*W(2).

[0196] In the direction of the depth dimension of the three-dimensional matrix, the image feature tC*H*W(3) is split into two three-dimensional matrices of the same dimension, namely image feature C / 2*H*W(5) and image feature C / 2*H*W(6). The depth of image feature C / 2*H*W(5) and image feature C / 2*H*W(6) is half the depth of C*H*W(3).

[0197] The image feature C / 2*H*W(2) is concatenated with the image feature C / 2*H*W(3) in the direction of the depth dimension to obtain a concatenated feature C*H*W(4).

[0198] The concatenated feature C*H*W(4) is input into the average pooling layer. The average pooling layer processes the concatenated feature C*H*W(4) to obtain the pooling feature C*1*1. The matrix C*1*1 has only one value in both the width dimension and the height dimension.

[0199] The pooled feature C*1*1 is processed through the convolution layer, batch normalization layer and ReLU activation function in sequence to obtain the reduced dimension feature C / 2*1*1.

[0200] The image feature C / 2*H*W(4) is concatenated with the image feature C / 2*H*W(5) in the direction of the depth dimension to obtain a concatenated feature C*H*W(5).

[0201] The concatenated feature C*H*W(5) is processed in turn through a convolution layer, a batch normalization layer, and a ReLU activation function, and then processed again through another convolution layer, another batch normalization layer, and another ReLU activation function to obtain a reduced dimension feature C / 2*H*W(7).

[0202] Multiply the reduced dimension feature C / 2*1*1 by the reduced dimension feature C / 2*H*W(7) to obtain the product feature C / 2*H*W(8).

[0203] The product feature C / 2*H*W(8) is input into the Sigmoid function so that the Sigmoid function processes the product feature C / 2*H*W(8) to obtain a processed feature C / 2*H*W(9).

[0204] The image feature C / 2*H*W(1) is multiplied by the processed feature C / 2*H*W(9) to obtain the product feature C / 2*H*W(10).

[0205] The image feature C / 2*H*W(6) is multiplied by the processed feature C / 2*H*W(9) to obtain the product feature C / 2*H*W(11).

[0206] The product feature C / 2*H*W(10) and the product feature C / 2*H*W(11) are concatenated in the depth dimension to obtain the concatenated feature C*H*W(6).

[0207] The splicing feature C*H*W(6) is input into the fully connected layer so that the fully connected layer processes the splicing feature C*H*W(6), and the processing result of the fully connected layer is input into the Sigmoid function so that the Sigmoid function processes the processing result of the fully connected layer to obtain the probability of camera abnormality.

[0208] In one embodiment of the present application, by Figure 1 to Figure 3 In the embodiment shown in the figure, the probability of whether the camera is abnormal at a certain moment can be obtained. However, sometimes the image has noise due to accidental factors, which leads to the Figure 1 to Figure 3 The probability of whether the camera is abnormal at a certain moment obtained in the embodiment shown is inaccurate, which may lead to inaccurate detection results of whether the camera is abnormal that are finally obtained.

[0209] To this end, in another embodiment of the present application, multiple images collected by the camera are transmitted through Figure 1 to Figure 3 After obtaining the probability of whether the camera is abnormal at a certain moment in the embodiment shown in the figure, the camera can also be used to collect multiple images to obtain the probability of abnormality. Figure 1 to Figure 3 The embodiment shown in the figure obtains the probability of whether the camera is abnormal at another time, and can also continue to use other multiple images collected by the camera to obtain the probability of whether the camera is abnormal at another time. Figure 1 to Figure 3 The embodiment shown obtains the probability of whether the camera is abnormal at other times, and then calculates the average value of the obtained multiple probabilities.

[0210] For example, the present application can obtain the probability of whether the camera is abnormal at at least two moments, and then calculate the average value of the at least two probabilities obtained.

[0211] For example, by using the images t-2, t-1 and t collected by the camera, Figure 1 to Figure 3 After obtaining the probability of whether the camera is abnormal at time t, the embodiment shown in the figure can also use the image t-1, image t and image t+1 collected by the camera to obtain the probability of whether the camera is abnormal at time t. Figure 1 to Figure 3 The illustrated embodiment obtains the probability of whether the camera is abnormal at time t+2.

[0212] Later, the image t, image t+1 and image t+2 collected by the camera can be used to Figure 1 to Figure 3 The illustrated embodiment obtains the probability of whether the camera is abnormal at time t+2, etc. ... and also obtains the probability of whether the camera is abnormal at time t+N, etc.

[0213] Then calculate the average between the probability of whether the camera is abnormal at time t... and the probability of whether the camera is abnormal at time t+N.

[0214] When the average value is greater than the preset threshold, it can be determined that the camera is abnormal, for example, the camera is dirty or blocked. Alternatively, when the average value is less than or equal to the preset threshold, it can be determined that the camera is normal, for example, the camera is not dirty or blocked.

[0215] Through this embodiment, a sliding window filtering processing mechanism is adopted to smooth the image features of images within a period of time, suppress / filter out the influence of short-term random noise, and improve the accuracy of the detection result of whether the camera is abnormal.

[0216] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0217] Reference Figure 6 , shows a structural block diagram of a camera anomaly detection device of the present application, the device comprising:

[0218] A first acquisition module 11 is used to acquire a plurality of target images that are continuous in time sequence according to a plurality of images that are continuously acquired in time sequence by a camera;

[0219] An extraction module 12, used to extract image features of each target image respectively;

[0220] The second acquisition module 13 is used to acquire the overall dynamic change characteristics of the multiple target images in time sequence according to the image characteristics of each target image;

[0221] The determination module 14 is used to determine whether the camera is abnormal according to the overall dynamic change characteristics of the multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

[0222] In an optional implementation, there are multiple cameras, and each camera collects multiple images continuously in time sequence;

[0223] The first acquisition module includes:

[0224] A transformation unit, used to perform inverse perspective transformation on the images collected by the cameras at the same time, to obtain images of bird's-eye view corresponding to the images collected by the cameras at the same time;

[0225] A stitching unit, used for stitching the bird's-eye view images corresponding to the images collected by the cameras at the same time, to obtain a wide-area bird's-eye view image corresponding to the same time;

[0226] The first acquisition unit is used to acquire a plurality of target images that are continuous in time sequence according to the wide-area bird's-eye view images corresponding to each moment.

[0227] In an optional implementation, the second acquisition module includes:

[0228] A second acquisition unit is used to acquire, for any two target images that are adjacent in time sequence among the plurality of target images, adjacent dynamic change features of the two target images in time sequence according to image features of the two target images;

[0229] The third acquisition unit is used to acquire the overall dynamic change characteristics of multiple target images in time sequence according to the adjacent dynamic change characteristics of each two adjacent target images in time sequence.

[0230] In an optional implementation, the second acquiring unit includes:

[0231] A first acquisition subunit, used for acquiring a part of image features of the first target image in the two adjacent target images;

[0232] A second acquisition subunit, used for acquiring a part of image features of the second target image in the two adjacent target images;

[0233] The third acquisition subunit is used to acquire adjacent dynamic change features of the two adjacent target images in time sequence according to a part of the image features of the first target image and a part of the image features of the second target image.

[0234] In an optional implementation, the third acquisition subunit is specifically used to: splice a part of the image features of the first target image with a part of the image features of the second target image to obtain a first splicing feature; and obtain adjacent dynamic change features of the two adjacent target images in time sequence based on the first splicing feature.

[0235] In an optional implementation, the third acquisition subunit is specifically used to: perform average pooling processing on the first splicing feature to obtain a pooling feature; perform feature extraction processing and dimensionality reduction processing on the pooling feature to obtain a first reduced dimensionality feature; and obtain the adjacent dynamic change features of the two adjacent target images in time series based on the first reduced dimensionality feature.

[0236] In an optional implementation, the third acquisition subunit is specifically used to: perform feature extraction and dimensionality reduction processing on the first splicing feature to obtain a second dimensionality reduction feature; perform feature extraction and dimensionality reduction processing on the second dimensionality reduction feature to obtain a third dimensionality reduction feature; and obtain the adjacent dynamic change features of the two adjacent target images in time series based on the third dimensionality reduction feature.

[0237] In an optional implementation, the third acquisition unit includes:

[0238] A calculation subunit, used for calculating the product between the adjacent dynamic change features of each two adjacent target images in time sequence to obtain a first product feature;

[0239] The fourth acquisition subunit is used to acquire the overall dynamic change feature according to the first product feature.

[0240] In an optional implementation, the determining module includes:

[0241] A first calculation unit is used to calculate the product of the overall dynamic change feature and at least part of the image features of the target image located first among the multiple target images to obtain a second product feature;

[0242] A second calculation unit is used to calculate the product of the overall dynamic change feature and at least part of the image features of the target image located at the last position among the multiple target images to obtain a third product feature;

[0243] A determination unit is used to determine whether the camera is abnormal based on the second product feature and the third product feature.

[0244] In an optional implementation, the determining unit includes:

[0245] A concatenation subunit, used for concatenating the second product feature with the third product feature to obtain a second concatenation feature;

[0246] The determination subunit is used to determine whether the camera is abnormal according to the second stitching feature.

[0247] In the present application, based on multiple images continuously captured by a camera in time sequence, multiple target images continuous in time sequence are obtained; image features of each target image are extracted respectively; based on the image features of each target image, overall dynamic change features of the multiple target images in time sequence are obtained; based on the overall dynamic change features of the multiple target images in time sequence, it is determined whether the camera is abnormal, and camera abnormalities include that the camera is dirty and / or the camera is blocked.

[0248] Through the present application, the image features of images collected at different time points are subjected to time-series fusion analysis processing, which can capture the dynamic changes of the image features of the images collected at each time point in the time series, as well as the dynamic changes of stains or obstructions on the camera in the time series. In this way, when detecting whether the camera is abnormal, not only the image features of the images collected at each time point are utilized, but also the dynamic changes of the image features of the images collected at each time point in the time series are utilized, so that richer information can be captured to assist in detecting whether the camera is abnormal, which can improve the accuracy of detecting whether the camera is abnormal and reduce misjudgments.

[0249] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0250] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0251] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0252] Figure 7 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0253] Reference Figure 7, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0254] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0255] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0256] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0257] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0258] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0259] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0260] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0261] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0262] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0263] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0264] Figure 8 1 is a block diagram of an electronic device 1900 shown in the present application. For example, the electronic device 1900 may be provided as a server.

[0265] Reference Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0266] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0267] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0268] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0269] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0270] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0271] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0272] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0273] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0274] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0275] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0276] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A camera anomaly detection method, characterized in that: The method comprises: Acquire multiple target images that are continuous in time sequence according to multiple images that are continuously captured by the camera in time sequence; Extract image features of each target image respectively; According to the image features of each target image, the overall dynamic change features of multiple target images in time sequence are obtained; Whether the camera is abnormal is determined based on the overall dynamic change characteristics of multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

2. The method according to claim 1, characterized in that: There are multiple cameras, and each camera collects multiple images continuously in time sequence; The method of acquiring a plurality of target images that are continuous in time sequence according to a plurality of images that are continuously captured by a camera in time sequence comprises: Performing inverse perspective transformation on the images collected by each camera at the same time, respectively, to obtain images from a bird's-eye view corresponding to the images collected by each camera at the same time; The bird's-eye view images corresponding to the images collected by the cameras at the same time are stitched together to obtain the wide-area bird's-eye view image corresponding to the same time; A plurality of target images that are continuous in time sequence are acquired based on the wide-area bird's-eye view images corresponding to each moment.

3. The method according to claim 1, characterized in that: The step of obtaining overall dynamic change characteristics of multiple target images in time sequence according to the image characteristics of each target image includes: For any two target images that are adjacent in time sequence among the plurality of target images, according to image features of the two adjacent target images, adjacent dynamic change features of the two adjacent target images in time sequence are acquired; According to the adjacent dynamic change characteristics of each two adjacent target images in time sequence, the overall dynamic change characteristics of multiple target images in time sequence are obtained.

4. The method according to claim 3, characterized in that The step of acquiring adjacent dynamic change features of the two adjacent target images in time sequence according to the image features of the two adjacent target images includes: Acquire a portion of image features of the first target image in the two adjacent target images; Acquire a portion of image features of the second target image in the two adjacent target images; Adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to a portion of image features in the image features of the first target image and a portion of image features in the image features of the second target image.

5. The method according to claim 4, characterized in that The step of obtaining adjacent dynamic change features of the two adjacent target images in time sequence according to a portion of the image features of the first target image and a portion of the image features of the second target image comprises: Splicing a portion of the image features of the first target image with a portion of the image features of the second target image to obtain a first spliced ​​feature; The adjacent dynamic change feature of the two adjacent target images in time sequence is obtained according to the first splicing feature.

6. The method according to claim 5, characterized in that The step of acquiring the adjacent dynamic change features of the two adjacent target images in time sequence according to the first splicing feature includes: Perform average pooling on the first concatenated features to obtain pooled features; Perform feature extraction and dimensionality reduction processing on the pooled features to obtain the first dimensionality reduction feature; The adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to the first dimensionality reduction feature.

7. The method according to claim 5, characterized in that The step of acquiring the adjacent dynamic change features of the two adjacent target images in time sequence according to the first splicing feature includes: Performing feature extraction and dimensionality reduction processing on the first concatenated features to obtain second dimensionality reduction features; Performing feature extraction and dimensionality reduction processing on the second dimensionality reduction feature to obtain a third dimensionality reduction feature; The adjacent dynamic change features of the two adjacent target images in time sequence are obtained according to the third dimensionality reduction feature.

8. The method according to claim 3, characterized in that The step of obtaining overall dynamic change characteristics of multiple target images in time sequence according to the adjacent dynamic change characteristics of each two adjacent target images in time sequence includes: Calculate the product of the adjacent dynamic change features of each two adjacent target images in time sequence to obtain a first product feature; The overall dynamic change feature is obtained according to the first product feature.

9. The method according to claim 4, characterized in that The determining whether the camera is abnormal according to the overall dynamic change characteristics of the multiple target images in time sequence includes: Calculate the product of the overall dynamic change feature and at least part of the image features of the target image located first among the multiple target images to obtain a second product feature; Calculate the product of the overall dynamic change feature and at least part of the image features of the target image located at the last position among the multiple target images to obtain a third product feature; Whether the camera is abnormal is determined based on the second product feature and the third product feature.

10. The method according to claim 9, characterized in that The determining whether the camera is abnormal according to the second product feature and the third product feature includes: Concatenate the second product feature with the third product feature to obtain a second concatenated feature; Check whether the camera is abnormal based on the second stitching feature.

11. A camera anomaly detection device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of target images that are continuous in time sequence according to a plurality of images that are continuously acquired in time sequence by a camera; An extraction module, used for respectively extracting image features of each target image; The second acquisition module is used to acquire the overall dynamic change characteristics of the multiple target images in time sequence according to the image characteristics of each target image; The determination module is used to determine whether the camera is abnormal based on the overall dynamic change characteristics of multiple target images in time sequence. The camera abnormality includes that the camera is dirty and / or the camera is blocked.

12. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by the processor.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images