Camera image anomaly detection method and system

Abstract

The invention discloses a camera image anomaly detection method and system, and the method comprises the steps: introducing a space-time reconstruction path and a space-time prediction path, and carrying out the parallel processing of image frame data, thereby obtaining a reconstruction error and a prediction error. The two kinds of errors respectively represent the space-time fidelity of the current image and the causality in the time sequence, and a more comprehensive anomaly criterion is provided for the system. Furthermore, an error vector formed by the two errors is input into a trained small decision model for comprehensive judgment, so that the system can more finely distinguish normal change of image content and real imaging system abnormity, and frequent false alarms in a high dynamic scene are effectively avoided. Through the mode, the accuracy and the robustness of anomaly detection are remarkably improved, so that the strict requirements of actual application on low false alarm rate and high detection rate are met, and the practicability and the reliability of a camera monitoring system are greatly improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent detection, and more specifically, to a camera image anomaly detection method and system. Background Art

[0002] With the rapid development of intelligent surveillance, smart cities, industrial automation, and autonomous driving, cameras have become core sensors for acquiring environmental information and ensuring system operation. In these applications, cameras are required to operate stably and continuously, providing high-quality image data. However, in real-world applications, image data captured by cameras often experiences various anomalies, such as image degradation or interruptions caused by device failure, environmental interference, or human sabotage. Therefore, timely and accurate detection of these anomalies is crucial to ensuring system reliability, reducing manual intervention, and improving the effectiveness of data analysis.

[0003] Various approaches have been proposed for detecting anomalies in camera images. For example, some methods based on simple inter-frame differencing or motion detection determine whether changes exist by calculating pixel differences between consecutive frames. While computationally efficient, these methods struggle to adapt to complex and dynamic backgrounds. Any object movement, lighting fluctuations, or even swaying leaves can be mistakenly flagged as an anomaly, resulting in a high false positive rate. More advanced methods, such as reconstruction models based on autoencoders or generative adversarial networks, rely on learning the distribution of image features under normal conditions and then using the reconstruction error between the input image and the model-reconstructed image as an anomaly criterion. In theory, inputs that deviate from normal conditions will result in high reconstruction error, thus being identified as an anomaly. However, in real-world high-dynamic scenes, this single error metric also has inherent limitations. When legitimate scene content changes, such as sudden increases in the number of people, dramatic changes in ambient brightness, or the entry of new legitimate objects into the field of view, the image complexity and information content increase, leading to significantly higher reconstruction errors. This high reconstruction error can be very similar to, or even higher than, the reconstruction error caused by true image quality anomalies such as lens occlusion, freezing, or signal loss. In other words, existing models often struggle to fully understand the inherent nature of change, making it difficult to effectively determine whether high errors stem from legitimate evolution of scene content or physical failures of the imaging system. This confusion persists in accuracy and robustness for anomaly detection in dynamic environments, making it difficult to meet the demands of practical applications for low false alarm rates and high detection rates.

[0004] Therefore, an optimized camera image anomaly detection method is expected. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, the present application is proposed. The embodiment of the present application provides a camera image anomaly detection method and system, which processes image frame data in parallel by introducing a spatiotemporal reconstruction path and a spatiotemporal prediction path, thereby obtaining a reconstruction error and a prediction error respectively. These two errors respectively characterize the spatiotemporal fidelity and causality of the current image in the time series, providing the system with a more comprehensive anomaly judgment criterion. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, so that the system can more finely distinguish between normal changes in image content and true imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thereby meeting the stringent requirements of practical applications for low false alarm rate and high detection rate, and greatly improving the practicality and reliability of the camera monitoring system.

[0006] According to one aspect of the present application, a camera image anomaly detection method is provided, which includes: Get the image frame timing window captured by the camera; Inputting the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; Calculating a reconstruction error based on the image frame timing window and the image frame reconstruction timing window; Calculating a prediction error based on a current image frame and a predicted image frame in the image frame timing window; The error vector consisting of the reconstruction error and the prediction error is input into the trained small decision model to obtain the diagnosis result.

[0007] According to another aspect of the present application, a camera image anomaly detection system is provided, comprising: An image frame timing window acquisition module is used to acquire the image frame timing window captured by the camera; A parallel spatiotemporal feature processing module, configured to input the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; A reconstruction error calculation module, used for calculating a reconstruction error based on an image frame timing window and an image frame reconstruction timing window; A prediction error calculation module, configured to calculate a prediction error based on a current image frame and a predicted image frame in an image frame timing window; The diagnosis module is used to input the error vector composed of the reconstruction error and the prediction error into the trained small decision model to obtain the diagnosis result.

[0008] Compared with the prior art, the present application provides a camera image anomaly detection method and system, which processes image frame data in parallel by introducing a spatiotemporal reconstruction path and a spatiotemporal prediction path, thereby obtaining a reconstruction error and a prediction error respectively. These two errors respectively characterize the spatiotemporal fidelity and causality of the current image in the time series, providing the system with a more comprehensive anomaly judgment criterion. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, so that the system can more finely distinguish between normal changes in image content and true imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thereby meeting the stringent requirements of practical applications for low false alarm rate and high detection rate, and greatly improving the practicality and reliability of camera monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 Flowchart of a camera image anomaly detection method according to an embodiment of the present application; Figure 2 Schematic diagram of data flow of a camera image anomaly detection method according to an embodiment of the present application; Figure 3 is a block diagram of a camera image anomaly detection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0012] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0013] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0014] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0016] In the technical solution of the present application, a camera image anomaly detection method is proposed. Figure 1 Flowchart of a camera image anomaly detection method according to an embodiment of the present application. Figure 2 FIG. 1 is a system architecture diagram of a camera image anomaly detection method according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the camera image anomaly detection method according to the embodiment of the present application includes the following steps: S1, obtaining an image frame timing window captured by a camera; S2, inputting the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; S3, calculating a reconstruction error based on the image frame timing window and the image frame reconstruction timing window; S4, calculating a prediction error based on the current image frame and the predicted image frame in the image frame timing window; S5, inputting an error vector composed of the reconstruction error and the prediction error into a trained small decision model to obtain a diagnosis result.

[0017] In particular, the S1 obtains the time sequence window of the image frames captured by the camera. The original image data captured by the camera is instantaneous, but the judgment of image abnormalities often needs to be based on the contextual information of the image change trend or image sequence over a period of time. A single image frame cannot effectively capture temporal anomalies such as screen freezing, stuttering, flickering, slow blurring or sudden image loss, and it is difficult to provide sufficient information to judge the degree and persistence of spatial anomalies such as partial occlusion and lens deviation. Therefore, in the technical solution of the present application, by constructing a sequence containing multiple continuous image frames, the necessary spatiotemporal context information is provided for the subsequent parallel spatiotemporal feature processing engine, so that the spatiotemporal characteristics of the image can be comprehensively analyzed, laying a data foundation for accurately identifying various types of anomalies.

[0018] In practice, the following steps are used to obtain a temporal window of image frames captured by a camera: First, real-time image frames are acquired. During this process, the system establishes a connection with the camera and continuously receives raw image data output from the camera sensor. This typically involves decoding the video stream, parsing the continuous video stream into a series of independent image frames. For example, an IP network camera sends a video stream via a network protocol (such as RTSP or HTTP). The system receives this data in real time and uses a corresponding decoder (such as the FFmpeg library) to convert the video encoding format into a sequence of image frames, each representing a visual snapshot at a specific moment in time. Next, the real-time image frames are scaled and normalized to produce processed frames. Scaling refers to adjusting the original image frames to a preset uniform resolution. For example, if the original image resolution output by the camera is 1920x1080 pixels, and the subsequent deep learning model uses a fixed input size of 256x256 pixels for computational efficiency or model architecture requirements, the original image will need to be downsampled or scaled. This can be achieved through various image interpolation algorithms, such as bilinear interpolation or bicubic interpolation. Normalization refers to normalizing the image's pixel values. Image pixel values ​​typically range from 0 to 255, but deep learning models often have certain requirements for the numerical range and distribution of input data. For example, pixel values ​​must be normalized to the interval [0, 1] or [-1, 1]. This can be achieved by dividing pixel values ​​by 255 (for 8-bit images) or performing mean-variance normalization (mean-variance normalization), which subtracts the pixel mean and divides by the pixel standard deviation. After resizing and normalization, the raw real-time image frames are converted into processed frames with a uniform size and numerical range suitable for further processing. These processed frames are then placed into a first-in-first-out (FIFO) queue frame buffer to form an image frame timing window. A frame buffer is a memory area used to temporarily store a certain number of image frames. A FIFO queue is a data structure that ensures that the data first entered into the buffer is the first to be retrieved. During this process, each processed frame is added to the end of the queue. If the number of image frames stored in the queue reaches the preset timing window size (for example, a preset window contains 16 frames), the oldest (first-entered) frame in the queue is removed (dequeued) to make room for the new frame. This mechanism allows the frame buffer to always maintain a fixed length (i.e., the timing window size) of the latest image frames, known as the image frame timing window. This window slides continuously with the arrival of new image frames, reflecting the camera's recent visual information in real time.

[0019] In particular, S2 inputs the image frame temporal window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction temporal window and a predicted image frame. The parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path. It should be understood that single-dimensional anomaly detection (focusing only on image reconstruction or only on image prediction) often has limitations. For example, a frozen image may be completely normal spatially, but its temporal dynamics may be abnormal, making it difficult to detect using reconstruction error alone. Static spatial anomalies such as lens blur or occlusion may, in some cases, have a minimal impact on the prediction of the next frame, potentially resulting in insufficient prediction error for effective judgment. Therefore, in the technical solution of the present application, by constructing the spatiotemporal reconstruction path and the spatiotemporal prediction path in parallel, the system can simultaneously analyze the image from two dimensions: structural reconstructability and behavioral predictability. This ensures that both spatial structural anomalies (such as image blur, occlusion, color distortion, etc.) and temporal dynamic anomalies (such as freezing, stuttering, severe jitter, flickering, etc.) can be effectively modeled and detected, thereby providing a comprehensive and accurate data foundation for subsequent error calculation and comprehensive diagnosis.

[0020] In specific implementation, the parallel processing engine includes the following two main paths: For the spatiotemporal reconstruction path, the feature encoding process includes: First, a temporal window of image frames is input into a spatiotemporal autoencoder to obtain a spatiotemporal feature vector for the image frame. It should be understood that traditional single-frame processing methods cannot effectively capture the temporal dynamic information in video data. However, by jointly modeling the spatial and temporal dimensions, the spatiotemporal autoencoder can learn more discriminative motion patterns and appearance change features, which are crucial for accurately detecting image anomalies. Here, the spatiotemporal autoencoder comprises an encoder composed of 3D convolutional layers and a decoder composed of 3D deconvolutional layers. Unlike traditional 2D convolutions that operate only within the image plane, 3D convolutions can simultaneously extract features in both spatial dimensions (height and width) and temporal dimensions (frame sequence). Through layer-by-layer 3D convolution, pooling, and other operations, the encoder learns and condenses the intrinsic, most representative spatiotemporal feature vector of the image frame from the input temporal window.

[0021] During this process, the encoder receives as input a temporal window of image frames, a high-dimensional tensor whose dimensions are typically expressed as (number of time steps T, image height HJ, image width W, number of image channels C). 3D convolutional layers then slide and compute convolutions simultaneously across the time, height, and width dimensions. In other words, each convolution operation not only considers spatial information within a local region of the image but also captures temporal correlations and motion patterns across consecutive frames. For example, a 3x3x3 convolution kernel simultaneously observes changes in a 3x3 pixel region across three frames. As data flows through the encoder's 3D convolutional layers at different depths, the network extracts progressively higher-level, more abstract spatiotemporal features. Initial 3D convolutional layers may capture edges, textures, and simple motion trajectories in the image sequence; deeper 3D convolutional layers can recognize more complex spatiotemporal patterns, such as object shape, motion direction, and changes in scene activity. After each 3D convolutional layer, a nonlinear activation function (such as ReLU, or Rectified Linear Unit) is typically applied to introduce nonlinear capabilities into the model, enabling it to learn and represent complex, nonlinear feature relationships. Ultimately, to effectively compress the high-dimensional temporal window of image frames into low-dimensional spatiotemporal feature vectors, the encoder employs dimensionality reduction strategies at each layer, such as 3D pooling layers. Through these operations, the size of the feature map (including time, height, and width) is gradually reduced, while the number of feature channels (determined by the number of convolution kernels) is typically gradually increased, allowing each pixel to contain richer semantic information.

[0022] Then, the image frame spatiotemporal feature vector is decoded and regressed to obtain the image frame reconstructed timing window. The decoder is composed of a series of 3D deconvolution layers. The decoder's task is to perform the opposite operation of the encoder, that is, to gradually upsample and restore the image frame spatiotemporal feature vector to an image frame reconstructed timing window with the same size and number of frames as the original input image frame timing window. Here, if the input timing window is normal and the autoencoder is fully trained, the reconstructed timing window should be highly similar to the original input at the pixel level, and any significant differences will be regarded as potential signals of abnormality.

[0023] For the spatiotemporal prediction path, the feature encoding process includes: First, all frames preceding the current frame are extracted from the image frame timing window to obtain the image frame timing history window. Specifically, the historical portion of the complete image frame timing window is filtered out, excluding the current (last) frame to obtain the image frame timing history window. This history window encompasses all consecutive image frames up to the current moment and serves as the basis for predicting future frames.

[0024] Next, the temporal history window of the image frames is input into the spatiotemporal prediction network to generate predicted image frames. It should be understood that video sequences in normal scenes typically exhibit temporal continuity and regular motion, while abnormal events often disrupt this regularity. By constructing an accurate spatiotemporal prediction model, these abnormal patterns can be effectively captured. Compared to traditional single-frame analysis methods, this temporal prediction-based approach is more capable of detecting motion-related and progressive anomalies.

[0025] In the technical solution of the present application, the specific process of inputting the image frame time sequence history window into the spatiotemporal prediction network includes: First, each image frame in the image frame time series history window is input into a convolutional encoder to obtain a sequence of image frame semantic feature vectors. It should be understood that the original image frame contains a large amount of redundant information and high-dimensional data. Direct time series prediction from the original pixel level will face problems such as high computational complexity and noise sensitivity. In addition, slight changes at the pixel level may not be meaningful for abnormality judgment, while the overall semantic changes in the image content (for example, whether there are new objects in the scene, whether the position of the object has moved, whether the lighting conditions have changed suddenly, etc.) more directly reflect potential abnormalities. In the technical solution of the present application, feature extraction is performed on each image frame through a convolutional encoder, which can not only effectively extract the key visual information in each image frame, but also significantly reduce the data dimension.

[0026] Among them, the convolutional encoder is a deep learning neural network structure, usually composed of multiple convolutional layers, activation functions, pooling layers, and possible batch normalization layers. Among them, the convolutional layer is its core component. Each convolutional layer extracts local features by sliding a learned convolution kernel over a small image area and calculating the weighted sum of the pixels in the area. Convolution kernels at different levels will learn features of varying complexity, from basic edges and textures to more advanced shapes and patterns; the activation function applies a nonlinear activation function after the convolution operation, introducing nonlinear capabilities that enable the network to learn more complex mapping relationships; the pooling layer is mainly used to reduce the spatial dimensions (length and width) of the feature map while retaining the main features and increasing robustness to small changes in position (i.e., translation invariance). For example, the maximum pooling layer selects the maximum value within a region as the output.

[0027] Next, the sequence of image frame semantic feature vectors is input into the image semantic temporal propagation recursive encoder to obtain the image frame semantic propagation implicit encoding vector. It should be understood that the complex environment faced by modern surveillance systems requires that anomaly detection not only focus on the visual integrity of individual frames but also grasp the dynamic evolution patterns inherent in video sequences. Traditional methods often use a uniformly decaying memory mechanism when processing temporal information, failing to fully consider the differences in the importance of image features at different moments. This performs poorly when faced with sudden anomalies (such as sudden image occlusion) or gradual anomalies (such as gradual camera drift). Therefore, in the technical solution of this application, by introducing a message passing mechanism based on temporal decay, the system can dynamically adjust the influence weight of historical frames on the current prediction, ensuring that frames with significant semantic changes (such as key frames containing abnormal objects) maintain a longer-lasting influence during temporal propagation, while relatively stable, ordinary frames gradually fade according to the natural decay pattern. This temporal encoding method significantly improves the system's ability to capture various abnormal patterns.

[0028] Specifically, first, the maximum eigenvalue of each image frame semantic feature vector in the sequence of image frame semantic feature vectors is extracted as a temporal salient identifier of the image frame semantic feature. It should be understood that in an actual monitoring environment, abnormal events are often accompanied by significant mutations in image features. Traditional temporal processing methods use equal weights for historical frames, making it difficult to distinguish between slowly changing normal states and abnormal signs that require special attention. Therefore, in the technical solution of the present application, the maximum eigenvalue of each image frame semantic feature vector in the sequence of image frame semantic feature vectors is extracted to capture the most active dimension in the image frame semantic feature vector to quantify the semantic saliency of each frame of image. This saliency identifier based on the maximum eigenvalue provides a content-aware basis for subsequent temporal modeling.

[0029] Specifically, in a specific example of the present application, the temporal salient identifier of the semantic feature of the image frame is extracted using the following formula:

[0030] in, is the timestamp, For the image frame semantic feature vectors, Indicates taking the maximum eigenvalue of all dimensions of the semantic feature vector of the image frame, It is a temporal salient identifier of semantic features of image frames.

[0031] Then, based on the timestamps of the semantic feature vectors of each image frame in the sequence of image frame semantic feature vectors, the time span of the semantic feature vectors of each image frame is calculated. It should be understood that in a real monitoring environment, the events captured by the camera do not occur uniformly. The traditional time series model assumes that the contribution of all historical frames to the current prediction is only monotonically decreasing with the time interval, ignoring the semantic significance differences of the events themselves. Therefore, in the technical solution of the present application, by calculating the time span of the semantic feature vectors of each image frame (that is, the physical time difference between the current prediction moment and the actual capture moment of the historical frame), the precise timestamp information attached to each frame of the image is used to calculate the actual time difference between adjacent frames. These time differences reflect the real rhythm of scene changes. This time series modeling based on real time spans significantly improves the robustness of the system in complex scenarios.

[0032] Specifically, in a specific example of the present application, the time span of the semantic feature vector of each image frame is calculated using the following formula:

[0033] in, is the timestamp of the semantic feature vector of the image frame, is the time span of the semantic feature vector of each image frame.

[0034] Furthermore, based on the time span of each image frame semantic feature vector in the sequence of image frame semantic feature vectors and the image frame semantic feature temporal saliency identifier, the sequence of image frame semantic feature vectors is subjected to temporal attenuation propagation to obtain the image frame semantic transfer implicit coding vector. It should be understood that traditional temporal processing methods often adopt a fixed-mode memory mechanism, which cannot adapt to the complex and changeable temporal dynamics and content importance differences in monitoring scenarios. This is particularly evident when dealing with sudden abnormal events or long-term monitoring tasks. Therefore, in the technical solution of the present application, by organically combining the two key dimensions of physical time span and semantic saliency, intelligent screening and weighted aggregation of historical information are achieved. This intelligent temporal propagation mechanism significantly improves the discrimination ability of the anomaly detection system in complex scenarios.

[0035] Specifically, in the technical solution of the present application, the specific process of performing temporal attenuation weighted transfer on the sequence of semantic feature vectors of image frames includes: First, the temporal attenuation weight of the semantic feature vector of each image frame is calculated. The process is expressed as follows: in, For the The temporal attenuation weight of the semantic feature vector of each image frame, is the learnable attenuation coefficient, and , controls the rate of time decay, which is optimized by backpropagation, is the Gaussian decay kernel, is the neighbor set of target node i, enhancing the contributions of valued neighbors; Here, by taking the maximum eigenvalue As a modulation factor of the attenuation amplitude, the salient features retain higher weight even over time; in addition, through the learnable parameters Dynamically adjust the attenuation curve shape to adapt to the time sensitivity requirements of different scenarios; Next, based on the temporal attenuation weights of the semantic feature vectors of each image frame, the sequence of the semantic feature vectors of the image frame is subjected to feature attenuation modulation to obtain a sequence of the semantic attenuation feature coding vectors of the image frame. The process is expressed as follows:

[0036]

[0037] in, for The corresponding image frame semantic attenuation feature encoding vector, are the first, second and third in the sequence of semantic attenuation feature encoding vectors of image frames respectively. image frame semantic attenuation feature encoding vector, A sequence of semantic attenuation feature encoding vectors for image frames; Finally, based on The temporal encoder of the model performs temporal encoding on the sequence of semantic attenuation feature encoding vectors of the image frame to obtain the implicit encoding vector of the semantic transfer of the image frame. The process is expressed as follows:

[0038] in, represents the timing code, Transferring implicit encoding vectors for image frame semantics.

[0039] Furthermore, the semantically transferred implicit encoding vector of the image frame is input into the image feature decoder to obtain the predicted image frame. The image feature decoder is usually composed of a series of deconvolution layers, upsampling layers (such as bilinear interpolation or nearest neighbor interpolation), convolution layers, and activation functions.

[0040] In specific implementation, image feature decoding of the implicit encoding vector of the image frame semantic transfer can be performed through the following steps: First, the image feature decoder receives the implicit encoding vector of the image frame semantic transfer from the image semantic temporal transfer recursive encoder as input and converts it into a small feature map with multiple channels (for example, a 4x4x512 feature map) through a fully connected layer or an initial deconvolution layer. Then, the decoder gradually expands the spatial size of the feature map through multiple layers of deconvolution operations, each layer of deconvolution uses a 3×3 convolution kernel and a ReLU activation function. During the deconvolution process, the system will upsample by a factor of 2, and at the same time, the low-level features are fused with the high-level features through a skip connection mechanism. Finally, the output layer uses a 1×1 convolution to adjust the number of channels to 3 (corresponding to the three RGB channels), and the pixel values ​​are normalized to the range [0, 1] through the Sigmoid activation function, thereby generating a predicted image frame with the same size as the input image.

[0041] In particular, S3 calculates the reconstruction error based on the image frame temporal window and the image frame reconstruction temporal window. It should be understood that during the training phase, the spatiotemporal autoencoder is typically trained on a large number of normal image frame temporal windows. Its goal is to learn how to effectively compress the spatiotemporal features of normal images and accurately reconstruct them. Therefore, for a normal image frame temporal window, the autoencoder should be able to generate a reconstructed image frame temporal window that is highly similar to the original window, resulting in a minimal reconstruction error. However, when the input image frame temporal window exhibits anomalies, such as blur, partial occlusion, lens shift, watermarks, image corruption, or pixel errors, these anomaly features do not conform to the patterns learned by the model on normal data. When faced with these unseen anomalies or those that do not conform to its normal patterns, the autoencoder is often unable to effectively compress and accurately reconstruct them, resulting in significant differences between the reconstructed image frame temporal window and the original anomaly input. This significant difference, or large reconstruction error, can serve as an indicator of anomalies in the spatial structure of the image. Therefore, in the technical solution of this application, the reconstruction error is calculated and analyzed to quantify the degree of difference between the original image and the reconstructed image, providing an important basis for subsequent abnormality diagnosis. Specifically, the calculation of the reconstruction error directly reflects the spatiotemporal autoencoder's ability to learn normal image patterns. When the input image is abnormal, the reconstruction error will increase significantly, thus becoming an effective indicator for abnormality detection. In this way, the system can effectively detect structural defects or interference that deviate from the normal image appearance pattern.

[0042] The reconstruction error refers to the mean square error between the current image frame and its corresponding current reconstructed image frame, which is used to measure whether the image structure or content deviates from the normal mode.

[0043] In specific implementation, the reconstruction error can be calculated through the following steps: First, the current image frame and the current reconstructed image frame are extracted from the image frame timing window and the image frame reconstruction timing window; in this process, first, the system extracts the current image frame from the image frame timing window, and at the same time extracts the corresponding current reconstructed image frame from the image frame reconstruction timing window; then, the system converts the two image frames into the same data format to ensure that they have the same size and pixel value range. Then, the mean square error between the current image frame and the current reconstructed image frame is calculated as the reconstruction error. That is, the mean square error (MSE) between the two image frames is calculated using a pixel-by-pixel comparison method, and the calculation process is expressed as follows: The image resolution is Pixels, is the original pixel value, To reconstruct pixel values, is the normalized reconstruction error scalar value.

[0044] Taking the solution of this application as an example, assume that the image resolution H×W=4×4 pixels (for simplified demonstration, the actual resolution is 256×256); Current original frame (The lower right area is blurred due to lens damage): [ [100, 100, 100, 100], / / Pixel value range 0~255 (after normalization) [100, 100, 100, 100], [100, 100,50,50], / / Damaged area (pixel value decreases) [100, 100,50,50] ] Current reconstructed frame ("normal" reconstruction of the model output): [ [100, 100, 100, 100], [100, 100, 100, 100], [100, 100, 100, 100], / / The model does not learn the defacement features and outputs a clear image [100, 100, 100, 100] ] The reconstruction error is calculated as follows: First, the square error of each pixel is calculated as follows: coordinate :

[0045] coordinate :

[0046] coordinate : / / Defaced area coordinate :

[0047] coordinate :

[0048] coordinate :

[0049] The errors of the remaining 12 pixels are all 0.

[0050] Then, the square error of all pixels is summed up and the calculation process is as follows: Finally, the normalized mean square error is calculated, and the calculation process is as follows: The reconstruction error is 625.

[0051] In particular, S4 calculates a prediction error based on the current image frame and the predicted image frame in the image frame time window. It should be understood that image sequences captured by a camera typically follow certain temporal patterns and motion patterns (e.g., a stable background, slowly moving objects, gradual changes in lighting, etc.). After training on a large number of normal image sequences, the spatiotemporal prediction network is able to learn and model these complex temporal dependencies, thereby accurately predicting the current frame based on historical frames. When there is a significant difference between the actual current image frame received and the predicted image frame predicted by the model based on historical frames, this indicates that the temporal evolution of the image sequence has deviated from the normal pattern. Such anomalies include, but are not limited to, freezes (sudden cessation of image change), image freezes or frame skips (discontinuous images), severe jitter, black or distorted screens due to signal loss, and abrupt scene changes. By calculating the prediction error, the system can effectively detect these unexpected, sudden, or prolonged visual events that violate temporal patterns, providing key information for comprehensive anomaly identification.

[0052] The prediction error is a quantitative measure of the difference between the actual observed current image frame and the predicted image frame generated by the model based on historical data. A high prediction error usually indicates unexpected temporal changes in the image sequence.

[0053] In specific implementation, similarly, the prediction error can be calculated by the following steps: first, extracting the current image frame from the image frame timing window, and extracting the predicted image frame; then, calculating the mean square error between the current image frame and the predicted image frame as the prediction error.

[0054] In particular, S5 inputs the error vector consisting of the reconstruction error and the prediction error into the trained small decision model to obtain a diagnosis result. It should be understood that, specifically, reconstruction error (spatial dimension anomalies, such as blur or occlusion) and prediction error (temporal dimension anomalies, such as freezing or jitter) can each only detect anomalies in a single dimension, while anomalies in real-world scenarios are often coupled in space and time (e.g., lens smudges accompanied by video freezes). By combining these two errors into a single vector and inputting it into the decision model, the system can integrate spatial and temporal information to generate a unified anomaly diagnosis result, avoiding misjudgments or missed detections caused by a single error source, thereby improving detection robustness and decision accuracy in complex environments.

[0055] The trained small decision model is a logistic regression classifier. During this process, the logistic regression classifier can weigh the importance of different errors and make more robust and accurate diagnoses under complex error combinations, ultimately outputting a clear diagnosis result (e.g., normal, abnormal).

[0056] In practice, the error vector consisting of the reconstruction error and the prediction error can be input into the trained small decision model through the following steps: First, the reconstruction error and the prediction error are concatenated to obtain the error vector, which can be expressed as:

[0057] in, is the reconstruction error scalar value, is the prediction error scalar value, is the error vector; Subsequently, the error vector is input into the trained small decision model to obtain the diagnosis result. In the technical solution of this application, the logistic regression classifier is a two-class model, the core of which is the sigmoid function, which outputs the abnormal probability , and its calculation process is expressed as follows:

[0058] in, and is the weight matrix, b is the bias term, is the abnormal probability.

[0059] It is worth mentioning that the model is trained on a historical dataset that contains error vectors and labels of normal and abnormal samples (0 = normal, 1 = abnormal), and the weights are optimized by maximum likelihood estimation. and and bias b, so that the predicted probability Approximate the true label; Finally, the output probability of the logistic regression classifier is , generate the final diagnosis result. In this process, a decision threshold (usually 0.5) can be set. If , diagnosed as "abnormal"; if ,, the diagnosis was "normal".

[0060] In summary, the camera image anomaly detection method according to the embodiment of the present application is explained, which processes the image frame data in parallel by introducing a spatiotemporal reconstruction path and a spatiotemporal prediction path, thereby obtaining a reconstruction error and a prediction error respectively. These two errors respectively characterize the spatiotemporal fidelity and causality of the current image in the time series, providing the system with a more comprehensive anomaly judgment criterion. Furthermore, the error vector composed of these two errors is input into a trained small decision model for comprehensive judgment, so that the system can more finely distinguish between normal changes in image content and true imaging system anomalies, effectively avoiding frequent false alarms in high-dynamic scenes. In this way, the accuracy and robustness of anomaly detection are significantly improved, thereby meeting the stringent requirements of practical applications for low false alarm rate and high detection rate, and greatly improving the practicality and reliability of the camera monitoring system.

[0061] Furthermore, a camera image anomaly detection system is also provided.

[0062] Figure 3 FIG is a block diagram of a camera image anomaly detection system according to an embodiment of the present application. Figure 3 As shown, the camera image anomaly detection system 300 according to an embodiment of the present application includes: an image frame timing window acquisition module 310, which is used to obtain the image frame timing window collected by the camera; a parallel spatiotemporal feature processing module 320, which is used to input the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, and the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; a reconstruction error calculation module 330, which is used to calculate the reconstruction error based on the image frame timing window and the image frame reconstruction timing window; a prediction error calculation module 340, which is used to calculate the prediction error based on the current image frame and the predicted image frame in the image frame timing window; a diagnosis module 350, which is used to input the error vector composed of the reconstruction error and the prediction error into a trained small decision model to obtain a diagnosis result.

[0063] As described above, the camera image anomaly detection system 300 according to an embodiment of the present application can be implemented in various wireless terminals, such as a server equipped with a camera image anomaly detection algorithm. In one possible implementation, the camera image anomaly detection system 300 according to an embodiment of the present application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the camera image anomaly detection system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal. Of course, the camera image anomaly detection system 300 can also be one of the many hardware modules of the wireless terminal.

[0064] Alternatively, in another example, the camera image anomaly detection system 300 and the wireless terminal may also be separate devices, and the camera image anomaly detection system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0065] While various embodiments of the present disclosure have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A camera image anomaly detection method, characterized in that: include: Get the image frame timing window captured by the camera; Inputting the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; Calculating a reconstruction error based on the image frame timing window and the image frame reconstruction timing window; Calculating a prediction error based on a current image frame and a predicted image frame in the image frame timing window; The error vector consisting of the reconstruction error and the prediction error is input into the trained small decision model to obtain the diagnosis result.

2. The camera image anomaly detection method according to claim 1, wherein: Get the timing window of the image frames captured by the camera, including: Acquire real-time image frames; resizing and normalizing the real-time image frame to obtain a processed frame; Fill the processed frames into the first-in-first-out queue frame buffer to obtain the image frame timing window.

3. The camera image anomaly detection method according to claim 2, wherein: The image frame time sequence window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction time sequence window and the predicted image frame, including: The image frame time sequence window is input into the spatiotemporal autoencoder to obtain the image frame spatiotemporal feature vector; The image frame spatiotemporal feature vector is decoded and regressed to obtain the image frame reconstruction timing window.

4. The camera image anomaly detection method according to claim 2, wherein: The image frame time sequence window is input into the parallel spatiotemporal feature processing engine to obtain the image frame reconstruction time sequence window and the predicted image frame, including: Extract all image frames between the current image frame from the image frame timing window to obtain an image frame timing history window; The image frame temporal history window is input into the spatiotemporal prediction network to obtain the predicted image frame.

5. The camera image anomaly detection method according to claim 3, characterized in that: The spatiotemporal autoencoder includes an encoder consisting of 3D convolutional layers and a decoder consisting of 3D deconvolutional layers.

6. The camera image anomaly detection method according to claim 5, characterized in that: The image frame temporal history window is input into the spatiotemporal prediction network to obtain the predicted image frame, including: Input each image frame in the image frame time sequence history window into a convolutional encoder to obtain a sequence of image frame semantic feature vectors; Inputting the sequence of image frame semantic feature vectors into the image semantic temporal transfer recursive encoder to obtain the image frame semantic transfer implicit coding vector; The image frame semantic transfer implicit coding vector is input into the image feature decoder to obtain the predicted image frame.

7. The camera image anomaly detection method according to claim 1, characterized in that: Based on the image frame timing window and the image frame reconstruction timing window, the reconstruction error is calculated, including: Extracting a current image frame and a current reconstructed image frame from the image frame timing window and the image frame reconstruction timing window; The mean square error between the current image frame and the current reconstructed image frame is calculated as the reconstruction error.

8. The camera image anomaly detection method according to claim 1, wherein: The trained small decision model is a logistic regression classifier.

9. The camera image anomaly detection method according to claim 6, characterized in that: The sequence of image frame semantic feature vectors is input into the image semantic temporal transfer recursive encoder to obtain the image frame semantic transfer implicit coding vector, including: Extracting the maximum eigenvalue of each image frame semantic feature vector in the sequence of image frame semantic feature vectors as an image frame semantic feature temporal salient identifier; Calculating a time span of each image frame semantic feature vector based on a timestamp of each image frame semantic feature vector in the sequence of image frame semantic feature vectors; Based on the time span of each image frame semantic feature vector in the sequence of image frame semantic feature vectors and the image frame semantic feature temporal saliency identifier, the sequence of image frame semantic feature vectors is subjected to temporal attenuation propagation to obtain the image frame semantic transfer implicit coding vector.

10. A camera image anomaly detection system, characterized in that: include: An image frame timing window acquisition module is used to acquire the image frame timing window captured by the camera; A parallel spatiotemporal feature processing module, configured to input the image frame timing window into a parallel spatiotemporal feature processing engine to obtain an image frame reconstruction timing window and a predicted image frame, wherein the parallel spatiotemporal feature processing engine includes a spatiotemporal reconstruction path and a spatiotemporal prediction path; A reconstruction error calculation module, used for calculating a reconstruction error based on an image frame timing window and an image frame reconstruction timing window; A prediction error calculation module, configured to calculate a prediction error based on a current image frame and a predicted image frame in an image frame timing window; The diagnosis module is used to input the error vector composed of the reconstruction error and the prediction error into the trained small decision model to obtain the diagnosis result.

Images